Systematic Reviews and Meta-Analyses: Synthesis & Discussion
One of the final steps in a systematic review is the synthesis of evidence and writing the discussion.
Your team began working toward this stage in the protocol when you clearly identified the comparisons of interest. The work you've done in data extraction and critical appraisal phases will feed directly into the synthesis.
Qualitative Synthesis in Systematic Reviews and/or Meta-Analyses
Selecting the best approach for synthesis will depend on your scope, included material, field of research, etc. Therefore, it is important to follow methodological guidance that best matches your scope and field (e.g., a heath-focused review guided by the Cochrane Handbook). It can also be helpful to check out the synthesis and discussion of systematic reviews published by journals to which you plan to submit your review.
In almost all cases, a qualitative synthesis of some kind will be part of your systematic review. A quantitative synthesis (e.g., meta-analysis) should only be pursued as appropriate.
Meta-synthesis and Qualitative Evidence Synthesis are term sometimes used to describe a systematic review with only a qualitative synthesis.
Guidance for Qualitative Synthesis
In some methodological guidance, this stage may effectively be described as a separate methodology altogether.
For example, the Cochrane Handbook, Part 2: Core Methods covers synthesis through the lens of conducting a meta-analysis and/or quantitative synthesis. In Part 3: Specific perspectives in reviews, Cochrane goes into more detail about qualitative evidence synthesis in Chapter 21: Qualitative Evidence. Similarly, the JBI Manual for Evidence Synthesis contains a stand-alone chapter, Chapter 2: Systematic Reviews of Qualitative Evidence
Considerations and Decisions
- How you will group data for your synthesis and how grouping decisions are made, whether you're pursuing just a qualitative synthesis or both a qualitative synthesis and a meta-analysis, is an important consideration prior to starting the synthesis.
- Assess heterogeneity between studies, even if you don't plan to pursue a meta-analysis. Consider variability in participants studied, the definitions/measurements/frequency/etc. of interventions, or exposures, or outcomes, etc. This is part of the process to determine which studies are reasonable to synthesize.
- Selection of a formal qualitative synthesis approach (optional)
Qualitative Synthesis Approaches
This is not a comprehensive list of approaches. However, it can be a jumping off point for your team as you plan. The selection of approaches listed here is partially informed by Barnett-Page & Thomas (2009)
Note: Many of these approaches are also stand-alone qualitative research methods.
"In the case of qualitative systematic reviews, raw data consist of qualitative research findings (i.e. text) that have been systematically extracted from existing research reports...The manner in which these findings are coded is largely guided by the research topic and questions and the data that are available for analysis." (Finfgeld-Connett, 2014)
- Identification of data segments
- Memoing & diagramming
Resources for Content Analysis
- Finfgeld-Connett D. Use of content analysis to conduct knowledge-building and theory-generating qualitative systematic reviews. Qualitative Research. 2014;14(3):341-352. doi:10.1177/1468794113481790
- Neuendorf, K. (2017). The content analysis guidebook. SAGE Publications, Inc, https://dx.
"Developed out of a need to conduct reviews that addressed questions relating to intervention need, appropriateness, acceptability, [and effectiveness] without compromising on key principles developed in systematic reviews"(Barnett-Paige & Thomas 2009)
According to Thomas & Harden (2008):
- Code text (line-by-line)
- Develop descriptive themes
- Generate analytic themes
Resources for Thematic Synthesis
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008 Jul 10;8:45. doi: 10.1186/1471-2288-8-45. PMID: 18616818; PMCID: PMC2478656.
The "rationale [behind framework synthesis] is that qualitative research produces large amounts of textual data in the form of transcripts, observational fieldnotes etc. The sheer wealth of information poses a challenge for rigorous analysis. Framework synthesis offers a highly structured approach to organising and analysing data (e.g. indexing using numerical codes, rearranging data into charts etc)." (Barnett-Page & Thomas, 2009)
According to Brunton & James (2020):
- Familiarization (with existing literature)
- Framework selection
- Indexing & charting
- Mapping & interpretation
Resources for Framework Synthesis
- Brunton, G., Oliver, S., & Thomas, J. (2020). Innovations in framework synthesis as a systematic review method. Research Synthesis Methods, 11(3), 316–330. https://doi.org/10.1002/jrsm.1399
- Dixon-Woods, M. Using framework-based synthesis for conducting reviews of qualitative studies. BMC Med 9, 39 (2011). https://doi.org/10.1186/1741-7015-9-39
Grounded theory is defined as "a specific methodology developed by Glaser and Strauss (1967) for the purpose of building theory from data. In this book the term grounded theory is used in a more generic sense to denote theoretical constructs derived from qualitative analysis of data." (Strauss & Corbin, 2008)
According to Barnett-Paige & Thomas, 2009, "key methods and assumptions...include":
- "simultaneous phases of data collection and analysis;
- inductive approach to analysis, allowing the theory to emerge from the theory;
- the use of constant comparison method;
- the use of theoretical sampling to reach theoretical saturation; and the generation of new theory"
Resources for Grounded Theory
- Glaser, B. G., & Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine.
- Corbin, J., & Strauss, A. (2008). Basics of qualitative research (3rd ed.): Techniques and procedures for developing grounded theory. SAGE Publications, Inc. https://dx.
This is proposed as an alternative to "Meta-Analysis" (Nolbit & Hare, 1998; Barnett-Paige & Thomas 2009) and "should be interpretive rather than aggregative. We make the case that is should take the form of reciprocal translations of studies into one another" (Nolbit & Hare, 1998)
- Reciprocal translational analysis (RTA) - translate concepts; evolve overarching concepts
- Refutational synthesis - explore and explain contradictions between studies
- Lines-of-argument (LOA) synthesis - building up a picture of a whole from the parts (the individual studies)
Improving reporting of meta-ethnography: The eMERGe reporting guidance (documents the development of eMERGe)
Resources for Meta-Ethnography
- Sattar, R., Lawton, R., Panagioti, M. et al. Meta-ethnography in healthcare research: a guide to using a meta-ethnographic approach for literature synthesis. BMC Health Serv Res 21, 50 (2021). https://doi-org.ezproxy.lib.vt.edu/10.1186/s12913-020-06049-w
- France, E.F., Wells, M., Lang, H. et al. Why, when and how to update a meta-ethnography qualitative synthesis. Syst Rev 5, 44 (2016). https://doi-org.ezproxy.lib.vt.edu/10.1186/s13643-016-0218-4
- Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography. SAGE Publications, Inc. https://dx.
- Barnett-Page, E., Thomas, J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodoly 9, 59 (2009). https://doi-org.ezproxy.lib.vt.edu/10.1186/1471-2288-9-59
- Flemming, K., & Noyes, J. (2021). Qualitative Evidence Synthesis: Where Are We at? International Journal of Qualitative Methods. https://doi.org/10.1177/1609406921993276
“The statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings.” (Glass, 1976)
“A statistical analysis which combines the results of several independent studies considered by the analyst to be ‘combinable’.” (Huque, 1988)
“Meta-analysis is the statistical combination of results from two or more separate studies.” (Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Chapter 10)
The Cochrane Handbook (Chapter 10.1) states:
"Do not start here!"
...results of meta-analyses can be very misleading if suitable attention has not been given to formulating the review question; specifying eligibility criteria; identifying and selecting studies; collecting appropriate data; considering risk of bias; planning intervention comparisons; and deciding what data would be meaningful to analyse.
Choosing to pursue a Meta-Analysis
Reasons to pursue a Meta-Analysis
Meta-analyses are a desirable end-goal as a this kind of synthesis can:
- Increase statistical power / improve precision
- Result in a summary estimate of the direction and size of the effect or association
- Determine consistent across studies and explore why studies found different results
- Address questions that can’t be addressed by the individual studies (related to factors that differ across studies)
- Potentially resolve uncertainties if disagreement in literature and identify areas where evidence is insufficient
Reasons not to pursue a Meta-Analysis
Despite the appeal of the meta-analytic approach, it is vital that studies in the meta-analysis measure the same thing in the same way - that the studies themselves are reasonable to combine statistically.
According to Cochrane Chapter 12.1, "Legitimate reasons [for not conducting a meta-analysis] include limited evidence; incompletely reported outcome/effect estimates, or different effect measures used across studies; and bias in the evidence." Table 12.1.a describes scenarios that may preclude meta-analyses, with possible solutions
Likewise, a synthesis is only as good as the studies included. In other words, a meta-analysis cannot improve poor quality studies.
Considerations and Decisions
This is not a comprehensive list - as with any analysis, you'll need to select specific approaches based on the kind of data you have.
- How you will group data for your synthesis and how grouping decisions are made, is an important consideration prior to starting the synthesis.
- Effect size measures must be comparable across included studies and/or computable given the information available in the primary studies. For example, in a review of weight loss studies, you may convert all effects to pounds of lost weight.
- Choose a fixed- or random-effects model. In most cases, random-effects model will be most appropriate as meeting fixed-effects assumptions is difficult. In fact, Cochrane and Campbell Collaborations say never report only fixed effects, either random or both.
- Fixed-Effects: "assumes (1) all studies are measuring the same common (true) effect size (why we call it fixed), [and] (2) the observed results would be identical expect for random (sampling error)" (Borenstein, 2009)
- Random-Effects: "assumes (1) there are multiple population effects that the studies are estimating - different effect sizes underlying different studies, [and] (2) variability between effect sizes is due to sampling error + variability in population of effects" (Borenstein, 2009)
- There are some additional analyses you'll need to run to determine heterogeneity (how different studies are from each other). A sensitivity analysis or meta-regression is used to evaluate the effects of including or excluding certain groups of studies in your analysis, for example studies rated as low quality or high-risk of bias during the critical appraisal. You can also consider publication bias in your sample using a funnel plot (although there are valid critiques of the reliability of this practice).
- Glass, Gene V. “Primary, Secondary, and Meta-Analysis of Research.” Educational Researcher, vol. 5, no. 10, 1976, pp. 3–8, https://doi.org/10.2307/1174772.
- Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470743386
- Pigott, T. D., & Polanin, J. R. (2020). Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review. Review of Educational Research, 90(1), 24–46. https://doi.org/10.3102/0034654319877153
Tools for Meta-Analyses
Several tools exist for running your own meta-analyses. If you need further support, check out the help tab in this box.
Graphical User Interface (no programming required)
- RevMan | Developed by the Cochrane Collaboration; good for beginners
- PyMeta | Built from PythonMeta package for command line interface in python
- Comprehensive Meta-Analysis | fee-based
- MedCalc | fee-based
Command Line Interface (programming required)
- Metafor | R package; introduction from creator, Wolfgang Viechtbauer
- xmeta | R package; toolbox for multivariate meta-analyses
- PythonMeta | Python package; graphical interface available as PyMeta
Present Meta-Analysis Results
A meta-analysis is most commonly presented as a Forest Plot.
If you are new to the concept of forest plots, check out Dr. Terry Shaneyfelt from UAB School of Medicine How to interpret a forest plot.
Alternative Quantitative Synthesis Methods
According to Cochrane Chapter 9.5, "There are circumstances under which a meta-analysis is not possible, however, and other statistical synthesis methods might be considered, so as to make best use of the available data."
Table 9.5.a from the Cochrane Handbook, represented below, outlines some alternative synthesis method (and one summary method in the first row).
|Methods||Questions addressed||Example plots|
Text / Tabular
(summary, not synthesis)
|narrative summary of evidence presented in either text or tabular form||
(plotting individual study effects without a combined effect estimate)
|Vote counting||Is there any evidence of an effect?||
Effect direction plot
|Combining P values||Is there evidence that there is an effect in at least one study?||Albatross plot|
|Summary of effect estimates||
What is the range and distribution of observed effects?
Box and whisker plot
What is the common intervention effect? (fixed-effects model)
What is the average intervention effect? (random-effects model)
|Network meta-analysis||Which intervention of multiple is most effective?||
|Subgroup analysis / meta-regression||What factors modify the magnitude of the intervention effects?||
Box and whiskey plot
While the Evidence Synthesis Services (ESS) team at the University Libraries is available to support the other stages of a systematic review and/or meta-analysis,
we recommend reaching out to the Statistical Applications and Innovations Group (SAIG) for support in the statistical synthesis / meta-analysis.
Chapter 9: Summarizing study characteristics and preparing for synthesis
- 9.2 A general framework for synthesis
- 9.3 Preliminary steps of a synthesis
- 9.4 Checking data before synthesis
- 9.5 Types of synthesis
Chapter 10: Analyzing data and undertaking meta-analyses
- 10.1 Do not start here!
- 10.2 Introduction to meta-analysis
- 10.3 A generic inverse-variance approach to meta-analysis
- 10.4 Meta-analysis of dichotomous outcomes
- 10.5 Meta-analysis of continuous outcomes
- 10.6 Combining dichotomous and continuous outcomes
- 10.7 Meta-analysis of ordinal outcomes and measurement scales
- 10.8 Meta-analysis of counts and rates
- 10.9 Meta-analysis of time-to-event outcomes
- 10.10 Heterogeneity
- 10.11 Investing heterogeneity
- 10.12 Missing data
- 10.13 Bayesian approaches to meta-analysis
- 10.14 Sensitivity analyses
- 10.S1 Supplementary material: Statistical algorithms in Review Manager 5.1
Chapter 12: Synthesizing and presenting findings using other methods
- 12.1 Why a meta-analysis of effect estimates may not be possible
- 12.2 Statistical synthesis when meta-analysis of effect estimates is not possible
- 12.3 Visual display and presentation of the data
Chapter 13: Assessing risk of bias due to missing results in a synthesis
- 13.2 Minimizing risk of bias due to missing results
- 13.3 A framework for assessing risk of bias due to missing results in a synthesis
Chapter 15: Interpreting results and drawing conclusions
- 15.2 Issues of indirectness and applicability
- 15.3 Interpreting results of statistical analyses
- 15.4 Interpreting results from dichotomous outcomes
- 15.5 Interpreting results from continuous outcomes (including standardized mean differences)
- 15.6 Drawing conclusions
Chapter 21: Qualitative Evidence
- 21.2 Designs for synthesizing and integrating qualitative evidence with intervention reviews
- 21.3 Defining qualitative evidence and studies
- 21.4 Planning qualitative evidence synthesis linked to an intervention review
- 21.5 Question development
- 21.13 Methods for integrating the qualitative evidence synthesis with an intervention review
Step 5. Data Synthesis
Conducting systematic reviews of intervention questions III: Synthesizing data from intervention studies using meta-analysis. O’Connor AM, Sargeant JM, Wang C. Zoonoses Public Health. 2014 Jun;61 Suppl 1:52-63. doi: 10.1111/zph.12123. PMID: 24905996
Meta-analyses including data from observational studies. O’Connor AM, Sargeant JM. Prev Vet Med. 2014 Feb 15;113(3):313-22. doi: 10.1016/j.prevetmed.2013.10.017. Epub 2013 Oct 31. PMID: 24268538
Conducting systematic reviews of intervention questions II: Relevance screening, data extraction, assessing risk of bias, presenting the results and interpreting the findings. Sargeant JM, O’Connor AM. Zoonoses Public Health. 2014 Jun;61 Suppl 1:39-51. doi: 10.1111/zph.12124. PMID: 24905995
C59. Addressing risk of bias / study quality in the synthesis (review / final manuscript)
C60. Incorporating assessments of risk of bias (review / final manuscript)
C61. Combining different scales (review / final manuscript)
C62. Ensuring meta-analyses are meaningful (review / final manuscript)
C63. Assessing statistical heterogeneity (protocol & review / final manuscript)
C64. Addressing missing outcome data (review / final manuscript)
C65. Addressing skewed data (review / final manuscript)
C66. Addressing studies with more than two groups (protocol & review / final manuscript)
C67. Comparing subgroups (protocol & review / final manuscript)
C68. Interpreting subgroup analyses (protocol & review / final manuscript)
C69. Considering statistical heterogeneity when interpreting the results (review / final manuscript)
C70. Addressing non-standard designs (protocol & review / final manuscript)
C71. Conducting sensitivity analysis (protocol & review / final manuscript)
C72. Interpreting results (review / final manuscript)
C73. Investigating reporting biases (review / final manuscript)
C77. Formulating implications for practice (review / final manuscript)
C78. Avoiding recommendations (review / final manuscript)
C79. Formulating implications for research (review / final manuscript)
Section 9. Data synthesis
9.1 Systematic Reviews
9.1.1 Narrative Synthesis
9.1.2 Quantitative Data Synthesis
9.1.3 Qualitative Data Synthesis
Section 10. Interpreting findings and reporting conduct
10.1 The interpretation of evidence syntheses
10.2 Reporting conduct of evidence synthesis
10.3 Reporting findings of evidence syntheses
Reporting in Protocol and Final Manuscript
In the Protocol | PRISMA-P
Data Synthesis (Item 15)
Qualitative Synthesis only
If quantitative synthesis is not appropriate, describe the type of summary planned (Item 15d)
all of the above plus:
Describe criteria under which study data will be quantitatively synthesised (Item 15a)...quantitative synthesis, describe planned summary measures, methods of handling data and methods of combining data from studies, including any planned exploration of consistency (such as I2 , Kendall’s τ) (Item 15b)...describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression) (Item 15c)
In the Final Manuscript | PRISMA
Synthesis Methods (Item 13; report in methods)
Qualitative Synthesis only
- Describe the processes used to decide which studies were eligible for each synthesis. (Item 13a)
- Report any methods required to prepare the data collected from studies for presentation or synthesis, such as handling of missing summary statistics or data conversions (Item 13b)
- Report chosen tabular structure(s) used to display results of individual studies and syntheses, along with details of the data presented (Item 13c)
- Report chosen graphical methods used to visually display results of individual studies and syntheses (Item 13c)
- If it was not possible to conduct a meta-analysis, describe and justify the synthesis methods...or summary approach used (Item 13d)
- If a planned synthesis was not considered possible or appropriate, report this and the reason for that decision (Item 13d)
- If studies are ordered or grouped within tables or graphs based on study characteristics (such as by size of the study effect, year of publication), consider reporting the basis for the chosen ordering/grouping (Item 13c)
- If non-standard graphs were used, consider reporting the rationale for selecting the chosen graph (Item 13c)
Meta-Analysis (or other quantitative methods used)
all of the above plus:
- ...reference the software, packages, and version numbers used to implement synthesis methods (such as metan in Stata metafor (version 2.1-0) in R118) (Item 13d)
- ...specify (Item 13d):
- the meta-analysis model (fixed-effect, fixed-effects, or random-effects) and provide rationale for the selected model.
- the method used (such as Mantel-Haenszel, inverse-variance).
- any methods used to identify or quantify statistical heterogeneity (such as visual inspection of results, a formal statistical test for heterogeneity, heterogeneity variance (τ2), inconsistency (such as I2), and prediction intervals)
- If a random-effects meta-analysis model was used, specify (Item 13d):
- the between-study (heterogeneity) variance estimator used (such as DerSimonian and Laird, restricted maximum likelihood (REML)).
- the method used to calculate the confidence interval for the summary effect (such as Wald-type confidence interval, Hartung-Knapp-Sidik-Jonkman)
- If a Bayesian approach to meta-analysis was used, describe the prior distributions about quantities of interest (such as intervention effect being analysed, amount of heterogeneity in results across studies) (Item 13d)
- If multiple effect estimates from a study were included in a meta-analysis...describe the method(s) used to model or account for the statistical dependency...(Item 13d)
- If methods were used to explore possible causes of statistical heterogeneity, specify the method used (such as subgroup analysis, meta-regression) (Item 13e)
- If subgroup analysis or meta-regression was performed, specify for each:
- which factors were explored, levels of those factors, and which direction of effect modification was expected and why (where possible) (Item 13e)
- whether analyses were conducted using study-level variables (where each study is included in one subgroup only), within-study contrasts (where data on subsets of participants within a study are available, allowing the study to be included in more than one subgroup), or some combination of the above (Item 13e)
- how subgroup effects were compared (such as statistical test for interaction for subgroup analyses) (Item 13e)
- If other methods were used to explore heterogeneity because data were not amenable to meta-analysis of effect estimates, describe the methods used (such as structuring tables to examine variation in results across studies based on subpopulation, key intervention components, or contextual factors) along with the factors and levels (Item 13e)
- If any analyses used to explore heterogeneity were not pre-specified, identify them as such (Item 13e)
- If sensitivity analyses were performed, provide details of each analysis (such as removal of studies at high risk of bias, use of an alternative meta-analysis model) (Item 13f)
- If any sensitivity analyses were not pre-specified, identify them as such (Item 13f)
If a random-effects meta-analysis model was used, consider specifying other details about the methods used, such as the method for calculating confidence limits for the heterogeneity variance (Item 13d)
Reporting Bias Assessment (Item 14; report in methods)
- Specify the methods...used to assess the risk of bias due to missing results in a synthesis (arising from reporting biases).
- If risk of bias due to missing results was assessed using an existing tool, specify the methodological components/domains/items of the tool, and the process used to reach a judgment of overall risk of bias.
- If any adaptations to an existing tool to assess risk of bias due to missing results were made (such as omitting or modifying items), specify the adaptations.
- If a new tool to assess risk of bias due to missing results was developed for use in the review, describe the content of the tool and make it publicly accessible.
- Report how many reviewers assessed risk of bias due to missing results in a synthesis, whether multiple reviewers worked independently, and any processes used to resolve disagreements between assessors.
- Report any processes used to obtain or confirm relevant information from study investigators.
- If an automation tool was used to assess risk of bias due to missing results, report how the tool was used, how the tool was trained, and details on the tool’s performance and internal validation
Results of Synthesis (Item 20; report in results)
Qualitative Synthesis only
- Provide a brief summary of the characteristics and risk of bias among studies contributing to each synthesis (meta-analysis or other). The summary should focus only on study characteristics that help in interpreting the results (especially those that suggest the evidence addresses only a restricted part of the review question, or indirectly addresses the question). If the same set of studies contribute to more than one synthesis, or if the same risk of bias issues are relevant across studies for different syntheses, such a summary need be provided once only (Item 20a)
- Indicate which studies were included in each synthesis (such as by listing each study in a forest plot or table or citing studies in the text) (Item 20a)
- Report results of all statistical syntheses described in the protocol and all syntheses conducted that were not pre-specified (Item 20b)
Meta-Analysis (or other quantitative methods used)
all of the above plus:
- ...report for each:
- the summary estimate and its precision (such as standard error or 95% confidence/credible interval).
- measures of statistical heterogeneity (such as τ2, I2, prediction interval).
- If other statistical synthesis methods were used (such as summarising effect estimates, combining P values), report the synthesised result and a measure of precision (or equivalent information, for example, the number of studies and total sample size) (Item 20b)
- If the statistical synthesis method does not yield an estimate of effect (such as when P values are combined), report the relevant statistics (such as P value from the statistical test), along with an interpretation of the result that is consistent with the question addressed by the synthesis method (for example, “There was strong evidence of benefit of the intervention in at least one study (P < 0.001, 10 studies)” when P values have been combined) (Item 20b)
- If comparing groups, describe the direction of effect (such as fewer events in the intervention group, or higher pain in the comparator group) (Item 20b)
- If synthesising mean differences, specify for each synthesis, where applicable, the unit of measurement (such as kilograms or pounds for weight), the upper and lower limits of the measurement scale (for example, anchors range from 0 to 10), direction of benefit (for example, higher scores denote higher severity of pain), and the minimally important difference, if known. If synthesising standardised mean differences and the effect estimate is being re-expressed to a particular instrument, details of the instrument, as per the mean difference, should be reported (Item 20b)
- If investigations of possible causes of heterogeneity were conducted:
- present results regardless of the statistical significance, magnitude, or direction of effect modification (Item 20c)
- identify the studies contributing to each subgroup (Item 20c)
- report results with due consideration to the observational nature of the analysis and risk of confounding due to other factors (Item 20c)
- If subgroup analysis was conducted, report for each analysis the exact P value for a test for interaction as well as, within each subgroup, the summary estimates, their precision (such as standard error or 95% confidence/credible interval) and measures of heterogeneity. Results from subgroup analyses might usefully be presented graphically (Item 20c)
- If meta-regression was conducted, report for each analysis the exact P value for the regression coefficient and its precision (Item 20c)
- If informal methods (that is, those that do not involve a formal statistical test) were used to investigate heterogeneity—which may arise particularly when the data are not amenable to meta-analysis—describe the results observed. For example, present a table that groups study results by dose or overall risk of bias and comment on any patterns observed (Item 20c)
- If any sensitivity analyses were conducted:
- report the results for each sensitivity analysis (Item 20d)
- comment on how robust the main analysis was given the results of all corresponding sensitivity analyses (Item 20d)
- If subgroup analysis was conducted, consider presenting the estimate for the difference between subgroups and its precision (Item 20c)
- If meta-regression was conducted, consider presenting a meta-regression scatterplot with the study effect estimates plotted against the potential effect modifier (Item 20c)
- If any sensitivity analyses were conducted, consider:
- presenting results in tables that indicate:
- the summary effect estimate, a measure of precision (and potentially other relevant statistics, for example, I2 statistic) and contributing studies for the original meta-analysis;
- the same information for the sensitivity analysis; and
- details of the original and sensitivity analysis assumptions (Item 20d)
- presenting results of sensitivity analyses visually using forest plots (Item 20d)
- presenting results in tables that indicate:
Reporting Biases (Item 21; report in results)
- Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed.
- If a tool was used to assess risk of bias due to missing results in a synthesis, present responses to questions in the tool, judgments about risk of bias, and any information used to support such judgments to help readers understand why particular judgments were made.
- If a funnel plot was generated to evaluate small-study effects (one cause of which is reporting biases), present the plot and specify the effect estimate and measure of precision used in the plot (presented typically on the horizontal axis and vertical axis respectively). If a contour-enhanced funnel plot was generated, specify the “milestones” of statistical significance that the plotted contour lines represent (P=0.01, 0.05, 0.1, etc).
- If a test for funnel plot asymmetry was used, report the exact P value observed for the test and potentially other relevant statistics, such as the standardised normal deviate, from which the P value is derived.
- If any sensitivity analyses seeking to explore the potential impact of missing results on the synthesis were conducted, present results of each analysis (see item #20d), compare them with results of the primary analysis, and report results with due consideration of the limitations of the statistical method.
- If studies were assessed for selective non-reporting of results by comparing outcomes and analyses pre-specified in study registers, protocols, and statistical analysis plans with results that were available in study reports, consider presenting a matrix (with rows as studies and columns as syntheses) to present the availability of study results.
- If an assessment of selective non-reporting of results reveals that some studies are missing from the synthesis, consider displaying the studies with missing results underneath a forest plot or including a table with the available study results (for example, see forest plot in Page et al)
Discussion (Item 23)
- Provide a general interpretation of the results in the context of other evidence (Item 23a)
- Discuss any limitations of the evidence included in the review (Item 23b)
- Discuss any limitations of the review processes used and comment on the potential impact of each limitation (Item 23c)
- Discuss implications of the results for practice and policy (Item 23d)
- Make explicit recommendations for future research (Item 23d)