Systematic Reviews and Meta-Analyses: Synthesis & Discussion

Qualitative Synthesis Approaches

This is not a comprehensive list of approaches. However, it can be a jumping off point for your team as you plan. The selection of approaches listed here is partially informed by Barnett-Page & Thomas (2009)

Note: Many of these approaches are also stand-alone qualitative research methods. 

Content Analysis

"In the case of qualitative systematic reviews, raw data consist of qualitative research findings (i.e. text) that have been systematically extracted from existing research reports...The manner in which these findings are coded is largely guided by the research topic and questions and the data that are available for analysis." (Finfgeld-Connett, 2014)

Process

• Identification of data segments
• Memoing & diagramming
• Reflection

Resources for Content Analysis

• Finfgeld-Connett D. Use of content analysis to conduct knowledge-building and theory-generating qualitative systematic reviews. Qualitative Research. 2014;14(3):341-352. doi:10.1177/1468794113481790
• Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x 
• Mayring, P. (2015). Qualitative content analysis: Theoretical background and procedures. In A. Bikner-Ahsbahs, C. Knipping, & N. Presmeg (Eds.), Approaches to Qualitative Research in Mathematics Education: Examples of Methodology and Methods (pp. 365–380). Springer Netherlands. https://doi.org/10.1007/978-94-017-9181-6_13

Thematic Synthesis 

"Developed out of a need to conduct reviews that addressed questions relating to intervention need, appropriateness, acceptability, [and effectiveness] without compromising on key principles developed in systematic reviews"(Barnett-Paige & Thomas 2009)

Process

According to Thomas & Harden (2008):

• Code text (line-by-line) 
• Develop descriptive themes
• Generate analytic themes

Resources for Thematic Synthesis 

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008 Jul 10;8:45. doi: 10.1186/1471-2288-8-45. PMID: 18616818; PMCID: PMC2478656.

Framework Synthesis

The "rationale [behind framework synthesis] is that qualitative research produces large amounts of textual data in the form of transcripts, observational fieldnotes etc. The sheer wealth of information poses a challenge for rigorous analysis. Framework synthesis offers a highly structured approach to organising and analysing data (e.g. indexing using numerical codes, rearranging data into charts etc)." (Barnett-Page & Thomas, 2009)

Process

According to Brunton & James (2020):

• Familiarization (with existing literature)
• Framework selection 
• Indexing & charting
• Mapping & interpretation

Resources for Framework Synthesis 

• Brunton, G., Oliver, S., & Thomas, J. (2020). Innovations in framework synthesis as a systematic review method. Research Synthesis Methods, 11(3), 316–330. https://doi.org/10.1002/jrsm.1399
• Dixon-Woods, M. Using framework-based synthesis for conducting reviews of qualitative studies. BMC Med 9, 39 (2011). https://doi.org/10.1186/1741-7015-9-39

Grounded Theory

Grounded theory is defined as "a specific methodology developed by Glaser and Strauss (1967) for the purpose of building theory from data. In this book the term grounded theory is used in a more generic sense to denote theoretical constructs derived from qualitative analysis of data." (Strauss & Corbin, 2008)

Process

According to Barnett-Paige & Thomas, 2009, "key methods and assumptions...include":

• "simultaneous phases of data collection and analysis;
• inductive approach to analysis, allowing the theory to emerge from the theory;
• the use of constant comparison method;
• the use of theoretical sampling to reach theoretical saturation; and the generation of new theory"

Resources for Grounded Theory

• Glaser, B. G., & Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine.
• Corbin, J., & Strauss, A. (2008). Basics of qualitative research (3rd ed.): Techniques and procedures for developing grounded theory. SAGE Publications, Inc. https://dx.doi.org/10.4135/9781452230153

Meta-Ethnography 

This is proposed as an alternative to "Meta-Analysis" (Nolbit & Hare, 1998; Barnett-Paige & Thomas 2009) and "should be interpretive rather than aggregative. We make the case that is should take the form of reciprocal translations of studies into one another" (Nolbit & Hare, 1998)

Approaches

• Reciprocal translational analysis (RTA) - translate concepts; evolve overarching concepts
• Refutational synthesis - explore and explain contradictions between studies 
• Lines-of-argument (LOA) synthesis - building up a picture of a whole from the parts (the individual studies) 

Reporting Guideline

eMERGe

Improving reporting of meta-ethnography: The eMERGe reporting guidance (documents the development of eMERGe)

Resources for Meta-Ethnography

• Sattar, R., Lawton, R., Panagioti, M. et al. Meta-ethnography in healthcare research: a guide to using a meta-ethnographic approach for literature synthesis. BMC Health Serv Res 21, 50 (2021). https://doi-org.ezproxy.lib.vt.edu/10.1186/s12913-020-06049-w
• France, E.F., Wells, M., Lang, H. et al. Why, when and how to update a meta-ethnography qualitative synthesis. Syst Rev 5, 44 (2016). https://doi-org.ezproxy.lib.vt.edu/10.1186/s13643-016-0218-4
• Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography. SAGE Publications, Inc. https://dx.doi.org/10.4135/9781412985000

Resources

• Barnett-Page, E., Thomas, J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodoly 9, 59 (2009). https://doi-org.ezproxy.lib.vt.edu/10.1186/1471-2288-9-59 
• Flemming, K., & Noyes, J. (2021). Qualitative Evidence Synthesis: Where Are We at? International Journal of Qualitative Methods. https://doi.org/10.1177/1609406921993276

Tools for Meta-Analyses

Several tools exist for running your own meta-analyses. If you need further support, check out the help tab in this box.

Graphical User Interface (no programming required) 

• RevMan | Developed by the Cochrane Collaboration; good for beginners
• PyMeta | Built from PythonMeta package for command line interface in python  
• Comprehensive Meta-Analysis | fee-based
• MedCalc | fee-based

Command Line Interface (programming required)

• Metafor | R package; introduction from creator, Wolfgang Viechtbauer
• xmeta | R package; toolbox for multivariate meta-analyses
• PythonMeta | Python package; graphical interface available as PyMeta

Resources

• Polanin, J. R., Hennessy, E. A., & Tanner-Smith, E. E. (2017). A Review of Meta-Analysis Packages in R. Journal of Educational and Behavioral Statistics, 42(2), 206–242. https://doi.org/10.3102/1076998616674315
• Video for using R for Meta package

Present Meta-Analysis Results

A meta-analysis is most commonly presented as a Forest Plot.

Forest Plot

If you are new to the concept of forest plots, check out Dr. Terry Shaneyfelt from UAB School of Medicine How to interpret a forest plot.

Alternative Quantitative Synthesis Methods

According to Cochrane Chapter 9.5, "There are circumstances under which a meta-analysis is not possible, however, and other statistical synthesis methods might be considered, so as to make best use of the available data."

Table 9.5.a from the Cochrane Handbook, represented below, outlines some alternative synthesis method (and one summary method in the first row).

While the Evidence Synthesis Services (ESS) team at the University Libraries is available to support the other stages of a systematic review and/or meta-analysis,

we recommend reaching out to the Statistical Applications and Innovations Group (SAIG) for support in the statistical synthesis / meta-analysis. 

SYREAF Tutorials

Step 5. Data Synthesis

Conducting systematic reviews of intervention questions III: Synthesizing data from intervention studies using meta-analysis. O’Connor AM, Sargeant JM, Wang C. Zoonoses Public Health. 2014 Jun;61 Suppl 1:52-63. doi: 10.1111/zph.12123. PMID: 24905996

Meta-analyses including data from observational studies. O’Connor AM, Sargeant JM. Prev Vet Med. 2014 Feb 15;113(3):313-22. doi: 10.1016/j.prevetmed.2013.10.017. Epub 2013 Oct 31. PMID: 24268538

Step 6. Presenting the results & Step 7. Reaching a conclusion

Conducting systematic reviews of intervention questions II: Relevance screening, data extraction, assessing risk of bias, presenting the results and interpreting the findings. Sargeant JM, O’Connor AM. Zoonoses Public Health. 2014 Jun;61 Suppl 1:39-51. doi: 10.1111/zph.12124. PMID: 24905995

Campbell - MECCIR

The Campbell Collaboration has combined their reporting and conduct (methodological) guidelines into a single document.

Section 6 - Synthesis methods (p. 4-5)

Conduct Guidance

• Heterogeneity uses appropriate metrics (e.g., Tau 2)

6d. Handle effect size dependencies 

6e. Examine heterogeneity if possible

6g. Explore publication and outcome selection bias

Reporting Guidance

• Justify synthesis and use established methods
• Describe synthesis and if any quantitative analysis was used
• Detail heterogeneity and effect measures
• Describe the handling of multiple effect sizes
• Describe plans for narrative synthesis
• Present findings with visualizations, like forest plots

6a. Justify effect measure(s)

6b. Justify model and estimation method

6c. Report effect sizes, variance, and heterogeneity

Section 7 - Discussion (p. 5)

Conduct Guidance

• Interpret results based on evidence across studies
• Interpret evidence on quality, heterogeneity, and relevance
• Assess relevance of effect size to review population

7a. Interpret effect size based on evidence quality and bias

7c. Interpret findings based on the alignment of study elements with the question

Reporting Guidance

7b. Discuss uncertainty in key findings (e.g., heterogeneity)

7d. Identify implications for practice and research

CEE - Guidelines and Standards for Evidence synthesis in Environmental Management

Section 9. Data synthesis

CEE Standards for conduct and reporting

9.1 Systematic Reviews

9.1.1 Narrative Synthesis

9.1.2 Quantitative Data Synthesis 

9.1.3 Qualitative Data Synthesis

Section 10. Interpreting findings and reporting conduct

10.1 The interpretation of evidence syntheses

10.2 Reporting conduct of evidence synthesis

10.3 Reporting findings of evidence syntheses

In the Final Manuscript | PRISMA

Synthesis Methods (Item 13; report in methods)

Qualitative Synthesis only

Essential Items

• Describe the processes used to decide which studies were eligible for each synthesis. (Item 13a)
• Report any methods required to prepare the data collected from studies for presentation or synthesis, such as handling of missing summary statistics or data conversions (Item 13b)
• Report chosen tabular structure(s) used to display results of individual studies and syntheses, along with details of the data presented (Item 13c)
• Report chosen graphical methods used to visually display results of individual studies and syntheses (Item 13c)
• If it was not possible to conduct a meta-analysis, describe and justify the synthesis methods...or summary approach used (Item 13d)
• If a planned synthesis was not considered possible or appropriate, report this and the reason for that decision (Item 13d)

Additional Items

• If studies are ordered or grouped within tables or graphs based on study characteristics (such as by size of the study effect, year of publication), consider reporting the basis for the chosen ordering/grouping (Item 13c)
• If non-standard graphs were used, consider reporting the rationale for selecting the chosen graph (Item 13c)

Meta-Analysis (or other quantitative methods used)

all of the above plus:

Essential Items

• ...reference the software, packages, and version numbers used to implement synthesis methods (such as metan in Stata metafor (version 2.1-0) in R118) (Item 13d)
• ...specify (Item 13d):
  • the meta-analysis model (fixed-effect, fixed-effects, or random-effects) and provide rationale for the selected model.
  • the method used (such as Mantel-Haenszel, inverse-variance).
  • any methods used to identify or quantify statistical heterogeneity (such as visual inspection of results, a formal statistical test for heterogeneity, heterogeneity variance (τ2), inconsistency (such as I2), and prediction intervals) 
• If a random-effects meta-analysis model was used, specify (Item 13d):
  • the between-study (heterogeneity) variance estimator used (such as DerSimonian and Laird, restricted maximum likelihood (REML)).
  • the method used to calculate the confidence interval for the summary effect (such as Wald-type confidence interval, Hartung-Knapp-Sidik-Jonkman) 
• If a Bayesian approach to meta-analysis was used, describe the prior distributions about quantities of interest (such as intervention effect being analysed, amount of heterogeneity in results across studies) (Item 13d)
• If multiple effect estimates from a study were included in a meta-analysis...describe the method(s) used to model or account for the statistical dependency...(Item 13d)
• If methods were used to explore possible causes of statistical heterogeneity, specify the method used (such as subgroup analysis, meta-regression) (Item 13e)
• If subgroup analysis or meta-regression was performed, specify for each:
  • which factors were explored, levels of those factors, and which direction of effect modification was expected and why (where possible) (Item 13e)
  • whether analyses were conducted using study-level variables (where each study is included in one subgroup only), within-study contrasts (where data on subsets of participants within a study are available, allowing the study to be included in more than one subgroup), or some combination of the above (Item 13e)
  • how subgroup effects were compared (such as statistical test for interaction for subgroup analyses) (Item 13e)
• If other methods were used to explore heterogeneity because data were not amenable to meta-analysis of effect estimates, describe the methods used (such as structuring tables to examine variation in results across studies based on subpopulation, key intervention components, or contextual factors) along with the factors and levels (Item 13e)
• If any analyses used to explore heterogeneity were not pre-specified, identify them as such (Item 13e)
• If sensitivity analyses were performed, provide details of each analysis (such as removal of studies at high risk of bias, use of an alternative meta-analysis model) (Item 13f)
• If any sensitivity analyses were not pre-specified, identify them as such (Item 13f)

Additional Items

If a random-effects meta-analysis model was used, consider specifying other details about the methods used, such as the method for calculating confidence limits for the heterogeneity variance (Item 13d)

Qualitative Synthesis only

Essential Items

• Provide a brief summary of the characteristics and risk of bias among studies contributing to each synthesis (meta-analysis or other). The summary should focus only on study characteristics that help in interpreting the results (especially those that suggest the evidence addresses only a restricted part of the review question, or indirectly addresses the question). If the same set of studies contribute to more than one synthesis, or if the same risk of bias issues are relevant across studies for different syntheses, such a summary need be provided once only (Item 20a)
• Indicate which studies were included in each synthesis (such as by listing each study in a forest plot or table or citing studies in the text) (Item 20a)
• Report results of all statistical syntheses described in the protocol and all syntheses conducted that were not pre-specified (Item 20b)

Meta-Analysis (or other quantitative methods used)

all of the above plus:

Essential Items

• ...report for each:
  • the summary estimate and its precision (such as standard error or 95% confidence/credible interval).
  • measures of statistical heterogeneity (such as τ2, I2, prediction interval).
• If other statistical synthesis methods were used (such as summarising effect estimates, combining P values), report the synthesised result and a measure of precision (or equivalent information, for example, the number of studies and total sample size) (Item 20b)
• If the statistical synthesis method does not yield an estimate of effect (such as when P values are combined), report the relevant statistics (such as P value from the statistical test), along with an interpretation of the result that is consistent with the question addressed by the synthesis method (for example, “There was strong evidence of benefit of the intervention in at least one study (P < 0.001, 10 studies)” when P values have been combined) (Item 20b)
• If comparing groups, describe the direction of effect (such as fewer events in the intervention group, or higher pain in the comparator group) (Item 20b)
• If synthesising mean differences, specify for each synthesis, where applicable, the unit of measurement (such as kilograms or pounds for weight), the upper and lower limits of the measurement scale (for example, anchors range from 0 to 10), direction of benefit (for example, higher scores denote higher severity of pain), and the minimally important difference, if known. If synthesising standardised mean differences and the effect estimate is being re-expressed to a particular instrument, details of the instrument, as per the mean difference, should be reported (Item 20b)
• If investigations of possible causes of heterogeneity were conducted:
  • present results regardless of the statistical significance, magnitude, or direction of effect modification (Item 20c)
  • identify the studies contributing to each subgroup (Item 20c)
  • report results with due consideration to the observational nature of the analysis and risk of confounding due to other factors (Item 20c)
• If subgroup analysis was conducted, report for each analysis the exact P value for a test for interaction as well as, within each subgroup, the summary estimates, their precision (such as standard error or 95% confidence/credible interval) and measures of heterogeneity. Results from subgroup analyses might usefully be presented graphically (Item 20c)
• If meta-regression was conducted, report for each analysis the exact P value for the regression coefficient and its precision (Item 20c)
• If informal methods (that is, those that do not involve a formal statistical test) were used to investigate heterogeneity—which may arise particularly when the data are not amenable to meta-analysis—describe the results observed. For example, present a table that groups study results by dose or overall risk of bias and comment on any patterns observed (Item 20c)
• If any sensitivity analyses were conducted:
  • report the results for each sensitivity analysis (Item 20d)
  • comment on how robust the main analysis was given the results of all corresponding sensitivity analyses (Item 20d)

Additional Items

• If subgroup analysis was conducted, consider presenting the estimate for the difference between subgroups and its precision (Item 20c)
• If meta-regression was conducted, consider presenting a meta-regression scatterplot with the study effect estimates plotted against the potential effect modifier (Item 20c)
• If any sensitivity analyses were conducted, consider:
  • presenting results in tables that indicate:
    1. the summary effect estimate, a measure of precision (and potentially other relevant statistics, for example, I2 statistic) and contributing studies for the original meta-analysis;
    2. the same information for the sensitivity analysis; and
    3. details of the original and sensitivity analysis assumptions (Item 20d)
  • presenting results of sensitivity analyses visually using forest plots (Item 20d)

Reporting Biases (Item 21; report in results)

Essential Items

• Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed.
• If a tool was used to assess risk of bias due to missing results in a synthesis, present responses to questions in the tool, judgments about risk of bias, and any information used to support such judgments to help readers understand why particular judgments were made.
• If a funnel plot was generated to evaluate small-study effects (one cause of which is reporting biases), present the plot and specify the effect estimate and measure of precision used in the plot (presented typically on the horizontal axis and vertical axis respectively). If a contour-enhanced funnel plot was generated, specify the “milestones” of statistical significance that the plotted contour lines represent (P=0.01, 0.05, 0.1, etc).
• If a test for funnel plot asymmetry was used, report the exact P value observed for the test and potentially other relevant statistics, such as the standardised normal deviate, from which the P value is derived.
• If any sensitivity analyses seeking to explore the potential impact of missing results on the synthesis were conducted, present results of each analysis (see item #20d), compare them with results of the primary analysis, and report results with due consideration of the limitations of the statistical method.

Additional Items

• If studies were assessed for selective non-reporting of results by comparing outcomes and analyses pre-specified in study registers, protocols, and statistical analysis plans with results that were available in study reports, consider presenting a matrix (with rows as studies and columns as syntheses) to present the availability of study results.
• If an assessment of selective non-reporting of results reveals that some studies are missing from the synthesis, consider displaying the studies with missing results underneath a forest plot or including a table with the available study results (for example, see forest plot in Page et al)

Discussion (Item 23)

Essential Items

• Provide a general interpretation of the results in the context of other evidence (Item 23a)
• Discuss any limitations of the evidence included in the review (Item 23b)
• Discuss any limitations of the review processes used and comment on the potential impact of each limitation (Item 23c)
• Discuss implications of the results for practice and policy (Item 23d)
• Make explicit recommendations for future research (Item 23d)