One of the final steps in a systematic review is the synthesis of evidence and writing the discussion.
Your team began working toward this stage in the protocol when you clearly identified the comparisons of interest. The work you've done in data extraction and critical appraisal phases will feed directly into the synthesis.
Selecting the best approach for synthesis will depend on your scope, included material, field of research, etc. Therefore, it is important to follow methodological guidance that best matches your scope and field (e.g., a heath-focused review guided by the Cochrane Handbook). It can also be helpful to check out the synthesis and discussion of systematic reviews published by journals to which you plan to submit your review.
In almost all cases, a qualitative synthesis of some kind will be part of your systematic review. A quantitative synthesis (e.g., meta-analysis) should only be pursued as appropriate.
Meta-synthesis and Qualitative Evidence Synthesis are term sometimes used to describe a systematic review with only a qualitative synthesis.
In some methodological guidance, this stage may effectively be described as a separate methodology altogether.
For example, the Cochrane Handbook, Part 2: Core Methods covers synthesis through the lens of conducting a meta-analysis and/or quantitative synthesis. In Part 3: Specific perspectives in reviews, Cochrane goes into more detail about qualitative evidence synthesis in Chapter 21: Qualitative Evidence. Similarly, the JBI Manual for Evidence Synthesis contains a stand-alone chapter, Chapter 2: Systematic Reviews of Qualitative Evidence
This is not a comprehensive list of approaches. However, it can be a jumping off point for your team as you plan. The selection of approaches listed here is partially informed by Barnett-Page & Thomas (2009)
Note: Many of these approaches are also stand-alone qualitative research methods.
"In the case of qualitative systematic reviews, raw data consist of qualitative research findings (i.e. text) that have been systematically extracted from existing research reports...The manner in which these findings are coded is largely guided by the research topic and questions and the data that are available for analysis." (Finfgeld-Connett, 2014)
"Developed out of a need to conduct reviews that addressed questions relating to intervention need, appropriateness, acceptability, [and effectiveness] without compromising on key principles developed in systematic reviews"(Barnett-Paige & Thomas 2009)
According to Thomas & Harden (2008):
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008 Jul 10;8:45. doi: 10.1186/1471-2288-8-45. PMID: 18616818; PMCID: PMC2478656.
The "rationale [behind framework synthesis] is that qualitative research produces large amounts of textual data in the form of transcripts, observational fieldnotes etc. The sheer wealth of information poses a challenge for rigorous analysis. Framework synthesis offers a highly structured approach to organising and analysing data (e.g. indexing using numerical codes, rearranging data into charts etc)." (Barnett-Page & Thomas, 2009)
According to Brunton & James (2020):
Grounded theory is defined as "a specific methodology developed by Glaser and Strauss (1967) for the purpose of building theory from data. In this book the term grounded theory is used in a more generic sense to denote theoretical constructs derived from qualitative analysis of data." (Strauss & Corbin, 2008)
According to Barnett-Paige & Thomas, 2009, "key methods and assumptions...include":
This is proposed as an alternative to "Meta-Analysis" (Nolbit & Hare, 1998; Barnett-Paige & Thomas 2009) and "should be interpretive rather than aggregative. We make the case that is should take the form of reciprocal translations of studies into one another" (Nolbit & Hare, 1998)
Improving reporting of meta-ethnography: The eMERGe reporting guidance (documents the development of eMERGe)
“The statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings.” (Glass, 1976)
“A statistical analysis which combines the results of several independent studies considered by the analyst to be ‘combinable’.” (Huque, 1988)
“Meta-analysis is the statistical combination of results from two or more separate studies.” (Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Chapter 10)
The Cochrane Handbook (Chapter 10.1) states:
"Do not start here!"
...results of meta-analyses can be very misleading if suitable attention has not been given to formulating the review question; specifying eligibility criteria; identifying and selecting studies; collecting appropriate data; considering risk of bias; planning intervention comparisons; and deciding what data would be meaningful to analyse.
Meta-analyses are a desirable end-goal as a this kind of synthesis can:
Despite the appeal of the meta-analytic approach, it is vital that studies in the meta-analysis measure the same thing in the same way - that the studies themselves are reasonable to combine statistically.
According to Cochrane Chapter 12.1, "Legitimate reasons [for not conducting a meta-analysis] include limited evidence; incompletely reported outcome/effect estimates, or different effect measures used across studies; and bias in the evidence." Table 12.1.a describes scenarios that may preclude meta-analyses, with possible solutions
Likewise, a synthesis is only as good as the studies included. In other words, a meta-analysis cannot improve poor quality studies.
This is not a comprehensive list - as with any analysis, you'll need to select specific approaches based on the kind of data you have.
Several tools exist for running your own meta-analyses. If you need further support, check out the help tab in this box.
A meta-analysis is most commonly presented as a Forest Plot.
If you are new to the concept of forest plots, check out Dr. Terry Shaneyfelt from UAB School of Medicine How to interpret a forest plot.
According to Cochrane Chapter 9.5, "There are circumstances under which a meta-analysis is not possible, however, and other statistical synthesis methods might be considered, so as to make best use of the available data."
Table 9.5.a from the Cochrane Handbook, represented below, outlines some alternative synthesis method (and one summary method in the first row).
Methods | Questions addressed | Example plots |
Text / Tabular (summary, not synthesis) |
narrative summary of evidence presented in either text or tabular form |
Forest plot (plotting individual study effects without a combined effect estimate) |
Vote counting | Is there any evidence of an effect? |
Harvest plot Effect direction plot |
Combining P values | Is there evidence that there is an effect in at least one study? | Albatross plot |
Summary of effect estimates |
What is the range and distribution of observed effects? |
Box and whisker plot Bubble plot |
Pairwise meta-analysis |
What is the common intervention effect? (fixed-effects model) What is the average intervention effect? (random-effects model) |
Forest plot |
Network meta-analysis | Which intervention of multiple is most effective? |
Forest plot Network diagram Rankogram plot |
Subgroup analysis / meta-regression | What factors modify the magnitude of the intervention effects? |
Forest plot Box and whiskey plot Bubble plot |
While the Evidence Synthesis Services (ESS) team at the University Libraries is available to support the other stages of a systematic review and/or meta-analysis,
we recommend reaching out to the Statistical Applications and Innovations Group (SAIG) for support in the statistical synthesis / meta-analysis.
Chapter III: Reporting the Review (specifically part III.III); Note: if you are not conducting a Cochrane Review, use this resource as a guidepost
Chapter 9: Summarizing study characteristics and preparing for synthesis
Chapter 10: Analyzing data and undertaking meta-analyses
Chapter 12: Synthesizing and presenting findings using other methods
Chapter 13: Assessing risk of bias due to missing results in a synthesis
Chapter 15: Interpreting results and drawing conclusions
Chapter 21: Qualitative Evidence
Conducting systematic reviews of intervention questions III: Synthesizing data from intervention studies using meta-analysis. O’Connor AM, Sargeant JM, Wang C. Zoonoses Public Health. 2014 Jun;61 Suppl 1:52-63. doi: 10.1111/zph.12123. PMID: 24905996
Meta-analyses including data from observational studies. O’Connor AM, Sargeant JM. Prev Vet Med. 2014 Feb 15;113(3):313-22. doi: 10.1016/j.prevetmed.2013.10.017. Epub 2013 Oct 31. PMID: 24268538
Conducting systematic reviews of intervention questions II: Relevance screening, data extraction, assessing risk of bias, presenting the results and interpreting the findings. Sargeant JM, O’Connor AM. Zoonoses Public Health. 2014 Jun;61 Suppl 1:39-51. doi: 10.1111/zph.12124. PMID: 24905995
C59. Addressing risk of bias / study quality in the synthesis (review / final manuscript)
C60. Incorporating assessments of risk of bias (review / final manuscript)
C61. Combining different scales (review / final manuscript)
C62. Ensuring meta-analyses are meaningful (review / final manuscript)
C63. Assessing statistical heterogeneity (protocol & review / final manuscript)
C64. Addressing missing outcome data (review / final manuscript)
C65. Addressing skewed data (review / final manuscript)
C66. Addressing studies with more than two groups (protocol & review / final manuscript)
C67. Comparing subgroups (protocol & review / final manuscript)
C68. Interpreting subgroup analyses (protocol & review / final manuscript)
C69. Considering statistical heterogeneity when interpreting the results (review / final manuscript)
C70. Addressing non-standard designs (protocol & review / final manuscript)
C71. Conducting sensitivity analysis (protocol & review / final manuscript)
C72. Interpreting results (review / final manuscript)
C73. Investigating reporting biases (review / final manuscript)
C77. Formulating implications for practice (review / final manuscript)
C78. Avoiding recommendations (review / final manuscript)
C79. Formulating implications for research (review / final manuscript)
CEE Standards for conduct and reporting
9.1 Systematic Reviews
9.1.1 Narrative Synthesis
9.1.2 Quantitative Data Synthesis
9.1.3 Qualitative Data Synthesis
10.1 The interpretation of evidence syntheses
10.2 Reporting conduct of evidence synthesis
10.3 Reporting findings of evidence syntheses
If quantitative synthesis is not appropriate, describe the type of summary planned (Item 15d)
all of the above plus:
Describe criteria under which study data will be quantitatively synthesised (Item 15a)...quantitative synthesis, describe planned summary measures, methods of handling data and methods of combining data from studies, including any planned exploration of consistency (such as I2 , Kendall’s τ) (Item 15b)...describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression) (Item 15c)
Essential Items
Additional Items
all of the above plus:
If a random-effects meta-analysis model was used, consider specifying other details about the methods used, such as the method for calculating confidence limits for the heterogeneity variance (Item 13d)
all of the above plus: