Systematic Reviews & Meta-analyses: Data Synthesis
& Meta-Analysis

Guidance on conducting systematic reviews and meta-analyses.

Consultations

Request a consultation:

  • Email: srconsultation-g@vt.edu
  • In your email, let us know your current project research focus, planning stage, timeline, and goals. 

Cozette Comer, Evidence Synthesis Librarian, Liaison Librarian: Statistics and Computational Modeling & Data Analytics, cozette@vt.edu

Kiri DeBose, Head, Veterinary Medicine Library & Liaison to Animal Sciences, kdebose@vt.edu

Ginny Pannabecker, Liaison Librarian: Biochemistry, Biocomplexity Institute, Biological Sciences, Biomedical Engineering and Mechanics, Neuroscience, and Systems Biology; Director, RCE, vpannabe@vt.edu

What is a Meta-Analysis?

Chapters 9, 10, 11, and 12 discuss meta-analyses and other synthesis methods.

Cochrane's glossary includes this definition for Meta-analysis:

The use of statistical techniques in a systematic review to integrate the results of included studies. Sometimes misused as a synonym for systematic reviews, where the review includes a meta-analysis. Meta-analysis is the statistical combination of results from two or more separate studies.

This Coursera course from Johns Hopkins University, "Introduction to Systematic Reviews and Meta-Analysis," Module 7, "Planning the Meta-Analysis" gives an overview of common meta-analysis methods.

Data Synthesis Methods and Recommendations

Cochrane Handbook - Key Section

Chapter 9: Preparing for Synthesis

Editors: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA

Key Points:

  • Synthesis is a process of bringing together data from a set of included studies with the aim of drawing conclusions about a body of evidence. This will include synthesis of study characteristics and, potentially, statistical synthesis of study findings.

  • A general framework for synthesis can be used to guide the process of planning the comparisons, preparing for synthesis, undertaking the synthesis, and interpreting and describing the results.

  • Tabulation of study characteristics aids the examination and comparison of PICO elements across studies, facilitates synthesis of these characteristics and grouping of studies for statistical synthesis.

  • Tabulation of extracted data from studies allows assessment of the number of studies contributing to a particular meta-analysis, and helps determine what other statistical synthesis methods might be used if meta-analysis is not possible.

9.1 Introduction

9.2 A general framework for synthesis

9.3 Preliminary steps of a synthesis

9.3.1 Summarize the characteristics of each study (step 2.1)

9.3.2 Determine which studies are similar enough to be grouped within each comparison (step 2.2)

9.3.3 Determine what data are available for synthesis (step 2.3)

9.3.4 Determine if modification to the planned comparisons or outcomes is necessary, or new comparisons are needed (step 2.4)

9.3.5 Synthesize the characteristics of the studies contributing to each comparison (2.5)

9.4 Checking data before synthesis

9.5 Types of synthesis

9.6 Chapter information

9.7 References

Chapter 10: Analysing data and undertaking meta-analysis

Editors: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA

Key Points:

  • Meta-analysis is the statistical combination of results from two or more separate studies.

  • Potential advantages of meta-analyses include an improvement in precision, the ability to answer questions not posed by individual studies, and the opportunity to settle controversies arising from conflicting claims. However, they also have the potential to mislead seriously, particularly if specific study designs, within-study biases, variation across studies, and reporting biases are not carefully considered.

  • It is important to be familiar with the type of data (e.g. dichotomous, continuous) that result from measurement of an outcome in an individual study, and to choose suitable effect measures for comparing intervention groups.

  • Most meta-analysis methods are variations on a weighted average of the effect estimates from the different studies.

  • Studies with no events contribute no information about the risk ratio or odds ratio. For rare events, the Peto method has been observed to be less biased and more powerful than other methods.

  • Variation across studies (heterogeneity) must be considered, although most Cochrane Reviews do not have enough studies to allow for the reliable investigation of its causes. Random-effects meta-analyses allow for heterogeneity by assuming that underlying effects follow a normal distribution, but they must be interpreted carefully. Prediction intervals from random-effects meta-analyses are a useful device for presenting the extent of between-study variation.

  • Many judgements are required in the process of preparing a meta-analysis. Sensitivity analyses should be used to examine whether overall findings are robust to potentially influential decisions.

10.1 Do not start here!

10.2 Introduction to a meta-analysis

10.2.1 Principals of meta-analysis

10.3 A generic inverse-variance approach to meta-analysis

10.3.1 Fixed-effect method for meta-analysis

10.3.2 Random-effects methods for meta-analysis

10.3.3 Performing inverse-variance meta-analyses 

10.4 Meta-analysis of dichotomous outcomes

10.4.1 Mantel-Haenszel methods

10.4.2 Peto odds ratio method

10.4.3 Which effect measure for dichotomous outcomes?

10.4.4 Meta-analysis of rare events

10.5 Meta-analysis of continuous outcomes

10.5.1 Which effect measure for continuous outcomes?

10.5.2 Meta-analysis of change scores

10.5.3 Meta-analysis of skewed data

10.6 Combining dichotomous and continuous outcomes

10.7 Meta-analysis of ordinal outcomes and measurement scales

10.8 Meta-analysis of counts and rates

10.9 Meta-analysis of time-to-event outcomes

10.10 Heterogeneity

10.10.1 What is heterogeneity?

10.10.2 Identifying and measuring heterogeneity

10.10.3 Strategies for addressing heterogeneity

10.10.4 Incorporating heterogeneity into random-effects models

10.11 Investigating

10.11.1 Interaction and effect modification

10.11.2 What are subgroup analyses

10.11.3 Undertaking subgroup analyses

10.11.4 Meta-regression

10.11.5 Selection of study characteristics for subgroup analyses and meta-regression

10.11.6 Interpretation of subgroup analyses and meta-regressions

10.11.7 Investigating the effect of underlying risk

10.11.8 Dose-response analyses

10.12 Missing data

10.12.1 Types of missing data

10.12.2 General principles for dealing with missing data

10.12.3 Dealing with missing outcome data from individual participants 

10.13 Bayesian approaches to meta-analysis

10.14 Sensitivity analysis

10.15 Chapter information

10.16 References 

Chapter 11: Undertaking network meta-analyses

Editors: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA

Key Points:

  • Network meta-analysis is a technique for comparing three or more interventions simultaneously in a single analysis by combining both direct and indirect evidence across a network of studies.

  • Network meta-analysis produces estimates of the relative effects between any pair of interventions in the network, and usually yields more precise estimates than a single direct or indirect estimate. It also allows estimation of the ranking and hierarchy of interventions.

  • A valid network meta-analysis relies on the assumption that the different sets of studies included in the analysis are similar, on average, in all important factors that may affect the relative effects.

  • Incoherence (also called inconsistency) occurs when different sources of information (e.g. direct and indirect) about a particular intervention comparison disagree.

  • Grading confidence in evidence from a network meta-analysis begins by evaluating confidence in each direct comparison. Domain-specific assessments are combined to determine the overall confidence in the evidence.

11.1 What is network meta-analysis?

11.1.1 Network diagrams

11.1.2 Advantages of network meta-analysis

11.1.3 Outline of this chapter

11.2 Important concepts

11.2.1 Indirect comparisons

11.2.2 Transitivity 

11.2.3 Indirect comparisons and the validity of network meta-analysis 

11.3 Planning a Cochrane Review to compare multiple interventions

11.3.1 Expertise required in the review team

11.3.2 The importance of a well-defined research question

11.3.3 Selecting outcomes to examine

11.3.4 Study designs to include

11.4 Synthesis of results

11.4.1 What does a network meta-analysis estimate?

11.4.2 Synthesizing direct and indirect evidence using meta-regression

11.4.3 Performing network meta-analysis 

11.4.4 Disagreement between evidence sources (incoherence)

11.5 Evaluating confidence in the results of a network meta-analysis 

11.5.1 Available approaches for evaluating confidence in the evidence

11.6 Presenting network meta-analyses

11.6.1 Presenting the evidence base of a network meta-analysis

11.6.2 Tabular presentation of the network structure

11.6.3 Presenting the flow of evidence in a network 

11.6.4 Presentation of results

11.7 Concluding remarks

11.8 Chapter information

11.9 References

Chapter 12: Synthesizing and presenting findings using other methods

Editors: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA

Key Points:

  • Meta-analysis of effect estimates has many advantages, but other synthesis methods may need to be considered in the circumstance where there is incompletely reported data in the primary studies.

  • Alternative synthesis methods differ in the completeness of the data they require, the hypotheses they address, and the conclusions and recommendations that can be drawn from their findings.

  • These methods provide more limited information for healthcare decision making than meta-analysis, but may be superior to a narrative description where some results are privileged above others without appropriate justification.

  • Tabulation and visual display of the results should always be presented alongside any synthesis, and are especially important for transparent reporting in reviews without meta-analysis.

  • Alternative synthesis and visual display methods should be planned and specified in the protocol. When writing the review, details of the synthesis methods should be described.

  • Synthesis methods that involve vote counting based on statistical significance have serious limitations and are unacceptable.

12.1 Why a meta-analysis of effect estimates may not be possible

12.2 Statistical synthesis when meta-analysis of effect estimates is not possible

12.2.1 Acceptable synthesis methods

12.2.2 Unacceptable synthesis methods

12.3 Visual display and presentation of the data

12.3.1 Structured tabulation of results across studies

12.3.2 Forest plots

12.3.3 Box-and-whisker plots and bubble plots

12.3.4 Albatross plot

12.3.5 Harvest and effect direction plots

12.4 Worked example

12.4.1 Scenario 1: structured reporting of effects

12.4.2 Overview of scenarios 2-4: synthesis approaches

12.5 Chapter information

12.6 References

 

Data Synthesis - Additional Methods and Resources

What is a Forest Plot? 

In this video, Dr. Terry Shaneyfelt from UAB School of Medicine quickly goes over how to interpret a forest plot: https://www.youtube.com/watch?v=py-L8DvJmDc&t=2s