Systematic Reviews and Meta-Analyses: Critical Appraisal
All relevant studies must undergo a critical appraisal to evaluate the risk of bias, or internal and external validity, of all relevant references.
Critical appraisal is also referred to as quality assessment, risk of bias assessment, and similar variations. Sometimes the critical appraisal phase is confused with the assessment of certainty of evidence - although related, these are independent stages of the systematic review process.
According to the Center for Evidence-Based Medicine (CEBM):
"Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability and relevance."
Systematic reviews require a formal, systematic, uniform appraisal of the quality - or risk of bias - of all relevant studies. In a critical appraisal, you are examining the methods not the results.
Use risk of bias tools for this stage - these tools are often formatted as checklists. You can find more about risk of bias tools in the next tab! If a refresher of some common biases, definitions, and examples is helpful, check out the Catalogue of Bias from the University of Oxford and CEBM.
Just like the other stages of a systematic review, 2 reviewers should assess risk of bias in each reference. As such, your team should calculate and report interrater reliability, deciding ahead of time how to resolve conflicts. Oftentimes the critical appraisal occurs at the same time as data extraction.
In addition to the formal risk of bias assessment, your team should also consider meta-biases like publication bias, selective reporting, etc. Search for errata and retractions related to included research, and consider other limitations of and concerns about the included studies and how this may impact the reliability of your review.
Note: Subjectivity of Critical Appraisal
The critical appraisal is inherently subjective, from the selection of the RoB tool(s) to the final assessment of each study. Therefore, it is important to consider how tools compare, and how this process may impact the results of your review. Check out these studies evaluating Risk of Bias Tools:
Page MJ, McKenzie JE, Higgins JPT Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review BMJ Open 2018;8:e019703. doi: 10.1136/bmjopen-2017-019703
Losilla, J.-M., Oliveras, I., Marin-Garcia, J. A., & Vives, J. (2018). Three risk of bias tools lead to opposite conclusions in observational research synthesis. Journal of Clinical Epidemiology, 101, 61–72. https://doi.org/10.1016/j.jclinepi.2018.05.021
Margulis, A. V., Pladevall, M., Riera-Guardia, N., Varas-Lorenzo, C., Hazell, L., Berkman, N., Viswanathan, M., & Perez-Gutthann, S. (2014). Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: The Newcastle-Ottawa Scale and the RTI item bank. Clinical Epidemiology, 359. https://doi.org/10.2147/CLEP.S66677
Select Risk of Bias Tool(s)
When you think of a critical appraisal in a systematic review and/or meta-analysis, think of assessing the risk of bias of included studies. The potential biases to consider will vary by study design. Therefore, risk of bias tool(s) should be selected based on the designs of included studies. If you include more than one study design, you'll include more than one risk of bias tool. Whenever possible, select tools developed for a discipline relevant to your topic.
Risk of Bias Tools
Risk of bias tools are simply checklists used to consider bias specific to a study design, and sometimes discipline.
- Cochrane Risk of Bias Tool | randomized trials, health
- Collaboration for Environmental Evidence (CEE) Critical Appraisal Tool, Prototype | environmental management focused
- Crowe Critical Appraisal Tool | mixed methods
- Meta-QAT | public health focused
- Meta-QAT Grey Literature Companion | grey literature
- Mixed Method Appraisal Tool (MMAT) | mixed method (more detail)
- Newcastle-Ottawa Scale | non-randomized studies
- RTI Item Bank | observational studies
- SYRCLE's Risk of Bias Tool | animal studies
- Quality Checklist for Blogs | blogs
- Quality Checklist for Podcasts | podcasts
Risk of Bias Toolsets
Risk of bias toolsets are a series of tools developed by the same group or organization, where each tool addresses a specific study design. The organization is usually discipline specific. Note that many also include a systematic review and/or meta-analysis quality assessment tool, but that these tools will not be useful during this stage as existing reviews will not be folded into your synthesis.
Critical Appraisal Skills Programme (CASP) Checklists include tools for:
- Randomized Controlled Trials
- Qualitative Studies
- Cohort Study
- Diagnostic Study
- Case Control Study
- Economic Evaluation
- Clinical Prediction Rule
National Institutes of Health (NIH) Study Quality Assessment Tools include tools for:
- Controlled intervention studies
- Observational cohort and cross-sectional studies
- Case-control studies
- Before-after (pre-post) studies without control
- Case series studies
Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) includes tools for:
- Conference abstracts
Joanna Briggs Institute (JBI) Manual for Evidence Synthesis includes the following tools found in respective relevant chapters:
- Qualitative research (appendix 2.1)
- Randomized controlled trials (appendix 3.1)
- Quasi-experimental studies (non-randomized experimental studies; appendix 3.3)
- Text and opinion (appendix 4.1) with explanation (Appendix 4.2)
- Prevalence studies (appendix 5.1)
- Cohort studies (appendix 7.1)
- Case-control studies (appendix 7.2)
- Case series (appendix 7.3)
- Case reports (appendix 7.4)
- Cross sectional studies (appendix 7.5)
- Diagnostic test accuracy (appendix 9.1)
Risk of Bias Tool Repositories
Risk of bias tool repositories are curated lists of existing tools - kind of like what we've presented above. Although we update this guide with new tools as we find them, these repositories may contain additional resources:
- Quality Assessment and Risk of Bias Tool Repository, from Duke University's Medical Center Library & Archives
- Interactive Tableau Dataset of 68 Risk of Bias Tools, from the National Toxicology Program
Presenting Critical Appraisal Results
Risk of bias within each reference should be presented in a table like the one seen below. Studies are presented along the y-axis and biases considered (what is addressed by the tool) along the x-axis, such that each row belongs to a study, and each column belongs to a bias (or domain/category of biases).
It is also best practice to present the bias across the included set of literature (seen below). Each bias or bias category is represented as a row and each row is associated with a bar showing the percentage of the total included literature that was rated as low risk, some risk, high risk, or unable to determine the risk.
The images above can be created using the ROBVIS package of metaverse for evidence synthesis in R. You can create your own graphics without using this software.
Chapter 7: Considering bias and conflicts of interest among the included studies
- 7.2 Empirical evidence of bias
- 7.3 General procedures for risk-of-bias assessment
- 7.4 Presentation of assessment of risk of bias
- 7.5 Summary assessments of risk of bias
- 7.6 Incorporating assessment of risk of bias into analyses
- 7.7 Considering risk of bias due to missing results
- 7.8 Considering source of funding and conflict of interest of authors of included studies
Chapter 8: Assessing risk of bias in randomized trial
- 8.2 Overview of RoB 2
- 8.3 Bias arising from the randomization process
- 8.4 Bias due to deviations from intended interventions
- 8.5 Bias due to missing outcome data
- 8.6 Bias in measurement of the outcome
- 8.7 Bias in selection of the reported result
- 8.8 Differences from the previous version of the tool
Chapter 25: Risk of bias in non-randomized studies
Step 3: Identifying eligible papers
Conducting systematic reviews of intervention questions II: Relevance screening, data extraction, assessing risk of bias, presenting the results and interpreting the findings. Sargeant JM, O’Connor AM. Zoonoses Public Health. 2014 Jun;61 Suppl 1:39-51. doi: 10.1111/zph.12124. PMID: 24905995
C51. Assessing risk of bias / study quality (protocol & review / final manuscript)
C52. Assessing risk of bias / study quality in duplicate (protocol & review / final manuscript)
C53. Supporting judgements of risk of bias / study quality (review / final manuscript)
C54. Providing sources of information for risk of bias / study quality assessments (review / final manuscript)
C55. Differentiating between performance bias and detection bias (protocol & review / final manuscript)
C56. If applicable, assessing risk of bias due to lack of blinding for different outcomes (review / final manuscript)
C57. If applicable, assessing completeness of data for different outcomes (review / final manuscript)
C58. If applicable, summarizing risk of bias when using the Cochrane Risk of Bias tool (review / final manuscript)
C59. Addressing risk of bias / study quality in the synthesis (review / final manuscript)
C60. Incorporating assessments of risk of bias (review / final manuscript)
Reporting in Protocol and Final Manuscript
In the Protocol | PRISMA-P
Risk of Bias Individual Studies (Item 14)
...planned approach to assessing risk of bias should include the constructs being assessed and a definition for each, reviewer judgment options (high, low, unclear), the number of assessors...training, piloting, previous risk of bias assessment experience...method(s) of assessment (independent or in duplicate)...
Protocol for reporting results
"...summarise risk of bias assessments across studies or outcomes..."
Protocol for reporting impact on synthesis
"...describe how risk of bias assessments will be incorporated into data synthesis (that is, subgroup or sensitivity analyses) and their potential influence on findings of the review (Item 15c) in the protocol..."
In the Final Manuscript | PRISMA
For the critical appraisal stage, PRISMA requires specific items to be addressed in both the methods and results section.
Study Risk of Bias Assessment (Item 11; report in methods)
- Specify the tool(s) (and version) used to assess risk of bias in the included studies.
- Specify the methodological domains/components/items of the risk of bias tool(s) used.
- Report whether an overall risk of bias judgment that summarised across domains/components/items was made, and if so, what rules were used to reach an overall judgment.
- If any adaptations to an existing tool to assess risk of bias in studies were made (such as omitting or modifying items), specify the adaptations.
- If a new risk of bias tool was developed for use in the review, describe the content of the tool and make it publicly accessible.
- Report how many reviewers assessed risk of bias in each study, whether multiple reviewers worked independently (such as assessments performed by one reviewer and checked by another), and any processes used to resolve disagreements between assessors.
- Report any processes used to obtain or confirm relevant information from study investigators.
- If an automation tool was used to assess risk of bias in studies, report how the automation tool was used (such as machine learning models to extract sentences from articles relevant to risk of bias88), how the tool was trained, and details on the tool’s performance and internal validation
Risk of Bias in Studies (Item 18; report in results)
- Present tables or figures indicating for each study the risk of bias in each domain/component/item assessed and overall study-level risk of bias.
- Present justification for each risk of bias judgment—for example, in the form of relevant quotations from reports of included studies.
If assessments of risk of bias were done for specific outcomes or results in each study, consider displaying risk of bias judgments on a forest plot, next to the study results, so that the limitations of studies contributing to a particular meta-analysis are evident (see Sterne et al86 for an example forest plot).