Systematic Reviews and Meta-Analyses: Assess Certainty
A final step of the systematic review and/or meta-analysis is to assess the certainty of evidence purported by your review.
Assess Certainty of Evidence
Assess Certainty of Evidence
This stage shares some characteristics with the critical appraisal phase, but is a separate, more robust appraisal and is described as its own phase in the systematic review. Sometimes this phase is also called referred to as assessing strength of recommendations, quality of evidence, or some combination thereof.
In contrast to the Critical Appraisal
In the critical appraisal stage of the review, you're assessing the risk of bias in each included study. This phase, the assessment of certainty of evidence, takes the critical appraisal a step further by considering the overall quality or reliability of evidence purported by your review. As a result, your team will identify whether the evidence is strong or weak, or some variation of this. The ultimate intention of this stage is to translate the findings of your review to action.
Why Assess Certainty of Evidence?
Reflecting on the importance of this stage, the GRADE Working Group (a health-focused initiative) state that "Systematic reviews of the effects of healthcare provide essential, but not sufficient information for making well informed decisions. Reviewers and people who use reviews draw conclusions about the quality of the evidence, either implicitly or explicitly. Such judgments guide subsequent decisions. For example, clinical actions are likely to differ depending on whether one concludes that the evidence that blood thinners reduces the risk of stroke in patients with atrial fibrillation is convincing (high quality) or that it is unconvincing (low quality)."
Tools for Assessing Certainty of Evidence - GRADE
The GRADE approach (Grading of Recommendations, Assessment, Development and Evaluation) is a method of grading the quality of evidence and the strength of recommendations in guidelines. This approach is being used by many international organisations to produce rigorous and transparent clinical practice guidelines and other health care recommendations.
Despite having been developed for health sciences in particular, GRADE is the most commonly referenced and used tool for assessing quality of evidence in systematic reviews and/or meta-analyses.
Is GRADE Right for your review?
The GRADE Working Group "discourage the use of 'modified GRADE approaches' that differ from the approach described by the GRADE Working Group." With this in mind, they propose authors meet the following criteria to use GRADE:
- "The certainty in the evidence...should be defined consistently with the definitions used by the GRADE Working Group.
- The strength of recommendations should be assessed using two categories (for or against an option)...
- Explicit consideration should be given to each of the GRADE domains...
- The overall certainty in the evidence should be assessed for each important outcome using four or three categories (such as high, moderate, low and/or very low)...
- Evidence summaries and evidence to decision criteria should be used as the basis for judgments about the certainty...and these should be based on systematic reviews...evidence that was assessed and the methods that were used to identify and appraise that evidence should be clearly described.
- Explicit consideration should be given to each of the GRADE criteria for determining the direction and strength of a recommendation or decision..."
Qualitative evidence
If your evidence is qualitative, you may consider using the GRADE CERQual tool. Conceptually, this tool is similar to the standard GRADE, but is built to address qualitative evidence in particular.
Using GRADE
The GRADEPro GDT web-application is available to help you conduct this analysis.
Other Tools
GRADE is not the only option, though it is most commonly referenced in the context of assessing certainty of evidence in systematic reviews and/or meta-analyses. There are many, very specific tools available to assess certainty of evidence. For example, the Strength of Recommendation Taxonomy (SORT): A Patient-Centered Approach to Grading Evidence in the Medical Literature.
If there is not a tool appropriate for your review, you should still reflect on the certainty of evidence through less formal means.
Methodological Guidance
Cochrane Handbook - Part 2: Core Methods
Chapter 14: Completing 'Summary of findings' tables and grading the certainty of evidence
Reporting in Protocol and Final Manuscript
In the Protocol | PRISMA-P
Specify any planned assessment of meta-bias(es)...(Item 16)
...pre-specify any methods used to explore the possibility that the data identified are biased due to non-study related processes. Such bias may result from non-publication of studies (publication or dissemination bias) and the reporting of a subset of measured outcomes and analyses within studies (outcome reporting bias)...both publication bias and outcome reporting bias may affect meta-analyses, and the effect can be unpredictable
Describe how the strength of the body of evidence will be assessed... (Item 17)
...summarize the confidence they have in the resulting body of evidence, ideally using an established and validated approach... plan for assessing the risk of bias across studies, inconsistency, imprecision, indirectness, publication bias, and factors that increase the confidence in an effect... for each outcome...in the PICO. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach is increasingly recommended.
In the Final Manuscript | PRISMA
Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed (Item 22)
Essential Items:
- Report the overall level of certainty in the body of evidence (such as high, moderate, low, or very low) for each important outcome
- Provide an explanation of reasons for rating down (or rating up) the certainty of evidence (such as in footnotes to an evidence summary table). Explanations for each judgment should be concise, informative, relevant to the target audience, easy to understand, and accurate (that is, addressing criteria specified in the methods guidance).
- Communicate certainty in the evidence wherever results are reported (that is, abstract, evidence summary tables, results, conclusions). Use a format appropriate for the section of the review. For example, in text, certainty might be reported explicitly in a sentence (such as “Moderate-certainty evidence (downgraded for bias) indicates that...”) or in brackets alongside an effect estimate (such as “[RR 1.17, 95% CI 0.81 to 1.68; 4 studies, 1781 participants; moderate certainty evidence]”). When interpreting results in “summary of findings” tables or conclusions, certainty might be communicated implicitly using standard phrases (such as “Hip protectors probably reduce the risk of hip fracture slightly”)
Additional Items:
- Consider including evidence summary tables, such as GRADE Summary of Findings tables.