VT Logo
VT Logo
Cozette Comer, Evidence Synthesis Librarian, Liaison Librarian: Statistics and Computational Modeling & Data Analytics, cozette@vt.edu
Kiri DeBose, Head, Veterinary Medicine Library & Liaison to Animal Sciences, kdebose@vt.edu
Ginny Pannabecker, Liaison Librarian: Biochemistry, Biocomplexity Institute, Biological Sciences, Biomedical Engineering and Mechanics, Neuroscience, and Systems Biology; Director, RCE, vpannabe@vt.edu
The critical appraisal process evaluates studies based on quality measures specific to your research question components, the discipline/s related to your research topic, and those specific to the study design, outcome measures, methodology, data analysis, and reporting methods that relate to your research question and identified outcomes.
During the critical appraisal process, you will be assessing each study's risk of bias. See many of the links below to tools often used to critically appraise research studies.
Critical Appraisal Planning:
This is the newly revised Cochrane Collaboration Risk of Bias tool, released October 2018. The Risk of Bias tool developed by the Cochrane Collaboration evaluates studies in several domains, including selection bias, performance bias, attrition bias, reporting bias, etc. It ONLY can be used for randomized control trials (RCTs). More information on this tool can be located here:
Center for Evidence-Based Medicine Critical Appraisal Tools
Mixed Methods Appraisal Tools
Non-Randomized Studies
The Newcastle-Ottawa Scale for Assessing the Quality of Nonrandomized Studies
Subject/Discipline-based tools
Meta Quality Appraisal Tool (MetaQAT)
SYRCLE's checklist for Animal Studies
This website contains a plethora of biases that you may want to consider for your systematic review. Remember, you don't have to use an established tool if it doesn't work for you. Try creating your own!
Here's a paper that evaluates 18 critical appraisal tools that are available for researchers to use.
Abstract
Background: Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.
Methods: We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.
Results: We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.
Conclusion: There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgments and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.
© 2017 Virginia Polytechnic Institute and State University
University Libraries (0434)
Carol M. Newman Library, Virginia Tech
560 Drillfield Drive
Blacksburg, VA 24061
(540) 231-6170
