Systematic Reviews and Meta-Analyses: Protocol
A protocol developed before starting the review is a defining feature of the systematic review and/or meta-analysis.
Reporting Guideline for Protocol Development
PRISMA-P is a reporting guideline for developing the protocol. It should be used in conjunction with a systematic review and/or meta-analysis reporting guideline (e.g., PRISMA). However, other systematic review and/or meta-analysis reporting guidelines can also support the development of your protocol (e.g., PRISMA, NIRO, ROSES).
It is important to know where you are headed, so a good first step is becoming familiar with each stage (e.g., tab in this guide, section in a handbook) of the systematic review before drafting the protocol.
About Protocols
Why start with a protocol?
Defining the protocol at the start of your review is one of the mechanisms used to reduce risk of bias. Think of the protocol as a map to your review journey - outlining the path you'll take before you begin will reduce the need for ad hoc decision-making mid-review. Ad hoc decision making is an invitation for bias!
Joanna Briggs Institute (JBI) Manual for Evidence Synthesis
An essential step in the early development of a systematic review is the development of a review protocol. A protocol pre-defines the objectives and methods of the systematic review which allows transparency of the process which in turns allows the reader to see how the findings and recommendations were arrived at. It must be done prior to conducting the systematic review as it is important in restricting the presence of reporting bias. The protocol is a completely separate document to the systematic review report.
Without review protocols, how can we be assured that decisions made during the research process aren’t arbitrary, or that the decision to include/exclude studies/data in a review aren’t made in light of knowledge about individual study findings?...Given their costly time and resource requirements, we simply cannot afford wasted efforts when it comes to systematic reviews.
In the context of systematic reviews and meta-analyses, a protocol is a document that presents an explicit scientific “road map” of a planned, uninitiated systematic review. The protocol details the rational and planned methodological and analytical approach of the review.
Develop the protocol before conducting review
According to PRISMA-P:
Arguably one of the most important functions of systematic review protocols is their role as a documentation of planned review methods, outcomes, and analyses that can be compared with completed reviews to detect whether unintended and undocumented changes were made. Bias related to selective reporting of outcomes (that is, when reporting is related to the statistical significance or direction of effect estimate) is a problem in clinical research...When reviewers selectively choose which information to include in a report based on the direction and significance of findings, they risk biasing the evidence base on which healthcare decisions and policies are made.
Although this excerpt mentions clinical research, research in any field is susceptible to selective reporting bias. The development of a protocol before beginning your review will help to hold your team accountable to what you set out to do. It is one of several measures taken to reduce the risk of biasing results from a systematic review and/or meta-analysis.
What goes in a protocol?
The PRISMA-P elaboration and explanation document details the items that should be addressed in a systematic review and/or meta-analysis protocol. These items are addressed throughout this library guide, as applicable.
Pilot the protocol
There are several approaches to piloting your review protocol. Regardless of how it's done, it is a vital part of the process. Piloting helps a team identify and address issues early on so that these issues don't cause a more serious problem (like having to redo a whole stage of the review) down the road.
Basic approaches
Your team may choose to pilot all of the stages of the review, from search to synthesis, before finalizing and registering the protocol. This process is described in Long, 2014 "Routine piloting in systematic reviews--a modified approach?"
At a minimum, your team should pilot each stage as you go through the review itself. For example, during the eligibility screening stage, start by having the two reviewers review only 5-10% of the unique references from your search. Calculate the interrater reliability. A low score indicates a lack of consistency between reviewers, which may mean the eligibility criteria needs clarification or to be refined. The pilot concludes when reviewers are screening articles with relative consistency and confidence in applying eligibility criteria.
Why register?
One of the first steps in your systematic review and/or meta-analysis journey was to identify in-progress reviews. Registering your protocol allows others to know that your review is underway. Registering will also help to keep your team accountable by having a public record of what you aim to accomplish and how.
Some journals use a pre-registration peer-review model, meaning you can receive feedback before starting the review by first registering your protocol. In some cases, if a journal publishes your protocol and your team does not deviate beyond reason, the journal will also publish the final systematic review and/or meta-analysis.
Where to register?
Submit for approval
- Campbell Systematic Reviews Journal | Social Sciences, systematic reviews and/or meta-analysis, scoping/mapping reviews; focused on intervention-based research & reviews with stakeholder engagement
- PROCEED | Environmental science, evidence synthesis focused
- PROSPERO | Health, systematic review and/or meta-analysis, restricted/rapid reviews focused
- Research Registry (Systematic Reviews/Meta-Analyses) | Health & medicine
No approval required
- Figshare | general-purpose repository
- Open Science Framework (OSF) | general-purpose repository
- VTechWorks | Virginia Tech's Institutional Repository (or another institutional repository), general-purpose
Deviation from the protocol
Deviation from the protocol is expected! But your team should only deviate from the protocol when it is necessary.
Deviations mid-review can be avoided by piloting first. If you are already mid-review, but need to deviate from your protocol, be sure to document all changes (and justification for those changes) thoroughly, and include a report (PRISMA-P recommends a chart or table format) of all deviations in the final published protocol.
The protocol can be a peer reviewed publication in and of itself, for example, as is with the Aetiology and outcome of non-traumatic coma in African children: protocol for a systematic review and meta-analysis. It contains all of the sections (e.g., background, methods, funding) required for a reader to understand the purpose of the review and the expected conduct of the team.
Protocols that have undergone peer review should also be accompanied by documentation of this process, including links to other relevant documents, as is the case in this example (below).
Methodological Guidance
SYREAF Tutorials
Step 1: Developing a protocol
Conducting systematic reviews of intervention questions I: Writing the review protocol, formulating the question and searching the literature. O’Connor AM, Anderson KM, Goodell CK, Sargeant JM. Zoonoses Public Health. 2014 Jun;61 Suppl 1:28-38. doi: 10.1111/zph.12125. PMID: 24905994
Campbell - MECCIR
MECCIR guidance describes all required and recommend items to be addressed in the Title (T), Protocol (P), and Review (R). These items are referenced throughout this guide where relevant.
To submit a Campbell review, authors are required, first, to register a title, then submit a protocol before submitting the final manuscript.
CEE - Guidelines and Standards for Evidence synthesis in Environmental Management
Section 4. Writing and registering a Protocol
4.1 Purpose of the Protocol
4.2 Developing and writing a protocol
Reporting in Final Manuscript
In the Final Manuscript | PRISMA
Registration and Protocol
Essential Items
- Provide registration information for the review, including register name and registration number, or state that the review was not registered (Item 24a)
- Indicate where the review protocol can be accessed (such as by providing a citation, DOI, or link) or state that a protocol was not prepared (Item 24b)
- Report details of any amendments to information provided at registration or in the protocol, noting: (a) the amendment itself, (b) the reason for the amendment, and (c) the stage of the review process at which the amendment was implemented (Item 24c)