Systematic Reviews and Meta-Analyses: Where to Search

Study Registries & Preprint Archives

For a comprehensive search, it's important to locate and include unpublished to the extent possible in order to reduce the risk of missing important emerging information and to avoid risk of publication bias.

Study registries contain unpublished research, which may (or may not) be considered a type of grey literature in your field. Research may be unpublished for several reasons - for example, the research is still in progress or not ready to publish. Sometimes research is unpublished for other reasons, such as negative or null results, or the study was not considered novel (e.g., replications). 

Some health-based study (and clinical trial) registeres include:

• clinicaltrials.gov from the US National Libraries of Medicine 
• International Clinical Trials Registry Platform (ICTP) from the World Health Organization
• ISRCTN Registry from BCM Springer Nature
• Australian Clinical Trials from the Australian Government

Preprints are studies that have not yet been peer-reviewed and/or published by a journal. Preprints exist to get research into the community more quickly; much of this material ends up going through the formal peer review and publishing process. Note: it is important to match preprints with their subsequent publication(s) and to not duplicate findings in your synthesis. 

Some archives to explore include:

• arxiv
• bioRxiv
• socarixv

Online Resources

It is best practice to supplement primary searching (done in academic databases) by searching other online resources and web browsing. This kind of searching may result in additional primary research and/or grey literature.

Web search engines and specific websites

While general search engines like Google should never be the primary resource for your search, it is a great way to supplement the results of academic database searches. These searches may yield primary research and grey literature.

Companies like Google tailor search results to individual users, which increase the risk of introducing bias, but there are ways to reduce this risk. For example, log out of your account, use 'incognito' mode (or similar), and/or opt for a search engine that doesn't track your data like DuckDuckGo or StartPage.

In contrast to searches in academic databases, it is more often appropriate to apply artificial limits in web searches, as yields tend to be unmanageable. For example, a team may only review the first 200 records (or 20 pages). Search strings themselves may also need to be refined to accommodate character limits in both general search engines and specific websites. Exactly how to search specific websites will vary. Note that you can also use general search engines to search within specific websites or even types of websites (e.g., government sites by specifying a .gov url).

Resources for web browsing

• Google Scholar Search Tips
• Bulk Downloading from Google Scholar 
• Google Syntax Tips (Sheridan College Library Guide)
• Google Custom/Programmable Search (instructions from WorldBank/IMF Library Guide)
• Set Google Alerts
• Alternatives to Google (that don't track your data):
  • StartPage
  • DuckDuckGo
  • MillionShort (can remove top hits to find what you're missing)
• Haddaway NR, Collins AM, Coughlin D, Kirk S (2015) The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS ONE 10(9): e0138237. https://doi-org.ezproxy.lib.vt.edu/10.1371/journal.pone.0138237 

• Piasecki, J., Waligora, M., & Dranseika, V. (2018). Google Search as an Additional Source in Systematic Reviews. Science and engineering ethics, 24(2), 809–810. https://doi.org/10.1007/s11948-017-0010-4 

Citation Searching

Citation searching, also referred to as pearl growing, snowballing, and citation chasing, chaining, or tracking, describes two processes: 

Backward citation chasing, where reviewers scan work cited by included references and other relevant reviews, and

Forward citation chasing, where reviewers search for references that cite the references included in your review and other relevant reviews.

In a comprehensive search, we often find existing reviews (e.g., traditional reviews, scoping reviews, restricted/rapid reviews) on the same topic and/or other systematic reviews on similar topics. Though it is not appropriate to synthesize the reviews themselves in your review, be sure to use them in your citation search.

Citation searching applied to included references (references your team has already identified as relevant).  As such, this step does not take place until after reviewing the full-Text of included references.

How to citation search?

Citation searching can be done manually (e.g., reviewing full text) and by using bibliographic databases that index both citations and standard bibliographic data. For example, Web of Science Core Collection provided by Clarivate Analytics, Scopus provided by Elsevier, and Google Scholar (for forward citation searching only).

There are also semi-automated tools available to support this stage, such as Citation Chaser and SpiderCite.

Personal Contacts

Contacting relevant authors and other stakeholders can occur during the:

Comprehensive searching phase to collect unpublished research or other grey literature, and

Eligibility screening, critical appraisal, and data extraction phases as follow-up when there is not enough information provided in an original publication to determine relevance, appraise quality, or extract data properly.

Also consider contacting groups, organizations, committees, listservs, etc. 

Example from: Curry, O. S., Rowland, L., Van Lissa, C. J., Zlotowitz, S., McAlaney, J., PhD, & Whitehouse, H. (2019, October 1). Happy to Help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor. Retrieved from osf.io/sey6x, Appendix https://osf.io/6pgzc/

Predatory Journals

"Predatory journals-also called fraudulent, deceptive, or pseudo-journals-are publications that claim to be legitimate scholarly journals but misrepresent their publishing practices. Some common forms of predatory publishing practices include falsely claiming to provide peer review, hiding information about article processing charges, misrepresenting members of the journal's editorial board, and other violations of copyright or scholarly ethics." (Elmore & Wetson, 2020) 

For more about predatory journals, check out the GRAD 5124 topic 11.4 Issues in Academic Publishing.

Predatory Journals in Systematic Reviews 

Given the description above, it's no surprise that whether or not to include and how to include publications from predatory publishers in systematic reviews and/or meta-analyses. However, the poor practices of a publisher does not necessarily mean the research published there was done poorly. There have been a few recommendations set forth over the years.

Poliani, et al. (2020) recommend to:

1. determine whether the journal or publisher is part of COPE, DOAJ, OASP, INASP Journals Online Platform, STM (global voice of scholarly publishing), and
2. critically appraise reports from uncertain journals or publishers using a tool like Think, Check, Submit

According to Munn, et al., (2021), "Options for systematic reviewers could include excluding all studies from suspected predatory journals, applying additional strategies to forensically examine the results of studies published in suspected predatory journals, setting stringent search limits, and applying analytical techniques (such as subgroup or sensitivity analyses) to investigate the impact of suspected predatory journals in a synthesis."

And Rice, et al. (2021) suggest to include "(1) detail methods for addressing predatory journal articles a priori in a study protocol, (2) determine whether included studies are published in open access journals and if they are listed in the directory of open access journals, and (3) conduct a sensitivity analysis with predatory papers excluded from the synthesis."

Resources

Elmore SA, Weston EH. Predatory Journals: What They Are and How to Avoid Them. Toxicol Pathol. 2020 Jun;48(4):607-610. doi: 10.1177/0192623320920209. Epub 2020 Apr 22. PMID: 32319351; PMCID: PMC7237319.

John D, Polani Chandrasekar R, Lohmann J, Dazy A. Identifying predatory journals in systematic reviews. In: Advances in Evidence Synthesis: special issue Cochrane Database of Systematic Reviews 2020;(9 Suppl 1). https://doi.org/10.1002/14651858.CD202001

Munn Z, Barker T, Stern C, Pollock D, Ross-White A, Klugar M, Wiechula R, Aromataris E, Shamseer L. Should I include studies from "predatory" journals in a systematic review? Interim guidance for systematic reviewers. JBI Evid Synth. 2021 Jun 28;19(8):1915-1923. doi: 10.11124/JBIES-21-00138. PMID: 34171895.

Rice, D.B., Skidmore, B. & Cobey, K.D. Dealing with predatory journal articles captured in systematic reviews. Syst Rev 10, 175 (2021). https://doi-org.ezproxy.lib.vt.edu/10.1186/s13643-021-01733-2