Systematic Reviews and Meta-Analyses: Grey Literature
This page provides more information about grey literature.
Grey Literature
Grey Literature
Grey Literature is a broad term that varies across discipline. In very general terms, grey literature is anything other than peer-reviewed, empirical research. Examples of grey literature include: unpublished research, government publications, theses and dissertations, statistics and datasets, news articles, trade and popular magazines, blogs, podcasts, social media posts (e.g., tweets), slide decks, or factsheets. Including grey literature in your systematic review can help mitigate publication bias.
According to GreyNet, 2013:
Grey Literature...deals with the production, distribution, and access to multiple document types produced on all levels of government, academics, business, and organization in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.
According to Paez, 2017:
Publication bias refers to the propensity for only studies reporting positive findings to be published, and may skew the results of the meta-analysis and systematic review. As gray literature may describe neutral or negative results, including it with commercially published research may provide a more balanced understanding of the evidence and a more accurate effect sizes...excluding unpublished studies may compromise the validity and reliability of meta-analyses and the specificity of systematic reviews.
Should you include grey literature?
Can a type (or several types) of grey literature help to answer your research question? Do you have time for additional searches and processing additional material? If so, you should consider including grey literature in your review!
However, not all questions will benefit from grey literature and you cannot include all kinds of grey literature.
Wilson (2009) states:
"Ignoring the grey literature on the basis that it is of inferior methodological quality is empirically and logically invalid. Furthermore, the use of publication status as a proxy measure for methodological quality abdicates the important role of carefully evaluating the robustness of the evidence to others using unknown criteria (i.e., the journal peer-review process)."
Resources
Ćurković M, Košec A. Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility. BMC Med Res Methodol. 2018 Nov 13;18(1):130. doi: 10.1186/s12874-018-0599-2. PMID: 30424741; PMCID: PMC6234590.
Godin, K., Stapleton, J., Kirkpatrick, S.I. et al. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev 4, 138 (2015). https://doi.org/10.1186/s13643-015-0125-0
Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods. 2020 Mar;11(2):181-217. doi: 10.1002/jrsm.1378. Epub 2020 Jan 28. PMID: 31614060; PMCID: PMC7079055
Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits: MAHOOD ET AL . Research Synthesis Methods, 5(3), 221–234. https://doi.org/10.1002/jrsm.1106
Paez A. Grey literature: An important resource in systematic reviews. J Evid Based Med. 2017 Dec 21. doi: 10.1111/jebm.12265. Epub ahead of print. PMID: 29266844.
Check out Appendix I for an annotated list of grey lit sources!
Scherer, R.W., Saldanha, I.J. How should systematic reviewers handle conference abstracts? A view from the trenches. Syst Rev 8, 264 (2019). https://doi-org.ezproxy.lib.vt.edu/10.1186/s13643-019-1188-0
Wilson, D.B. Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. J Exp Criminol 5, 429–440 (2009). https://doi-org.ezproxy.lib.vt.edu/10.1007/s11292-009-9085-5
Conference Proceedings
Conference proceedings are some of the most common types of grey literature to consider in systematic reviews. Many of the databases you'll search will include conference proceedings. But there are databases that contain only conference proceedings such as ProceedingsFirst, as well as databases containing the conference proceedings for a specific conference organization (which may require affiliation to access). If you plan to include a specific type of grey literature, like conference proceedings, it is not enough to include what you happen to find through your primary search. You should also search purposefully for this type of literature by targeting focused databases, just as you did when searching for peer-reviewed empirical research.
According to PRISMA-S, Item 4:
Including conference proceedings in a systematic review search helps minimize bias...specify the conference names, the dates of conferences included, and the method used to search the proceedings...
Scherer & Saldanha, 2019 offer a more pragmatic approach, suggesting reviewers consider their capacity and corpus:
If available evidence is sparse or conflicting, it may be worthwhile to include conference abstracts. If results from conference abstracts are included, then it is necessary to make diligent attempts to contact the authors of the abstract and examine study registers and published protocols to obtain further and confirmatory information on methods and results.
This is important as, in a Cochrane Network Meta-Analysis, Hackenbroich, Kranke, Meybohm, & Weibel (2022) found:
Most conference abstracts on PONV were insufficiently reported regarding review's narrow inclusion criteria and could not be included in NMA. The resource-intensive search and evaluation of abstracts did not substantially extent the full-text evidence base of the review, given the few adequately reported abstracts. Conferences should oblige authors to adhere to CONSORT for abstracts.