Evidence Synthesis: Research Question Development Tools
Research Question Development Tools
What is a Concept Developer?
'Concept developers', also called 'research question formulation frameworks' are simply acronyms that help you consider the details behind your research question.
For example, the intervention-focused tool PICO stands for Population, Intervention, Comparator, and Outcome. To use this tool, you would to clearly state the population, intervention, comparator (as it relates to the intervention), and outcome of your research question. Because the systematic review methodology stems from health sciences, PICO is most commonly cited in the context of systematic reviews.
However, there are many options (see the tab labeled "More") to choose from! When in doubt, you can always use the 5Ws & H - who, what, where, when, why, and how. The point of using a concept developer is to consistently and thoroughly define the boundaries of your research question.
The PICO (T) is a widely known strategy for framing an intervention-based research question, particularly in health sciences (Sackett et al.,1997). This tool is also used in other fields (e.g., medicine, social sciences). Sometimes "Intervention" is replace with "Exposure" as PECO alternative.
Worksheet for Developing PICO(T) Questions from the AAACN (American Academy of Ambulatory Care Nursing); permission granted on the PDF file for educational and research uses of the template.
The important thing to take away from PICO is that you need to develop the concepts behind your research question to fully define and illustrate your goals, which will inform every stage of your review. You can include the details defined through these concept builders in the research question directly, but it's not necessary - it is just necessary that you have explicitly stated them.
SPIDER can be a suitable tool for qualitative and mixed methods research in several disciplines. The following table shows the breakdown of the format with an example in qualitative research
Research Question: What are the challenges Australian refugees experience in achieving the four dimensions of food security?
Example from: Lawlis, T., Islam, W., & Upton, P. (2018). Achieving the four dimensions of food security for resettled refugees in Australia: A systematic review: Refugee food security: A systematic review. Nutrition & Dietetics, 75(2), 182–192. https://doi.org/10.1111/1747-0080.12402
Well-formulated research questions in business, management and organization studies need to take into account how and why relationship occurs in certain circumstances for which CIMO has been established. The table below shows the format of CIMO (Denyer & Tranfield, 2009)
Research Question: How can socio-technical systems theory be used to design a sustainable architecture for integration in Industry 4.0?
Example from: Sony, M., & Naik, S. (2020). Industry 4.0 integration with socio-technical systems theory: A systematic review and proposed theoretical model. Technology in Society, 61, 101248. https://doi.org/10.1016/j.techsoc.2020.101248
|1||3WH*||Who; What; When; How [study conducted]||Sandelowski M, Barroso J. Searching for and retrieving qualitative research reports. In: Sandelowski M, Barroso J, eds. New York: Springer 2007:35-74.|
|2||BeHEMoTh*||Behavior; Health context; Exclusions; Models or Theories||Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J 2015;32(3):220-35. doi: 10.1111/hir.12108|
|3||CHIP*||Context; How [study conducted]; Issues; People||Shaw RL. Conducting literature reviews. In: Forrester MA, ed. Doing qualitative research in psychology: A practical guide. London: Sage 2010:39-56.|
|4||CIMO*||Context; Intervention; Mechanism; Outcomes||Denyer D, Tranfield D. Producing a systematic review. In: Buchanan DA, Bryman A, eds. The SAGE handbook of organizational research methods. London: SAGE Publications Ltd 2009|
|5||CoCoPop*||Condition; Context; Population||Munn Z, Moola S, Lisy K, et al. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc 2015;13(3):147-53. doi: 10.1097/XEB.0000000000000054|
|6||CPTM*||Construct of interest or the name of the measurement instrument(s); Population; Type of measurement instrument; Measurement properties||Terwee CB, Mokkink L. Protocol for systematic reviews of measurement properties: COSMIN: Knowledgecenter Measurement Instruments 2011.|
|7||ECLIPSe*||Expectations (improvement, innovation, or information); Client group (recipients of service); Location (where service is housed); Impact (change in service and how measured); Professionals involved; Service||Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Info Libr J 2002;19 doi: 10.1046/j.1471- 1842.2002.00378.x|
|8||EPICOT*||Evidence; Population; Intervention; Comparison; Outcome; Timestamp||Brown P, Brunnhuber K, Chalkidou K, et al. How to formulate research recommendations. BMJ (Clinical research ed) 2006;333(7572):804-6. doi: 10.1136/bmj.38987.492014.94 [published Online First: 2006/10/14]|
|9||MIP*||Method; Issues; Participants||Strech D, Synofzik M, Marckmann G. Systematic reviews of empirical bioethics. J Med Ethics 2008;34(6):472-77|
|10||PCC*||Population; Concept; Context||Peters MD, Godfrey CM, Khalil H, et al. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc 2015;13(3):141-6. doi: 10.1097/XEB.0000000000000050|
Population; Exposure; Adverse Health Effects; Negative reactions
alternative to PICOC
Problem; Experiment; Alternative; Setting
alternative to PIE
Patient/Population; Exposure; Comparison; Outcomes
|Collaboration for Environmental Evidence. Guidelines for systematic review and evidence synthesis in environmental Management: Environmental Evidence; 2013 [Version 4.2. :[Available from: http://www.environmentalevidence.org/wpcontent/uploads/2014/06/Review-guidelines-version-4.2-final.pdf|
|14||PECODR*||Population; Exposure; Comparison; Outcome; Duration; Results||Dawes M, Pluye P, Shea L, et al. The identification of clinically important elements within medical journal abstracts: Patient_Population_Problem, Exposure_Intervention, Comparison, Outcome, Duration and Results (PECODR). Journal of Innovation in Health Informatics 2007;15(1):9-16.|
|15||PEICO(S)*||Person; Environment; Intervention; Comparison; Outcomes; (Stakeholders)||Howell Major C, M S-B. Designing the synthesis In: Major, C. H., Savin-Baden, M. An introduction to qualitative research synthesis: managing the information explosion in social science research. London: Routledge; 2010. p. 43–55.|
|16||PEO*||Population and their problems; Exposure; Outcomes or Themes||Moola S, Munn Z, Sears K, et al. Conducting systematic reviews of association (etiology): The Joanna Briggs Institute's approach. Int J Evid Based Healthc 2015;13(3):163- 9. doi: 10.1097/XEB.0000000000000064|
|17||PESICO*||Person; Environment; Intervention; Comparison; Outcomes; (Stakeholders)||Schlosser RW, Koul R, Costello J. Asking well-built questions for evidence-based practice in augmentative and alternative communication. J Commun Disord 2007;40(3):225- 38. doi: 10.1016/j.jcomdis.2006.06.008|
|18||PFO*||Population; Prognostic Factors (or models of interest); Outcome||Dretzke J, Ensor J, Bayliss S, et al. Methodological issues and recommendations for systematic reviews of prognostic studies: an example from cardiovascular disease. Syst Rev 2014;3:140. doi: 10.1186/2046-4053-3-140 [published Online First: 2014/12/04]|
|19||PICO*||Patient/Population; Intervention; Comparison; Outcomes||Richardson WS, Wilson MC, Nishikawa J, et al. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995;123|
|20||PICo*||Population; Phenomenon of Interest; Context||Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. Am J Nurs 2014;114(4):53-6. doi: 10.1097/01.NAJ.0000445689.67800.86|
|21||PICo*||Population; Intervention or Phenomenon of Interest; Context||. McArthur A, Klugarova J, Yan H, et al. Innovations in the systematic review of text and opinion. Int J Evid Based Healthc 2015;13(3):188-95. doi: 10.1097/xeb.0000000000000060 [published Online First: 2015/07/25]|
|22||PICOC*||Patient/Population; Intervention; Comparison; Outcomes Context||Petticrew M, Roberts H. Systematic Reviews in the Social Sciences: A practical guide. Oxford: Blackwell Publishing 2006.|
|23||PICOCPRRST*||Population or problem; Intervention or exposure; Comparison; Outcome; Context or environment or setting; Professionals; Results; Research - incorporating type of question and type of study design; STakeholder or perspective or potential users; Timeframe or duration||Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evid Based Libr Info Pract 2011;6 doi: 10.18438/b8ws5n|
|24||PICOS*||Patient/Population; Intervention; Comparison; Outcomes; Study Type||Methley AM, Campbell S, Chew-Graham C, et al. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 2014;14 doi: 10.1186/s12913-014-0579-0|
|25||PICOT*||Patient/Population; Intervention; Comparison; Outcomes; Timeframe||
Riva JJ, Malik KM, Burnie SJ, et al. What is your research question? An introduction to the PICOT format for clinicians. The Journal of the Canadian Chiropractic Association 2012;56(3):167.
Fineout-Overholt E, Johnston L. Teaching EBP: asking searchable, answerable clinical questions. Worldviews on evidence-based nursing 2005;2(3):157-60. doi: 10.1111/j.1741- 6787.2005.00032.x [published Online First: 2006/10/17]
Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP journal club, 123(3), A12-A12
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell Publishers.
|26||PICOT-D*||Population; Intervention; Comparison; Outcome; Time; Digital-data||Elias BL, Polancich S, Jones C, et al. Evolving the PICOT Method for the Digital Age: The PICOT-D. The Journal of nursing education 2015;54(10):594-9. doi: 10.3928/01484834- 20150916-09 [published Online First: 2015/10/03]|
|27||PICOt*||Patient/Population; Intervention; Comparison; Outcomes; timing||Stillwell SB, Fineout-Overholt E, Melnyk BM, et al. Evidence-based practice, step by step: asking the clinical question: a key step in evidence-based practice. AJN The American Journal of Nursing 2010;110(3):58-61.|
|28||PICOT*||Population; Intervention; Comparison; Outcome; Timeframe||Abbade LP, Wang M, Sriganesh K, et al. Framing of research question using the PICOT format in randomised controlled trials of venous ulcer disease: a protocol for a systematic survey of the literature. BMJ open 2016;6(11):e013175. doi: 10.1136/bmjopen-2016-013175 [published Online First: 2016/11/12]|
|29||PICOTS*||Patient/Population; Intervention; Comparison; Outcomes; Timing; Setting||Kelly MP, Noyes J, Kane RL, et al. AHRQ series on complex intervention systematic reviews-paper 2: defining complexity, formulating scope, and questions. J Clin Epidemiol 2017 doi: 10.1016/j.jclinepi.2017.06.012 [published Online First: 2017/07/20]|
|30||PICOTT*||Patient/Population; Intervention; Comparison; Outcomes; Type of Question; Type of Study Design||Schardt C, Adams M, Owens T, et al. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Medical Informatics and Decision Making 2007;7(1):16. doi: doi: 10.1186/1472-6947-7-16|
|31||PIE*||Patient; Intervention/Interest; Evaluation||Anonymous. Easy as PIE. Nursing 1999;29(4):25|
|32||PIPOH*||Population [receiving intervention]; Intervention; Professionals [delivering intervention]; Outcome; Setting [in which Guidance is to be implemented]||Adapte Collaboration. The ADAPTE process: resource toolkit for guideline adaptation. Version 2.0. Available at:)(Accessed November 13, 2014) 2009|
|33||PIPOS*||Population [receiving intervention]; Intervention; Professionals [delivering intervention]; Outcome; Setting [in which Guidance is to be implemented]||Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evid Based Libr Info Pract 2011;6 doi: 10.18438/b8ws5n|
|34||PIRD*||Population; Index test; Reference Test; Diagnosis of Interest||Campbell JM, Klugar M, Ding S, et al. Diagnostic test accuracy: methods for systematic review and meta-analysis. Int J Evid Based Healthc 2015;13(3):154-62. doi: 10.1097/XEB.0000000000000061|
|35||PO*||Population/Phenomenon; Outcome||Pach B, Massarella S, Sharma M. To PICO or not to PICO: What is the question? Frameworks for developing answerable research questions 2016 [Available from: https://www.publichealthontario.ca/en/LearningAndDevelopment/EventPresentations/PICO_ or_not_to_PICO_Pach_Massarella_Sharma_2016.pdf.|
|36||PS*||Population; Situation||DiCenso A, Guyatt G, Ciliska D. Evidence-based Nursing: A Guide to Clinical Practice: Elsevier Health Sciences 2005.|
|37||ProPheT*||Problem; Phenomenon of interest; Timing||Booth A, Sutton A, Papaioannou D. Systematic approaches to a successful literature review. Second edition. ed. London: Sage 2016.|
Setting; Approach; Primary Outcomes
alternative to PICO
|39||SDMO*||Types of Studies; Types of Data; Types of Methods; Outcomes||Clarke M, Oxman A, Paulsen E, et al. Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for Systematic Reviews of Interventions 2009|
|40||SPICE*||Setting; Perspective; (Intervention/Interest); [Comparison]; Evaluation||Booth A. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech 2006;24(3):355-68. doi: 10.1108/07378830610692127|
|41||SPIDER*||Sample; Phenomenon of Interest; Design; Evaluation; Research type||Cooke A, Smith D, Booth A. Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research 2012;22(10):1435-43. doi: 10.1177/1049732312452938|
Suggested Approach; Best Alternative; Best compromise
alternative to SPICE
*These items have been curated from Formulating questions to explore complex interventions within qualitative evidence synthesis
**These items have been curated from Dr. Andrew Booth's Collection of PICO alternatives
More Resources for RQ Development
- Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(1), 579.
- Booth A, Noyes J, Flemming K, et al (2019) Formulating questions to explore complex interventions within qualitative evidence synthesis BMJ Global Health;4:e001107.
- Andrew Booth's Collection of PICO alternatives