Evidence Synthesis: Research Question Development Tools

Research Question Development Tools

What is a Concept Developer?

'Concept developers', also called 'research question formulation frameworks' are simply acronyms that help you consider the details behind your research question.

For example, the intervention-focused tool PICO stands for Population, Intervention, Comparator, and Outcome. To use this tool, you would to clearly state the population, intervention, comparator (as it relates to the intervention), and outcome of your research question. Because the systematic review methodology stems from health sciences, PICO is most commonly cited in the context of systematic reviews.

However, there are many options (see the tab labeled "More") to choose from! When in doubt, you can always use the 5Ws & H - who, what, where, when, why, and how. The point of using a concept developer is to consistently and thoroughly define the boundaries of your research question.

PICO(T)

The PICO (T) is a widely known strategy for framing an intervention-based research question, particularly in health sciences (Sackett et al.,1997). This tool is also used in other fields (e.g., medicine, social sciences). Sometimes "Intervention" is replace with "Exposure" as PECO alternative.

Research Question: How does the influenza vaccine influence risk of contracting pneumonia during flu season in elderly populations?
P	I	C	O	T
Population of Interest	Intervention	Control	Outcome	Time
Population description, important characteristics, demographics, etc.	Exposure to be considered, tests, treatments, etc.	Control or Comparison as compared to the Intervention.	Outcome is the effect of the intervention. For example, effectiveness of the new therapy.	Time period of the study to be considered. This component is optional, like "(S) Study Design"
Patients 65 years of age and older	Influenza vaccine	No influenza vaccine	Develop pneumonia	Flu season only

Worksheet for Developing PICO(T) Questions from the AAACN (American Academy of Ambulatory Care Nursing); permission granted on the PDF file for educational and research uses of the template.

The important thing to take away from PICO is that you need to develop the concepts behind your research question to fully define and illustrate your goals, which will inform every stage of your review. You can include the details defined through these concept builders in the research question directly, but it's not necessary - it is just necessary that you have explicitly stated them.

SPIDER

SPIDER can be a suitable tool for qualitative and mixed methods research in several disciplines. The following table shows the breakdown of the format with an example in qualitative research

Research Question: What are the challenges Australian refugees experience in achieving the four dimensions of food security?

S	PI	D	E	R
Sample	Phenomenon of Interest	Design	Evaluation	Research Type
Sample represents smaller group of participants from the population more relevant to qualitative research	Aims to understand the how and why of certain behaviors, decisions and individual experiences	The research method that is used in the study will be derived from the theoretical framework used in the study.	Similar to Outcome in PICO, Evaluation determines the outcome measures of the study which might include attitudes, views, etc. in qualitative research	Research types include qualitative, quantitative, and mixed methods which could be used.
In resettled refugees in Australia	Studies identifying the predictors of food insecurity and challenges to food availability and access	No specific study design	Views and experiences of the participant group, not there searchers, formed the basis for study selection	Both qualitative and quantitative research types

Example from: Lawlis, T., Islam, W., & Upton, P. (2018). Achieving the four dimensions of food security for resettled refugees in Australia: A systematic review: Refugee food security: A systematic review. Nutrition & Dietetics, 75(2), 182–192. https://doi.org/10.1111/1747-0080.12402

CIMO

Well-formulated research questions in business, management and organization studies need to take into account how and why relationship occurs in certain circumstances for which CIMO has been established. The table below shows the format of CIMO (Denyer & Tranfield, 2009)

Research Question: How can socio-technical systems theory be used to design a sustainable architecture for integration in Industry 4.0?

C	I	M	O
Context	Intervention	Mechanisms	Outcomes
Context of the study including individuals, relationships, institutional settings, or wider systems used	The effects of what event, actions or activity are being studied	Explains the relationship between intervention and outcome	The effects of the Intervention
Manufacturing	Socio-technical systems theory	Architecture design for three types of Industry 4.0 integration	Sustainable Industry 4.0 Implementation

Example from: Sony, M., & Naik, S. (2020). Industry 4.0 integration with socio-technical systems theory: A systematic review and proposed theoretical model. Technology in Society, 61, 101248. https://doi.org/10.1016/j.techsoc.2020.101248

	Framework	Elements	Source
1	3WH*	Who; What; When; How [study conducted]	Sandelowski M, Barroso J. Searching for and retrieving qualitative research reports. In: Sandelowski M, Barroso J, eds. New York: Springer 2007:35-74.
2	BeHEMoTh*	Behavior; Health context; Exclusions; Models or Theories	Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J 2015;32(3):220-35. doi: 10.1111/hir.12108
3	CHIP*	Context; How [study conducted]; Issues; People	Shaw RL. Conducting literature reviews. In: Forrester MA, ed. Doing qualitative research in psychology: A practical guide. London: Sage 2010:39-56.
4	CIMO*	Context; Intervention; Mechanism; Outcomes	Denyer D, Tranfield D. Producing a systematic review. In: Buchanan DA, Bryman A, eds. The SAGE handbook of organizational research methods. London: SAGE Publications Ltd 2009
5	CoCoPop*	Condition; Context; Population	Munn Z, Moola S, Lisy K, et al. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc 2015;13(3):147-53. doi: 10.1097/XEB.0000000000000054
6	CPTM*	Construct of interest or the name of the measurement instrument(s); Population; Type of measurement instrument; Measurement properties	Terwee CB, Mokkink L. Protocol for systematic reviews of measurement properties: COSMIN: Knowledgecenter Measurement Instruments 2011.
7	ECLIPSe*	Expectations (improvement, innovation, or information); Client group (recipients of service); Location (where service is housed); Impact (change in service and how measured); Professionals involved; Service	Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Info Libr J 2002;19 doi: 10.1046/j.1471- 1842.2002.00378.x
8	EPICOT*	Evidence; Population; Intervention; Comparison; Outcome; Timestamp	Brown P, Brunnhuber K, Chalkidou K, et al. How to formulate research recommendations. BMJ (Clinical research ed) 2006;333(7572):804-6. doi: 10.1136/bmj.38987.492014.94 [published Online First: 2006/10/14]
9	MIP*	Method; Issues; Participants	Strech D, Synofzik M, Marckmann G. Systematic reviews of empirical bioethics. J Med Ethics 2008;34(6):472-77
10	PCC*	Population; Concept; Context	Peters MD, Godfrey CM, Khalil H, et al. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc 2015;13(3):141-6. doi: 10.1097/XEB.0000000000000050
11	PEAHEN**	Population; Exposure; Adverse Health Effects; Negative reactions alternative to PICOC
12	PEAS**	Problem; Experiment; Alternative; Setting alternative to PIE
13	PECO*	Patient/Population; Exposure; Comparison; Outcomes	Collaboration for Environmental Evidence. Guidelines for systematic review and evidence synthesis in environmental Management: Environmental Evidence; 2013 [Version 4.2. :[Available from: http://www.environmentalevidence.org/wpcontent/uploads/2014/06/Review-guidelines-version-4.2-final.pdf
14	PECODR*	Population; Exposure; Comparison; Outcome; Duration; Results	Dawes M, Pluye P, Shea L, et al. The identification of clinically important elements within medical journal abstracts: Patient_Population_Problem, Exposure_Intervention, Comparison, Outcome, Duration and Results (PECODR). Journal of Innovation in Health Informatics 2007;15(1):9-16.
15	PEICO(S)*	Person; Environment; Intervention; Comparison; Outcomes; (Stakeholders)	Howell Major C, M S-B. Designing the synthesis In: Major, C. H., Savin-Baden, M. An introduction to qualitative research synthesis: managing the information explosion in social science research. London: Routledge; 2010. p. 43–55.
16	PEO*	Population and their problems; Exposure; Outcomes or Themes	Moola S, Munn Z, Sears K, et al. Conducting systematic reviews of association (etiology): The Joanna Briggs Institute's approach. Int J Evid Based Healthc 2015;13(3):163- 9. doi: 10.1097/XEB.0000000000000064
17	PESICO*	Person; Environment; Intervention; Comparison; Outcomes; (Stakeholders)	Schlosser RW, Koul R, Costello J. Asking well-built questions for evidence-based practice in augmentative and alternative communication. J Commun Disord 2007;40(3):225- 38. doi: 10.1016/j.jcomdis.2006.06.008
18	PFO*	Population; Prognostic Factors (or models of interest); Outcome	Dretzke J, Ensor J, Bayliss S, et al. Methodological issues and recommendations for systematic reviews of prognostic studies: an example from cardiovascular disease. Syst Rev 2014;3:140. doi: 10.1186/2046-4053-3-140 [published Online First: 2014/12/04]
19	PICO*	Patient/Population; Intervention; Comparison; Outcomes	Richardson WS, Wilson MC, Nishikawa J, et al. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995;123
20	PICo*	Population; Phenomenon of Interest; Context	Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. Am J Nurs 2014;114(4):53-6. doi: 10.1097/01.NAJ.0000445689.67800.86
21	PICo*	Population; Intervention or Phenomenon of Interest; Context	. McArthur A, Klugarova J, Yan H, et al. Innovations in the systematic review of text and opinion. Int J Evid Based Healthc 2015;13(3):188-95. doi: 10.1097/xeb.0000000000000060 [published Online First: 2015/07/25]
22	PICOC*	Patient/Population; Intervention; Comparison; Outcomes Context	Petticrew M, Roberts H. Systematic Reviews in the Social Sciences: A practical guide. Oxford: Blackwell Publishing 2006.
23	PICOCPRRST*	Population or problem; Intervention or exposure; Comparison; Outcome; Context or environment or setting; Professionals; Results; Research - incorporating type of question and type of study design; STakeholder or perspective or potential users; Timeframe or duration	Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evid Based Libr Info Pract 2011;6 doi: 10.18438/b8ws5n
24	PICOS*	Patient/Population; Intervention; Comparison; Outcomes; Study Type	Methley AM, Campbell S, Chew-Graham C, et al. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 2014;14 doi: 10.1186/s12913-014-0579-0
25	PICOT*	Patient/Population; Intervention; Comparison; Outcomes; Timeframe	Riva JJ, Malik KM, Burnie SJ, et al. What is your research question? An introduction to the PICOT format for clinicians. The Journal of the Canadian Chiropractic Association 2012;56(3):167. Fineout-Overholt E, Johnston L. Teaching EBP: asking searchable, answerable clinical questions. Worldviews on evidence-based nursing 2005;2(3):157-60. doi: 10.1111/j.1741- 6787.2005.00032.x [published Online First: 2006/10/17] Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP journal club, 123(3), A12-A12 Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell Publishers.
26	PICOT-D*	Population; Intervention; Comparison; Outcome; Time; Digital-data	Elias BL, Polancich S, Jones C, et al. Evolving the PICOT Method for the Digital Age: The PICOT-D. The Journal of nursing education 2015;54(10):594-9. doi: 10.3928/01484834- 20150916-09 [published Online First: 2015/10/03]
27	PICOt*	Patient/Population; Intervention; Comparison; Outcomes; timing	Stillwell SB, Fineout-Overholt E, Melnyk BM, et al. Evidence-based practice, step by step: asking the clinical question: a key step in evidence-based practice. AJN The American Journal of Nursing 2010;110(3):58-61.
28	PICOT*	Population; Intervention; Comparison; Outcome; Timeframe	Abbade LP, Wang M, Sriganesh K, et al. Framing of research question using the PICOT format in randomised controlled trials of venous ulcer disease: a protocol for a systematic survey of the literature. BMJ open 2016;6(11):e013175. doi: 10.1136/bmjopen-2016-013175 [published Online First: 2016/11/12]
29	PICOTS*	Patient/Population; Intervention; Comparison; Outcomes; Timing; Setting	Kelly MP, Noyes J, Kane RL, et al. AHRQ series on complex intervention systematic reviews-paper 2: defining complexity, formulating scope, and questions. J Clin Epidemiol 2017 doi: 10.1016/j.jclinepi.2017.06.012 [published Online First: 2017/07/20]
30	PICOTT*	Patient/Population; Intervention; Comparison; Outcomes; Type of Question; Type of Study Design	Schardt C, Adams M, Owens T, et al. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Medical Informatics and Decision Making 2007;7(1):16. doi: doi: 10.1186/1472-6947-7-16
31	PIE*	Patient; Intervention/Interest; Evaluation	Anonymous. Easy as PIE. Nursing 1999;29(4):25
32	PIPOH*	Population [receiving intervention]; Intervention; Professionals [delivering intervention]; Outcome; Setting [in which Guidance is to be implemented]	Adapte Collaboration. The ADAPTE process: resource toolkit for guideline adaptation. Version 2.0. Available at:)(Accessed November 13, 2014) 2009
33	PIPOS*	Population [receiving intervention]; Intervention; Professionals [delivering intervention]; Outcome; Setting [in which Guidance is to be implemented]	Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evid Based Libr Info Pract 2011;6 doi: 10.18438/b8ws5n
34	PIRD*	Population; Index test; Reference Test; Diagnosis of Interest	Campbell JM, Klugar M, Ding S, et al. Diagnostic test accuracy: methods for systematic review and meta-analysis. Int J Evid Based Healthc 2015;13(3):154-62. doi: 10.1097/XEB.0000000000000061
35	PO*	Population/Phenomenon; Outcome	Pach B, Massarella S, Sharma M. To PICO or not to PICO: What is the question? Frameworks for developing answerable research questions 2016 [Available from: https://www.publichealthontario.ca/en/LearningAndDevelopment/EventPresentations/PICO_ or_not_to_PICO_Pach_Massarella_Sharma_2016.pdf.
36	PS*	Population; Situation	DiCenso A, Guyatt G, Ciliska D. Evidence-based Nursing: A Guide to Clinical Practice: Elsevier Health Sciences 2005.
37	ProPheT*	Problem; Phenomenon of interest; Timing	Booth A, Sutton A, Papaioannou D. Systematic approaches to a successful literature review. Second edition. ed. London: Sage 2016.
38	SAPO**	Setting; Approach; Primary Outcomes alternative to PICO
39	SDMO*	Types of Studies; Types of Data; Types of Methods; Outcomes	Clarke M, Oxman A, Paulsen E, et al. Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for Systematic Reviews of Interventions 2009
40	SPICE*	Setting; Perspective; (Intervention/Interest); [Comparison]; Evaluation	Booth A. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech 2006;24(3):355-68. doi: 10.1108/07378830610692127
41	SPIDER*	Sample; Phenomenon of Interest; Design; Evaluation; Research type	Cooke A, Smith D, Booth A. Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research 2012;22(10):1435-43. doi: 10.1177/1049732312452938
42	SugABABes**	Suggested Approach; Best Alternative; Best compromise alternative to SPICE

*These items have been curated from Formulating questions to explore complex interventions within qualitative evidence synthesis

**These items have been curated from Dr. Andrew Booth's Collection of PICO alternatives

More Resources for RQ Development

Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(1), 579.
Booth A, Noyes J, Flemming K, et al (2019) Formulating questions to explore complex interventions within qualitative evidence synthesis BMJ Global Health;4:e001107.
Andrew Booth's Collection of PICO alternatives