Systematic Reviews

What is a Systematic Review?

“A systematic review uses systematic and explicit methods to identify, select, critically appraise, and extract and analyze data from relevant research” – Cochrane Handbook

Systematic Review Process Diagram

Welcome to the interactive process map.

This tool was developed by the St. Michael’s hospital KT program. We have a defined and rigorous process for all our systematic reviews.

This process map takes you through the steps to conduct a systematic review. Click on the individual steps for more information about each step.

Develop Research Question using PICOST

• Population of interest

• Intervention to examine

• Comparator(s)

• Outcome of interest

• Study design(s)

• Time limitations

Additional Resources

Cochrane Handbook Section 2.5: Defining the review question and developing criteria for including studies

Popping the (PICO) question in research and evidence- based practice. Stone PW. 2002. Appl Nurs Res.

The well-built clinical question: a key to evidence based decisions. Richardson WS et al. 1995. ACP J Club.

Formulating questions and locating primary studies for inclusion in systematic reviews. Counsell C. 1997. Ann Intern Med.

Developing the review question and inclusion criteria. Stem C et al. 2014. Am J Nurs.

Posing the research question: not so simple. Thabane L et al. 2009. Can J Anaesth.

Determine Eligibility Criteria

• Develop a research question based on PICOST

Additional Resources

Cochrane Handbook Section 2.5: Defining the review question and developing criteria for including studies

Centre for Reviews and Dissemination: 1.2.2.2 Review question and inclusion criteria and 1.2.2.3 Defining inclusion criteria

AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program. Whitlock EP et al. 2010. J Clin Epidemiol.

Plan Literature Search

• Ask a librarian to develop the search to ensure relevant keywords and search strategies are used for a comprehensive search; peer-review by another librarian

Additional Resources

Cochrane Handbook Section 2.6.4: Designing search strategies

PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. McGowan J et al. 2016. J Clin Epidemiol.

Perform Literature Search

• Search at least two databases, difficult to locate or “grey” literature, hand search key journals, ask experts, and scan reference lists

Additional Resources

Cochrane Handbook Section 2.6.2: Sources to search

CADTH: Grey matters: a practical search tool for evidence-based medicine

Analysis of the reporting of search strategies in Cochrane systematic reviews. Yoshii A et al. 2009. J Med Libr Assoc.

Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Golder et al. 2008. J Clin Epidemiol.

Influences on the outcome of literature searches for integrative research reviews. Cooper et al. 1989. Sci Commun.

PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. McGowan J et al. 2016. J Clin Epidemiol.

Errors in search strategies were identified by type and frequency. Sampson M et al. 2006. J Clin Epidemiol.

Starting a Systematic Review

• Identify knowledge gap and research question with knowledge users

• Involve key stakeholders during the review process (i.e. integrated knowledge translation)

Additional Resources

Cochrane Handbook Section 1.2.1: The need for systematic reviews

Cochrane Handbook Section 1.2.2: What is a systematic review

Centre for Reviews and Dissemination: 1.1.1 Is a review required?

CIHR Guide to Knowledge Translation

The art and science of knowledge synthesis. Tricco AC et al. 2011. J Clin Epidemiol.

Systematic reviews from astronomy to zoology: myths and misconceptions. Petticrew M. 2001. BMJ.

Stakeholder engagement opportunities in systematic reviews: knowledge transfer for policy and practice. Keown K et al. 2008. J Contin Educ Health Prof.

Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? Bastian H et al. 2010. PLoS Med.

Systematic reviews need systematic searchers. McGowan J et al. 2005. J Med Libr Assoc.

Level 1 Screening: Titles and Abstracts

• Pilot test to assess accuracy and usability of eligibility criteria developed for the Level 1 screening process with a small sample of studies

• Screening of all titles and abstracts by two independent reviewers. Conflicts usually resolved by discussion

Additional Resources

Cochrane Handbook Section 2.7.2: Selecting studies

Centre for Reviews and Dissemination: 1.3.2 Study Selection

Identifying relevant studies for systematic reviews. Dickersin et al. 1994. BMJ.

The science of reviewing research. Oxman et al. 1993. Ann N Y Acad Sci.

The art and science of study identification: a comparative analysis of two systematic reviews. Rosen et al. 2016. BMC Med Res Methodol.

Level 2 Screening: Full Text Articles

• Pilot test to assess accuracy and usability of eligibility criteria developed for the Level 2 screening process with a small sample of studies

• Screening of all full text articles by two independent reviewers. Conflicts usually resolved by discussion

Additional Resources

Cochrane Handbook Section 2.7.2: Selecting studies

Identification of randomized controlled trials in systematic reviews: Accuracy and reliability of screening records. Edwards P et al. 2002. Statist Med.

Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. Page MJ et al. 2013. J Clin Epidemiol.

Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students. Ng L et al. 2014. Syst Rev.

Develop Data Abstraction Form

• Pilot test the form on a small sample of studies to ensure it captures all relevant information from the studies. Revise forms as necessary

Additional Resources

Cochrane Handbook Section 2.7.3: What data to collect

Cochrane Handbook Section 2.7.5: Data collection forms

Centre for Reviews and Dissemination: 1.3.3 Data extraction

Innovations in data collection, management, and archiving for systematic reviews. Li et al. 2015. Ann Intern Med.

Abstract Data

• Two reviewers abstract data independently and assess study quality. Conflicts usually resolved by discussion

Additional Resources

Cochrane Handbook Section 2.7.6: Extracting data from reports

Cochrane Handbook Section 2.7.7: Extracting study results and converting to the desired format

Cochrane Handbook: Part 2, Chapter 7, Sections 7.3-7.9: Collecting data

Single data extraction generated more errors than double data extraction in systematic reviews. Buscemi N et al. 2006. J Clin Epidemiol.

Choice of data extraction tools for systematic reviews depends on resources and review complexity. Elamin MB et al. 2009. J Clin Epidemiol.

Data extraction errors in metaanalyses that use standardized mean differences. Gøtzsche et al. 2007. JAMA.

Systematic review data extraction: Cross-sectional study showed that experience did not increase accuracy. Horton J et al. 2010. J Clin Epidemiol.

Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. Tendal B et al. 2009. BMJ.

Critical Appraisal and Assessment

• Critical appraisal is the assessment of evidence by systematically reviewing its relevance, validity, and results.

Additional Resources

Cochrane Handbook Section 2.12.2: Assessing the quality of a body of evidence

Cochrane Handbook Section 2.8: Assessing risk of bias in included studies

Centre for Reviews and Dissemination: 1.3.4 Quality Assessment

Critical Appraisals Skills Programme (CASP)

Cochrane Effective Practice and Organisation of Care Group

Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. Dixon WM et al. 2007. J Health Serv Res Policy.

Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Sanderson S et al. 2007. Int J Epidemiol.

The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. Higgins JP et al. 2011. BMJ.

The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Wells G et al. 2000.

AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Shea BJ et al. 2009. J Clin Epidemiol.

Few systematic reviews exist documenting the extent of bias: a systematic review. Tricco AC et al. 2008. J Clin Epidemiol.

AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. Brouwers MC et al. 2010. J Clin Epidemiol.

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Sterne et al. 2016. BMJ.

GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. Guyatt et al. 2011. J Clin Epidemiol.

Summarize Study and Patient Characteristics

• Use tables and text to describe study and patient characteristics

Additional Resources

Cochrane Handbook Section 2.11.2.2: Characteristics of included studies tables

Synthesize the Results

• Based on your research question and the available data, choose the appropriate method to synthesize your findings

• This may include:

- Meta-analysis

- Network meta-analysis (NMA)

- Individual patient data NMA

- Qualitative synthesis methods

Additional Resources

Cochrane Handbook Section 2.9: Analysing data and undertaking meta-analysis

Cochrane Handbook Section 3.16.6: Indirect comparisons and multiple-treatments meta-analysis

Cochrane Handbook Section 3.20: Qualitative research and Cochrane reviews

Centre for Reviews and Dissemination: 1.3.5 Data Synthesis Module 12.4

The statistical basis of meta-analysis. Fleiss JL. 1993. Stat Methods Med Res.

Meta-analysis: formulating, evaluating, combining, and reporting. S-L. Normand. 1999. Stat Med.

Bias due to changes in specified outcomes during the systematic review process. Kirkham JJ et al. 2010. PLoS One.

Bias in meta-analytic research. Felson DT. 1992. J Clin Epidemiol.

GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. Guyatt GH et al. 2008. BMJ.

Presentation of Results

• Present, summarize, discuss and contextualize findings

• Use summary tables and figures

Additional Resources

Cochrane Handbook Section 2.11: Presenting results and ‘Summary of findings’ tables

Cochrane Handbook: Part 2, Chapter 12: Interpreting results and drawing conclusions

Centre for Reviews and Dissemination: Box 1.11 Suggested structure of a systematic review report

Centre for Reviews and Dissemination: 1.3.6.4 Conclusions, implications, recommendations

How can we improve the interpretation of systematic reviews? Tricco et al. 2011. BMC Med.

Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. Rosenbaum SE et al. 2010. J Clin Epidemiol.

A catalogue of reporting guidelines for health research. Simera I et al. 2010. Eur J Clin Invest.

Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Page MJ et al. 2014. Cochrane Database Syst Rev.

Dissemination and publication of research findings: an updated review of related biases. Song F et al. 2010. Health Technol Assess.

Prepare Manuscript

• Include background, methods, results, and discussion; may involve key stakeholders/audience

Additional Resources

PRISMA Statement

Cochrane Handbook Section 2.11: Presenting results and ‘Summary of findings’ tables

Cochrane Handbook Section 2.12: Interpreting results and drawing conclusions

Centre for Reviews and Dissemination: 1.3.6.3 Formulating the discussion

End-of-Review Knowledge Translation

• For example: Conference presentations, Key opinion meetings, Peer-reviewed publications

Additional Resources

CIHR Guide to Knowledge Translation

Moving knowledge to action through dissemination and exchange. Gagnon ML. 2011. J Clin Epidemiol.

Effectiveness and efficiency of guideline dissemination and implementation strategies. Grimshaw JM et al. 2004. Health Technol Assess.

Bridging research and practice: models for dissemination and implementation research. Tabak RG et al. 2012. Am J Prev Med.

Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. Tricco AC et al. 2016. Implement Sci.

Full publication of results initially presented in abstracts. Scherer RW et al. 2007. Cochrane Database Syst Rev.

Systematic Review Protocol Development

Systematic Review Protocol Development based on the PRISMA-P statement

Additional Resources

Cochrane Handbook Section 2.1: Rationale for protocols

Cochrane Handbook Section 2.2.2 Outline of a protocol for a Cochrane review

The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Booth A et al. 2012. Syst Rev.

Download: Systematic Review Process Map (.pdf, 198kb)

Examples of methodology projects conducted by our team

Robson, R. C., Pham, B., Hwee, J., Thomas, S. M., Rios, P., Page, M. J., Tricco, A. C. (2019). Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review. Journal of Clinical Epidemiology, 106, 121-135. doi: 10.1016/j.jclinepi.2018.10.003.

Tricco, A. C., Cogo, E., Page, M. J., Polisena, J., Booth, A., Dwan, K., … & Moher, D. (2016). A third of systematic reviews changed or did not specify the primary outcome: A PROSPERO register study. Journal of Clinical Epidemiology, 79, 46-54. doi: 10.1016/j.jclinepi.2016.03.025.

Tricco, A. C., Straus, S. E., & Moher, D. (2011). How can we improve the interpretation of systematic reviews? BMC Medicine, 9(31). doi: 10.1186/1741-7015-9-31.

Examples of Systematic Reviews conducted by our team

Ashoor, H. M., Lillie, E., Zarin, W., Pham, B., Khan, P. A., Nincic, V., … & Tricco, A. C. (2017). Effectiveness of different compression-to-ventilation methods for cardiopulmonary resuscitation: A systematic review. Resuscitation, 118, 112-125. doi: 10.1016/j.resuscitation.2017.05.032.

Tricco, A. C., Soobiah, C., Hui, W., Antony, J., Struchkov, V., Hutton, B., … & Straus, S. E. (2015). Interventions to decrease the risk of adverse cardiac events for patients receiving chemotherapy and serotonin (5-HT3) receptor antagonists: A systematic review. BMC Pharmacology and Toxicology, 16(1). doi: 10.1186/2050-6511-16-1.