This is a version of the LibGuide I started while employed at Temple University Health Sciences Library from July 2015-September 12, 2022. I plan to maintain it here going forward.
What is a Systematic Review?
Systematic Review vs. Literature (Narrative) Review
Traditional literature review / narrative review:
- Describes and appraises previous work but does not describe specific methods by which the reviewed studies were identified, selected and evaluated
- Overviews, discussions, critiques of previous work and the current gaps in knowledge
- Often used as rationale for new research
- To scope the types of interventions available to include in a review
- The writers assumptions and agenda often unknown
- Biases that occur in selecting and assessing the literature are unknown
- Cannot be replicated
- The scope of the review is identified in advance (eg review question and sub‐questions and/or sub‐group analysis to be undertaken)
- Comprehensive search to find all relevant studies
- Use of explicit criteria to include / exclude studies
- Application of established standards to critically appraise study quality
- Explicit methods of extracting and synthesizing study findings (qualitative or quantitative)
- May include a meta-analysis (quantitative synthesis) *optional
- Identifies, appraises and synthesizes all available research that is relevant to a particular review question
- Collates all that is known on a given topic and identifies the basis of that knowledge
- Comprehensive report using explicit processes so that rationale, assumptions and methods are open to scrutiny by external parties
- Can be replicated / updated
- Systematic reviews with narrowly defined review questions provide specific answers to specific questions
- Alternative questions that have not been answered usually need to be reconstructed by the reader
Source: Cochrane. Background to Systematic Reviews
This link will open a PDF document.
What is a Systematic Review?
A systematic review is defined as “a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” The methods used must be reproducible and transparent.
Source: Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.
Image: EBM Pyramid and EBM Page Generator, copyright 2006 Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.
When is a Systematic Review the most appropriate study design?
When answering questions of effectiveness comparing two different treatments or interventions.
Choosing a Review Type: This guide explains other comprehensive literature reviews of similar methodology to the systematic review.
A helpful article about review types. (Meeting the Review Family: Exploring review types and associated information retrieval requirements, 2019,Sutton et al.)
7 Stages of Conducting a Systematic Review
1. Gathering your team (Minimum of two reviewers with a third to serve as a tiebreaker)
A systematic review must have a team of two or greater. A systematic review cannot be completed by one person. Choose team members wisely and based on areas of expertise. A third team member is sometimes called a tiebreaker. They are to resolve disagreements for reviewers 1 and 2 for stages of the review that are blinded (screening, data extraction, critical appraisal) and are completed by two independent reviewers.
2. Questioning (Define a narrow question, may use PICO)
The PICO format is commonly used to define the research question into one that is a searchable question. In some cases, the PICO format may not work and another format can be used. The P in PICO is Patient/Problem or Person. I is for Intervention/Exposure/Therapy or Treatment. C is optional and is for Comparison (such as a placebo or another drug/therapy) and O is for Outcome(s), what is the expected or anticipated outcome you will find in the literature? A systematic review question should also be narrow in scope in order to be able to conduct the review in one year. The purpose of a systematic review is to draw conclusions based on the evidence to answer that one well-defined and narrow question.
3. Planning (Create a protocol, plan methods & strategies, register protocol) * This course focuses on the planning stage
Having a plan in place is essential to a good quality review and by spending more time planning before the review takes place, you could avoid issues or errors that may slow down the process or be detrimental to the review. Planning includes seeing if the review is feasible, checking to make sure there are no conflicting reviews and also ensuring that there is a plan to carry out each stage of the review. Setting goals and timelines for the review is important as well as mapping out how the review project will be managed. This is also put into a document called a protocol. Registering the protocol is optional but highly recommended. The protocol also includes defining a priori what the selection criteria will be for the review in terms of inclusion and exclusion criteria for what studies should be screened by for inclusion in the review. This is usually the stage where the librarian becomes (or should become) involved in the systematic review, unless they are conducting their own systematic review and are involved from the start.
4. Searching/Screening (Exhaustive, transparent & repeatable searching for evidence/selecting studies)
Includes searching multiple databases, grey literature/clinical trial registries and handsearching of the literature (performed by the subject matter expert). Librarians are involved in creating the comprehensive search, translating the search for databases or grey literature, documenting the search and are often involved in deduplicating the repeating references in a citation manager and writing the search methods of the review. However, librarians are usually not involved in grey literature searching unless they are an expert in the subject matter. The review team member with the most subject matter expertise is the one who is best equipped to handsearch. The search stage may also include contacting other experts in the field to identify publications that have not been published yet. Systematic reviews include both published and unpublished literature to avoid a type of publication bias, called positive outcomes bias since positive outcomes are more likely to be published. Screening is done in two phases. The first phase is screening titles/abstracts (together) and the second phase is screening full texts. Screening is done independently by two reviewers, with a third reviewer sometimes serving as a tiebreaker. The process for resolving conflicts should be recorded. If conflicts were resolved by discussion, that should be reported. Reviewers should not move on to the full text screening phase until they have screened all of the titles and abstracts and each is a clear Yes or No without maybes remaining. Once they are ready to screen full texts, they must acquire and read all of the full texts and screen them based on the studies selection criteria. Only Yes’s are included in the review but all No’s must have a reason listed for exclusion. The new PRISMA 2020 requires reporting of study Near Misses too. Near misses are any studies that did not meet inclusion in the review but were very close to being included. Refer to the PRISMA 2020 http://www.prisma-statement.org/PRISMAStatement/ for more guidance on this stage. There are tools designed specifically to assist with the systematic review screening phase such as Covidence or Rayyan.
5. Managing & reporting
All methods must be fully reported, transparent and reproducible. The methods reported must also follow the recommended reporting guidance such as the PRISMA 2020. Reporting guidance can be identified by searching the Equator Network https://www.equator-network.org/. Reporting guidance may be modified for review types similar to the systematic review. Refer to the many PRISMA 2020 extensions http://www.prisma-statement.org/Extensions/ for more information.
6. Data Extraction/Synthesizing the evidence
This stage includes appraising the evidence, interpreting results, performing a qualitative (narrative analysis) and/or a quantitative/meta-analysis. A meta-analysis is optional and is only done if it is feasible. A biostatistician or advanced training in statistics is recommended if doing a meta-analysis. There are many tools designed to assist with this process. Evidence from studies are assessed using critical appraisal or Risk of Bias tools and/or checklists by study design.
Critical appraisal checklists from my website Systematic Review Librarian
More tools for critical appraisal and other stages of the review from the National Collaborating Centre for Methods and Tools
Data from all studies must also be extracted and put into tables/charts such as the Summary of Findings (SOF) table and is reported as a narrative synthesis. Data is collected from all studies if conducting a meta-analysis and its numerical findings are reported.
Synthesis: Provide a narrative synthesis of the included studies individually and when combined (What are the differences and the commonality between studies?) or what can be demonstrated from the research when combining the studies together? A meta-analysis is optional. Create a data abstraction/extraction form for the purposes of collecting data that is similar across all included studies, include a ‘Characteristics of Studies’ table to show this data (see table example). Another example of a table is available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6483779/pdf/CD011479.pdf (see page 48). Summary of Findings tables are provided starting on page 8 of the same document. Data extraction must be done using data extraction forms and conducted by two independent reviewers. There should be a clear process for how to handle disagreements. If disagreements were resolved by discussion, that should be reported. A third reviewer can be called in to resolve any disagreements. Unresolved disagreements should be fully reported.
7. Drawing Conclusions, Writing & Publishing
After completing these steps, the results of the review must be shared. What is the level of evidence? Is there evidence in support of the question or are more studies needed to draw conclusions? What are your recommendations for future studies? What are the limitations to your systematic review? How do these findings from your review change what is known on the topic or question?
Discuss what contribution this review makes and how your review answers or addresses the original question. Discuss any gaps found in the research. Make recommendations for needed research to address these gaps and the importance of addressing them. Discuss the overall strength of evidence in support of your original question (strong, moderate or weak).
For more guidance on the systematic review stages, refer to the Cochrane Handbook (medicine/health sciences), the JBI Manual (health sciences/nursing) or the methods guides for Campbell Collaboration Systematic Reviews (Business, education, social welfare, criminal & justice topics and more).
Where to register your Systematic Review Protocol?
- PROSPERO -registry for medical/health systematic reviews
- OSF Preregistration Register a systematic review protocol, including non-medical/health topics. Choose to affiliate with Temple University.
- SYREAF-systematic reviews for animals and food
- Joanna Briggs Institute Systematic Review Register
- Inclusive Systematic Review Registration Form This form, therefore, is a fall-back for more specialized forms and can be used if no specialized form or registration platform is available. Can be used for a variety of disciplines and review types.
- Moller AM, Myles PS. What makes a good systematic review and meta-analysis? BJA. 2016. 117(4):428-430.
- How to do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses
- JBI Manual: Systematic Reviews of Effectiveness
- JBI Manual: Systematic Reviews of Prevalence and Incidence
- JBI Manual: Systematic Reviews for Etiology and Risk
- JBI Manual: Systematic Reviews of Measurement Properties
- JBI manual: Systematic Reviews of Text and Opinion
- How to review the evidence: Systematic identification and review of the scientific literature
- BMJ OPEN Systematic Reviews Systematic Review articles published in the BMJ Open (Open-Access) medical journal.
- Lockwood, Geum. Systematic reviews: Guidelines, tools and checklists for authors (2017)
Systematic Review Standards
- Finding What Works in Healthcare: Standards for Systematic Reviews National Academies of Science, Engineering, & Medicine (Formerly Institute of Medicine (IOM)) Standards for systematic reviews.
- Equator Reporting Guidelines Decision Tree A tool to select the appropriate reporting guideline for your review. This link will open a PDF document.
- PRISMA: Transparent Reporting of Reviews and Meta-Analyses Reporting guidelines for reviews of Interventions.
- PRISMA Checklist
- PRISMA Flow Diagram
- PRISMA-S extension for search reporting
- MOOSE (Meta-analysis of Observational Studies in Epidemiology) Guidelines Reporting guidelines for observational reviews of cohort, case-control or cross-sectional studies. This link will open a PDF document.
- Cochrane Handbook of Systematic Reviews for Interventions A guide for Cochrane reviewers, explains the Cochrane systematic review process in great detail.
- MECIR-Updated Cochrane Methodological Expectations of Cochrane Intervention Reviews
- JBI Manual for Evidence Synthesis
- Campbell Collaboration
- The Campbell Collaboration MECCIR standards These standards are adapted from the Cochrane MECIR standards and are for disciplines beyond medicine such as crime, justice, education, international development, methods, social welfare, disability, business knowledge translation/implementation and food security.
- AHRQ Methods Guide for Effectiveness of Comparative Effectiveness Reviews
- Centre for Reviews and Dissemination (Univ. of York) Includes guides on how to conduct systematic reviews and more.
- BEME: Best Evidence Medical Education
- Guidelines for Systematic Reviews of Health Promotion and Public Health This link will open a PDF document.
Oops, if you find any broken links, please contact me!