Systematic Reviews, Scoping Reviews, and Other Knowledge Syntheses

McGill University Library has a subscription to Covidence, a useful tool for importing database records, removing duplicate records, screening, documenting critical appraisal/risk of bias and data extraction, and exporting data. Covidence also allows members of the McGill community to invite external reviewers to join their review team.

For access and support for Covidence, please consult: https://support.covidence.org/help/mcgill-university-library

There are many resources that explain how to go about screening. One place to start is the guide on Screening Studies created by the librarians at the University of Toronto.

These librarians have also compiled example screening templates available in Open Science Framework.

Software packages specifically designed for knowledge synthesis (e.g., Covidence) will typically include a record screening/study selection function. This allows more than one reviewer to independently screen the records without seeing other reviewers' decisions to include or exclude studies, and thus reduces bias.

Some software tools also include screening prioritization, which allows you to re-sort the records left to screen by relevance, based on past decisions to include or exclude records already screened. 

Partial list of software tools using machine learning/AI for screening prioritization:

• AI and text mining for searching and screening the literature: Tools for screening (study selection)

The Systematic Review Tool Box is another useful resource for keeping up to date with screening tools:

• The Systematic Review Tool Box - Click on "Advanced Search" > Select Tool Type and Review Family (or leave both blank) > Select Review Stage: Screen

Critical appraisal should involve an assessment of the risk of bias in the relevant studies and may also involve an assessment of how the studies were reported.

As librarians, we are generally not involved in the appraisal process, but we can provide guidance on finding critical appraisal tools if needed. They are generally specific to a given study design or research methodology. The following are some suggested tools but this list is not exhaustive and they may not have been validated.

Assessment of tools:

• CATevaluation: Assessment of critical appraisal tools

• Validity assessment tools for evidence synthesis: Your one-stop-shop (LATITUDES Network) 

  • See Whiting P, Wolff R, Savovic J, Devine B, Mallett S. Introducing the LATITUDES network: A library of assessment tools and training to improve transparency, utility and dissemination in evidence synthesis. J Clin Epidemiol. 2024;174. 

Repositories and collections of tools:

• Quality Assessment and Risk of Bias Tool Repository
• Critical appraisal tools (Joanna Briggs Institute) - Includes checklists for a wide range of studies including case reports, economic evaluations, incidence/prevalence, quasi-experimental studies, and text/opinion
• CASP checklists (Critical Appraisal Skills Programme) - Includes checklists for wide range of studies. Useful for novice appraisers as checklists provide prompting questions and identify different elements of the study to assess
• SIGN checklists (Healthcare Improvement Scotland) - Includes systematic reviews & meta-analyses, randomised controlled trials, cohort studies, case-control studies, diagnostic studies, economic studies
• The Registry of Methods and Tools for Evidence-Informed Decision Making: Resources filtered to Appraisal (National Collaborating Centre for Methods and Tools)

Other tools for specific contexts not necessarily covered above:

• Collaboration for Environmental Evidence Critical Appraisal Tool Version 0.3 [Prototype]
• Grey literature: AACODS checklist
• Narrative reviews: Baethge C, Goldbeck-Wood S, Mertens S. SANRA—a scale for the quality assessment of narrative review articles. Res Integr Peer Rev. 2019;4(1):5. doi:10.1186/s41073-019-0064-8 (Open access)
• Studies reporting on measurement properties, patient-reported outcome measures: COSMIN
• New measure development and validation studies: Holmbeck GN, Devine KA. Editorial: an author's checklist for measure development and validation manuscripts. J Pediatr Psychol. 2009;34(7):691-696.

Reporting guidelines:

• Reporting guidelines: See the EQUATOR Network for guidelines relevant to specific study designs such as randomized controlled trials (CONSORT), systematic reviews (PRISMA)

Resources on analyzing/synthesizing findings

Narrative synthesis of quantitative effect data

Thomson H, Campbell M. “Narrative synthesis” of quantitative effect data in Cochrane reviews: Current issues and ways forward [Internet]. Cochrane Learning Live Webinar Series 2020 Feb.  https://training.cochrane.org/resource/narrative-synthesis-quantitative-effect-data-cochrane-reviews-current-issues-and-ways. 

• Part 1 helps navigate some of the confusion over the concepts of "narrative synthesis" or "qualitative review of (quantitative) data" versus the ambiguous use of the terms "narrative review" or "qualitative review"

Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., Welch, V., & Thomson, H. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: Reporting guideline. BMJ, 368, l6890. doi:10.1136/bmj.l6890

Network meta-analysis

Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 2. Value Health. 2011;14(4):429-37.

Quantitative evidence synthesis in environmental sciences

Nakagawa S, Yang Y, Macartney EL, Spake R, Lagisz M. Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences. Environmental Evidence. 2023;12(1):8.
 

Find a librarian in your subject area to help you with your knowledge synthesis project.

Or contact the librarians at the
Schulich Library of Physical Sciences, Life Sciences, and Engineering
schulich.library@mcgill.ca