Systematic Reviews, Scoping Reviews, and Other Knowledge Syntheses

Finding studies relevant to your question should not depend solely on database searching: Other search techniques are recommended in order to avoid different forms of bias in what studies are ultimately included in the review.

Examples of other search techniques:

• Finding grey literature, e.g., theses & dissertations, preprint servers, clinical trials registers, conference proceedings, using Google Scholar
• Searching for citation relationships (reference list searching, scanning the included studies in other relevant knowledge syntheses, similar articles searching, co-cited articles searching, or forward citation searching)
• Handsearching
• Contacting authors

Grey literature, according to the Cochrane Handbook, is usually understood to be literature not formally published in books or journals. This can include theses or dissertations, conference proceedings, clinical trials registries, white papers, government reports, and more. Some grey literature will be retrievable through database searching, but it depends on the databases you have chosen to search and what kind of content the databases index. For example, MEDLINE does not index much grey literature, whereas you can retrieve some conference proceedings indexed in Web of Science Core Collection databases available through McGill.

You may be interested in finding grey literature available on websites. One suggestion is to identify associations, organisations, institutions, etc. that are likely to make documents or reports of relevance to your question available on their websites, and to then selectively search or browse those sites.

Summarized evidence on searching and documenting approaches to grey literature:

Severn M, Mierzwinski-Urban M, Farrah K, Walter M, Spry C,  Argáez C. Grey literature.  Last updated 15 March 2023. In: SuRe Info: Summarized Research in Information Retrieval for HTA.  Available from: https://www.sure-info.org//grey-literature

Resources to help you identify grey literature in public health and health technology assessment:

• La boussole grise, produced by the Institut national de santé publique du Québec (French interface)
• Grey Matters, produced by the Canadian Agency for Drugs and Technologies in Health (CADTH)

Resources to help you identify grey literature for interventions in crime & justice, education, international development, and social welfare:

• Searching for Studies: A Guide to Information Retrieval for Campbell Systematic Reviews, 2017

Resources to help you identify Canadian health policy:

• A Guide to Governmental and Grey Literature Sources for Public Policy Research, University of Toronto

For more information on searching the grey literature, the University of Toronto Gerstein Science Information Centre has a comprehensive guide on searching the grey literature, including lists of potential resources to search.

Theses and dissertations are potentially rich sources of grey literature given the depth of research involved in writing a thesis or dissertation. They are also usually considered grey literature in and of themselves as well.

See also:

• McGill guide on finding theses & dissertations

Preprint servers are free online archives that allow researchers to identify studies in rapidly developing fields; they can be used to identify studies that have not gone through the more formalized and traditional peer review process​. Depending on the site, searching and exporting functionality may be more or less available.

Hoy, M. B. (2020, Jan-Mar). Rise of the Rxivs: How preprint servers are changing the publishing process. Medical Reference Services Quarterly, 39(1), 84-89.

Trials may go unreported in the published literature. One useful method to identify unpublished clinical trials is to search clinical trials registries. The results may be available within the registries or you may need to contact the researchers associated with the trial for further information.

ClinicalTrials.gov

European Union Clinical Trials Register

ICTRP (International Clinical Trials Registry Platform) Portal (WHO)

ISRCTN Registry - "primary clinical trial registry recognised by WHO and ICMJE that accepts all clinical research studies (whether proposed, ongoing or completed), providing content validation and curation and the unique identification number necessary for publication"

Trials Register of Promoting Health Interventions (TRoPHI)

Clinical Trials Registers

• List updated by Julie Glanville and Carol Lefebvre

For more information on what and how to search for randomized controlled trials of new drugs, see:

Knelangen M, Hausner E, Metzendorf M-I, Sturtz S, Waffenschmidt S. Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity. J Clin Epidemiol. 2018;94:69-75. doi:10.1016/j.jclinepi.2017.11.003

Conference proceedings are, technically speaking, collections of conference abstracts, papers, or presentations. They are usually published in a book or journal format or available online through conference websites.

Some databases index conference abstracts, papers, and presentations at the individual level (e.g., Conference Proceedings Citation Index via Web of Science Core Collection, Embase+Embase Classic via Ovid, APA PsycInfo via Ovid, Scopus), whereas others may only index proceedings as a whole/collection/compilation (e.g., MEDLINE via Ovid or PubMed). 

Conference documents can be confusing to researchers who might expect to be able to access the full-text of a paper when in fact it is often the case in health sciences that only an abstract is available. It may be necessary to contact the authors for more information.

Google Scholar can be used to identify grey literature, but there are a few issues to keep in mind:

• Use advanced search operators if needed
• You may have better success finding grey literature by using title searching (allintitle: or intitle:) - this will depend on your specific search though
• Decide on a cutoff point: How many records will you screen? Up to 50? 100? 200? 400?
• It is unclear what the character limit is in Google Scholar; check your search after you run it to see if any terms were cut off in the Google Scholar search box
• Capitalize the operators: Google prioritizes OR over the implied AND (the latter which is not necessary to enter) in the execution of the search logic
  • You can save space by using | (without spaces around it) instead of a capitalized OR between terms being used for the same concept 
  • Google Scholar ignores brackets and you don't have to type AND -- it is automatically inserted between terms when you are not using OR
    • Related to this: You cannot use a nested AND in a set of OR terms, e.g., ((wuhan [AND] coronavirus) OR "covid 19" OR "sars-cov-2") [AND] (diagnosis OR assay OR testing): the brackets are ignored, so the search will be executed as wuhan [AND] coronavirus|"covid 19"|"sars cov 2" [AND]  diagnosis|assay|testing 
• Google Scholar does not support word truncation, e.g., "family practi*" will not work as intended; instead, spell out "family practice"|"family practitioners"
• You can only see the first 1,000 results
• You can use phrase searching by adding quotation marks around the phrase of interest (quotation marks may turn off other features though)
• You can use * to find phrasal variations, e.g., "primary * care" can retrieve primary care OR primary health care OR primary medical care
• You can use proximity searching, e.g., "lung|pulmonary AROUND 3 cancer|neoplasms"
• The order of your search terms will affect the results
• It will be difficult to logically explain what is happening with Google Scholar searches: It is a "black box"
  • That said, you can keep a record of the searches as you executed them, the dates you ran them, the location you ran them from, and how many of the results you screened, for your search documentation
• You can use Publish or Perish, free software, to apply many of these suggestions, and to batch export records from Google Scholar

See also:

Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One. 2015;10(9):e0138237. 

Russell DM. Advanced search operators. 2022 [cited 2023 Jan 16].

Publish or Perish:

Harzing AW. Using Publish or Perish to do a literature review. 2017 [cited 2021 Sep 3].

Adams D. Exporting your data. 2016 [updated 2021 Mar 22; cited 2021 Sep 3]. 

Hickner, A. Tip #1: Bulk export from Google Scholar. 2022 [updated 2022 Oct 10; cited 2022 Dec 7].

Searching for citation relationships includes at least six approaches to finding references that you may not have picked up through your database searching. This can be especially important if it is difficult to comprehensively search for your topic, if the literature on your topic is dispersed across multiple databases or subject areas, or if the vocabulary used by authors is highly heterogenous for a given idea.

1. Reference list checking or backward citation searching

• Reference list checking is probably the most widely known supplementary approach to finding literature on a given topic; it is usually limited to searching the reference lists of included studies, and therefore takes place after the initial screening. This process can also be semi-automated using the citationchaser tool, in which case all of the cited references are exported and screened in a more rigorous process called backward citation searching.

2. Checking included studies in other relevant knowledge syntheses

• This is another accepted method of searching for relevant literature; this process can also be semi-automated using the citationchaser tool

3. Similar articles feature

• PubMed includes a "similar articles" feature that can be used to potentially identify additional relevant studies based on included studies (information on the computation of similar articles). Other databases may also have this functionality

4. Co-cited citation searching

• Method which involves retrieving and screening co-cited references (Hirt et al., 2024)

5. Co-citing citation searching

• Method which involves retrieving and screening co-citing references (Hirt et al., 2024)

6. Forward citation searching

• This method involves, for example, copying and pasting the titles or PMIDs (PubMed identifiers) of included studies into a citation database such as Scopus or into a tool like citationchaser to see who has cited the included studies of interest; these citing references are then exported and screened to potentially identify additional studies for inclusion in a review. 

Forward citation searching is available as a feature in the following resources:

See also:

Gusenbauer, M. (2024). Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices. Research Synthesis Methods, 15(5), 802-817.

Hirt, J., Nordhausen, T., Fuerst, T., Ewald, H., & Appenzeller-Herzog, C. (2024). Guidance on terminology, application, and reporting of citation searching: The TARCiS statement. BMJ, 385, e078384.

Literature mapping includes tools that allow you to visualize citation networks/connections/associations as well as tools that use artificial intelligence/machine learning to identify similar articles. These tools are useful if you have one or a few seed references to work with, or if you would like to map included studies to additional citations that may have been missed by other search methods, for example.

• Tay, A. (2024). List of literature mapping tools. Musings about librarianship.

Some suggestions:

• Connected Papers - Uses data from Semantic Scholar. Free and paid versions.
• Inciteful - Uses data from Semantic Scholar, Microsoft Academic Graph, Crossref, OpenCitations. Beta version
• Litmaps - Uses data from Semantic Scholar, OpenAlex, Crossref. Free and paid versions.
• PubMed Similar Articles
• ResearchRabbit - Uses data from Semantic Scholar and OpenAlex. Option to donate to help fund this tool.

See also: Tay, A. (2024, Jun 25). All about citation chasing and tools that [do] citation chasing like Citation Gecko, Connected papers, Research Rabbit, LitMaps and more. Musings about librarianship.

Handsearching is a term that predates online tables of contents and generally involves reading the tables of contents of journals that are highly likely to publish literature on your topic. This may pick up studies that were missed by the database searches, for example because they used terms that you did not include in your search strategy or because the journal is not indexed in the databases you chose to search (that said, it is a good idea to search databases that index the journals relevant to your topic).

We recommend the Cochrane free online course on Hand Searching to help you find:

• Conference proceedings
• Major journals
• Relevant, non-indexed journals

Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766-2777.

Bonato S. Searching the Grey Literature: A Handbook for Searching Reports, Working Papers, and Other Unpublished Research. Lanham, Maryland: Rowman & Littlefield; 2018.

Cooper, C., Booth, A., Britten, N., & Garside, R. (2017). A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Systematic Reviews, 6(1), 234.

Gusenbauer, M. (2024). Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices. Research Synthesis Methods, 15(5), 802-817.  

Haddaway, N.R., Collins, A.M., Coughlin, D, & Kirk, S. (2015). The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One, 10(9), e0138237.

Hirt, J., Nordhausen, T., Fuerst, T., Ewald, H., & Appenzeller-Herzog, C. (2024). Guidance on terminology, application, and reporting of citation searching: The TARCiS statement. BMJ, 385, e078384.

Hoy, M. B. (2020, Jan-Mar). Rise of the Rxivs: How preprint servers are changing the publishing process. Medical Reference Services Quarterly, 39(1), 84-89.

Knelangen, M., Hausner, E., Metzendorf, M.-I., Sturtz, S., Waffenschmidt, S. (2018). Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity. Journal of Clinical Epidemiology, 94, 69-75.

Tay, A. (2024, Jun 25). All about citation chasing and tools that [do] citation chasing like Citation Gecko, Connected papers, Research Rabbit, LitMaps and more. Musings about librarianship.

Find a librarian in your subject area to help you with your knowledge synthesis project.

Or contact the librarians at the
Schulich Library of Physical Sciences, Life Sciences, and Engineering
schulich.library@mcgill.ca