Systematic Reviews, Scoping Reviews, and other Knowledge Syntheses

What are supplementary search methods?

Finding studies relevant to your question should not depend solely on database searching: Supplementary search methods are recommended in order to avoid different forms of bias in what studies are ultimately included in the review.

Examples of supplementary search methods:

• Finding grey literature, e.g., theses & dissertations, preprint servers, clinical trials registers, using Google Scholar
• Searching for citation relationships (reference list searching, scanning the included studies in other relevant knowledge syntheses, similar articles searching, co-cited articles searching, or forward citation searching)
• Handsearching
• Contacting authors

Grey literature, according to the Cochrane Handbook, is usually understood to be literature not formally published in books or journals. This can include theses or dissertations, conference proceedings, clinical trials registries, white papers, government reports, and more. Some grey literature will be retrievable through database searching, but it depends on the databases you have chosen to search and what kind of content the databases index. For example, MEDLINE does not index much grey literature, whereas you can retrieve some conference proceedings indexed in Web of Science Core Collection databases available through McGill.

You may be interested in finding grey literature available on websites. One suggestion is to identify associations, organisations, institutions, etc. that are likely to make documents or reports of relevance to your question available on their websites, and to then selectively search or browse those sites.

Resources to help you identify grey literature:

Grey Matters, produced by the Canadian Agency for Drugs and Technologies in Health (CADTH)

Resources to help you identify grey literature for interventions in crime & justice, education, international development, and social welfare:

Searching for Studies: A Guide to Information Retrieval for Campbell Systematic Reviews, 2017

Resources to help you identify Canadian health policy:

A Guide to Governmental and Grey Literature Sources for Public Policy Research, University of Toronto

For more information on searching the grey literature, the University of Toronto Gerstein Science Information Centre has a comprehensive guide on searching the grey literature, including lists of potential resources to search.

Theses and dissertations are potentially rich sources of grey literature given the depth of research involved in writing a thesis or dissertation. They are also usually considered grey literature in and of themselves as well.

See also:

• McGill guide on finding theses & dissertations

Preprint servers are free online archives that allow researchers to identify studies in rapidly developing fields and can be used to identify studies that have not gone through the more formalized and traditional peer review process​. Depending on the site, searching and exporting functionality may be more or less available.

Hoy, M. B. (2020, Jan-Mar). Rise of the Rxivs: How preprint servers are changing the publishing process. Medical Reference Services Quarterly, 39(1), 84-89.

Clinical trials may go unreported in the published literature. One useful method to identify unpublished clinical trials is to search clinical trials registries. The results may be available within the registries or you may need to contact the researchers associated with the trial for further information.

ClinicalTrials.gov

• EndNote software compatibility: Download filter

International Clinical Trials Registry Platform (WHO)

• EndNote software compatibility: Download filter

Clinical Trials Registers

• List updated by Julie Glanville and Carol Lefebvre

For more information on what and how to search for randomized controlled trials of new drugs, see:

Knelangen M, Hausner E, Metzendorf M-I, Sturtz S, Waffenschmidt S. Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity. J Clin Epidemiol. 2018;94:69-75. doi:10.1016/j.jclinepi.2017.11.003

Google Scholar can be used to identify grey literature, but there are a few issues to keep in mind:

• Use advanced search operators if needed [5]
• You may have better success finding grey literature by using title searching (allintitle: or intitle:) [1] - this will depend on your specific search though
• You may need to screen the first 400 results rather than the suggested practice of screening the first 50-100 [1]
• Google Scholar only allows 256 characters in the search line [2]
• Capitalize the operators: Google prioritizes OR over the implied AND (the latter which is not necessary to enter) in the execution of the search logic
  • You can save space by using | (without spaces around it) instead of a capitalized OR between terms being used for the same concept 
  • Google Scholar ignores brackets and you don't have to type AND -- it is automatically inserted between terms when you are not using OR
    • Related to this: You cannot use a nested AND in a set of OR terms, e.g., ((wuhan [AND] coronavirus) OR "covid 19" OR "sars-cov-2") [AND] (diagnosis OR assay OR testing): the brackets are ignored, so the search will be executed as wuhan [AND] coronavirus|"covid 19"|"sars cov 2" [AND]  diagnosis|assay|testing 
• Google Scholar does not support word truncation, e.g., "family practi*" will not work as intended; instead, spell out "family practice"|"family practitioners"
• You can only see the first 1,000 results [2]
• You can use phrase searching by adding quotation marks around the phrase of interest (quotation marks may turn off other features though)
• You can use * to find phrasal variations, e.g., "primary * care" can retrieve primary care OR primary health care OR primary medical care
• You can use proximity searching, e.g., "lung|pulmonary AROUND 3 cancer|neoplasms" [3]
• The order of your terms will affect the results
• It will be difficult to logically explain what is happening with Google Scholar searches: It is a bit of a black box
• That said, you can keep a record of the searches as you executed them, the dates you ran them, the location you ran them from, and how many of the results you screened, for your search documentation
• You can use Publish or Perish, free software, to apply many of these suggestions [4], and to batch export records from Google Scholar [5, 6]

[1] Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One. 2015;10(9):e0138237. doi:10.1371/journal.pone.0138237

[2] Grey literature for health sciences. 

[3] Russell DM. Advanced search operators [Internet]. 2022 [cited 2023 Jan 16].

[4] Harzing AW. Using Publish or Perish to do a literature review [Internet]. 2017 [cited 2021 Sep 3].

[5] Adams D. Exporting your data [Internet]. 2016 [updated 2021 Mar 22; cited 2021 Sep 3]. 

[6] Hickner, A. Tip #1: Bulk export from Google Scholar [Internet]. 2022 [updated 2022 Oct 10; cited 2022 Dec 7].

Searching for citation relationships

Searching for citation relationships includes at least five approaches to finding references that you may not have picked up through your database searching. This can be especially important if your database search strategies were not thorough, if the literature on your topic is dispersed across multiple databases or subject areas, or if the vocabulary used by authors is highly heterogenous for a given idea.

1. Reference list searching

• This method is probably the most widely known supplementary approach to finding literature on a given topic; it is usually limited to searching the reference lists of included studies, and therefore takes place after screening

2. Checking included studies in other relevant knowledge syntheses

• This is another accepted method of searching for relevant literature

3. Similar articles feature

• PubMed includes a similar articles feature that can be used to identify more studies (information on the computation of similar articles). Other databases may also have this functionality

4. Co-cited article searching

• Searching the reference lists of articles citing included studies may also be beneficial

5. Forward citation searching

• This method involves copying and pasting, for example, the titles or PMIDs of included studies into a citation database to see who has cited the included studies of interest

Forward citation searching is available as a feature in the following resources:

Literature mapping includes tools that allow you to visualize citation networks/connections/associations as well as tools that use artificial intelligence/machine learning to identify similar articles. These tools are useful if you have one or a few seed articles to work with, or if you would like to map included studies to additional citations that may have been missed by other search methods, for example.

• List of innovative literature mapping tools, by Aaron Tay (Singapore Management University)

Some suggestions:

• Connected Papers - free and paid versions
• Inciteful - beta version
• Litmaps - free and paid versions
• ResearchRabbit - option to donate to help fund this tool

Handsearching

Handsearching is a term that predates online tables of contents and generally involves reading the tables of contents of journals that are highly likely to publish literature on your topic. This may pick up studies that were missed by the database searches, for example because they used terms that you did not include in your search strategy or because the journal is not indexed in the databases you chose to search (that said, it is a good idea to search databases that index the journals relevant to your topic).

We recommend the Cochrane free online course on Hand Searching to help you find:

• Conference proceedings
• Major journals
• Relevant, non-indexed journals

Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766-2777. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.23605. doi:10.1002/asi.23605

Cooper, C., Booth, A., Britten, N., & Garside, R. (2017). A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev, 6(1), 234. doi:10.1186/s13643-017-0625-1

Find a librarian in your subject area to help you with your knowledge synthesis project.

Or contact the librarians at the
Schulich Library of Physical Sciences, Life Sciences, and Engineering
schulich.library@mcgill.ca