AI and Text Mining for Searching and Screening the Literature

Stansfield, O'Mara-Eves, and Thomas (2017) report five ways in which text mining tools can assist in search strategy development:

1. Improving the precision of searches (i.e., the proportion of retrieved records that are relevant), for example, by identifying more precise phrasal terms instead of using single-word terms in a search
2. Improving the sensitivity of searches (i.e., the proportion of relevant studies retrieved by the search over the total number of relevant studies in the database) by identifying additional search terms (validating this requires the development of a gold standard/quasi-gold standard/reference set, which is often used when developing search filters or hedges)
3. Assisting in the translation of search strategies from one database and/or platform to another
4. Searching and screening within an integrated system
5. Developing objectively derived search strategies

Using text mining techniques to increase the objectivity of search strategies requires a more sophisticated use of tools that librarians or other searchers may or may not be prepared to implement. Decisions about cutoffs for high frequency terms, for example, and calculations to establish high frequencies require somewhat large sets of relevant references (which can be derived based on the included studies of relevant systematic reviews, for example) as well as a population set of random records against which one can test whether a term is high frequency across documents in general (for example, words that are high-frequency due to common check tags such as 'human') or in the relevant documents only. 

Text mining, like data science in general, also involves a great deal of preprocessing, which tools may or may not handle. Preprocessing includes data cleaning and normalization techniques such as:

• Changing all characters to lower case
• Removing punctuation
• Stripping whitespace
• Removing numbers
• Removal of stopwords
• Stemming
• Lemmatization

Some of the tools listed allow for customization of these procedures, while some are preconfigured. Programming tools such as the tm package in R or quanteda allow for much more flexibility than some of the tools covered here, but they are also much more difficult to use if one is not accustomed to programming.

Digital Evidence Synthesis Tool (DEST) Evaluations - Examines automation tools in evidence synthesis, including tools available for the searching stage

• Use filters in the left column to limit to, e.g., Evidence synthesis stage: Searching/Deduplication, then click on "List records" button at the top of the column
• Focuses on tools for health and climate change syntheses, but tools are often agnostic or applicable to fields more generally speaking, like health sciences

These tools should be used with caution: They may apply the correct syntax to translate one strategy to another database/interface and make it seem that the subject headings have also been mapped correctly when in fact they simply change the syntax but do not adjust the subject headings to the corresponding vocabulary (e.g., when translation from PubMed or Ovid MEDLINE to Embase, they will continue to use MeSH terms instead of EMTREE terms). They are useful if you understand the fundamentals of searching within the applicable databases and how the databases/platforms work, but the searches will require reviewing and editing.

Useful repository of articles:

• Bond, M., Finnerty, A., O'Mara-Eves, A., O'Driscoll, P., Thomas, J., Minx, J., Callaghan, M., & Scheelbeek, P. (2024). Digital Evidence Synthesis Tool Evaluations. EPPI Visualiser database.
  • In left column, under "Evidence Synthesis Stage": Select "Searching/Deduplicating" > Click "List Records"

Selected articles:

• Adam, G. P., DeYoung, J., Paul, A., Saldanha, I. J., Balk, E. M., Trikalinos, T. A., & Wallace, B. C. (2024). Literature search sandbox: A large language model that generates search queries for systematic reviews. JAMIA Open, 7(3), ooae098.
• Grames EM, Stillman AN, Tingley MW, Elphick CS. (2019). An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods in Ecology and Evolution, 10(10), 1645-54.
• Hatami, N., Zarenezhad, M., Javdani, F., Keshavarz, P., Kalani, N., Sadeghinikoo, A., Ghaedi, M., Nobari, M. N., Feily, A., & Hashemi, S. A. (2023). Meta-phill: Feasibility of a new metadata repository for evidence-based practice in literature review. Research Square.
• Hausner, E., Knelangen, M., & Waffenschmidt, S. (2022). Use of text mining tools in the development of search strategies – Comparison of different approaches. Journal of Clinical Epidemiology, 149, 254-256.  
• Kapp, C., Fujita-Rohwerder, N., Lilienthal, J., Sieben, W., Waffenschmidt, S., & Hausner, E. (2024). The searchbuildr shiny app: A new implementation of the objective approach for search strategy development in systematic reviews. Cochrane Evidence Synthesis and Methods, 2(6), e12078.   
• Lefebvre, C., Glanville, J., Briscoe, S., Featherstone, R., Littlewood, A., Metzendorf, M.-I., Noel-Storr, A., Paynter, R., Rader, T., Thomas, J., Wieland, L. S. (2024, Sep). Technical Supplement to Chapter 4: Searching for and selecting studies. 3.2.3 Text mining, machine learning and artificial intelligence for term selection and strategy building. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane. Available from www.training.cochrane.org/handbook.
• O'Keefe, H., Rankin, J., Wallace, S. A., & Beyer, F. (2023). Investigation of text-mining methodologies to aid the construction of search strategies in systematic reviews of diagnostic test accuracy—A case study. Research Synthesis Methods, 14(1), 79-98.
• Paynter, R. A., Featherstone, R., Stoeger, E., Fiordalisi, C., Voisin, C., & Adam, G. P. (2021). A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools. Journal of Clinical Epidemiology.
• Smalheiser, N. R., Fragnito, D. P., & Tirk, E. E. (2021). Anne O’Tate: Value-added PubMed search engine for analysis and text mining. PloS One, 16(3), e0248335.  
• Stansfield, C., O'Mara-Eves, A., & Thomas, J. (2017). Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods, 8(3), 355-365.
• Staudinger, M., Kusa, W., Piroi, F., Lipani, A., & Hanbury, A. (2024). A reproducibility and generalizability study of large language models for query generation. Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, Tokyo, Japan.