AI and Text Mining for Searching and Screening the Literature

Background references on text mining

• Maceli, M. (2016). Introduction to text mining with R for information professionals. code{4}lib Journal(33).
• Miner, G., Elder Iv, J., Fast, A., Hill, T., Nisbet, R., & Delen, D. (2012). Practical text mining and statistical analysis for non-structured text data applications. Burlington: Elsevier Science.
• Kwartler, T. (2017). Text mining in practice with R. Hoboken, NJ: John Wiley & Sons.
• Welbers, K., Van Atteveldt, W., & Benoit, K. (2017). Text analysis in R. Communication Methods and Measures, 11(4), 245-265. 

Background references on AI and text mining in knowledge syntheses

• Ananiadou, S., Rea, B., Okazaki, N., Procter, R., & Thomas, J. (2009). Supporting systematic reviews using text mining. Social Science Computer Review, 27(4), 509-523.
• Bond, M., Finnerty, A., O'Mara-Eves, A., O'Driscoll, P., Thomas, J., Minx, J., Callaghan, M., & Scheelbeek, P. (2024). Digital Evidence Synthesis Tool Evaluations. EPPI Visualiser database. 
• O'Connor AM, Clark J, Thomas J, Spijker R, Kusa W, Walker VR, et al. (2024). Large language models, updates, and evaluation of automation tools for systematic reviews: a summary of significant discussions at the eighth meeting of the International Collaboration for the Automation of Systematic Reviews (ICASR). Systematic Reviews, 13(1),290.
• Paynter, R., Banez, L. L., Berliner, E., Erinoff, E., Lege-Matsuura, J., Potter, S., & Uhl, S. (2016). EPC methods: An exploration of the use of text-mining software in systematic reviews. In AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US).
  • Paynter R, Banez LL, Erinoff E, Lege-Matsuura J, Potter S. Commentary on EPC methods: An exploration of the use of text-mining software in systematic reviews. (2017). Journal of Clinical Epidemiology, 84, 33-6. 
• Roth, S., & Wermer-Colan, A. (2023). Machine learning methods for systematic reviews: A rapid scoping review. Delaware Journal of Public Health, 9(4), 40-47. 
• Stansfield, C., O'Mara-Eves, A., & Thomas, J. (2017). Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods, 8(3), 355-365.
• Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., . . . Elliott, J. (2017). Living systematic reviews: 2. Combining human and machine effort. Journal of Clinical Epidemiology, 91, 31-37. 

Useful repository of articles:

• Bond, M., Finnerty, A., O'Mara-Eves, A., O'Driscoll, P., Thomas, J., Minx, J., Callaghan, M., & Scheelbeek, P. (2024). Digital Evidence Synthesis Tool Evaluations. EPPI Visualiser database.
  • In left column, under "Evidence Synthesis Stage": Select "Searching/Deduplicating" > Click "List Records"

Selected articles:

• Adam, G. P., DeYoung, J., Paul, A., Saldanha, I. J., Balk, E. M., Trikalinos, T. A., & Wallace, B. C. (2024). Literature search sandbox: A large language model that generates search queries for systematic reviews. JAMIA Open, 7(3), ooae098.
• Grames EM, Stillman AN, Tingley MW, Elphick CS. (2019). An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods in Ecology and Evolution, 10(10), 1645-54.
• Hatami, N., Zarenezhad, M., Javdani, F., Keshavarz, P., Kalani, N., Sadeghinikoo, A., Ghaedi, M., Nobari, M. N., Feily, A., & Hashemi, S. A. (2023). Meta-phill: Feasibility of a new metadata repository for evidence-based practice in literature review. Research Square.
• Hausner, E., Knelangen, M., & Waffenschmidt, S. (2022). Use of text mining tools in the development of search strategies – Comparison of different approaches. Journal of Clinical Epidemiology, 149, 254-256.  
• Kapp, C., Fujita-Rohwerder, N., Lilienthal, J., Sieben, W., Waffenschmidt, S., & Hausner, E. (2024). The searchbuildr shiny app: A new implementation of the objective approach for search strategy development in systematic reviews. Cochrane Evidence Synthesis and Methods, 2(6), e12078.   
• Lefebvre, C., Glanville, J., Briscoe, S., Featherstone, R., Littlewood, A., Metzendorf, M.-I., Noel-Storr, A., Paynter, R., Rader, T., Thomas, J., Wieland, L. S. (2024, Sep). Technical Supplement to Chapter 4: Searching for and selecting studies. 3.2.3 Text mining, machine learning and artificial intelligence for term selection and strategy building. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane. Available from www.training.cochrane.org/handbook.
• O'Keefe, H., Rankin, J., Wallace, S. A., & Beyer, F. (2023). Investigation of text-mining methodologies to aid the construction of search strategies in systematic reviews of diagnostic test accuracy—A case study. Research Synthesis Methods, 14(1), 79-98.
• Paynter, R. A., Featherstone, R., Stoeger, E., Fiordalisi, C., Voisin, C., & Adam, G. P. (2021). A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools. Journal of Clinical Epidemiology.
• Smalheiser, N. R., Fragnito, D. P., & Tirk, E. E. (2021). Anne O’Tate: Value-added PubMed search engine for analysis and text mining. PloS One, 16(3), e0248335.  
• Stansfield, C., O'Mara-Eves, A., & Thomas, J. (2017). Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods, 8(3), 355-365.
• Staudinger, M., Kusa, W., Piroi, F., Lipani, A., & Hanbury, A. (2024). A reproducibility and generalizability study of large language models for query generation. Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, Tokyo, Japan.
   

In addition to the references below, you can use the following search strategy in Google Scholar to identify more literature on these and other tools (this is not a comprehensive search): 

intitle:"abstract screening"|abstrackr|intitle:"artificial intelligence"|intitle:ChatGPT|intitle:"large language models"|intitle:"machine learning"|asreview|intitle:"citation screening"|colandr|covidence|distillerai|distillersr|"eppi-reviewer"|"pico portal"|rayyan|robotanalyst|intitle:"study identification"|"swift review" screening|eligibility|"study identification"|"study selection"|prioritization|"stopping criteria"|"stopping rules" "systematic reviews"|"systematic literature reviews"|"knowledge syntheses"|”evidence syntheses”|”rapid reviews”|”practice guidelines”

Useful repository of articles

• Bond, M., Finnerty, A., O'Mara-Eves, A., O'Driscoll, P., Thomas, J., Minx, J., Callaghan, M., & Scheelbeek, P. (2024). Digital Evidence Synthesis Tool Evaluations. EPPI Visualiser database.
  • In left column, under "Evidence Synthesis Stage": Select "Screening" > Click "List Records"

Selected articles

• Boetje, J., & van de Schoot, R. (2024). The Safe Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses. Systematic Reviews, 13(1), 81.  

• Chan, Y. T., Abad, J. E., Dibart, S., & Kernitsky, J. R. (2024). Assessing the Article Screening Efficiency of Artificial Intelligence for Systematic Reviews. Journal of Dentistry, 149, 105259.  

• Gartlehner, G., Wagner, G., Lux, L., Affengruber, L., Dobrescu, A., Kaminski-Hartenthaler, A., & Viswanathan, M. (2019). Assessing the Accuracy of Machine-Assisted Abstract Screening with DistillerAI: A User Study. Systematic Reviews, 8(1), 277.

• Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019, Nov 15). Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. Systematic Reviews, 8(1), 278.

• Hamel, C., Hersi, M., Kelly, S. E., Tricco, A. C., Straus, S., Wells, G., Pham, B., & Hutton, B. (2021, Dec 20). Guidance for Using Artificial Intelligence for Title and Abstract Screening While Conducting Knowledge Syntheses. BMC Medical Research Methodology, 21(1), 285.

• Li, M., Sun, J., & Tan, X. (2024). Evaluating the Effectiveness of Large Language Models in Abstract Screening: A Comparative Analysis. Systematic Reviews, 13(1), 219.

• O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using Text Mining for Study Identification in Systematic Reviews: A Systematic Review of Current Approaches. Systematic Reviews, 4, 5.

• Olorisade, B. K., Quincey, E. d., Brereton, P., & Andras, P. (2016). A Critical Analysis of Studies That Address the Use of Text Mining for Citation Screening in Systematic Reviews. Paper presented at the Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, Limerick, Ireland.

• Przybyla, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M. A., McNaught, J., von Elm, E., . . . Ananiadou, S. (2018). Prioritising References for Systematic Reviews with Robotanalyst: A User Study. Research Synthesis Methods.

• van de Schoot, R. (2023). Comprehensive Guide to Machine Learning Software for Text Screening. GitHub. 

• van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An Open Source Machine Learning Framework for Efficient and Transparent Systematic Reviews. Nature Machine Intelligence, 3(2), 125-133. 

• Waffenschmidt, S., Sieben, W., Jakubeit, T., Knelangen, M., Overesch, I., Buhn, S., Pieper, D., Skoetz, N., & Hausner, E. (2023). Increasing the Efficiency of Study Selection for Systematic Reviews Using Prioritization Tools and a Single-Screening Approach. Systematic Reviews, 12(1), 161.

• Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020). Error Rates of Human Reviewers During Abstract Screening in Systematic Reviews. PLoS One, 15(1), e0227742.