AI and Text Mining for Searching and Screening the Literature

Given the high sensitivity/recall of most knowledge synthesis search strategies, researchers are investigating the feasibility of using text mining and machine learning in the record screening phase, to reduce the burden on reviewers while still capturing relevant studies from the search set. I recommend that librarians understand what software is available for this, to allow them to advise users on their options.

The approaches that have been explored can be generally categorized into the following:

1. Improving workflow through screening prioritisation - for process parallelisation, allowing reviewers to perform tasks in parallel (e.g., prioritising relevant records in screening phase so that full texts, data extraction, and synthesis can begin earlier)
2. Using software as a second reviewer
3. Speeding up the screening process

Hamel et al. (2021) provide guidance on using artificial intelligence (including text mining) for title and abstract screening in systematic reviews and other knowledge syntheses.

Screening: AI and text mining tools for non-programmers

Digital Evidence Synthesis Tool (DEST) Evaluations - Examines automation tools in evidence synthesis, including tools available for the screening stage

• Use filters in the left column to limit to, e.g., Evidence synthesis stage: Screening, then click on "List records" button at the top
• Focuses on tools for health and climate change syntheses, but tools are often agnostic or applicable to fields more generally speaking, like health sciences

• Requires installing Python and running commands, but instructions are easy to follow
• Uploaded references should already be deduplicated
• See: van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2), 125-133. https://doi.org/10.1038/s42256-020-00287-7 

• See: Gartlehner, G., Wagner, G., Lux, L., Affengruber, L., Dobrescu, A., Kaminski-Hartenthaler, A., & Viswanathan, M. (2019). Assessing the accuracy of machine-assisted abstract screening with DistillerAI: A user study. Systematic Reviews, 8(1), 277. https://doi.org/10.1186/s13643-019-1221-3

• See: Olofsson, H., Brolund, A., Hellberg, C., Silverstein, R., Stenström, K., Österberg, M., & Dagerhamn, J. (2017). Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Research synthesis methods, 8(3), 275-280. https://doi.org/10.1002/jrsm.1237

• See: Howard, B.E., Phillips, J., Miller, K. et al. (2016). SWIFT-Review: A text-mining workbench for systematic review. Systematic Reviews, 5, 87. https://doi.org/10.1186/s13643-016-0263-z

In addition to the references below, you can use the following search strategy in Google Scholar to identify more literature on these and other tools (this is not a comprehensive search): 

intitle:"abstract screening"|abstrackr|intitle:"artificial intelligence"|intitle:"machine learning"|asreview|intitle:"citation screening"|colandr|covidence|distillerai|distillersr|"eppi-reviewer"|"pico portal"|rayyan|robotanalyst|intitle:"study identification"|"swift review" screening|eligibility|"study identification"|"study selection"|prioritization|"stopping criteria"|"stopping rules" "systematic reviews"|"knowledge syntheses"|”evidence syntheses”|”rapid reviews”|”practice guidelines”

Suggested references

• Boetje, J., & van de Schoot, R. (2024). The Safe Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses. Systematic Reviews, 13(1), 81.  

• Chan, Y. T., Abad, J. E., Dibart, S., & Kernitsky, J. R. (2024). Assessing the Article Screening Efficiency of Artificial Intelligence for Systematic Reviews. Journal of Dentistry, 149, 105259.  

• Gartlehner, G., Wagner, G., Lux, L., Affengruber, L., Dobrescu, A., Kaminski-Hartenthaler, A., & Viswanathan, M. (2019). Assessing the Accuracy of Machine-Assisted Abstract Screening with DistillerAI: A User Study. Systematic Reviews, 8(1), 277.

• Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019, Nov 15). Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. Systematic Reviews, 8(1), 278.

• Hamel, C., Hersi, M., Kelly, S. E., Tricco, A. C., Straus, S., Wells, G., Pham, B., & Hutton, B. (2021, Dec 20). Guidance for Using Artificial Intelligence for Title and Abstract Screening While Conducting Knowledge Syntheses. BMC Medical Research Methodology, 21(1), 285.

• Li, M., Sun, J., & Tan, X. (2024). Evaluating the Effectiveness of Large Language Models in Abstract Screening: A Comparative Analysis. Systematic Reviews, 13(1), 219.

• O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using Text Mining for Study Identification in Systematic Reviews: A Systematic Review of Current Approaches. Systematic Reviews, 4, 5.

• Olorisade, B. K., Quincey, E. d., Brereton, P., & Andras, P. (2016). A Critical Analysis of Studies That Address the Use of Text Mining for Citation Screening in Systematic Reviews. Paper presented at the Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, Limerick, Ireland.

• Przybyla, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M. A., McNaught, J., von Elm, E., . . . Ananiadou, S. (2018). Prioritising References for Systematic Reviews with Robotanalyst: A User Study. Research Synthesis Methods.

• van de Schoot, R. (2023). Comprehensive Guide to Machine Learning Software for Text Screening. GitHub. 

• van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An Open Source Machine Learning Framework for Efficient and Transparent Systematic Reviews. Nature Machine Intelligence, 3(2), 125-133. 

• Waffenschmidt, S., Sieben, W., Jakubeit, T., Knelangen, M., Overesch, I., Buhn, S., Pieper, D., Skoetz, N., & Hausner, E. (2023). Increasing the Efficiency of Study Selection for Systematic Reviews Using Prioritization Tools and a Single-Screening Approach. Systematic Reviews, 12(1), 161.

• Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020). Error Rates of Human Reviewers During Abstract Screening in Systematic Reviews. PLoS One, 15(1), e0227742.