Given the high sensitivity/recall of most knowledge synthesis search strategies, researchers are investigating the feasibility of using text mining and machine learning in the record screening phase to reduce the burden on reviewers while still capturing relevant studies from the search set. I recommend that librarians understand what software is available for this, to allow them to advise users on their options.
The approaches that have been explored can be generally categorized into the following:
O'Mara-Eves, Thomas, McNaught, Miwa, and Ananiadou (2015) systematically reviewed the literature on text mining in the record screening (i.e., study selection or study identification) phase and concluded that prioritising records for screening could be considered a safe method in live reviews, but that using screening software as a second reviewer should be done with caution and that using text mining to automatically eliminate studies needs more investigation. Below are some of the tools they explored as well as others that have appeared on the scene since they published their review. Librarians and other advanced searchers are well situated to advise research groups on the use of these tools in the knowledge synthesis workflow.
Hamel et al. (2021) provide updated guidance on using artificial intelligence (including text mining) for title and abstract screening in systematic reviews and other knowledge syntheses.
In addition to the references below, you can use the following search strategy in Google Scholar to identify more literature on these and other tools (this is not a comprehensive search):
Gartlehner, G., Wagner, G., Lux, L., Affengruber, L., Dobrescu, A., Kaminski-Hartenthaler, A., & Viswanathan, M. (2019). Assessing the Accuracy of Machine-Assisted Abstract Screening with DistillerAI: A User Study. Systematic Reviews, 8(1), 277. doi:10.1186/s13643-019-1221-3
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019, Nov 15). Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. Systematic Reviews, 8(1), 278. doi:10.1186/s13643-019-1222-2
Hamel, C., Hersi, M., Kelly, S. E., Tricco, A. C., Straus, S., Wells, G., Pham, B., & Hutton, B. (2021, Dec 20). Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses. BMC Medical Research Methodology, 21(1), 285. doi:10.1186/s12874-021-01451-2
O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using Text Mining for Study Identification in Systematic Reviews: A Systematic Review of Current Approaches. Systematic Reviews, 4, 5. doi:10.1186/2046-4053-4-5
Olorisade, B. K., Quincey, E. d., Brereton, P., & Andras, P. (2016). A Critical Analysis of Studies That Address the Use of Text Mining for Citation Screening in Systematic Reviews. Paper presented at the Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, Limerick, Ireland. doi:10.1145/2915970.2915982
Przybyla, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M. A., McNaught, J., von Elm, E., . . . Ananiadou, S. (2018). Prioritising References for Systematic Reviews with Robotanalyst: A User Study. Research Synthesis Methods. doi:10.1002/jrsm.1311
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An Open Source Machine Learning Framework for Efficient and Transparent Systematic Reviews. Nature Machine Intelligence, 3(2), 125-133. doi:10.1038/s42256-020-00287-7
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLoS One, 15(1), e0227742. doi:10.1371/journal.pone.0227742
McGill Library • Questions? Ask us!
Privacy notice