Systematic Reviews, Scoping Reviews, and other Knowledge Syntheses

Removing duplicate records with Covidence

McGill University Library has an institutional license for Covidence. For access and support for Covidence, please consult: https://support.covidence.org/help/mcgill-university-library

Covidence removes duplicate records automatically. The Covidence support pages explain the process in detail.

Please note: It has been reported that exporting records from Ovid databases in RIS format results in better removal of duplicates than exporting records from Ovid databases in EndNote format.

Study supporting the use of Covidence: McKeown S, Mir ZM. Considerations for Conducting Systematic Reviews: Evaluating the Performance of Different Methods for De-Duplicating References. Syst Rev. 2021;10(1):38. doi:10.1186/s13643-021-01583-y

Removing duplicate records with EndNote

You can use one of the following methods to remove duplicates from a merged EndNote library.

Before deduplicating, you will need a merged EndNote library containing the records from all your separate EndNote libraries (if applicable) for the individual database searches, if you had previously exported records from each database into separate libraries:

• Create a new EndNote library that will contain the records from all the databases you searched (I like to put DEDUPING in the EndNote library name)
• Import the records from each EndNote library you created for the individual database searches:
  • Go to EndNote menu > File > Import > File
  • Next to "Import File", browse to choose the .enl file (NOT the .enlx file -- you may need to open that beforehand to "create" the .enl file) for each library of downloaded records from your searches and select "EndNote Library" as the "Import Option"
  • Once all the records have been added to this new library, check to make sure the final number of records, before removing duplicates, matches the sum of the records found for all the database searches

Using this merged library of records from your individual database searches, you are now ready to remove duplicates. Here are two methods you can use:

1) Earlier version of "Bramer method" for deduplicating, with steps provided in Word document format:

2) Paper describing more advanced configuration options for removing duplicates in EndNote: Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240-243. doi:10.3163/1536-5050.104.3.014

After deduplication

Create a compressed library for backup after having removed as many duplicates as possible, with a filename like SearchTerms-yyyymmdd-Deduplicated—xRecords.enlx. This will be the library for screening. You can then export this deduplicated record set from EndNote to Covidence:

• In EndNote, select all the records for screening (Ctrl-A on a PC or Cmd-A on a Mac)
• From the EndNote menu, click on File > Export > Save as type: Text File (*.txt) and Output Style: RefMan (RIS Export), note/change the location to which you are saving the file if needed, then save the file
• Import the text file into Covidence.

Removing duplicate records with the IEBH SR-Accelerator Deduplicator 

Use the online Deduplicator tool.

• Help using Deduplicator

Large sets of records

The offline DeDuplicator (old version) may be useful for large sets of records (which used to be considered ≥ 2000). Download the SRA-dedupe-UI application from GitHub. As of October 2020, there were only Linux and Windows versions available.

• Help using DeDuplicator Offline
• When exporting your records to an XML file, don't forget to select them all (Help Importing/Exporting EndNote records)
• If you choose to use the stand alone executable version for Windows, you may get a message that "Windows protected your PC": Click on More info and then a Run anyway button will appear, which you should click if you feel comfortable trusting the software developers

Keep a copy of the RIS or XML file for your records.

See also: Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2015). Better Duplicate Detection for Systematic Reviewers: Evaluation of Systematic Review Assistant-Deduplication Module. Systematic Reviews, 4(1), 6. doi:10.1186/2046-4053-4-6

Find a librarian in your subject area to help you with your knowledge synthesis project.

Or contact the librarians at the
Schulich Library of Physical Sciences, Life Sciences, and Engineering
schulich.library@mcgill.ca