Skip to main content

Systematic Reviews, Scoping Reviews, and other Knowledge Syntheses

Deduplicating in EndNote

Removing duplicates

Systematic review software may offer this functionality, or you can use one of the following methods to remove duplicates from a merged EndNote library.

Before deduplicating, you will need a merged EndNote library containing the records from all your separate EndNote libraries for the individual database searches, if you had previously exported records from each database into separate libraries:

  • Create a new EndNote library that will contain the records from all the databases you searched (I like to put DEDUPING in the EndNote library name)
  • Import the records from each EndNote library you created for the individual database searches:
    • Go to EndNote menu > File > Import > Import file
    • Next to "Import file", browse to find the .enl file (NOT the .enlx file) for each library and select "EndNote library" as the "Import option"
    • Once all the records have been added to this new library, check to make sure the final number of records, before removing duplicates, matches the sum of the records found in each database search

Using this merged library of records from your individual database searches, you are now ready to remove duplicates. Here are two methods you can use:

1) Earlier version of "Bramer method" for deduplicating, with steps provided in Word document format:

2) Paper describing more advanced configuration options for removing duplicates in EndNote: Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240-243. doi:10.3163/1536-5050.104.3.014

After deduplication

Create a compressed library for backup after having removed as many duplicates as possible, with a filename like SearchTerms-yyyymmdd-Deduplicated—xRecords.enlx. This will be the library for screening. You can export the deduplicated library to Rayyan (see Rayyan guide for information on proper format for the EndNote file), for example, or other synthesis software, to facilitate the screening process, which usually involves more than one reviewer.

Deduplicating with SRA DeDuplicator

Deduplicating with SRA DeDuplicator 

< 2000 records:

Create a free account with CREBP SRA, log in and import your library (ideally with < 2000 if performing this online), and use the Deduplicate tool.

≥ 2000 records:

The offline DeDuplicator is useful for large sets of records (≥ 2000). Download the SRA-dedupe-UI application from GitHub. As of May 3, 2019, there were only Linux and Windows versions available.

  • Help using DeDuplicator Offline
  • When exporting your records to an XML file, don't forget to select them all (Help Importing/Exporting EndNote records)
  • If you choose to use the stand alone executable version for Windows, you may get a message that "Windows protected your PC": Click on More info and then a Run anyway button will appear, which you should click if you feel comfortable trusting the software developers

Keep a copy of the RIS or XML file for your records.

See also: Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2015). Better Duplicate Detection for Systematic Reviewers: Evaluation of Systematic Review Assistant-Deduplication Module. Systematic Reviews, 4(1), 6. doi:10.1186/2046-4053-4-6

Contact us

Find a librarian in your subject area to help you with your knowledge synthesis project.

 

Or contact the librarians at the
Schulich Library of Physical Sciences, Life Sciences, and Engineering
schulich.library@mcgill.ca
514-398-4769

Need help? Ask us!

McGill LibraryQuestions? Ask us!
Privacy notice