reference-deduplication

Deduplicating Reference Libraries Across Databases

Anyone who has searched more than one bibliographic database knows the problem. The same article appears in MEDLINE, Embase, Scopus, Web of Science and a preprint server, each time with slightly different metadata. A search that returns 6,000 records may contain only 3,500 unique studies. Screening the duplicates wastes time, and handling them inconsistently can distort counts in your flow diagram or even lead to the same study being included twice in a synthesis. This guide explains how to deduplicate a reference library reliably, what tools can and cannot do, and how to document the process.

Why duplicates are harder than they look

If every database recorded the same title, authors, year and DOI in the same format, deduplication would be trivial. In practice, records differ in many small ways:

• titles with different capitalisation, punctuation or special characters;
• author names with or without initials, diacritics or full first names;
• journal names abbreviated in one database and spelled out in another;
• page numbers formatted as "123-9" or "123-129";
• missing DOIs, or DOIs recorded with and without the resolver prefix;
• conference abstracts, preprints and final articles that describe the same study but are different publications.

Automated tools handle the first few cases well. The last case, related but distinct publications, needs human judgement.

Duplicates versus related reports

It is important to distinguish two situations.

True duplicates are the same publication recorded more than once. They should be removed so that each publication is screened only once.

Multiple reports of one study are different publications describing the same underlying study: a protocol, a conference abstract, a main results paper and a long-term follow-up. These are not duplicates at the record level. They should all be screened, and then linked together so that the study is counted once in your synthesis. Merging them too early can lose useful information, such as additional outcomes reported only in the follow-up paper.

A practical deduplication workflow

1. Export consistently

Export records from every source in a format your reference manager or deduplication tool handles well, such as RIS, and include abstracts and DOIs. Keep the raw export files unchanged in a project folder; you may need them later to show what each search retrieved.

2. Record counts before you start

Note the number of records from each source and the total. This number goes into your flow diagram and lets you check the arithmetic later.

3. Run automated matching

Reference managers and dedicated tools use combinations of fields, such as DOI, title, authors, year, journal and pages, to identify likely duplicates. Start with conservative settings that require several fields to match. Review the proposed duplicate pairs rather than deleting them blindly, at least until you trust the tool's behaviour on your dataset.

4. Run a second, looser pass

After removing clear duplicates, run a looser match, for example on title and year only, and review the pairs manually. This catches records with missing or garbled metadata.

5. Choose which record to keep

When you remove a duplicate, keep the most complete record: the one with an abstract, a DOI and full author names. Some tools let you merge metadata from both records, which is ideal.

6. Spot-check the result

Sort the remaining library by title and skim it. Duplicates with small differences often sit next to each other. Also sort by first author and year. A few minutes of skimming usually finds a handful of missed duplicates.

7. Record the final count

The number of duplicates removed is simply the difference between the totals before and after. Record it, along with the tool and settings used.

Common traps

• Deleting related reports as duplicates. A conference abstract and the later full paper are not duplicates. Keep both and link them later.
• Errata and corrections. These often have titles almost identical to the original article. Keep them, because they may change the data you extract.
• Retractions. A retraction notice may appear as a separate record. Treat it as a warning, not a duplicate.
• Translated publications. The same study published in two languages may look like two different papers. Check authors and sample sizes.
• Updates of searches. When you re-run searches later, deduplicate the new records against the existing library, not just among themselves.

Linking reports to studies

After screening, group included reports by study. Useful signals are trial registration numbers, identical sample sizes, the same author team and setting, and explicit cross-references. Assign each study an identifier and attach all of its reports to it. During extraction, take each data item from the most complete or most recent report and note which report it came from.

This linking step matters for evidence traceability. When a number in your evidence table comes from a follow-up paper rather than the main results paper, readers checking it need to know where to look. Workbenches built around traceable evidence, such as Yila (https://yila.ai), keep each extracted claim connected to the passage it came from, which makes multi-report studies far less confusing when a reviewer asks where a value originated.

Documenting deduplication

Your methods section should state:

• the tool or reference manager used;
• whether matching was automatic, manual or both;
• the number of duplicates removed;
• how multiple reports of the same study were handled.

Keep a copy of the deduplicated library at the point screening began. If questions arise later, you can show exactly which records were screened.

Deduplication in team projects

When several people search different sources, agree on one person or one tool to own the master library. Merging libraries independently deduplicated by different team members almost always creates new duplicates and inconsistent records. Share the deduplicated master library in a read-only form for screening, and keep any edits, such as corrected metadata or merged records, in a single place. If you use a screening platform, import the deduplicated library once rather than importing each database export separately, unless the platform's own deduplication has been tested on your data. A short note in the project log recording who deduplicated, when and with which settings is enough to answer most later questions.

How much effort is enough?

Perfect deduplication is not necessary. A small number of missed duplicates will simply be screened twice and excluded or merged later. What matters is that the process is systematic, conservative (it should not remove unique records) and documented. For most reviews, an automated pass, a manual review of proposed pairs and a quick skim of the sorted library are enough.

Conclusion

Deduplication is unglamorous, but it underpins accurate counts, efficient screening and a correct synthesis. Export consistently, record counts, use automated matching with review, distinguish true duplicates from related reports, link reports to studies and document what you did. Done carefully, it takes an afternoon and spares you much bigger problems later in the review.