Keep a small research reference list in CSV, normalize DOI text for comparison, and generate a review file for duplicate and missing DOI values. The workflow preserves the original CSV and does not claim that a DOI is valid merely because it is present.
Who this is forThis guide is for researchers who maintain reference lists in spreadsheets or CSV files and want a simple duplicate and missing-DOI check.
This workflow checks two practical problems: a reference has no DOI value, or two rows contain the same DOI after simple text normalization. It does not contact a DOI registry and therefore does not prove that a DOI exists, resolves correctly, or belongs to the cited article.
The script removes common textual prefixes such as https://doi.org/ and doi:, trims whitespace, and converts the remaining DOI text to lowercase for comparison. The original DOI text is preserved in the output beside the normalized value.
The following reference list is synthetic and was written specifically for this article. The DOI-like strings are demonstration data and are not presented as real or resolvable publications. Save the file as references.csv.
ref_id,title,year,doi
R001,Synthetic bracket study,2024,10.0000/demo.alpha
R002,Synthetic vibration study,2025,10.0000/demo.beta
R003,Synthetic bracket study copy,2024,https://doi.org/10.0000/demo.alpha
R004,Synthetic fatigue note,2023,
R005,Synthetic surrogate study,2026,doi:10.0000/DEMO.GAMMA
R006,Synthetic optimization study,2026,10.0000/demo.delta
There are 6 reference rows. Five contain DOI text and one, R004, is missing a DOI. R001 and R003 use different textual forms but normalize to the same value. R005 demonstrates that capitalization and a doi: prefix can also be normalized for comparison.
| ref_id | Original DOI | Normalized DOI | Expected issue |
|---|---|---|---|
| R001 | 10.0000/demo.alpha | 10.0000/demo.alpha | DUPLICATE_DOI |
| R002 | 10.0000/demo.beta | 10.0000/demo.beta | None |
| R003 | https://doi.org/10.0000/demo.alpha | 10.0000/demo.alpha | DUPLICATE_DOI |
| R004 | MISSING_DOI | ||
| R005 | doi:10.0000/DEMO.GAMMA | 10.0000/demo.gamma | None |
| R006 | 10.0000/demo.delta | 10.0000/demo.delta | None |
The expected counts are 6 references, 5 rows with DOI text, 1 row with a missing DOI, and 1 duplicate DOI group containing 2 rows. After normalization, the five nonblank DOI entries contain 4 distinct DOI values.
Save the following script as reference_list_check.py. It writes a normalized copy of the entire reference list and a separate issues.csv containing only rows that need review. The original references.csv is read only.
import csv
from pathlib import Path
SOURCE = Path("references.csv")
OUTPUT_DIR = Path("outputs") / "reference_list_result"
NORMALIZED = OUTPUT_DIR / "normalized_references.csv"
ISSUES = OUTPUT_DIR / "issues.csv"
REQUIRED = {"ref_id", "title", "year", "doi"}
def normalize_doi(value: str) -> str:
text = value.strip()
lowered = text.lower()
prefixes = (
"https://doi.org/",
"http://doi.org/",
"doi:",
)
for prefix in prefixes:
if lowered.startswith(prefix):
text = text[len(prefix):].strip()
break
return text.lower()
def main() -> None:
if not SOURCE.is_file():
raise FileNotFoundError(f"Source CSV not found: {SOURCE}")
if OUTPUT_DIR.exists():
raise FileExistsError(f"Output folder already exists: {OUTPUT_DIR}")
rows = []
with SOURCE.open("r", encoding="utf-8-sig", newline="") as stream:
reader = csv.DictReader(stream)
if reader.fieldnames is None or not REQUIRED.issubset(reader.fieldnames):
raise ValueError("CSV is missing a required column.")
seen_ids = set()
for row in reader:
ref_id = row["ref_id"].strip()
if not ref_id:
raise ValueError("ref_id must not be blank.")
if ref_id in seen_ids:
raise ValueError(f"Duplicate ref_id: {ref_id}")
seen_ids.add(ref_id)
rows.append({
"ref_id": ref_id,
"title": row["title"],
"year": row["year"],
"doi": row["doi"],
"normalized_doi": normalize_doi(row["doi"]),
})
by_doi = {}
for row in rows:
doi = row["normalized_doi"]
if doi:
by_doi.setdefault(doi, []).append(row["ref_id"])
duplicate_dois = {
doi: ids for doi, ids in by_doi.items() if len(ids) > 1
}
issue_rows = []
for row in rows:
doi = row["normalized_doi"]
if not doi:
issue_rows.append({
"ref_id": row["ref_id"],
"issue": "MISSING_DOI",
"normalized_doi": "",
"related_ref_ids": "",
})
elif doi in duplicate_dois:
issue_rows.append({
"ref_id": row["ref_id"],
"issue": "DUPLICATE_DOI",
"normalized_doi": doi,
"related_ref_ids": ";".join(duplicate_dois[doi]),
})
OUTPUT_DIR.parent.mkdir(parents=True, exist_ok=True)
OUTPUT_DIR.mkdir()
normalized_fields = ["ref_id", "title", "year", "doi", "normalized_doi"]
with NORMALIZED.open("x", encoding="utf-8", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=normalized_fields)
writer.writeheader()
writer.writerows(rows)
issue_fields = ["ref_id", "issue", "normalized_doi", "related_ref_ids"]
with ISSUES.open("x", encoding="utf-8", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=issue_fields)
writer.writeheader()
writer.writerows(issue_rows)
missing_count = sum(row["issue"] == "MISSING_DOI" for row in issue_rows)
print(f"References checked: {len(rows)}.")
print(f"Missing DOI rows: {missing_count}.")
print(f"Duplicate DOI groups: {len(duplicate_dois)}.")
print(f"Issue rows: {len(issue_rows)}.")
print(f"Output folder: {OUTPUT_DIR.as_posix()}")
if __name__ == "__main__":
main()
issues.csv should contain three data rows: R001 and R003 for the duplicated normalized DOI, plus R004 for the missing DOI.
ref_id,issue,normalized_doi,related_ref_ids
R001,DUPLICATE_DOI,10.0000/demo.alpha,R001;R003
R003,DUPLICATE_DOI,10.0000/demo.alpha,R001;R003
R004,MISSING_DOI,,
The expected console output below was derived by hand from the synthetic list and script. It is not a captured execution log.
References checked: 6.
Missing DOI rows: 1.
Duplicate DOI groups: 1.
Issue rows: 3.
Output folder: outputs/reference_list_result| Problem | What to check |
|---|---|
| Same paper entered twice with different DOI formatting | Normalize common DOI URL and doi: prefixes before comparison. |
| Blank DOI treated as a duplicate | Keep missing DOI detection separate from duplicate DOI grouping. |
| Different papers share a similar title | Do not use title similarity alone as proof that records are duplicates. |
| A DOI-looking string is present | Presence is not proof of validity; verify important references against a trusted metadata source. |
| Existing result is overwritten | Use a fresh output folder so previous review evidence remains available. |
DOI matching is useful when DOI metadata is present, but a missing DOI does not mean a reference is invalid or incomplete. Some publications or document types may not have a DOI, and older records may require other identifiers or manual bibliographic checks.
This script performs text normalization only. It does not resolve DOI links, query Crossref or another registration agency, verify titles or authors, detect retractions, or determine whether two different DOI values refer to related versions of a work.
For a real research database, keep fields such as authors, title, year, journal, volume, pages, DOI, and another local reference ID. Treat automatic duplicate detection as a review aid rather than an instruction to delete bibliographic records.
2026-09-20 · hand-checked example · target: Python 3.12 · standard library: csv, pathlib · no execution
The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.