Dedup references by external identity (DOI+arXiv) across namespaces at add time

[chenpeizhi]

Background

sci-brain's download-ref checks whether a reference is already present by filename within one namespace (.raw/arxiv/<id>.json or .raw/doi/<safe>.json). But one paper often has two identities — a DOI and an arXiv id — so adding it by the identity it wasn't originally stored under duplicates it (a second rendered .md, a second bib entry). papis prevents this by checking uniqueness against the set of external identifiers (doi, isbn, eprint, url) before any insert. Motivation and full papis analysis: #30.

Objective

Identity-based dedup at add time: before fetching, resolve the candidate's DOI and arXiv id and refuse to create a second entry for a paper already present under either identity in either namespace.

Interface (Input → Output)

In: the existing Step-2 presence check in skills/download-ref/SKILL.md backed by a helper (new kb_identity.py or a function in an existing helper): --kb <path> --arxiv <id> / --doi <doi>.
Out: for each candidate either missing (safe to fetch) or present <existing rendered .md path> (matched via <doi|arxiv_id>). The SKILL.md workflow treats present as skip-by-default (user can override). Identity index is built from the rendered files' frontmatter (doi:, arxiv_id:, canonical_id:) — the tracked source of truth — falling back to .raw/*.json when the .md is absent.

Technical recommendations (non-binding)

Resolving "the DOI's arXiv id" (and vice versa) may need one Semantic Scholar lookup for candidates not yet in the KB; reuse fetch_metadata.py's batch client. Keep the check advisory (print + nonzero per-candidate status), not a hard abort — bulk --from-bib mode should skip-and-continue.

Verification

Setup: scratch KB containing DOI 10.22331/q-2018-08-06-79 (Preskill's NISQ paper, whose record carries externalIds.ArXiv = 1801.00862), added via the normal DOI flow.

1. Run the identity check for arXiv 1801.00862 → output present, naming the existing 10-22331-…md and the matching key — which proves cross-namespace identity is detected.
2. Follow the updated SKILL.md Step 2→4 for that arXiv id → no new .md, no new bib entry (ls "$KB"/*.md | wc -l and bib key count unchanged) — the naive current flow fails exactly this (it would mint 1801.00862_*.md).
3. Negative control: run the check for arXiv 1806.08734 (genuinely absent) → missing, and the normal flow fetches it — proving the check doesn't over-block.

Dependencies

Part of #30. kb_doctor's duplicate-identity check (companion issue) detects existing duplicates; this issue prevents new ones. #29's kb_sync should route through the same identity index when rebuilding.

Out of scope

Merging duplicates that already exist in a KB (manual, or a future kb_doctor --fix extension); ISBN identities (companion book-support issue).

🤖 Generated with Claude Code