Deduplicate the corpus before the AI reads it

The document assistant answered the compliance question with total confidence and six citations. Impressive, until someone noticed the six citations were the same vendor agreement six times: the signed copy, the countersigned scan, "FINAL_v2," an email attachment of the draft, and two copies from a folder migration in 2023. One of the six was an early draft with a liability clause that never made it into the signed version, and the assistant, seeing the clause "corroborated" across what it took to be independent sources, presented it as settled fact.

The retrieval system had done its job. The corpus was the lie.

Enterprise document stores are roughly one-third echo. Files get attached, exported, migrated, "finalized" into a second final version, and every copy drifts a little: an OCR pass here, a headers-stripped export there, a page of signatures appended. To a RAG pipeline every copy is a fresh document. The echoes crowd out genuinely relevant material in the retrieval window, they let a stale draft outvote the signed truth by sheer repetition, and they bill you for embedding and storing the same knowledge five times.

A retrieval system cannot tell you the corpus is wrong. It can only repeat it with confidence.

The fix runs at ingestion, before anything is chunked, embedded, or indexed, and the core technique is older than the AI it now protects: near-duplicate detection with MinHash.

The mechanism is elegant enough to explain at lunch. Each document's text is shredded into overlapping word sequences, shingles, and hashed many times; the set of minimum hash values forms a compact fingerprint with a lovely property, namely that the overlap between two documents' fingerprints estimates the Jaccard similarity of the documents themselves. Locality-sensitive hashing then buckets similar fingerprints together so you find all near-matches in a corpus of millions without comparing every pair, which is the difference between an overnight job and a heat-death-of-the-universe job. The open-source datasketch library implements the whole pattern in Python; text extraction aside, a working dedup pass over a million documents is a few hundred lines and a few hours of compute, and it is dramatically cheaper than embeddingbased comparison because it never calls a model at all.

Detection is half the method. The other half is canonicalization: each cluster of near-duplicates gets one canonical ID, and a policy picks the winner, signed beats countersigned scan beats draft, latest date breaks ties, native PDF beats OCR. The losers are not deleted; they are recorded as aliases pointing at the canonical version, so provenance survives and the RAG index carries exactly one copy of each real document.

Every question you ask a document pile is first a question about which documents are actually there.

Retrieval quality improves without touching the model, the prompt, or the vector database, because the top-k results stop being spent on echoes. Contradictions between draft and signed versions stop masquerading as corroboration. Embedding and storage bills drop by whatever your duplication rate was, which is rarely a small number. And the audit story changes register entirely: "the assistant read the signed agreement, here is its lineage" is a sentence you can say to a regulator.

We ran this pass for a firm before indexing roughly 800,000 documents for retrieval; 31 percent collapsed into aliases, and the draft-versus-signed hazard, the exact failure from the opening scene, surfaced eleven times in the first week of canonicalization. Eleven quiet lawsuits, arguably, caught by a hashing trick from the 1990s.

Model quality gets the headlines, but corpus hygiene is where document AI is actually won or lost, and it is the part nobody demos. At TensorLabs we build the ingestion layer, dedup, canonical IDs, provenance, as the first stage of every document system we ship, because it is far cheaper to give the AI a clean library than to argue with its answers afterward. The assistant can only be as honest as the shelf it reads from.