Recipe 3.1: Duplicate Claim Detection ⭐

Complexity: Simple · Phase: MVP · Estimated Cost: ~$0.002-0.01 per claim screened (mostly compute; rule layer is nearly free)

The Problem

Picture a claims operations supervisor at a mid-size payer on a Monday morning. Her team of twenty-two examiners processes roughly 180,000 claims per month. Her SIU (Special Investigations Unit) pulled a report last week showing that somewhere between 1% and 3% of paid claims in the prior quarter were duplicates. That's money that went out the door for services the payer had already paid for. Some of it was fraud (the kind of bad actor who submits the same claim to three clearinghouses and hopes one of them pays). Most of it was not. Most of it was bog-standard operational noise.

Here's what "operational noise" actually looks like on her desk:

The orthopedic group that re-submitted a batch of claims after their practice management software choked on an 837 transmission and they got no 277 acknowledgment back. They thought the first batch got lost. It didn't. They sent it twice. Both batches paid before anyone noticed.

The hospital that bills primary care and inpatient on two different tax IDs. Same physician, same patient, same date of service, same CPT for a critical care hour. Two payments. The physician doesn't even know; her billing is handled by a service.

The lab where a tech re-ran a CBC because the first draw was hemolyzed. The second run got a new accession number. The billing system swept both accessions into the next claim batch. Same patient, same CPT 85025, same date, two line items that look suspiciously like a single-run claim that was paid, plus a duplicate. Except it isn't. It's two legitimate runs, only one of which should be billed.

The skilled nursing facility that submits claims weekly and, when a resident transitions levels of care mid-week, generates two claims covering overlapping dates. Not an exact duplicate. But the date ranges overlap by three days, and the revenue codes partly overlap, and the question of whether this is a legitimate adjustment or a duplicate takes someone with SNF billing expertise to resolve.

Now the supervisor is looking at four-thousand-odd suspected duplicates in her review queue, generated by the payer's existing duplicate-detection rules. Her examiners clear about a hundred of these per day. At that rate, the queue never drains. Meanwhile, true duplicates are slipping through because the existing rules are too narrow: they catch exact matches on claim number, patient ID, date, and procedure. Anything with a minor variation (different place of service code, claim number typo, subscriber versus dependent identifier, provider NPI vs. taxonomy-paired billing NPI) sails past the check and into autoadjudication.

This is the duplicate claim detection problem, and it's the simplest interesting problem in the whole anomaly detection category. Simple because the outcome is clean: a claim either is or isn't a duplicate of a previously submitted one. Interesting because "is a duplicate" turns out to hide an enormous amount of structural variation, and the cost of getting it wrong in either direction is real. False positives irritate providers (nobody likes their clean claim sent to review for the fourth time this quarter). False negatives lose money.

The problem you actually have to solve: given a stream of incoming claims, identify which ones are candidate duplicates of claims already in your system, rank them by the likelihood that they're genuinely duplicate rather than a legitimate related claim, and hand the top of that ranked list to a human to adjudicate. That's it. It's not magic. It's just careful engineering of the "match" definition plus a feedback loop that lets the detection improve as examiners give you labels.

Let's get into how.

The Technology

What "Duplicate" Actually Means

The word "duplicate" carries more weight than it looks like it should. Three different definitions are all in common use, and they drive different technical choices:

Exact duplicates. The same claim submitted twice, character-for-character (or close to it). Payer trace number collision. Provider's billing system retransmitted after a comm failure. These are easy. A hash of the key fields catches them at ingestion.

Semantic duplicates. Two claims that describe the same actual service event, but the representation differs. One claim used CPT 99213; the other used the deprecated predecessor code. One listed the patient's subscriber ID; the other listed the dependent suffix. One had a typo in the date of service. Same clinical event, different encoding. These are the hard ones.

Overlapping or adjustment claims. This could be two claims that cover the same date range and patient but represent related, but legitimately distinct, billing events. A correction to a previously paid claim. An inpatient stay billed in two segments because it crossed a month boundary. Or, a split bill between professional and facility components. Not duplicates. But they'll look like duplicates to any naive matching rule, and treating them as duplicates is worse than missing them.

A duplicate detection system that doesn't cleanly separate these three categories ends up flagging adjustment claims as duplicates, which infuriates providers (their legitimate corrections get denied), or ignoring semantic duplicates because the rule was calibrated to reject the adjustments, which loses money. The design goal is to identify the semantic duplicates with high confidence while letting the adjustments through and catching the exact duplicates for free.

The Three Layers of Detection

Almost every serious duplicate detection system ends up with a three-layer architecture, whether the team designed it that way or discovered it by accident. The layers are:

Layer 1: Deterministic blocking. A cheap, fast first pass that rejects the obvious non-candidates and groups the rest into blocks where potential duplicates can live. You don't compare every incoming claim against every claim in your history. You use indexed keys (patient ID, provider NPI, date of service, CPT) to narrow the search space to a handful of candidates per incoming claim. This is the same "blocking" concept that entity resolution systems use, and it's the technique that makes the whole thing tractable at scale. Without it, you're looking at O(N²) comparisons against a database of hundreds of millions of historical claims, and your compute bill will make you cry.

Layer 2: Similarity scoring. Within each block, you compute a similarity score between the candidate pairs. This is where the interesting engineering lives. The score is typically a weighted combination of field-level similarity measures:

Exact match on patient ID, provider NPI, claim type
Date of service proximity (same day? within 1 day? overlapping ranges?)
Procedure code similarity (exact? synonym in a known code-family lookup? same HCPCS crosswalk?)
Billed amount similarity
Diagnosis code overlap
Modifier code comparison
Place of service comparison
Rendering provider comparison (some duplicates are "same provider billing under two NPIs")

Each field gets a weight. The weights can come from a rules engine (start here), a logistic regression trained on historical labels (better), or a gradient-boosted classifier if you have enough labeled examples (best, eventually). The output is a similarity score in [0, 1] where 1 is "certainly duplicate" and 0 is "certainly not."

Layer 3: Decision and routing. A score isn't a decision. You need thresholds: above some score, auto-reject the claim as a duplicate. Below some other score, auto-accept. In between, route to human review. Most production systems tune the upper and lower thresholds separately based on their tolerance for false positives and false negatives. The middle range is where the human examiners live, and the size of that range is effectively a budget decision: how many claims per day can your team review?

The neat thing about this architecture is that each layer is independently tunable. You can improve blocking recall without touching scoring. You can retrain the scorer without rebuilding the routing logic. And the feedback from the review step (examiners labeling flagged claims as true or false duplicates) flows directly back into retraining the scorer.

Fuzzy Matching, Demystified

The core of Layer 2 is "fuzzy" field matching. Let's dive into that, because it's where most teams either overcomplicate things or, more commonly, oversimplify them.

Consider just the claim number field. Two claims land with numbers C-2026-0487291 and C-2026-0487219. Are those the same claim with a typo, or two different claims? You compute an edit distance (Levenshtein, for example): the two strings differ by two character transpositions, edit distance 2 out of length 13. A common heuristic is a similarity threshold like 1 - (edit_distance / max_length). Here that gives 0.846. High enough to warrant a look.

Now consider the patient name field. "John Smith" vs "Jon Smyth." Edit distance is 2. By the same formula you'd get about 0.8. Same score as the claim number typos. But those two names are almost certainly the same person, while "C-2026-0487291" and "C-2026-0487219" might be two genuinely distinct claims. The pure string-edit-distance metric throws away useful information about what each field represents.

Which is why real duplicate detection uses field-specific comparison functions. Some options:

Claim numbers, patient IDs, NPIs: edit distance with a low tolerance (typos are rare in system-generated identifiers). Above a high threshold (say 0.95), suggest a match; below that, don't.
Patient names: a phonetic algorithm (Soundex, Double Metaphone, or the newer Beider-Morse) combined with edit distance. "Jon Smyth" and "John Smith" share the same phonetic signature, which is a much stronger duplicate signal than their string distance.
Dates of service: numerical distance in days. Same day is strongest; adjacent days are suspicious; more than a week apart is almost never a duplicate unless you're looking at a long inpatient stay.
CPT/HCPCS codes: a lookup against known synonyms, crosswalks, and hierarchical relationships. 99213 (established patient, level 3 office visit) and 99214 (level 4) are in the same family but represent different services; a duplicate claim is more likely to use the same code than to escalate the level.
Billed amounts: relative difference with a tolerance band (2% off is suspicious; 20% off is probably a correction, not a duplicate).

The weighting problem. Once you have per-field similarities, you need to combine them. A weighted sum with handcrafted weights is where most teams start. It works. It's interpretable. It's easy to explain to compliance and to adjust when the SIU calls and asks why a claim was flagged. You can graduate to a learned weighting (logistic regression on historical labels) once you have a decent labeled dataset, and to a non-linear model (gradient boosting, typically XGBoost or LightGBM) once the labeled set is large enough and the rule-based system has hit its ceiling.

The Embedding Shortcut (and Its Limits)

Over the last few years there's been a push toward using embedding models for duplicate detection. The pitch: convert each claim into a dense vector (using a sentence transformer or a claim-specific embedding model), and duplicate detection becomes a nearest-neighbor search in vector space. One index, no handcrafted field comparisons, learns similarity automatically.

This works, but only partially. Where it shines is on the unstructured parts of a claim (diagnosis narratives, service descriptions, clinician notes attached to the claim) where handcrafted string matching is fragile. Where it falls down is on the structured parts, and structured parts are most of what a claim is. The patient ID 12345678 and the patient ID 12345687 are a potential duplicate (typo). An embedding model will encode them as nearly identical strings because they look nearly identical. But if the patient IDs refer to two different patients, that's not a duplicate, it's a collision, and an embedding-only model has no way to distinguish "similar-looking identifiers for different entities" from "same entity, different encoding."

Practical guidance: use embeddings as one feature in a composite score, not as the whole score. Run embeddings on the unstructured fields (diagnosis narratives, notes). Use exact or edit-distance matching on the identifiers. Combine them with a classifier. This hybrid is uniformly better than either approach alone.

The Label Problem

Any supervised approach to duplicate detection needs labeled data. "Was this pair of claims genuinely a duplicate, or not?" The answer comes from your claims examiners' historical decisions: the denial codes, the review notes, the recovery actions. Three gotchas:

Selection bias. Your existing labels are labels on claims that your existing rules already flagged. If your rules flag exact CPT+DOS+patient matches, your labeled dataset is dominated by exact matches, and a model trained on it will learn to detect exact matches. It won't learn the edge cases because it never saw them. Mitigation: periodically sample some claims below the current flag threshold, ship them to review anyway, and feed the labels back in. It's expensive but it's a necessary part of the labelling and training step.

Decision drift. Your examiners' interpretations of "duplicate" drift over time. New billing codes appear. Rule changes from CMS arrive. Payer policy updates. A claim that was labeled a duplicate in 2022 might have been labeled an adjustment in 2024 under new guidance. Don't train on all-time historical labels indiscriminately; weight recent labels more heavily or filter to a recent window.

Adversarial dynamics. Some of the duplicates in your data are fraud attempts. Fraudsters adapt. Patterns that were common a year ago get replaced. If your model is trained primarily on old fraud patterns, it'll miss the current ones. This is less of a concern for duplicate detection than for general fraud detection (because most duplicates are operational noise, not fraud), but it's still worth monitoring.

Batch vs. Real-Time

Duplicate detection can run in either mode, and the right choice depends on when you want to catch the duplicate:

Batch, pre-adjudication. Nightly or hourly batch job that screens the day's submitted claims against the historical claim set before adjudication runs. Catches duplicates before money goes out the door. Easiest to build. Most payers run it this way.

Real-time, at submission. As each 837 transaction arrives, check against history in milliseconds and flag or reject before it enters the adjudication queue. Tighter integration with the submission pipeline. Higher engineering cost. Primary benefit: faster feedback to providers, which can be worth it for provider-relations reasons.

Retrospective. Run the detector against already-paid claims to identify duplicates that slipped through, then recover payments. Some of the biggest payback comes from this mode, but it's slow and it's a different operational workflow (recovery, not prevention).

Most payers end up running batch for prevention and retrospective for recovery, sometimes with a real-time layer for specific high-risk claim types. All three modes share the same scoring logic; what differs is the trigger and the action. For this recipe, we'll build the batch pre-adjudication pattern, because it covers the common case and it's the pattern that maps most cleanly to managed cloud services. The real-time variant is a straightforward adaptation (swap the batch trigger for an event stream; keep everything else).

General Architecture Pattern

At a conceptual level, the pipeline has three stages plus a feedback loop. The key architectural insight is that none of the stages are particularly complex on their own. The design work is in making them compose cleanly and in making the feedback loop fast enough that the system gets smarter over time.

┌────────────────── DETECTION PIPELINE ──────────────────┐
│                                                        │
│  [Incoming Claim Stream]                               │
│           │                                            │
│           ▼                                            │
│  [Ingestion + Normalization]                           │
│   (parse 837, canonicalize IDs, hash key fields)       │
│           │                                            │
│           ▼                                            │
│  [Layer 1: Blocking]                                   │
│   (lookup: patient + provider + date-window +          │
│    claim-type. Returns candidates in historical        │
│    store.)                                             │
│           │                                            │
│           ▼                                            │
│  [Layer 2: Similarity Scoring]                         │
│   (field-level fuzzy match → weighted combination →    │
│    score in [0, 1] per candidate pair)                 │
│           │                                            │
│           ▼                                            │
│  [Layer 3: Decision + Routing]                         │
│   score ≥ high_threshold  → auto-suspend (duplicate)   │
│   score ≤ low_threshold   → auto-accept (unique)       │
│   between thresholds      → human review queue         │
│           │                                            │
└───────────┼────────────────────────────────────────────┘
            │
┌───────────┼────────────────────────────────────────────┐
│           ▼                                            │
│  [Examiner Workstation]                                │
│   (adjudicates queue items; labels: duplicate,         │
│    adjustment, unique; records reasoning)              │
│           │                                            │
│           ▼                                            │
│  [Label Store]                                         │
│           │                                            │
│           ▼                                            │
│  [Periodic Retraining]                                 │
│   (update weights / model; monitor drift; refresh      │
│    thresholds)                                         │
│                                                        │
└──────────────────── FEEDBACK LOOP ─────────────────────┘

Ingestion and normalization. 837 EDI transactions get parsed into a canonical claim record. IDs get normalized (leading-zero padding on patient IDs, NPI format validation, date format standardization). A stable hash is computed over the subset of fields that define a "claim identity" for exact-duplicate detection (patient ID + provider NPI + DOS + CPT + modifiers + billed amount). Exact hash collisions get flagged immediately; this is effectively free duplicate detection for the easy cases.

Blocking. The incoming claim's blocking keys are used to query the historical claim store for candidate matches. A common blocking strategy: patient ID + provider organization + date window (for example, plus-or-minus 14 days) + claim type. This narrows the candidate set from "all history" to typically 0 to a few dozen candidates. If you have a claim type for which this isn't selective enough (for example, frequent high-volume lab claims for the same patient), you add more blocking dimensions for that type.

Similarity scoring. For each candidate pair, compute per-field similarity, combine into a total score. The scoring function is pluggable: the same pipeline can call into a rule-based scorer, a logistic regression, or a gradient-boosted model. Start simple; replace with learned models as labeled data accumulates.

Decision and routing. Thresholds are applied. Auto-suspend and auto-accept actions are executed immediately. The middle-band claims go to a queue with the top-N candidates attached to each review item, so the examiner isn't starting from scratch.

Feedback loop. Every examiner decision is a label. The label store captures the claim pair, the examiner's verdict, their reasoning (free text or a structured code), and timing metadata. Periodic retraining re-fits the scoring model on a recent window of labels and evaluates against a held-out set before the new model is promoted. Threshold tuning is a separate concern: the thresholds can be adjusted to target a specific review-queue size regardless of what the model is doing.

Historical claim store. Somewhere in the middle of all this is a store of historical claims. The access pattern is point lookups on blocking keys, which you want to be fast. The data volume grows forever (you don't delete claim history). Retention policies apply (CMS typically requires 10 years for Medicare). The store needs to support both the detection path (blocking queries) and the retraining path (bulk scans for training data). These are different access patterns and the design should anticipate both.

The AWS build lives in a companion page. This recipe covers the problem, the underlying technology, and the vendor-agnostic architecture. For the AWS services, architecture diagram, prerequisites, and the step-by-step pseudocode walkthrough, see the Architecture and Implementation companion. The Python example is linked from there.

The Honest Take

The blocking layer is the secret sauce, not the scorer. Teams new to this problem tend to obsess over the scoring model: which algorithm? XGBoost or LightGBM? Can we use embeddings? Most of the practical win comes from getting blocking right. A sloppy blocker that misses 20% of true duplicate pairs puts a ceiling on your recall that no scorer can recover. A careful blocker with multi-key lookup pushes that ceiling above 95%, at which point the scorer's job is easy. Budget accordingly.

The rule-based scorer is much harder to beat than you expect. A thoughtfully weighted rule-based scorer with field-specific similarity functions typically has high precision at a reasonable recall on the first day of production. The learned model you build six months later will outperform it on recall at the margin, but the absolute improvement is often 5-10 percentage points, not an order of magnitude. This is not a flaw; it means the rules are doing a lot of the work, and the learned model is catching the subtler cases. Don't skip the rules phase to rush to a learned model. You won't have labels yet anyway.

The feedback loop is the thing that makes the system durably good. A duplicate detector without a feedback loop decays: new billing codes appear, new provider organizations emerge, new fraud patterns develop, and the detector keeps running the rules it was given at deploy time. A detector with a working feedback loop gets smarter over time. The engineering work on the loop (label capture, label storage, retraining pipeline, monitoring) is straightforward but non-trivial and needs to be budgeted from day one, not bolted on in year two.

Your examiners are really good at their job, and the review queue interface is what determines whether the system helps them or fights them. Show the examiner the matched claim, the similarity components (which fields matched, which didn't), and one-click buttons for the common verdicts ("duplicate of X," "adjustment to X," "unique, not duplicate," "unclear, escalate"). Do not make them type. Do not make them re-fetch the matched claim from the claims system. The time-per-review is the rate-limit on the whole operation, and shaving 30 seconds off each review, at 100 reviews per examiner per day, at 20 examiners, is 1,000 minutes per day of examiner capacity freed. That math is worth more than an accuracy improvement on the scorer.

What I recommend to any team looking to embark on this project, start with a deterministic exact-duplicate check running in production before anything else. It's a one-week project. It catches the low-hanging fruit. It generates the first labels you'll need to train the fuzzy scorer. And it demonstrates value to the organization before you've invested in the harder pieces. Starting with the full ML pipeline before you've shipped the exact-match check is a classic overbuild, and I've seen multiple teams do it.

Recipe 3.3 (Billing Code Anomalies): Extends the pattern here to provider-specific baselines and detects claim patterns that don't appear to be duplicates but are still unusual. Shares the review-queue and feedback-loop infrastructure.
Recipe 3.6 (Healthcare Fraud/Waste/Abuse Detection): A superset of duplicate detection that adds adversarial dynamics, cross-entity graph analysis, and investigation workflows. Start with 3.1 and graduate to 3.6 as organizational maturity grows.
Recipe 5.1 (Provider Identity Resolution): The "which NPIs belong to the same billing organization?" problem this recipe mentions. A dedicated entity-resolution pipeline for provider hierarchies feeds cleaner organization IDs back into the blocking function, which improves both recall and precision.
Recipe 5.2 (Patient Record Linkage): The same "find near-duplicates" pattern applied to patient records rather than claims. Architecturally very similar; share as much of the fuzzy-matching and blocking infrastructure as possible.
Recipe 1.5 (Claims Attachment Processing): Upstream of this recipe: once you've identified a claim as a duplicate candidate, Recipe 1.5's extracted attachment data can support or refute the duplicate determination. Cross-reference where possible.