Recipe 5.1 Architecture and Implementation: Internal Duplicate Patient Detection

Companion to Recipe 5.1: Internal Duplicate Patient Detection. This page covers the AWS architecture, services, prerequisites, and pseudocode. For the problem framing and the conceptual approach, start with the main recipe.

The AWS Implementation

Why These Services

Amazon S3 for the source-record landing, the normalized-records store, and the candidate-pair archive. Patient registration extracts land in an S3 raw zone; the normalized records (after the ingest stage) land in a curated zone partitioned by ingest date; the candidate pairs and their per-field scores land in a derived zone partitioned by blocking pass. S3 is HIPAA-eligible under BAA, supports SSE-KMS at rest, and is the natural staging layer for batch matching workloads. The candidate-pair archive is also the audit substrate; every pair the system considered is preserved with its scores, regardless of routing decision, so a future investigation can trace exactly what the system saw.

Amazon DynamoDB for the master patient identity table, the cross-reference table, and the review queue. Three tables, each with a clear role. mpi-master keyed on mpi_id holds the resolved identity record (the "golden" record after survivorship). mpi-xref keyed on (source_system, source_record_id) holds the cross-reference from each source record to its current mpi_id, with version history of identity reassignments. review-queue keyed on (queue_id, candidate_pair_id) holds candidate pairs awaiting human decision, with the score, the contributing field scores, the snapshot of both records at the time of queuing, and the reviewer's eventual decision. DynamoDB's single-digit-millisecond reads support real-time identity lookups (which mpi_id does this source record map to right now?), and on-demand capacity handles the bursty review-queue write pattern without capacity-planning headaches.

AWS Glue for the batch matching pipeline. Glue jobs (Spark-backed) handle the normalization, the blocking-pass generation, and the per-pair scoring at scale. The Splink open-source library runs on Spark and is a natural fit for the Fellegi-Sunter combiner; it handles the EM-based m/u estimation, the per-field comparators, and the composite score computation with healthcare-scale workloads. Glue Data Catalog tracks schemas across the raw, normalized, candidate-pair, and audit zones. Athena (over the Glue catalog) provides the ad-hoc SQL access for cohort-stratified accuracy monitoring and HIM-team analytics.

AWS Step Functions for orchestration. Two workflows: a batch-matching workflow (run nightly or as a backfill, full pipeline from normalize to score to route) and a real-time-matching workflow (run on every new registration, normalize the single record, generate candidates against the existing index, score, and route). Step Functions provides the per-stage retry, timeout, and dead-letter-queue semantics that the matching pipeline needs.

AWS Lambda for the per-stage logic. Normalization, candidate generation against the index, single-pair scoring, threshold routing, merge-application with survivorship, and audit-event emission each run as Lambdas. Lambdas are in VPC with VPC endpoints for downstream services. The split between Glue (batch, high-throughput) and Lambda (real-time, single-record) lets each workload run on the right substrate without compromising the other.

Amazon OpenSearch Service for the candidate-generation index. For real-time matching, you need to find candidate pairs for a single new record in milliseconds. Scanning a million records is too slow; iterating blocking passes against a relational store is awkward. OpenSearch with custom analyzers (lowercase, ASCII-folding, phonetic encoders) is a good fit: index each normalized record, query for candidates using the equivalents of the blocking passes (multi-field bool queries with phonetic and prefix matches), and rank-limit the results before scoring. OpenSearch is HIPAA-eligible under BAA and supports KMS-encrypted indices, fine-grained access control, and VPC deployment.

Amazon EventBridge for the merge-event bus. When a merge happens (auto or after review), an event flows out to downstream consumers: the EHR for chart linkage, the data warehouse for analytics deduplication, the patient communication system for de-duplicated outreach, the billing system for account reconciliation. EventBridge provides the loose coupling, retry semantics, and per-consumer filtering that this fan-out pattern needs.

AWS Step Functions plus a simple web app (API Gateway + Lambda + a static S3-hosted SPA) for the review queue UI. The review queue UI is the operational face of the system. Most production deployments use a purpose-built internal tool that pulls candidates from review-queue, presents the side-by-side comparison, and writes the decision back. API Gateway exposes the read/write endpoints, Cognito (or the institution's identity provider) handles HIM-team authentication, and Lambda backs the API. Audit events fire on every reviewer action.

Amazon QuickSight for the operational and equity dashboards. Daily duplicate-detection rates, review-queue depth by reviewer, auto-match vs review-routing rates, false-positive feedback from unmerges, and cohort-stratified accuracy by race / ethnicity / language / age band / geographic region. QuickSight on Athena over the Glue-cataloged audit data, with row-level security for cohort-specific access where institutional policy requires it.

AWS KMS, CloudTrail, CloudWatch. Customer-managed keys for the S3 buckets, the DynamoDB tables, the OpenSearch domain, and the Lambda log groups. CloudTrail data events on the MPI tables and the audit S3 buckets (every read of these is a PHI access and needs to be audited). CloudWatch alarms on review-queue depth (excessive growth signals reviewer staffing problems), on auto-match rate drift (significant changes in rate signal upstream data quality issues), on cohort-stratified accuracy disparity threshold crossings (equity guard rail), and on Lambda or Glue job failures.

Identity Boundaries and Access Control

Kinesis registration-events stream. The producer IAM role (arn:aws:iam::<account>:role/registration-source-<env>) is the only principal authorized to PutRecord. Each event carries a signed envelope (HMAC-SHA256 with a shared secret rotated quarterly); the normalize Lambda validates the signature before processing and rejects unsigned or mis-signed events to a rejected-events DLQ. An allowed-list of source_system values gates acceptance. An idempotency window (DynamoDB TTL-backed, 24 hours) prevents duplicate-event processing from Kinesis retry behavior.

Review-queue API. Amazon Cognito authenticates HIM-team members (SAML federation with the institution's identity provider). An authorization layer enforces: (1) reviewer-to-queue assignment (a reviewer can only act on pairs assigned to their queue segment), (2) conflict-of-interest checks (the reviewer is not a relative of or the patient in the candidate pair, verified against an exclusion list maintained by HIM leadership). Every API invocation is logged at both the API Gateway and Lambda layers with the authenticated principal, the action taken, and the candidate-pair ID.

apply_merge caller validation. The invocation_source enum (auto_match_pipeline, review_queue_decision, backfill_pipeline) maps to a per-source caller IAM role. The merge Lambda validates the caller's assumed role ARN against the expected role for the declared invocation source; mismatches are rejected with a identity_boundary_rejection CloudWatch metric emission and a logged security event.

Unmerge governance. Unmerge requires: (1) an institution-defined reason from a controlled vocabulary, (2) second-operator approval for merges that affected medication records or recent clinical notes (gated on a DynamoDB flag set at merge time), (3) routing to the privacy-office restricted-review track for any unmerge involving no-link-flag cohorts (see No-Link Flags below). The unmerge event emits to EventBridge so downstream consumers can react.

Networking and Egress

API Gateway for the review-queue API. Deployed as a private API with a VPC endpoint; the VPC endpoint resource policy restricts access to the institutional VPC. AWS WAF rules enforce SQL/command-injection protection and rate limiting per authenticated principal. Optional mTLS available for institutional environments that require certificate-based client authentication. The HIM Review UI is served from S3 plus CloudFront with a WAF web ACL; if the UI is publicly addressable, API calls route through the institutional VPN or PrivateLink.

Kinesis producer connectivity. Producers access Kinesis through a PrivateLink endpoint within the institutional VPC. PutRecord calls use SIGv4 signing with a dedicated IAM role. Cross-account delivery (if the registration system runs in a separate AWS account) uses PrivateLink with explicit endpoint policies. TLS 1.2+ is enforced on all connections. Server-side encryption uses customer-managed KMS keys.

External egress for address standardization. Outbound HTTPS to address-standardization vendors (SmartyStreets, Melissa, USPS API) routes through a dedicated outbound proxy in VPC with allow-listed destination domains. VPC Flow Logs and CloudWatch Logs capture every outbound connection from the proxy. Each vendor must be BAA-covered before any PHI flows to it; review the BAA list annually.

Data Governance: Lake Formation and Retention

Lake Formation access control. Lake Formation is the access-control layer over the Glue Data Catalog. Column-level permissions restrict cohort_features, field_comparisons, and per_field_log_ratios columns to entity-resolution analytics roles. Demographic-snapshot columns are restricted to HIM-leadership for forensic investigation. Row-level filtering supports cohort-stratified analytics (analysts see only their assigned cohorts). QuickSight inherits Lake Formation grants; direct Athena access uses the same grants. CloudTrail logs Athena query execution; a periodic review ensures access patterns match authorized roles.

Retention floor. Audit logs are retained for the longer of: (1) 7 years (clinical-record minimum), (2) the institution's documented medical-record retention policy, (3) the applicable state-specific medical-record retention statute, (4) the state-specific minor-records floor for pediatric patients. Audit logs reside in a dedicated S3 bucket with Object Lock in Compliance mode for immutability; a lifecycle policy transitions objects to S3 Glacier Deep Archive after 90 days. CloudTrail data events are forwarded to a dedicated audit AWS account in the institution's organization. The retention floor is enforced at the bucket-policy and Object-Lock-configuration level. Reference: HIPAA 45 CFR section 164.530(j).

Cohort Equity Metrics

Metric emission privacy. When emitting cohort dimensions on CloudWatch metrics, use bucketed non-reversible cohort labels (cohort_race_eth_bucket = A, B, C, D, E, unknown) rather than raw demographic attributes. The cohort-label-to-attribute mapping lives in a separate access-controlled DynamoDB table loaded only at dashboard-render time by authorized QuickSight roles. This prevents demographic inference from CloudWatch metric streams.

Disparity alert thresholds. The equity monitoring system computes per-cohort ratios on a daily cadence and alerts when any ratio crosses threshold:

Per-axis override is available via the equity-review committee (documented in a governance policy table). When a cohort sample size falls below MIN_COHORT_SAMPLE_SIZE = 50, chronic suppression of alerts for that cohort is itself a fairness signal: the system logs cohort_below_minimum_sample and the equity-review committee examines whether the cohort's representation in the gold set is adequate. A cohort-stratified gold-set construction discipline ensures each cohort has proportional (or over-sampled) representation.

When an alert fires, the documented diagnose-and-address workflow activates: (1) identify the specific comparator or blocking pass contributing to the gap, (2) evaluate cohort-specific m/u tuning or supplementary blocking passes, (3) submit a remediation plan to the equity-review committee, (4) implement and validate against the cohort-stratified gold set, (5) promote after committee approval.

No-Link Flags

The no_link_flags table is keyed on (mpi_id_or_record_id, flag_type) and covers safety-sensitive populations where linking records could endanger the patient or violate legal protections:

The pipeline consults no-link flags at three stages:

1. Candidate generation (filter). Records carrying any no-link flag are excluded from the standard blocking passes. They participate only in a restricted matching path visible to the privacy office.
2. Threshold routing. Any candidate pair containing a flagged record bypasses the auto-match path entirely, regardless of score, and routes to a privacy-office restricted-review track.
3. Review-queue assignment. Flagged-record pairs route to a separate restricted queue staffed exclusively by privacy-office and HIM-leadership roles.

Flags are write-protected to privacy-office and HIM-leadership IAM roles. The flag table uses a separate customer-managed KMS key with a restricted key policy. CloudTrail data events on every read/write of the flag table provide forensic visibility.

Family-aware blocking. The family_relationship_explicit flag drives a supplementary blocking-pass rule: when two records are explicitly flagged as family members (twin, parent-infant), the pipeline skips their comparison entirely in standard passes and down-weights comparator scores on shared-family fields (address, phone, insurance) for any pair that does enter scoring through other blocking passes. This prevents the common failure mode where twins with the same birthday, address, and last name auto-merge.

Architecture Diagram

flowchart LR
    subgraph Sources
      A1[Patient Registration<br/>System / EHR]
      A2[Historical Patient<br/>Database backfill]
    end

    A1 -->|Real-time event| KS1[Kinesis Streams<br/>registration-events]
    A2 -->|Bulk export| S1[S3<br/>raw-records]

    KS1 --> L1[Lambda<br/>normalize-record]
    S1 --> GL1[Glue Job<br/>batch-normalize]
    L1 --> S2[S3<br/>normalized-records]
    GL1 --> S2

    L1 --> OS1[OpenSearch<br/>candidate-index]
    GL1 -->|Bulk index| OS1

    subgraph Real-Time Matching
        L1 --> L2[Lambda<br/>candidate-generator]
        OS1 --> L2
        L2 --> L3[Lambda<br/>pair-scorer]
        L3 --> L4[Lambda<br/>threshold-router]
    end

    subgraph Batch Matching
        S2 --> GL2[Glue Job<br/>block-and-score<br/>Splink on Spark]
        GL2 --> S3[S3<br/>candidate-pairs]
        S3 --> GL3[Glue Job<br/>threshold-router]
    end

    L4 -->|Auto-match| L5[Lambda<br/>apply-merge]
    L4 -->|Review| D1[DynamoDB<br/>review-queue]
    L4 -->|Auto-non-match| S4[S3<br/>audit-archive]

    GL3 -->|Auto-match| L5
    GL3 -->|Review| D1
    GL3 -->|Auto-non-match| S4

    D1 --> AG1[API Gateway<br/>review API]
    AG1 --> CG1[Cognito / IdP]
    CG1 --> UI1[HIM Review UI<br/>S3-hosted SPA]
    AG1 --> L6[Lambda<br/>review-decision]
    L6 -->|Match| L5
    L6 -->|Not match| S4

    L5 --> D2[DynamoDB<br/>mpi-master]
    L5 --> D3[DynamoDB<br/>mpi-xref]
    L5 --> S4
    L5 --> EB1[EventBridge<br/>merge-events]

    EB1 -->|FanOut| C1[EHR<br/>chart-linkage]
    EB1 -->|FanOut| C2[Data Warehouse]
    EB1 -->|FanOut| C3[Patient Outreach]
    EB1 -->|FanOut| C4[Billing]

    S4 --> GC1[Glue Catalog]
    GC1 --> AT1[Athena]
    AT1 --> QS1[QuickSight<br/>Ops + Equity Dashboards]

    style D1 fill:#9ff,stroke:#333
    style D2 fill:#9ff,stroke:#333
    style D3 fill:#9ff,stroke:#333
    style S1 fill:#cfc,stroke:#333
    style S2 fill:#cfc,stroke:#333
    style S3 fill:#cfc,stroke:#333
    style S4 fill:#cfc,stroke:#333
    style OS1 fill:#fc9,stroke:#333
    style KS1 fill:#f9f,stroke:#333
    style EB1 fill:#f9f,stroke:#333

Prerequisites

Ingredients

Code

Reference implementations: Useful aws-samples and open-source patterns for this recipe:

• Splink: a probabilistic record linkage library that runs on Spark, DuckDB, and Athena backends; produces interpretable Fellegi-Sunter outputs with EM-based parameter estimation. Healthcare-applicable.
• dedupe: a Python library for accurate and scalable fuzzy matching, record deduplication, and entity resolution; uses active learning to bootstrap from small labeled samples.
• amazon-glue-developer-guide: Glue patterns for Spark-based ETL applicable to the batch-matching pipeline.

Walkthrough

Step 1: Normalize each patient record. Aggressive normalization is the single biggest lever for matching accuracy. Skip this step and downstream comparators will spend their time on case differences, whitespace, formatting variations, and diacritics rather than the substantive differences that should drive the match decision.

FUNCTION normalize_record(raw_record):
    normalized = {}

    // Step 1A: name normalization. Case-fold, trim, strip diacritics
    // (or preserve them per institutional locale strategy), expand
    // common abbreviations, separate compound names.
    normalized.first_name = normalize_name(raw_record.first_name)
        // example: "  María-José " -> "maria jose"
    normalized.last_name = normalize_name(raw_record.last_name)
        // handle hyphenated last names: "García-López" -> "garcia lopez"
        // store both the combined form and individual components
    normalized.middle_name = normalize_name(raw_record.middle_name)
    normalized.suffix = normalize_suffix(raw_record.suffix)
        // map "Jr.", "Jr", "JR", "Junior" to canonical "jr"

    // Step 1B: nickname expansion. Maintain a curated nickname-to-
    // legal-name dictionary; for matching purposes, expand "Bob" to
    // also match "Robert" via a separate "first_name_expanded"
    // field that holds all plausible legal-name equivalents.
    normalized.first_name_expanded = expand_nicknames(normalized.first_name)
        // returns a set, not a string: {"bob", "robert", "rob", "bobby"}

    // Step 1C: phonetic encoding. Compute double metaphone codes
    // for use as blocking keys and as comparator inputs.
    normalized.first_name_metaphone = double_metaphone(normalized.first_name)
    normalized.last_name_metaphone = double_metaphone(normalized.last_name)

    // Step 1D: date of birth normalization. Parse to canonical
    // YYYY-MM-DD form, validate (no impossible dates), apply
    // century-windowing for two-digit years.
    normalized.dob = parse_date(raw_record.dob)
        // accept common formats: MM/DD/YYYY, DD-MM-YYYY, YYYYMMDD,
        // "March 14, 1972", etc.
        // century-windowing: a year of "72" becomes 1972 if today's
        // year minus 1972 is plausible for a living patient
    IF NOT is_plausible_dob(normalized.dob):
        // common garbage values: 01/01/1900, 12/31/9999, 01/01/0001
        // flag for review; do not use as a matching field
        normalized.dob_quality_flag = "implausible_dob"

    // Step 1E: address normalization. USPS-standardize where
    // possible; preserve both the input and the standardized form.
    normalized.address_input = raw_record.address
    normalized.address_usps = usps_standardize(raw_record.address)
        // returns canonical form: "123 MAIN ST APT 4 ANYTOWN ST 12345-6789"
        // or null if the address is undeliverable / unparseable

    // Step 1F: phone normalization. Strip formatting; preserve
    // last-7 and last-4 for partial-match comparators.
    normalized.phone = normalize_phone(raw_record.phone)
        // "(555) 123-4567 ext 89" -> "5551234567" (extension stored
        // separately if present)
    normalized.phone_last_7 = normalized.phone[-7:]
    normalized.phone_last_4 = normalized.phone[-4:]

    // Step 1G: SSN normalization. Strip formatting; validate length
    // and non-fake-pattern (no 000-00-0000, no 999-99-9999).
    normalized.ssn = normalize_ssn(raw_record.ssn)
    IF NOT is_valid_ssn_pattern(normalized.ssn):
        normalized.ssn = null
        normalized.ssn_quality_flag = "invalid_pattern"

    // Step 1H: email normalization.
    normalized.email = normalize_email(raw_record.email)
        // lowercase, trim, validate basic pattern

    // Step 1I: provenance. Preserve the source-record reference
    // so we can trace back from any normalized record to the raw.
    normalized.source_system = raw_record.source_system
    normalized.source_record_id = raw_record.source_record_id
    normalized.normalized_at = current UTC timestamp
    normalized.normalizer_version = NORMALIZER_VERSION

    RETURN normalized

Step 2: Generate candidate pairs through multiple blocking passes. Single-pass blocking misses real duplicates. Multi-pass blocking dramatically improves recall at modest cost. Skip this and the matcher will look great on easy duplicates and silently miss the harder ones.

FUNCTION generate_candidate_pairs(normalized_records):
    candidate_pairs = set()  // deduplicating set; same pair from
                              // multiple passes counts once

    // Pass 1: last-name metaphone + DOB year. Catches most direct
    // duplicates with name spelling variations.
    blocks_1 = group_by(normalized_records,
                        key = lambda r: (r.last_name_metaphone, year(r.dob)))
    FOR each block in blocks_1:
        FOR each pair in pairs_within_block(block):
            candidate_pairs.add(pair)

    // Pass 2: first-name metaphone + last-initial + DOB year.
    // Catches duplicates with last-name change (marriage, divorce).
    blocks_2 = group_by(normalized_records,
                        key = lambda r: (r.first_name_metaphone,
                                          r.last_name[0],
                                          year(r.dob)))
    FOR each block in blocks_2:
        FOR each pair in pairs_within_block(block):
            candidate_pairs.add(pair)

    // Pass 3: last-name initial + full DOB. Catches duplicates with
    // significant first-name variation (nickname mismatches).
    blocks_3 = group_by(normalized_records,
                        key = lambda r: (r.last_name[0], r.dob))
    FOR each block in blocks_3:
        FOR each pair in pairs_within_block(block):
            candidate_pairs.add(pair)

    // Pass 4: ZIP code + last-name initial. Catches duplicates with
    // DOB data quality issues.
    blocks_4 = group_by(normalized_records,
                        key = lambda r: (zip_code(r.address_usps),
                                          r.last_name[0]))
    FOR each block in blocks_4:
        IF len(block) > MAX_BLOCK_SIZE:
            // skip oversized blocks; they're not useful and they
            // explode the comparison count
            log_oversized_block(...)
            CONTINUE
        FOR each pair in pairs_within_block(block):
            candidate_pairs.add(pair)

    // Pass 5: phone last-4 + DOB year. Catches duplicates where the
    // name was entered very differently but the phone number is
    // stable.
    blocks_5 = group_by(normalized_records,
                        key = lambda r: (r.phone_last_4, year(r.dob)))
    FOR each block in blocks_5:
        FOR each pair in pairs_within_block(block):
            candidate_pairs.add(pair)

    // Add more passes as needed based on recall measurement against
    // the labeled gold set. Each pass costs candidate-pair count;
    // add only if it materially improves recall.

    RETURN candidate_pairs

Step 3: Score each candidate pair with field-specific comparators and the probabilistic combiner. Per-field comparators tuned to the specific field types. Probabilistic combination via Fellegi-Sunter with EM-estimated m and u probabilities. Skip the per-field tuning and you'll get bad scores on common patterns (transposed DOB digits, nicknames, hyphenated names). Skip the probabilistic combination and you'll be doing ad-hoc weighted scoring that doesn't reflect the actual information value of each field.

FUNCTION score_pair(record_a, record_b, model):
    // model contains the EM-estimated m and u probabilities per
    // field (these are estimated once, periodically re-estimated as
    // data accumulates).

    // Step 3A: per-field comparison.
    field_comparisons = {}

    field_comparisons.first_name = compare_first_name(record_a, record_b)
        // returns one of: exact, jaro_winkler_high, jaro_winkler_medium,
        // nickname_match, metaphone_match, mismatch, both_null
    field_comparisons.last_name = compare_last_name(record_a, record_b)
        // returns: exact, damerau_levenshtein_high, damerau_levenshtein_medium,
        // metaphone_match, hyphenated_partial_match, mismatch, both_null
    field_comparisons.dob = compare_dob(record_a, record_b)
        // returns: exact, year_only_match, month_day_swap, one_digit_off,
        // year_off_by_one, mismatch, one_null, both_null
        // month_day_swap is a common entry error worth catching
    field_comparisons.sex = compare_categorical(record_a.sex, record_b.sex)
    field_comparisons.address = compare_address(record_a.address_usps,
                                                  record_b.address_usps)
        // returns: exact, same_zip_plus_4, same_street_different_apt,
        // same_zip_different_street, mismatch, one_null, both_null
    field_comparisons.phone = compare_phone(record_a, record_b)
        // returns: exact, last_7_match, last_4_match, mismatch,
        // one_null, both_null
    field_comparisons.ssn = compare_ssn(record_a, record_b)
        // returns: exact, one_digit_off, mismatch, one_null, both_null
    field_comparisons.email = compare_email(record_a.email, record_b.email)
        // returns: exact, local_part_match, mismatch, one_null, both_null

    // Step 3B: Fellegi-Sunter combination. For each field, look up
    // the m and u probabilities for the observed comparison level,
    // compute the log-likelihood ratio, sum across fields.
    log_likelihood_ratio = 0
    FOR each field, comparison_level in field_comparisons:
        m = model.m_probabilities[field][comparison_level]
        u = model.u_probabilities[field][comparison_level]
        IF m > 0 AND u > 0:
            log_likelihood_ratio += log(m / u)
        // null and undefined cases handled per the model's null-handling
        // policy; typically they contribute zero (no information).

    // Step 3C: convert to a match probability if useful for routing
    // / display purposes.
    match_probability = sigmoid(log_likelihood_ratio)

    // Step 3D: package the score with full traceability so the
    // review queue can show why the pair scored as it did.
    pair_score = {
        record_a_id: record_a.source_record_id,
        record_b_id: record_b.source_record_id,
        composite_score: log_likelihood_ratio,
        match_probability: match_probability,
        field_comparisons: field_comparisons,
        per_field_log_ratios: compute_per_field_log_ratios(field_comparisons,
                                                              model),
        scored_at: current UTC timestamp,
        model_version: model.version
    }

    RETURN pair_score

Step 4: Route by threshold. Three buckets: auto-match, review, auto-non-match. Thresholds set by clinical leadership in consultation with HIM. Skip the conservative-thresholds discipline and you'll either auto-merge wrong patients (patient safety hazard) or flood the review queue with garbage (reviewer burnout, system collapse).

FUNCTION route_pair(pair_score, thresholds):
    // thresholds is a clinical-leadership-approved configuration:
    //   - HIGH_THRESHOLD: above this, auto-match
    //   - LOW_THRESHOLD: below this, auto-non-match (no action)
    //   - everything in between goes to review

    IF pair_score.composite_score >= thresholds.HIGH_THRESHOLD:
        // Auto-match path. Even auto-matches go through the merge
        // application step that records full provenance for
        // reversibility.
        emit_event("auto_match_decided", pair_score)
        return_decision = "auto_match"

    ELIF pair_score.composite_score <= thresholds.LOW_THRESHOLD:
        // Auto-non-match path. The pair is preserved in the audit
        // archive so a future investigation can trace what the
        // system saw, but no action is taken.
        emit_event("auto_non_match_decided", pair_score)
        return_decision = "auto_non_match"

    ELSE:
        // Review path. The pair goes to the review queue with a
        // priority based on the score (closer to high threshold
        // scored higher; reviewers can work the queue in priority
        // order to maximize impact per unit time).
        DynamoDB.PutItem("review-queue", {
            queue_id: assign_queue_id(pair_score),
                // queue assignment can route by clinical area,
                // by reviewer team, or by priority tier
            candidate_pair_id: new UUID,
            record_a_snapshot: deep_copy(record_a),
            record_b_snapshot: deep_copy(record_b),
                // store snapshots so the review surface shows what
                // the system saw at the time of queuing, even if
                // either source record is updated later
            score: pair_score.composite_score,
            match_probability: pair_score.match_probability,
            field_comparisons: pair_score.field_comparisons,
            per_field_log_ratios: pair_score.per_field_log_ratios,
            priority: compute_priority(pair_score, thresholds),
            queued_at: current UTC timestamp,
            review_status: "pending"
        })
        emit_event("review_queued", pair_score)
        return_decision = "review"

    // All decisions are written to the audit archive regardless of
    // routing. The archive is partitioned by date and routing
    // decision for efficient cohort-stratified analytics.
    write_to_audit_archive(pair_score, return_decision)

    RETURN return_decision

Step 5: Apply the merge with survivorship rules and full audit. The merge is the action that actually links the source records into a single patient identity. Survivorship rules decide which fields win on the golden record. Full provenance preserves the path back to the source records and supports unmerge. Skip the survivorship discipline and the merged record will be missing fields or carrying stale ones; skip the provenance and you cannot unmerge cleanly when a wrong merge surfaces.

FUNCTION apply_merge(record_a, record_b, decision_metadata):
    // decision_metadata is the audit context: who or what decided
    // (auto-match with score X, or human reviewer with ID Y), the
    // score, the routing path, the model version, the timestamp.

    // Identity-boundary validation: verify the caller is authorized
    // for this merge path before any state-changing operation.
    VALIDATE caller_role IN ALLOWED_ROLES[decision_metadata.invocation_source]
    // invocation_source is one of: auto_match_pipeline,
    // review_queue_decision, backfill_pipeline
    IF decision_metadata.invocation_source == "review_queue_decision":
        VALIDATE decision_metadata.reviewer_id is assigned to the queue
            containing this candidate pair
        VALIDATE decision_metadata.reviewer_id NOT IN conflict_of_interest_list
            for either record (reviewer is not a relative or the patient)
    IF validation fails:
        emit_metric("identity_boundary_rejection", 1,
                    dimensions={source: decision_metadata.invocation_source})
        REJECT with logged identity-boundary violation

    // Step 5A: identify the existing MPI assignments for both source
    // records. There are three cases:
    //   - both records already point to the same mpi_id (idempotent;
    //     no work)
    //   - both records point to different mpi_ids (a "cluster
    //     merge"; both clusters now combine under a single mpi_id)
    //   - at least one record has no mpi_id yet (assign one or
    //     adopt the other's)
    xref_a = DynamoDB.GetItem("mpi-xref",
                                key = (record_a.source_system,
                                        record_a.source_record_id))
    xref_b = DynamoDB.GetItem("mpi-xref",
                                key = (record_b.source_system,
                                        record_b.source_record_id))

    IF xref_a.mpi_id == xref_b.mpi_id AND xref_a.mpi_id IS NOT null:
        // Already linked; this is a re-confirmation, not a new merge.
        emit_event("merge_idempotent", ...)
        RETURN xref_a.mpi_id

    // Step 5B: pick the surviving mpi_id. If both records have one,
    // pick the older (more linkages typically accumulated under it)
    // or follow the institution's canonical-mpi-id rule.
    surviving_mpi_id = pick_surviving_mpi_id(xref_a, xref_b)

    // Step 5C: load the master records for both clusters.
    cluster_a_members = DynamoDB.Query("mpi-xref",
                                         IndexName = "mpi-id-index",
                                         KeyCondition = "mpi_id = :a")
    cluster_b_members = DynamoDB.Query("mpi-xref",
                                         IndexName = "mpi-id-index",
                                         KeyCondition = "mpi_id = :b")
    master_a = DynamoDB.GetItem("mpi-master", key = xref_a.mpi_id)
    master_b = DynamoDB.GetItem("mpi-master", key = xref_b.mpi_id)

    // Step 5D: apply survivorship rules to produce the merged
    // master record. Survivorship is field-by-field per the
    // institution's documented rules.
    merged_master = {
        mpi_id: surviving_mpi_id,
        names: merge_names(master_a.names, master_b.names,
                            rule = "preserve_all_with_primary_most_recent"),
        dob: merge_with_rule(master_a.dob, master_b.dob,
                              rule = "non_null_consistent_or_flag"),
        sex: merge_with_rule(master_a.sex, master_b.sex,
                              rule = "non_null_consistent_or_flag"),
        address_history: merge_address_lists(master_a.address_history,
                                                master_b.address_history),
        phone_history: merge_phone_lists(master_a.phone_history,
                                            master_b.phone_history),
        ssn: merge_with_rule(master_a.ssn, master_b.ssn,
                              rule = "non_null_consistent_or_flag"),
        email_history: merge_email_lists(master_a.email_history,
                                            master_b.email_history),
        insurance_records: merge_lists(master_a.insurance_records,
                                          master_b.insurance_records),
        merged_from_clusters: [xref_a.mpi_id, xref_b.mpi_id],
        last_merge_at: current UTC timestamp,
        active: true
    }

    // Step 5E: persist the merged master and update all cross-
    // references in the deprecated cluster to point to the survivor.
    //
    // Use TransactWriteItems to make the master write, xref updates,
    // and deprecated-cluster tombstone atomic. For clusters exceeding
    // 95 members (above the 100-item TransactWriteItems cap), use a
    // staging-table pattern: write intended state to merge-staging,
    // apply changes in atomic batches with progress tracking, and
    // reconcile staged-but-not-committed merges via a periodic job.
    // Audit-archive write happens after the master+xref state is
    // consistent; EventBridge merge event follows the audit-archive
    // write. Each step has DLQ routing for terminal failures so
    // half-applied merges surface for engineering investigation
    // rather than silently producing inconsistent state.
    DynamoDB.PutItem("mpi-master", merged_master)
    FOR each member in cluster_a_members + cluster_b_members:
        DynamoDB.UpdateItem("mpi-xref",
                              key = (member.source_system,
                                      member.source_record_id),
                              update = {
                                mpi_id: surviving_mpi_id,
                                previous_mpi_id_history: member.previous_mpi_id_history
                                                          + [member.mpi_id],
                                last_reassigned_at: current UTC timestamp
                              })

    // Mark the deprecated cluster as merged-into-survivor.
    IF surviving_mpi_id == master_a.mpi_id:
        DynamoDB.UpdateItem("mpi-master", key = master_b.mpi_id,
                              update = {
                                active: false,
                                merged_into: surviving_mpi_id,
                                merged_at: current UTC timestamp
                              })
    ELSE:
        DynamoDB.UpdateItem("mpi-master", key = master_a.mpi_id,
                              update = {
                                active: false,
                                merged_into: surviving_mpi_id,
                                merged_at: current UTC timestamp
                              })

    // Step 5F: write the full merge audit record. This is the
    // artifact that supports unmerge if the decision proves wrong.
    audit_record = {
        merge_id: new UUID,
        surviving_mpi_id: surviving_mpi_id,
        deprecated_mpi_ids: [master_a.mpi_id, master_b.mpi_id]
                             - [surviving_mpi_id],
        source_records_in_merge: cluster_a_members + cluster_b_members,
        decision_metadata: decision_metadata,
        survivorship_decisions: capture_field_level_survivorship(...),
        pre_merge_master_a: master_a,
        pre_merge_master_b: master_b,
        merge_at: current UTC timestamp
    }
    write_to_audit_archive(audit_record, "merge")

    // Step 5G: emit the merge event for downstream consumers.
    EventBridge.PutEvents([{
        source: "mpi-deduplication",
        detail_type: "patient_records_merged",
        detail: {
            surviving_mpi_id: surviving_mpi_id,
            deprecated_mpi_ids: audit_record.deprecated_mpi_ids,
            source_records: [r.source_record_id for r in
                             cluster_a_members + cluster_b_members],
            merge_id: audit_record.merge_id,
            merged_at: audit_record.merge_at
        }
    }])

    RETURN surviving_mpi_id

FUNCTION unmerge(merge_id, reason, operator_id):
    // Look up the audit record for the merge being reversed.
    audit_record = lookup_audit_record(merge_id)

    // Restore the pre-merge master records.
    FOR each pre_merge_master in [audit_record.pre_merge_master_a,
                                    audit_record.pre_merge_master_b]:
        DynamoDB.PutItem("mpi-master", pre_merge_master)

    // Restore the cross-references to their pre-merge mpi_ids.
    FOR each source_record_ref in audit_record.source_records_in_merge:
        DynamoDB.UpdateItem("mpi-xref",
                              key = (source_record_ref.source_system,
                                      source_record_ref.source_record_id),
                              update = {
                                mpi_id: source_record_ref.previous_mpi_id,
                                last_reassigned_at: current UTC timestamp
                              })

    // Mark the survivor as no-longer-active-as-survivor (it remains
    // available; it just isn't carrying the additional cluster anymore).
    DynamoDB.UpdateItem("mpi-master", key = audit_record.surviving_mpi_id,
                          update = {
                            unmerged_at: current UTC timestamp,
                            unmerge_reason: reason
                          })

    // Write the unmerge audit record.
    write_to_audit_archive({
        unmerge_id: new UUID,
        original_merge_id: merge_id,
        operator_id: operator_id,
        reason: reason,
        unmerged_at: current UTC timestamp
    }, "unmerge")

    // Emit the unmerge event for downstream consumers.
    EventBridge.PutEvents([{
        source: "mpi-deduplication",
        detail_type: "patient_records_unmerged",
        detail: {...}
    }])

Curious how this looks in Python? The pseudocode above covers the concepts. If you'd like to see sample Python code that demonstrates these patterns using boto3, check out the Python Example. It walks through each step with inline comments and notes on what you'd need to change for a real deployment.

Expected Results

Sample candidate-pair score (truncated for readability):

{
  "candidate_pair_id": "cp-2026-04-22-00018345",
  "record_a_id": "src-ehr1-MRN-009315",
  "record_b_id": "src-ehr1-MRN-018747",
  "composite_score": 9.42,
  "match_probability": 0.997,
  "field_comparisons": {
    "first_name": "exact",
    "last_name": "damerau_levenshtein_high",
    "dob": "exact",
    "sex": "exact",
    "address": "same_zip_different_street",
    "phone": "last_4_match",
    "ssn": "both_null",
    "email": "exact"
  },
  "per_field_log_ratios": {
    "first_name": 1.85,
    "last_name": 2.10,
    "dob": 4.20,
    "sex": 0.18,
    "address": -0.45,
    "phone": 0.92,
    "ssn": 0.00,
    "email": 0.62
  },
  "scored_at": "2026-04-22T03:14:18Z",
  "model_version": "fs-v2.3.1",
  "routing_decision": "auto_match",
  "routing_threshold_high": 8.5,
  "routing_threshold_low": -2.0,
  "snapshots": {
    "record_a": {
      "first_name": "maria",
      "last_name": "garcia",
      "dob": "1972-03-14",
      "sex": "F",
      "address_usps": "1421 ELM ST APT 4 ANYTOWN ST 12345-6789",
      "phone": "5551234567",
      "ssn": null,
      "email": "mgarcia@example.com"
    },
    "record_b": {
      "first_name": "maria",
      "last_name": "garcia-lopez",
      "dob": "1972-03-14",
      "sex": "F",
      "address_usps": "789 OAK AVE ANYTOWN ST 12345-6543",
      "phone": "5559994567",
      "ssn": null,
      "email": "mgarcia@example.com"
    }
  }
}

Sample merge audit record (truncated):

{
  "merge_id": "merge-2026-04-22-00018345",
  "surviving_mpi_id": "mpi-000000128347",
  "deprecated_mpi_ids": ["mpi-000000284901"],
  "source_records_in_merge": [
    {"source_system": "ehr1", "source_record_id": "MRN-009315",
      "previous_mpi_id": "mpi-000000128347"},
    {"source_system": "ehr1", "source_record_id": "MRN-018747",
      "previous_mpi_id": "mpi-000000284901"}
  ],
  "decision_metadata": {
    "decision_type": "auto_match",
    "score": 9.42,
    "score_threshold_high": 8.5,
    "model_version": "fs-v2.3.1",
    "decided_at": "2026-04-22T03:14:18Z"
  },
  "survivorship_decisions": {
    "primary_name": {
      "winner": "garcia-lopez",
      "rule": "most_recent_legal_name",
      "loser_preserved_as_alias": "garcia"
    },
    "current_address": {
      "winner": "789 OAK AVE ANYTOWN ST 12345-6543",
      "rule": "most_recent",
      "address_history_combined": true
    },
    "current_phone": {
      "winner": "5559994567",
      "rule": "most_recent",
      "phone_history_combined": true
    },
    "email": {
      "winner": "mgarcia@example.com",
      "rule": "exact_match_no_decision_needed"
    },
    "insurance_records": {
      "winner": "combined",
      "rule": "merge_lists_with_dedup"
    },
    "problem_list": {
      "winner": "combined",
      "rule": "merge_clinical_lists_for_downstream_review"
    }
  },
  "merged_at": "2026-04-22T03:14:19Z"
}

Performance benchmarks (illustrative, your mileage varies):

Where it struggles:

• Common-name + missing-DOB combinations. A "John Smith" with no DOB and no SSN matched against another "John Smith" with no DOB is essentially unidentifiable from the demographic data alone. The pipeline correctly flags these as low-information matches; it cannot fabricate confidence the data does not support. The mitigation is upstream: improve DOB and SSN capture at the registration desk so the matching system has the data it needs.
• Twin and family-member confounding. Twins share birthday, last name, address, sometimes first-name initial (especially with family-naming traditions). Mothers and infants are co-registered with overlapping fields. Spouses share insurance and phone. Probabilistic matchers built without family-aware logic regularly merge these. The mitigation is family-aware blocking and per-field comparators that down-weight matches on shared-family fields when other fields suggest different individuals.
• Pediatric and frequent-mover populations. Pediatric patients change addresses with their parents, change phone numbers as the household phone changes, and have less independent identity-anchoring data. Frequent movers have address-history fragmentation that breaks address-based matching. The mitigation is leaning more heavily on stable identifiers (DOB, SSN where available) and less on changing ones for these cohorts; this is a per-cohort tuning question rather than a one-size-fits-all algorithm.
• Recently-merged-system populations. When two health systems merge, the combined database carries every cross-system duplicate as a new entity-resolution problem. The blocking-pass design needs to be reviewed because the cross-system population shifts the m and u distributions; the threshold needs to be re-tuned because the score distribution shifts; the review-queue staffing needs to ramp because the backlog is large. Plan a deliberate cleanup project on system mergers; the steady-state matcher will not absorb the backlog quickly enough on its own.
• Names from naming conventions outside the dominant culture. Hispanic surnames with multiple components (paternal-maternal patterns), Asian names with order variations (family name first vs given name first), Arabic names with transliteration variations all match worse on average than dominant-culture names because the standard string-similarity heuristics were tuned on the dominant-culture cases. Cohort-stratified accuracy monitoring surfaces this; the mitigation is per-cohort comparator tuning, supplementary cohort-specific blocking passes, and (where feasible) a cohort-specific m/u model. This is an equity issue, not a fancy edge case.
• Records that should not be merged but score high. Same-name same-DOB different-people happen, particularly with common names. The pipeline relies on additional fields (SSN, address, phone, email) to discriminate; when those fields are also missing or coincidentally similar, the score can be misleadingly high. The mitigation is human review on borderline cases (set the high threshold conservatively enough that genuinely ambiguous cases route to review rather than auto-match), and structured rationale capture in the review interface so the HIM team's decisions inform model retuning.
• Active patients vs historical patients. A historical record from 2002 with a long-disconnected phone, an address from a building that was demolished, and a maiden name will not match against the current registration of the same person. The pipeline treats these as separate patients because the data does not connect them. The mitigation is a periodic deeper sweep that compares active records against historical records using looser blocking and human review for everything; this is a separate, slower process from the always-on real-time matcher.
• Low-information SSN matches. SSN match is one of the most informative fields, but data-quality issues (mistyped SSNs, SSNs reused on dependent records, fake SSNs entered when ID was not available) introduce false matches. The pipeline validates SSN patterns and flags suspicious values (000-00-0000, 999-99-9999, sequential digits) for manual review; never use a single-field SSN match as auto-match evidence on its own.

Why This Isn't Production-Ready

The pseudocode and architecture above demonstrate the pattern. A production deployment needs to close several gaps that are intentionally out of scope for a recipe.

Threshold tuning against a labeled gold set. The high and low thresholds are clinical-leadership decisions that depend on the score distribution in your specific data. Build the labeled gold set first (1,000 to 5,000 candidate pairs, reviewed by HIM specialists, with match / not-match labels), use it to characterize the score distribution under known matches and known non-matches, and pick thresholds that produce the auto-match precision and recall the institution can defend. Re-tune the thresholds at least annually and after any major data-quality change (registration system upgrade, new acquisition, change in nickname dictionary).

M/U probability estimation and re-estimation cadence. The m and u probabilities are the core of the Fellegi-Sunter scoring. EM-based estimation works but produces probabilities that drift as the underlying data drifts (population changes, registration practices change, new fields added or deprecated). Build a re-estimation pipeline that runs on a documented cadence (quarterly is typical), validates the new probabilities against the held-out gold set before promotion, and emits a "model version updated" event that downstream consumers can react to.

Survivorship rule design and patient-facing implications. The survivorship rules in the pseudocode are illustrative. The real survivorship policy is a cross-functional decision involving HIM, clinical informatics, the privacy office, and (ideally) patient-advisory representation. Decisions like "the merged record uses the most recent legal name" need to handle gender transitions, name changes that were specifically requested to be hidden, and patient-stated preferences about historical-name handling. The policy should be documented, reviewed annually, and implemented in code with the policy version recorded alongside every merge.

Review queue UX and reviewer training. The review interface in the architecture is a placeholder. A production interface needs side-by-side record display with field-level diff highlighting, a clear summary of the score and its contributors, single-keystroke advance/decision, bulk-action support for obvious clusters, escalation to a senior reviewer for ambiguous cases, and a search interface so a reviewer can pull up the candidate pair for a specific patient on demand. Reviewer training on the decision criteria, on edge cases (twins, family members, intentional name changes, suspected identity fraud), and on documentation standards is a one-to-three-week onboarding investment per reviewer.

Real-time matching latency budget and caching. Real-time matching at registration time has a tight latency budget; the registration clerk cannot wait fifteen seconds for the result. Architect for sub-second response: in-memory blocking key indices for the most common access patterns, OpenSearch query optimization, candidate-set capping (return at most N candidates per blocking pass), and asynchronous follow-up scoring for borderline cases that did not resolve in time. The asynchronous-resolution pattern lets the registration desk proceed while the system continues working in the background; if a duplicate is detected after the patient leaves, the merge happens in the standard review/auto-match flow.

Cross-system identity reconciliation when MPI assignments diverge. In a multi-EHR institution, the MPI must be the source of truth and downstream systems must adopt the surviving mpi_id. The reconciliation when source systems disagree (the EHR has linked records as the same patient but the billing system has them separate, or vice versa) requires a cross-system reconciliation process that surfaces the divergence and pushes the resolved identity to all systems. Build this from day one, because cross-system divergence accumulates silently and is much harder to fix later.

Identity-fraud detection. The same techniques that detect duplicate records also detect potential identity-fraud cases (same demographics with very different SSNs or different DOBs). The pipeline should route suspected fraud cases to the institution's fraud-investigation team rather than to the standard merge flow. Define the fraud-detection rules in consultation with the compliance and security teams.

Data quality feedback to the registration desk. The matcher learns where the source data quality is poor. Names that are routinely mistyped, DOBs that are routinely entered as 01/01/1900, addresses that fail USPS standardization, phones that are routinely stored with extension formatting issues. Feed this back to the registration system and the front-desk training program. The matcher cannot fix bad input; it can identify the patterns that need upstream fixes.

Equity instrumentation and cohort-specific tuning. The cohort-stratified accuracy dashboard is necessary but not sufficient. When the dashboard surfaces a cohort-specific accuracy gap, there must be a documented process for diagnosing and addressing it: cohort-specific m/u tuning, supplementary blocking passes, comparator adjustments for naming conventions, or (in some cases) HIM-team training on culturally-specific name handling. The process is operational; the dashboard alone does not close the gap.

Idempotency and retry semantics. Real-time matching is event-driven; the pipeline must handle duplicate-event-delivery without producing duplicate merges. Use the (source_system, source_record_id) as the idempotency key for normalize-and-route operations; use the merge_id as the idempotency key for merge-application operations. Lambda invocations should be idempotent at these keys. Step Functions Catch should distinguish retryable infrastructure failures from terminal logic failures and route terminal failures to a DLQ for human investigation.

Audit-log retention and access control. The audit log is highly sensitive: it contains every patient-identification decision the system has made, including the historical states of source records and the rationales. Apply the institution's clinical-record retention policy to the audit log (typically several years to decades). Apply tighter access control than for general analytics: the audit log is a forensic-grade artifact and should be queryable only by named auditors, compliance staff, and HIM leadership, with every read logged.

Backfill strategy for historical records. When the matcher launches, it has to process the existing patient base (potentially millions of records) before steady-state operation begins. The backfill is a separate engineering and operational project: generate the candidate pairs in batch, score them all, route through the review queue, ramp HIM-team capacity for the initial review wave, and accept that the cleanup will take weeks to months. Plan the backfill explicitly; do not assume the matcher will absorb it as part of normal operation.

Operational Architecture: M/U Re-estimation Pipeline

The m/u probabilities drift as the underlying population and registration practices change. A Step Functions workflow runs on a quarterly cadence (plus on-demand triggers after major data-quality events such as system mergers or registration-system upgrades). The workflow schedules a Glue job that re-estimates the m and u parameters using the current labeled gold set plus unlabeled data via expectation-maximization. The validation gate before promotion: per-cohort precision and recall reported on the held-out gold set, no per-cohort regression beyond a documented threshold (typically 2% absolute), equity-review committee approval. Approved models are promoted by writing a new MODEL_VERSION configuration entry to a DynamoDB config table; the threshold-router Lambda and the batch Glue job read the current model version on each invocation. The pipeline emits a model_version_updated event on EventBridge with prior and new version IDs so analytics dashboards can partition queries on model_version rather than computing across the version boundary.

Operational Architecture: Latency Primitives

Real-time matching at registration has a sub-second latency budget. The architecture uses:

• ElastiCache (Redis) in-memory blocking-key index for the most common access patterns, seeded by a periodic Glue job that refreshes the index from the normalized-records store.
• Per-pass candidate cap (N=50 typical, configurable) so no single blocking pass produces an unbounded candidate set.
• SQS-queued asynchronous-follow-up Lambda for borderline pairs that did not resolve within the latency budget. The registration desk proceeds; the follow-up Lambda scores and routes asynchronously.
• OpenSearch failover: when OpenSearch is unavailable, the candidate-generator falls back to ElastiCache alone with a logged metric opensearch_fallback_invoked. If both are unavailable, the record routes to a "queued for matching" DLQ that the next batch-matching cycle processes. A CloudWatch alarm fires on chronic fallback so degraded matching freshness does not go unnoticed.

Operational Architecture: Cross-System Reconciliation

Downstream systems emit reconciliation events back to the matcher:

A reconciliation Lambda compares the downstream linkage state to the MPI's mpi_id assignment. Divergence routes to a cross-system-divergence-queue with the source-system linkage evidence preserved. HIM-leadership roles adjudicate; resolution either updates the MPI to match downstream reality or pushes a corrective linkage event back to the source system. Cross-system divergence rates per system are dashboarded; chronic divergence indicates either a matcher gap or a downstream data-quality issue that needs investigation.

Operational Architecture: Data Quality Feedback Loop

The normalize Lambda emits data_quality_observed events on EventBridge for each record's quality flags (implausible DOB, invalid SSN pattern, USPS-unstandardizable address, phone with extension formatting issues). The registration system subscribes to these events and surfaces aggregate patterns on a front-desk training dashboard. The feedback is aggregate (weekly summaries by registration-desk location and time-of-day), not individual-patient, so it does not create a PHI exposure. The goal is upstream quality improvement: every percentage-point improvement in DOB capture reduces the matcher's unresolvable-pair rate by more than a percentage point.

Operational Architecture: DLQ and Idempotency Coverage

Every Lambda in the pipeline has a configured DLQ (SQS) for terminal failures. The DLQ triggers a CloudWatch alarm after one message; engineering investigates DLQ items within 24 hours as a standard operating procedure.

Idempotency keys on every write path:

The fall-back-to-no-action pattern: when matching fails partway through (e.g., OpenSearch timeout during candidate generation), the pipeline records the incomplete attempt, does not produce a partial routing decision, and either retries (for transient failures) or routes to DLQ (for terminal failures). A partial merge is never committed; the TransactWriteItems pattern ensures all-or-nothing semantics.

Operational Architecture: Cross-Recipe Orchestration Contract

This recipe establishes the shared substrate for downstream Chapter 5 recipes:

1. MPI-as-shared-substrate contract. Every downstream consumer holding an mpi_id from a prior write resolves it through the mpi-xref cross-reference table at read time to handle subsequent merges. Stale mpi_id references are common and expected; the resolution pattern is: query mpi-xref for the current mpi_id, follow the merged_into chain if the master is inactive.

2. Matching-primitives library boundary. Blocking passes, per-field comparators, and the Fellegi-Sunter combiner are packaged as a Lambda layer (or Glue Python library) with explicit version coupling. Downstream recipes (5.5 cross-facility, 5.6 claims-to-clinical, 5.7 longitudinal, 5.8 privacy-preserving, 5.9 TEFCA, 5.10 deceased-patient resolution) import the primitives at a pinned version rather than duplicating them.

3. Audit-archive shared-substrate. The same S3 partition scheme (audit/{decision_type}/{YYYY}/{MM}/{DD}/{uuid}.json) is consumed by all Chapter 5 recipes. Downstream recipes write to their own decision-type partitions but read from the shared archive for lineage queries.

Operational Architecture: Backfill Primitives

The backfill pipeline uses a separate review queue (review-queue-backfill) distinct from the real-time queue. Routing priority: real-time pairs are reviewed first; backfill pairs are served only when real-time queue depth permits. Backfill routing pauses when combined queue depth exceeds the configured ceiling (typically 2x daily decision throughput) and resumes below the hysteresis threshold (typically 1.5x). A backfill dashboard tracks projected completion timeline against HIM throughput; ramp-up and ramp-down decisions are made on dashboard signal rather than calendar. The transition to baseline staffing happens when projected steady-state queue depth at baseline FTE ratio falls below the maintenance threshold.

Operational Architecture: Patient-Facing Data Disclosure

Before surfacing stored demographic data to the patient in a self-service portal, gate on a data-disclosure policy review. Data sourced from the patient directly (registration self-attestation, portal submissions) is appropriate to display. Data sourced from third-party feeds (insurance eligibility responses, claims, partner-organization shares) may carry separate disclosure-and-consent requirements. The policy framework must address these before the self-service feature surfaces third-party-sourced data to the patient. Implementation: a data_source attribute on each field in mpi-master drives the disclosure-eligibility check at render time.

Variations and Extensions

Active-learning-driven labeling for the gold set. Rather than randomly sampling pairs for HIM review during gold-set construction, prioritize pairs near the decision boundary (medium scores) or pairs in cohorts where the model is least certain. Active learning can reduce the labeling effort to build a useful gold set by 50% or more, which materially reduces the upfront HIM-team load. Libraries like dedupe implement this pattern. The tradeoff is that active-learning samples are biased toward the boundary, so the gold set is not directly usable for unbiased accuracy measurement; you need a separate, randomly-sampled validation set for that.

Per-cohort m/u models. Rather than a single m/u model fitted across the entire dataset, fit separate models per cohort (defined by demographics, language, geographic region, or other relevant axes). Per-cohort models capture cohort-specific patterns (Hispanic surnames have different m/u distributions than non-Hispanic surnames) and substantially improve cohort fairness. The tradeoff is that smaller cohorts have less data for parameter estimation; production deployments typically use cohort-specific models for the larger cohorts and the population-level model as a fallback for smaller cohorts.

Embedding-augmented similarity. Add learned string embeddings (sentence-transformer or character-level embeddings) as additional similarity features in the Fellegi-Sunter framework. Useful for handling transliteration, abbreviation, and complex naming conventions. The embeddings can be domain-specific (trained on healthcare-name corpora) or general-purpose. This is an incremental enhancement to the probabilistic core, not a replacement.

Graph-based clustering for ambiguous chains. When a candidate pair connects to other candidate pairs through transitive relationships ("A matches B, B matches C, A does not match C"), the simple pair-based decision logic produces inconsistencies. Graph-based clustering algorithms (connected components on the "auto-match" subgraph, with conflict-resolution rules for transitivity violations) produce coherent identity clusters. This is a more sophisticated routing layer on top of the per-pair scoring; useful when the data is messy enough to produce many transitive ambiguities.

Streaming continuous matching. Rather than nightly batch refresh plus real-time at registration, run the matcher continuously as a streaming pipeline. Every source-record update flows through the candidate-generation, scoring, and routing pipeline within seconds, regardless of whether the update came from a new registration or an edit to an existing record. This catches duplicates that emerge from edits (a phone number update suddenly makes two records look the same) and supports near-real-time downstream consumers. Slightly more complex to operate; useful for institutions with high registration-edit volumes.

Privacy-preserving matching (related to recipe 5.8). When the matching needs to span organizational boundaries, the techniques in recipe 5.8 (Bloom-filter-based matching, secure hash chains) apply. The internal-duplicate matcher is the natural foundation; the privacy-preserving extensions add cryptographic layers that let the matcher operate without exposing raw demographic data across organizational boundaries.

Risk-tier-aware thresholding. Different patient populations have different stakes for false-merge errors. A pediatric oncology patient is higher-stakes than an episodic urgent-care patient. The pipeline can apply different thresholds to different patient populations, with tighter thresholds for high-stakes populations. The tradeoff is operational complexity; the institution must define and govern the population tiers and the per-tier threshold policies.

Continuous comparator updating from review-queue feedback. Every reviewer decision (match / not-match) is an additional labeled example. Pipe the labels back into the m/u re-estimation and into the comparator-tuning process. Over time the matcher learns from the reviewers' decisions and the m/u probabilities reflect the institution's specific population. The pattern requires careful versioning so that retraining cycles do not silently reroute previously-decided pairs differently.

Patient-facing identity self-service. A portal feature that lets patients see (and request corrections to) the demographic data the institution has on file for them. A patient who notices that their address is wrong, their phone number is stale, or their preferred name is missing can update the record directly, which improves the data quality at the source. This is a downstream-of-matching feature but it directly affects the matcher's input quality.

Additional Resources

AWS Documentation:

• Amazon DynamoDB Developer Guide
• Amazon OpenSearch Service Developer Guide
• AWS Glue Developer Guide
• Amazon Athena User Guide
• AWS Step Functions Developer Guide
• AWS Lambda Developer Guide
• Amazon EventBridge User Guide
• Amazon Kinesis Data Streams Developer Guide
• Amazon API Gateway Developer Guide
• Amazon Cognito Developer Guide
• Amazon QuickSight User Guide
• AWS HIPAA Eligible Services
• Architecting for HIPAA on AWS (Whitepaper)

AWS Sample Repos:

• aws-samples/aws-glue-samples: Glue ETL patterns applicable to the batch-matching pipeline
• aws-samples/amazon-opensearch-developer-guide: OpenSearch patterns including custom analyzers for phonetic search applicable to the candidate-generation index
• aws-samples/serverless-patterns: Step Functions + Lambda + DynamoDB orchestration patterns applicable to the real-time matching pipeline

AWS Solutions and Blogs:

• AWS Solutions Library (filter Healthcare and Life Sciences): browse for healthcare data quality, master data management, and patient matching reference architectures
• AWS for Industries: Healthcare and Life Sciences Blog: search "patient matching," "master data management," and "EMPI" for relevant deep-dives
• AWS Big Data Blog: search "entity resolution," "fuzzy matching," and "Splink" for relevant pipeline patterns

External References (Methodology):

• Fellegi I., Sunter A. A Theory for Record Linkage: the foundational 1969 paper that grounds probabilistic record linkage
• Christen P. Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection: comprehensive textbook covering the methodology
• Splink documentation: a probabilistic record linkage library with healthcare-applicable patterns
• dedupe documentation: library for fuzzy matching and entity resolution

External References (Healthcare Practice):

• ONC Patient Identification and Matching Final Report
• AHIMA Best Practices for Patient Identity Integrity: operational guidance on MPI management, survivorship, and review processes
• The Sequoia Project Patient Matching White Papers: industry-collaborative work on patient matching standards
• The Pew Charitable Trusts Patient Matching Reports: policy and operational research on the state of patient matching in the United States
• ECRI Top 10 Patient Safety Concerns: annual report consistently flagging patient identification as a top concern

External References (Equity):

• Obermeyer Z. et al. 2019, Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations: the canonical cautionary tale for fairness failures in healthcare AI; required reading for anyone building entity resolution systems
• RAND Corporation reports on patient matching disparities: research on demographic-cohort accuracy gaps in patient matching

Estimated Implementation Time

← Main Recipe 5.1 · Python Example · Chapter Preface