Recipe 4.8 Architecture and Implementation: Treatment Response Prediction

Companion to Recipe 4.8: Treatment Response Prediction. This page covers the AWS architecture, services, prerequisites, and pseudocode. For the problem framing and the conceptual approach, start with the main recipe.

The AWS Implementation

Why These Services

Amazon SageMaker for the causal-modeling stack. Three model families per treatment-comparator pair: the propensity score model, the outcome model, and the CATE estimator (typically run as an ensemble of two or more methods). With a treatment catalog of ten to thirty treatment-comparator pairs in scope, the model count multiplies; SageMaker Pipelines orchestrates the per-pair retraining workflow, and the SageMaker Model Registry holds versioned artifacts with the governance metadata (cohort version, training-cohort archive, calibration-test results, fairness-test results, medical-director sign-off). Inference runs through Batch Transform for the daily cohort scoring path and through a low-latency real-time endpoint for the point-of-care path when a clinician opens a patient chart and requests on-demand scoring. SageMaker is HIPAA-eligible under BAA.

Amazon SageMaker Feature Store for patient features and treatment-history features. Patient-level features (demographics, problem list, medication history, lab trajectories, vitals trajectories, comorbidity profile, SDOH proxies, prior treatment exposures, prior treatment responses) reused from earlier recipes and enriched with treatment-response-specific features (time since diagnosis for the index condition, prior medication-class exposures, prior discontinuation reasons, baseline disease severity). The offline store powers cohort construction and batch scoring; the online store supports the low-latency lookups when a clinician opens a chart at the point of care.

Amazon DynamoDB for the treatment catalog, scoring results, decision records, and feedback events. New table treatment-catalog keyed on treatment_id and version. New table treatment-comparison-pairs keyed on (treatment_id, comparator_id) and version, holding the modeling specification and the latest model registry pointer per pair. New table scoring-results keyed on (patient_id, scoring_run_id) for the structured scoring output that feeds the clinician view. New table decision-records keyed on (patient_id, decision_id) for the clinician's recorded decision and rationale, including the predictions presented at the time of decision (frozen for audit). New table prediction-outcome-pairs for the matched-outcome data feeding feedback and calibration monitoring. The patient-profile table from prior recipes is extended with treatment-response-relevant attributes.

Amazon S3 for the data lake, training cohorts, evaluation outputs, and longitudinal storage. Source data feeds (claims, EHR, lab, pharmacy, vitals, PROMs) land in S3 via Kinesis Firehose for streaming and Glue for batch. Offline feature store backed by S3. Cohort construction outputs land in S3 as Parquet, partitioned by treatment-comparator pair and version, and archived alongside the trained model artifacts so that any model can be reproduced. Per-treatment-comparator calibration plots, fairness reports, and sensitivity analyses live in S3 with the model artifacts.

AWS Glue and Amazon Athena for cohort construction, propensity matching, and outcome evaluation. Cohort construction is SQL-shaped at scale: the target-trial-protocol filters (eligibility, washout, exposure assignment, censoring) translate directly into SQL on the data lake. Glue jobs run the cohort construction nightly or weekly per treatment-comparator pair. Athena powers the per-cohort exploration, the calibration analyses, and the post-deployment outcome surveillance. Propensity matching for sensitivity analyses runs in Glue with logistic-regression or gradient-boosted propensity models and outputs the matched-pair datasets to S3.

AWS Step Functions for batch orchestration. Same pattern as 4.5 through 4.7. The nightly cohort-construction-and-feature-pipeline workflow is a Step Functions state machine. The weekly retraining-and-recalibration workflow is a separate state machine that runs the per-pair retraining, executes the calibration and fairness tests, gates on the governance review, and promotes new models in the registry. The monthly post-deployment surveillance workflow runs the prediction-to-outcome matching, the calibration drift detection, and the cohort-stratified performance monitoring.

Amazon EventBridge for scheduling and event-driven triggers. EventBridge schedules the nightly cohort run, the weekly retraining run, and the monthly surveillance run. EventBridge rules also handle event-driven triggers: a clinician opening an index-condition chart at the point of care triggers an on-demand scoring request; an adverse-event report triggers a surveillance check against recent predictions; a treatment-catalog update (new entry, tier change, supply constraint) triggers downstream re-evaluation of affected scoring outputs.

Amazon Bedrock for clinician-facing comparison briefings, patient-facing summaries, and disagreement-investigation narratives. Three distinct LLM use cases:

1. Clinician-facing comparison briefing. A structured-output prompt takes the per-treatment CATE estimates with confidence intervals, the cohort sizes and match quality, the OOD flags, the formulary statuses, the guideline references, and the patient summary, and returns a paragraph the clinician reads at the point of care. The validator strictly enforces no-recommendation language: the briefing describes the comparison without selecting a treatment.

2. Patient-facing summary. When the clinician chooses to share, a patient-facing summary translates the comparison into lay language. The patient version uses cohort-based phrasing ("among patients similar to you, treatment A was associated with a larger reduction in blood sugar than treatment B, with about this much uncertainty"), avoids probabilistic statements that may be misread as guarantees, and includes the explicit framing that the final decision is shared between the patient and the clinician. Reading-level matching applies the same pattern as Recipe 4.2.

3. Disagreement-investigation narrative. When the multiple CATE estimators disagree more than a defined threshold, an LLM-generated narrative lists the points of disagreement, the candidate explanations (model misspecification, unmeasured confounding, treatment-effect heterogeneity, data quality issues), and the recommended sensitivity-analysis follow-up for the clinical informatics team. This is internal-facing decision support for the modeling team, not patient-facing or clinician-facing.

Bedrock is HIPAA-eligible under BAA. Confirm in service terms that prompts and completions are not used to train the underlying foundation models.

AWS Lambda for per-stage glue logic. The cohort-construction dispatcher, the propensity-score-invocation orchestrator, the CATE-estimator-ensemble runner, the uncertainty-quantification computer, the calibration-drift detector, the OOD-flag computer, the on-demand-scoring orchestrator, the briefing-generator, the validator, the decision-recorder, and the prediction-to-outcome matcher all run as Lambdas.

Amazon API Gateway and Amazon Cognito for the on-demand scoring API. The clinician-facing decision-support component is exposed as an authenticated API consumed by the EHR integration layer. API Gateway provides the endpoint, Cognito (or the institution's identity provider via SAML or OIDC) provides authentication, and per-clinician audit logs go to CloudTrail. The EHR integration is typically through a SMART on FHIR app or a CDS Hooks endpoint, both of which the API supports.

Amazon Kinesis Data Streams for clinical events, scoring requests, decisions, and outcome updates. Same engagement-event bus pattern as prior chapters, with new event types: treatment_scoring_requested, treatment_scoring_completed, treatment_scoring_oodflag_triggered, treatment_decision_recorded, treatment_outcome_observed, prediction_calibration_alert, disagreement_alert. The state-machine worker consumes these and updates the relevant DynamoDB tables.

AWS HealthLake (recommended for this recipe). Treatment response prediction benefits substantially from FHIR-native clinical data. The condition list, medication list, observation history, and procedure history feed directly into the phenotyping and outcome construction. HealthLake provides FHIR-native storage with built-in normalization across multiple EHR vendors, which matters here because the model's accuracy depends on consistent representation of clinical concepts across data sources.

Amazon QuickSight for governance and surveillance dashboards. Per-treatment-comparator calibration dashboards (predicted versus observed treatment effects, with confidence intervals, stratified by cohort). Per-cohort fairness dashboards (calibration parity, estimate parity at equipoise, adverse-event parity). Post-deployment surveillance dashboards (prediction error trends over time, OOD-flag rates by cohort, override rates by clinician). Treatment-catalog governance dashboards (per-treatment usage, per-treatment outcome trends, model-version lineage). QuickSight on Athena, with row-level security for cohort-specific access.

AWS KMS, CloudTrail, CloudWatch. Same PHI infrastructure pattern as prior recipes, with elevated controls for the prediction artifacts. Customer-managed keys, CloudTrail data events on the prediction tables and the scoring API, CloudWatch alarms on calibration drift, OOD-flag rate spikes, and disagreement-alert rate spikes. Treatment recommendations are sensitive enough that the audit posture is closer to clinical-record audit than typical analytics audit.

Architecture Diagram

flowchart LR
    subgraph Sources
      A1[Claims Feed]
      A2[EHR / FHIR via HealthLake]
      A3[Lab Results]
      A4[Pharmacy / Adherence from 4.5]
      A5[Vitals and PROMs]
      A6[Patient Registries]
      A7[Patient Self-Report]
    end

    A1 -->|Streaming| B1[Kinesis Firehose\ntrx-source-events]
    A2 -->|FHIR| HL1[AWS HealthLake]
    HL1 -->|Export| B1
    A3 -->|Streaming| B1
    A4 -->|Daily| B1
    A5 -->|Streaming| B1
    A6 -->|Daily| B1
    A7 -->|Portal/app| B1

    B1 --> S1[S3\ntrx-data-lake]

    PR1[Lambda\ncatalog-sync] --> D1[DynamoDB\ntreatment-catalog]
    D1 --> C1[Glue\ncohort-construction]
    S1 --> C1
    C1 --> S2[S3\ncohorts and labels]
    C1 --> F1[SageMaker Feature Store\npatient and treatment-history features]

    SF1[Step Functions\nweekly-retrain] --> H1[SageMaker Pipelines\nper treatment-comparator pair]
    F1 --> H1
    S2 --> H1
    H1 --> H2[SageMaker Training\npropensity model]
    H1 --> H3[SageMaker Training\noutcome model]
    H1 --> H4[SageMaker Training\nCATE ensemble]
    H2 --> MR1[SageMaker Model Registry]
    H3 --> MR1
    H4 --> MR1

    MR1 --> H5[Lambda\ncalibration and fairness tests]
    H5 --> S3[S3\nmodel evaluation reports]
    H5 --> GR1[Lambda\ngovernance gate]
    GR1 --> MR1
    GR1 --> D1

    AG1[API Gateway\nscoring API] --> AU1[Cognito or IdP]
    AU1 --> SC1[Lambda\nscoring orchestrator]
    SC1 --> F1
    SC1 --> D1
    SC1 --> H6[SageMaker Real-Time\nCATE inference]
    SC1 --> H7[SageMaker Batch Transform\ncohort retrieval]
    SC1 --> SR1[DynamoDB\nscoring-results]
    SR1 --> BR1[Lambda\nbriefing-generator]
    BR1 --> BD1[Bedrock\ncomparison briefing]
    BD1 --> VL1[Lambda\nvalidator]
    VL1 --> SR2[DynamoDB\nbriefings]
    SR2 --> EHR1[EHR via SMART on FHIR\nor CDS Hooks]

    EHR1 -.Decision recorded.-> DR1[DynamoDB\ndecision-records]
    EHR1 -.Decision event.-> KS1[Kinesis\ntrx-events]

    KS1 --> SM1[Lambda\nstate-machine-worker]
    SM1 --> DR1
    SM1 --> PO1[DynamoDB\nprediction-outcome-pairs]

    SF2[Step Functions\nmonthly-surveillance] --> SU1[Lambda\nprediction-outcome-matcher]
    SU1 --> PO1
    SU1 --> S4[S3\nsurveillance outputs]
    S4 --> CD1[Lambda\ncalibration-drift-detector]
    S4 --> FA1[Lambda\nfairness-monitor]
    CD1 --> AL1[CloudWatch alarms]
    FA1 --> AL1

    S4 --> QS1[QuickSight\ngovernance and surveillance dashboards]

    KS1 --> KF1[Kinesis Firehose]
    KF1 --> S5[S3\ntrx-event-lake]
    S5 --> QS1

    style F1 fill:#9ff,stroke:#333
    style D1 fill:#9ff,stroke:#333
    style SR1 fill:#9ff,stroke:#333
    style SR2 fill:#9ff,stroke:#333
    style DR1 fill:#9ff,stroke:#333
    style PO1 fill:#9ff,stroke:#333
    style S1 fill:#cfc,stroke:#333
    style S2 fill:#cfc,stroke:#333
    style S3 fill:#cfc,stroke:#333
    style S4 fill:#cfc,stroke:#333
    style S5 fill:#cfc,stroke:#333
    style KS1 fill:#f9f,stroke:#333
    style HL1 fill:#fc9,stroke:#333
    style MR1 fill:#fc9,stroke:#333

Prerequisites

Ingredients

Code

Reference implementations: Useful aws-samples and open-source patterns for this recipe:

• amazon-sagemaker-examples: SageMaker Pipelines, Model Registry, and Real-Time Inference patterns that mirror per-treatment-comparator model lifecycles.
• amazon-sagemaker-feature-store-end-to-end-workshop: End-to-end Feature Store usage applicable to the per-patient and per-treatment-history feature pipelines.
• amazon-bedrock-workshop: Demonstrates structured-output prompting applicable to clinician-facing comparison briefings.
• EconML: Microsoft Research's heterogeneous-treatment-effect estimation library, the methodological foundation for the CATE ensemble.
• DoWhy: Causal-inference framework for structuring assumptions and running sensitivity analyses.

Walkthrough

Step 1: Construct the cohort and outcome labels per treatment-comparator pair using target trial emulation. Cohort construction is the foundation of every downstream causal estimate. The target-trial protocol is specified explicitly: who is eligible, when the index date is, what the washout window excludes, what the treatment exposure is, what the comparator is, what the outcomes are with explicit timing, and what censoring rules apply. Skip the explicit protocol and you build a cohort with implicit decisions that bias the resulting estimates in ways nobody can audit.

FUNCTION construct_cohort(treatment_pair, run_date):
    // Load the target-trial protocol from the treatment catalog. The
    // protocol is a structured spec with eligibility, washout,
    // exposure, comparator, outcome, follow-up, and censoring
    // definitions. The protocol is versioned; cohorts are tagged
    // with the protocol version so any cohort can be reproduced.
    protocol = DynamoDB.GetItem(
        "treatment-comparison-pairs",
        key = (treatment_pair.treatment_id, treatment_pair.comparator_id)
    ).protocol

    // Step 1A: identify candidate patients. The candidate population
    // is the patients with the index condition active at the index
    // date, who are also eligible for both the treatment and the
    // comparator (so the comparison is between actually-substitutable
    // options).
    candidates = Athena.Query(
        protocol.candidate_query,
        params = { :run_date = run_date }
    )

    // Step 1B: apply the washout window. Patients with relevant prior
    // exposures (the same treatment, the comparator, or other
    // confounding treatments) within the washout window are excluded.
    // The washout is specified per-pair because the relevant prior
    // exposures depend on what is being compared.
    eligible = Athena.Query(
        protocol.washout_query,
        params = { :run_date = run_date,
                   :washout_start = run_date - protocol.washout_days,
                   :candidates = candidates }
    )

    // Step 1C: assign treatment exposure. For each eligible patient,
    // determine which treatment arm they fall into based on what
    // they were actually prescribed at the index date. Patients who
    // received neither the treatment nor the comparator are excluded
    // from this pair's cohort (they may belong to another pair's
    // cohort).
    cohort_treated = Athena.Query(
        protocol.treated_assignment_query,
        params = { :eligible_population = eligible }
    )
    cohort_comparator = Athena.Query(
        protocol.comparator_assignment_query,
        params = { :eligible_population = eligible }
    )
    cohort = union(cohort_treated, cohort_comparator)

    // Step 1D: construct outcome labels. For each cohort member,
    // compute the outcome at the protocol-specified timing. Multiple
    // outcomes per cohort member are supported (primary, secondary,
    // safety). Censoring rules handle patients lost to follow-up,
    // patients who switched treatments mid-follow-up, and competing
    // events (death, plan disenrollment).
    cohort_with_outcomes = []
    FOR each member in cohort:
        outcomes = {}
        FOR each outcome_def in protocol.outcomes:
            outcome_value = compute_outcome(member, outcome_def, protocol)
                // returns { value, observed, censored_at, censor_reason }
            outcomes[outcome_def.outcome_id] = outcome_value

        cohort_with_outcomes.append({
            patient_id:        member.patient_id,
            treatment_arm:     member.treatment_arm,
            index_date:        member.index_date,
            covariates:        member.covariates,
                // patient features at index date; this is the conditioning
                // set for the CATE estimators
            outcomes:          outcomes,
            protocol_version:  protocol.version
        })

    // Step 1E: persist the cohort. Cohort artifacts are partitioned
    // by treatment-pair and protocol version, archived alongside the
    // model artifacts for reproducibility, and never overwritten
    // (each retraining run produces a new versioned cohort).
    cohort_path = "s3://trx-cohorts/pair=" + treatment_pair.pair_id +
                  "/protocol_version=" + protocol.version +
                  "/run_date=" + run_date + "/cohort.parquet"
    write_parquet(cohort_with_outcomes, cohort_path)

    // Persist cohort metadata for traceability.
    DynamoDB.PutItem("cohort-metadata", {
        cohort_id:         build_cohort_id(treatment_pair, protocol.version, run_date),
        treatment_pair:    treatment_pair.pair_id,
        protocol_version:  protocol.version,
        run_date:          run_date,
        cohort_path:       cohort_path,
        size_treated:      count_arm(cohort_with_outcomes, "treated"),
        size_comparator:   count_arm(cohort_with_outcomes, "comparator"),
        outcome_definitions: list_outcomes(protocol),
        feature_set_version: protocol.feature_set_version
    })

    Kinesis.PutRecord(stream = "trx-events", record = {
        event_type:     "cohort_constructed",
        cohort_id:      build_cohort_id(treatment_pair, protocol.version, run_date),
        treatment_pair: treatment_pair.pair_id,
        size_treated:   count_arm(cohort_with_outcomes, "treated"),
        size_comparator: count_arm(cohort_with_outcomes, "comparator"),
        timestamp:      current UTC timestamp
    })

    RETURN cohort_with_outcomes

Step 2: Train the propensity, outcome, and CATE-ensemble models per treatment-comparator pair. The propensity model estimates the probability of receiving each treatment given covariates; this anchors the doubly-robust methods. The outcome model estimates outcome given covariates and treatment; this feeds the meta-learners. The CATE ensemble runs at least two estimators from different method families (causal forest, DR-learner, BART, or equivalent) so that disagreement among them surfaces model misspecification. Skip the ensemble and you cannot tell whether your point estimate is robust or a methodological artifact.

FUNCTION train_pair_models(cohort_id, run_date):
    cohort_metadata = DynamoDB.GetItem("cohort-metadata", cohort_id)
    cohort = read_parquet(cohort_metadata.cohort_path)
    treatment_pair = lookup_treatment_pair(cohort_metadata.treatment_pair)
    protocol = lookup_protocol(treatment_pair, cohort_metadata.protocol_version)

    // Stage A: propensity score model. Predicts P(treatment = treated |
    // covariates X). Trained on the full cohort with treatment_arm as
    // the binary label. Logistic regression with regularization is a
    // reasonable baseline; gradient-boosted trees often perform better
    // and are the practical default. Use cross-validation for
    // hyperparameter selection; check propensity-score overlap (the
    // distribution of propensity scores in the treated and comparator
    // arms should overlap substantially for the cohort to support
    // causal estimation).
    propensity_job = SageMaker.CreateTrainingJob(
        training_job_name = "propensity-" + treatment_pair.pair_id + "-" + run_date,
        algorithm         = "xgboost-with-isotonic-calibration",
        training_data     = cohort_metadata.cohort_path,
        target            = "treatment_arm_treated",
        feature_set       = protocol.feature_set_version,
        hyperparameters   = protocol.propensity_hyperparameters
    )
    propensity_model_arn = SageMaker.RegisterModel(propensity_job, "propensity")

    // Quick check: propensity overlap. If overlap is poor (many
    // treated patients have propensity > 0.95 or many comparator
    // patients have propensity < 0.05), the cohort does not support
    // reliable CATE estimation. Flag for governance review and
    // suspend further training for this pair until the cohort can
    // be re-scoped.
    overlap_check = assess_propensity_overlap(propensity_model_arn, cohort)
    IF overlap_check.severe:
        DynamoDB.UpdateItem("treatment-comparison-pairs",
            key = treatment_pair.pair_id,
            updates = {
                training_status: "suspended_propensity_overlap",
                last_overlap_check: overlap_check
            })
        Kinesis.PutRecord(stream = "trx-events", record = {
            event_type:     "training_suspended",
            cohort_id:      cohort_id,
            reason:         "propensity_overlap_severe",
            timestamp:      current UTC timestamp
        })
        RETURN { status: "suspended", reason: "propensity_overlap" }

    // Stage B: outcome model(s). Predicts E[Y | X, T] for each
    // outcome of interest. For the CATE ensemble that follows, we
    // train at least two specifications: a flexible regression (XGBoost
    // or random forest) and a simpler interpretable specification
    // (regularized linear) so that the CATE estimators can leverage
    // both. Trained on the full cohort with outcome as the label
    // and treatment as a feature.
    outcome_models = {}
    FOR each outcome_def in protocol.outcomes:
        outcome_job = SageMaker.CreateTrainingJob(
            training_job_name = "outcome-" + treatment_pair.pair_id +
                                 "-" + outcome_def.outcome_id + "-" + run_date,
            algorithm         = "xgboost-regression-or-classification",
            training_data     = cohort_metadata.cohort_path,
            target            = outcome_def.outcome_id,
            feature_set       = protocol.feature_set_version + "+treatment_arm",
            hyperparameters   = protocol.outcome_hyperparameters[outcome_def.outcome_id]
        )
        outcome_models[outcome_def.outcome_id] = SageMaker.RegisterModel(outcome_job, "outcome")

    // Stage C: CATE estimator ensemble. At least two methods from
    // different families. The Causal Forest provides built-in
    // confidence intervals and handles non-linear heterogeneity;
    // the DR-learner is doubly robust; BART (or equivalent Bayesian
    // method) provides posterior-based uncertainty.
    cate_estimators = []

    cf_job = SageMaker.CreateTrainingJob(
        training_job_name = "cate-cf-" + treatment_pair.pair_id + "-" + run_date,
        algorithm         = "causal-forest-econml",
            // BYOC container; econml CausalForestDML or grf-equivalent
        training_data     = cohort_metadata.cohort_path,
        propensity_model  = propensity_model_arn,
        outcome_model     = outcome_models["primary"],
        feature_set       = protocol.feature_set_version,
        hyperparameters   = protocol.cate_hyperparameters.causal_forest
    )
    cate_estimators.append(SageMaker.RegisterModel(cf_job, "cate_causal_forest"))

    dr_job = SageMaker.CreateTrainingJob(
        training_job_name = "cate-dr-" + treatment_pair.pair_id + "-" + run_date,
        algorithm         = "dr-learner-econml",
            // BYOC container; econml DRLearner
        training_data     = cohort_metadata.cohort_path,
        propensity_model  = propensity_model_arn,
        outcome_model     = outcome_models["primary"],
        feature_set       = protocol.feature_set_version,
        hyperparameters   = protocol.cate_hyperparameters.dr_learner
    )
    cate_estimators.append(SageMaker.RegisterModel(dr_job, "cate_dr_learner"))

    bart_job = SageMaker.CreateTrainingJob(
        training_job_name = "cate-bart-" + treatment_pair.pair_id + "-" + run_date,
        algorithm         = "bcf-bart",
            // BYOC container; bcf or bartCause R package via wrapper
        training_data     = cohort_metadata.cohort_path,
        feature_set       = protocol.feature_set_version,
        hyperparameters   = protocol.cate_hyperparameters.bart
    )
    cate_estimators.append(SageMaker.RegisterModel(bart_job, "cate_bart"))

    DynamoDB.UpdateItem("treatment-comparison-pairs",
        key = treatment_pair.pair_id,
        updates = {
            training_status:        "trained_pending_evaluation",
            propensity_model_arn:   propensity_model_arn,
            outcome_model_arns:     outcome_models,
            cate_estimator_arns:    cate_estimators,
            cohort_id:              cohort_id,
            run_date:               run_date
        })

    RETURN {
        status:               "trained_pending_evaluation",
        propensity_model_arn: propensity_model_arn,
        outcome_model_arns:   outcome_models,
        cate_estimator_arns:  cate_estimators
    }

Step 3: Run the calibration and fairness tests, and gate model promotion through governance. A trained model is not a production model. Calibration tests check that predicted treatment effects match observed effects in held-out cohorts and in subgroups. Fairness tests check that calibration is consistent across protected subpopulations. The governance gate is a human review of the test results before the new model artifacts are promoted to production. Skip this and you ship a model with unknown calibration on the populations it will most affect.

FUNCTION evaluate_and_gate_pair_models(treatment_pair_id, run_date):
    pair = DynamoDB.GetItem("treatment-comparison-pairs", treatment_pair_id)
    cohort = read_parquet(lookup_cohort_path(pair.cohort_id))

    // Stage A: held-out calibration. The cohort is split into a
    // training set and a held-out test set (typically temporally,
    // so the test set is more recent than the training set). Train
    // on the training set, predict on the test set, and compare
    // predicted treatment effects to observed effects within
    // covariate-defined subgroups. The calibration plot per
    // estimator and per outcome is the primary evidence.
    calibration_results = {}
    FOR each estimator in pair.cate_estimator_arns:
        predicted_effects = SageMaker.BatchTransform(
            model_arn   = estimator.arn,
            input_path  = cohort.held_out_path,
            output_path = "s3://trx-eval/" + treatment_pair_id +
                          "/calibration/" + estimator.name + "/"
        )
        // Compute calibration in groups of similar predicted effect.
        // Within each group, the average predicted effect should be
        // close to the average observed effect.
        cal = compute_calibration_in_groups(predicted_effects, cohort.held_out)
        calibration_results[estimator.name] = cal
        IF cal.calibration_slope < 0.7 OR cal.calibration_slope > 1.3:
            calibration_results[estimator.name].calibration_warning = true

    // Stage B: cohort-stratified fairness. Repeat the calibration
    // within each protected cohort (defined by demographic and SDOH
    // axes from the treatment-catalog fairness specification).
    // Calibration parity across cohorts is the bar: the model should
    // be similarly accurate for each cohort.
    fairness_results = {}
    FOR each cohort_axis in CATALOG.fairness_axes[treatment_pair_id]:
        FOR each cohort_value in cohort_axis.values:
            cohort_subset = filter(cohort.held_out,
                                    cohort_features[cohort_axis.name] == cohort_value)
            IF len(cohort_subset) < MIN_FAIRNESS_SAMPLE_SIZE:
                fairness_results[(cohort_axis.name, cohort_value)] = {
                    status: "insufficient_sample"
                }
                CONTINUE
            cohort_predicted = predicted_effects.filter(
                patient_id IN cohort_subset.patient_ids)
            cohort_calibration = compute_calibration_in_groups(
                cohort_predicted, cohort_subset)
            fairness_results[(cohort_axis.name, cohort_value)] = cohort_calibration

    // Stage C: estimator agreement. The CATE estimators should
    // produce similar estimates on the same held-out patients.
    // Disagreement at the patient level is a signal of model
    // misspecification or unmeasured confounding.
    agreement_results = compute_estimator_agreement(
        pair.cate_estimator_arns, cohort.held_out)
        // returns per-patient correlation, percent-of-patients with
        // estimator-spread above threshold, and a global agreement
        // score per outcome

    // Stage D: sensitivity analysis. How strong would unmeasured
    // confounding need to be to overturn the conclusion? E-values
    // (VanderWeele and Ding) and Rosenbaum bounds are the standard
    // tools. The result is a structural-uncertainty bound that is
    // reported alongside the model's statistical confidence intervals.
    sensitivity_results = run_sensitivity_analyses(
        pair.cate_estimator_arns, cohort, protocol = lookup_protocol(pair))

    // Stage E: persist the evaluation report.
    eval_report = {
        treatment_pair_id:    treatment_pair_id,
        run_date:             run_date,
        calibration_results:  calibration_results,
        fairness_results:     fairness_results,
        agreement_results:    agreement_results,
        sensitivity_results:  sensitivity_results,
        evaluation_status:    classify_overall(calibration_results, fairness_results,
                                                 agreement_results, sensitivity_results)
            // values: 'green' (clear pass), 'yellow' (warnings, governance review
            // required), 'red' (clear fail, suspend)
    }
    write_json(eval_report,
               "s3://trx-eval/" + treatment_pair_id + "/run=" + run_date + "/report.json")

    // Stage F: governance gate. A human review committee evaluates
    // the report. The system creates a review task; the committee's
    // decision is recorded; only on approval are the new model
    // artifacts promoted to production.
    DynamoDB.PutItem("governance-review-tasks", {
        task_id:               new UUID,
        treatment_pair_id:     treatment_pair_id,
        run_date:              run_date,
        evaluation_report_path: "s3://trx-eval/" + treatment_pair_id +
                                 "/run=" + run_date + "/report.json",
        evaluation_status:     eval_report.evaluation_status,
        reviewer_assignments:  CATALOG.governance_reviewers[treatment_pair_id],
        status:                "pending_review",
        created_at:            run_date,
        sla_review_by:         run_date + GOVERNANCE.review_sla_days
    })

    Kinesis.PutRecord(stream = "trx-events", record = {
        event_type:           "governance_review_pending",
        treatment_pair_id:    treatment_pair_id,
        evaluation_status:    eval_report.evaluation_status,
        timestamp:            current UTC timestamp
    })

    // The governance review's decision is processed in a separate
    // Lambda triggered by a review-decision event; on approval, the
    // promotion logic moves the new artifacts to the production
    // alias in the model registry and the production pointers in
    // the treatment-comparison-pairs table. On rejection, the
    // training-status is set to evaluation-failed and the team
    // investigates.

FUNCTION sweep_pending_governance_tasks(run_date):
    // Runs daily via EventBridge. Enforces SLA escalation so model
    // artifacts do not sit in pending_review indefinitely while
    // drifted prior models continue to serve.
    //
    // SLA policy (per catalog model-risk tier):
    //   Tier 1 (high-risk SaMD pathway):  14 days review SLA
    //   Tier 2 and above (lower-risk):    28 days review SLA
    //
    // Escalation progression:
    //   75% of SLA elapsed:  notification to assigned reviewers
    //   100% of SLA elapsed: escalation to governance-committee chair
    //                         plus clinical informatics lead
    //   First SLA expiry +7: default-deferral (status = "deferred",
    //                         pair continues serving prior model)
    //   Second SLA expiry:   default-retirement (pair status =
    //                         "retired_governance_timeout"; pair stops
    //                         serving; never auto-promote)
    //
    // The system never auto-promotes; unreviewed models either get
    // human approval or the pair degrades gracefully.

    pending_tasks = DynamoDB.Query("governance-review-tasks",
        filter = "status = :s",
        params = { :s = "pending_review" })

    FOR each task in pending_tasks:
        pair = DynamoDB.GetItem("treatment-comparison-pairs", task.treatment_pair_id)
        sla_days = 14 IF pair.model_risk_tier == "tier_1" ELSE 28
        elapsed_days = days_between(task.created_at, run_date)
        elapsed_fraction = elapsed_days / sla_days

        IF elapsed_fraction >= 2.0:
            // Second SLA expiry: retire the pair. Never auto-promote.
            DynamoDB.UpdateItem("treatment-comparison-pairs",
                key = task.treatment_pair_id,
                updates = { training_status: "retired_governance_timeout" })
            DynamoDB.UpdateItem("governance-review-tasks",
                key = task.task_id,
                updates = { status: "retired_timeout" })
            Kinesis.PutRecord(stream = "trx-events", record = {
                event_type:        "governance_pair_retired_timeout",
                treatment_pair_id: task.treatment_pair_id,
                elapsed_days:      elapsed_days,
                timestamp:         current UTC timestamp
            })
        ELSE IF elapsed_fraction >= 1.0 + (7.0 / sla_days):
            // First SLA expiry +7 days: default-deferral.
            DynamoDB.UpdateItem("governance-review-tasks",
                key = task.task_id,
                updates = { status: "deferred" })
            Kinesis.PutRecord(stream = "trx-events", record = {
                event_type:        "governance_review_deferred",
                treatment_pair_id: task.treatment_pair_id,
                elapsed_days:      elapsed_days,
                timestamp:         current UTC timestamp
            })
        ELSE IF elapsed_fraction >= 1.0:
            // 100% SLA: escalate to committee chair.
            notify(CATALOG.governance_escalation_contacts[task.treatment_pair_id],
                   "governance_sla_expired", task)
        ELSE IF elapsed_fraction >= 0.75:
            // 75% SLA: reminder to assigned reviewers.
            notify(task.reviewer_assignments, "governance_sla_warning", task)

    // Equity instrumentation: per-cohort review latency. If reviews
    // for pairs affecting certain demographic cohorts take
    // systematically longer, that latency is itself a fairness signal.
    CloudWatch.PutMetricData(
        Namespace = "TrxGovernance",
        MetricData = compute_review_latency_by_cohort(pending_tasks)
    )

Curious how this looks in Python? The pseudocode above covers the concepts. If you'd like to see sample Python code that demonstrates these patterns using boto3, check out the Python Example. It walks through each step with inline comments and notes on what you'd need to change for a real deployment.

Step 4: Score an index patient on demand at the point of care, retrieve the similar-patient cohort, and compute uncertainty. When the clinician opens the patient's chart and requests treatment guidance, the system identifies eligible treatment-comparator pairs from the catalog, invokes each pair's CATE ensemble, retrieves the similar-patient cohort underlying the estimate, computes uncertainty across all sources, and flags out-of-distribution cases. Skip the OOD flag and the system silently extrapolates predictions to patients who are not represented in the training data, which is exactly the case where the prediction is least reliable.

FUNCTION score_patient(patient_id, request_context):
    // request_context includes the requesting clinician, the
    // index condition, the visit context, and any clinician-supplied
    // constraints (formulary preference, contraindication overrides).
    patient_features = SageMaker.FeatureStore.GetRecord(
        feature_group_name = "patient-features-online",
        record_identifier  = patient_id
    )

    // Step 4A: identify eligible treatment-comparator pairs. The
    // treatment catalog determines which pairs are in scope for this
    // patient given the index condition, the patient's covariates,
    // and any contraindications. Pairs whose model is not in the
    // production state are excluded.
    eligible_pairs = []
    FOR each pair in CATALOG.list_pairs_for_condition(request_context.index_condition):
        IF NOT pair.is_production:
            CONTINUE
        IF NOT meets_pair_eligibility(patient_features, pair):
            CONTINUE
        IF has_contraindication(patient_features, pair):
            CONTINUE
        eligible_pairs.append(pair)

    IF len(eligible_pairs) == 0:
        // No eligible pairs. Return a structured response that tells
        // the clinician we have nothing to add for this patient. Do
        // not silently produce a low-confidence estimate.
        RETURN {
            patient_id:        patient_id,
            scoring_run_id:    new UUID,
            scoring_status:    "no_eligible_pairs",
            scoring_reason:    diagnose_no_eligibility(patient_features,
                                                       request_context.index_condition),
            timestamp:         current UTC timestamp
        }

    // Step 4B: per-pair scoring. For each eligible pair, invoke the
    // CATE ensemble (one inference per estimator), retrieve the
    // similar-patient cohort, compute uncertainty, and flag OOD.
    pair_results = []
    FOR each pair in eligible_pairs:
        // Run each estimator. The Real-Time endpoints return the
        // point estimate and the estimator-specific confidence interval.
        estimator_outputs = {}
        FOR each estimator in pair.cate_estimator_arns:
            estimator_response = SageMaker.InvokeEndpoint(
                endpoint_name = estimator.endpoint,
                input         = patient_features.serialize_for(estimator.feature_set)
            )
            estimator_outputs[estimator.name] = parse_json(estimator_response)
                // { point_estimate, ci_low, ci_high, internal_metadata }

        // Step 4C: ensemble uncertainty. Combine the per-estimator
        // estimates and CIs into ensemble uncertainty: the union of
        // the estimator CIs is the sampling-plus-model uncertainty.
        // Disagreement among estimators (the spread of point estimates)
        // is a structural-uncertainty signal.
        ensemble_estimate = combine_ensemble_estimates(estimator_outputs)
            // { point_estimate, ci_low, ci_high,
            //   estimator_agreement_score, disagreement_flag }

        // Step 4D: similar-patient cohort retrieval. The cohort
        // underlying this estimate is the patients in the training
        // cohort with covariates similar to the index patient. The
        // retrieval uses the patient embedding (or hand-engineered
        // similarity metric) defined per pair. The retrieved cohort
        // is summarized (size, demographic match, average observed
        // outcome by arm) rather than returned in full; full-cohort
        // retrieval would be PHI-leaking.
        cohort_summary = retrieve_similar_patient_summary(
            patient_features, pair, k = SIMILAR_PATIENT_K)
            // {
            //   total_size, treated_size, comparator_size,
            //   demographic_match_score, condition_match_score,
            //   average_treated_outcome, average_comparator_outcome,
            //   treated_outcome_distribution_summary,
            //   comparator_outcome_distribution_summary,
            //   training_data_recency
            // }

        // Step 4E: out-of-distribution flag. If the patient is not
        // well-represented in the training cohort (low density of
        // similar patients, propensity score in the tails, large
        // extrapolation distance from the training distribution),
        // flag the estimate as out-of-distribution. OOD-flagged
        // estimates are presented with explicit warnings or
        // suppressed depending on policy.
        ood_flag = compute_ood_flag(patient_features, pair, cohort_summary)
            // { is_ood, severity, reasons }
            //
            // OOD severity bands (policy):
            //   severity < 0.50:  present normally with standard
            //     uncertainty caveats. The patient is within the
            //     training distribution; confidence intervals reflect
            //     the statistical uncertainty of the estimate.
            //   0.50 <= severity < 0.85:  present with explicit OOD
            //     warning text. The briefing must include: "This
            //     estimate is based on a limited number of similar
            //     patients. Treat with additional clinical skepticism."
            //     The validator's uncertainty-completeness layer
            //     enforces the warning presence.
            //   severity >= 0.85:  suppress the pair. Mark
            //     scoring_status = "suppressed_oodflag", exclude from
            //     the briefing's numeric comparison, render an
            //     "estimate suppressed: insufficient similar patients"
            //     line in the briefing. Suppression is the chapter's
            //     most defensive clinical-safety primitive: it is
            //     better to say nothing than to present an estimate
            //     the training data cannot support.
            //   Per-pair catalog overrides: allowed when clinically
            //     warranted (e.g., a pair with strong RCT backing
            //     may tolerate higher OOD severity because the
            //     external evidence backstops the estimate). Overrides
            //     require governance-committee approval and are
            //     logged as catalog change events.

        // Apply OOD suppression policy before appending to results.
        IF ood_flag.severity >= 0.85 AND NOT pair.ood_override_approved:
            pair_results.append({
                treatment_pair_id:    pair.pair_id,
                treatment_id:         pair.treatment_id,
                comparator_id:        pair.comparator_id,
                scoring_status:       "suppressed_oodflag",
                ood_flag:             ood_flag,
                suppression_reason:   "severity " + str(ood_flag.severity) +
                                      " exceeds 0.85 threshold"
            })
            Kinesis.PutRecord(stream = "trx-events", record = {
                event_type:        "treatment_scoring_oodflag_triggered",
                patient_id:        patient_id,
                treatment_pair_id: pair.pair_id,
                ood_severity:      ood_flag.severity,
                action:            "suppressed",
                timestamp:         current UTC timestamp
            })
            CONTINUE  // skip the rest of this pair

        // Step 4F: sensitivity-analysis bounds. The pre-computed
        // sensitivity results from Step 3 bound how much unmeasured
        // confounding could change the conclusion. Apply them to
        // the index patient's estimate to widen the reported CI
        // appropriately.
        adjusted_estimate = apply_sensitivity_bounds(
            ensemble_estimate, pair.sensitivity_results)

        pair_results.append({
            treatment_pair_id:    pair.pair_id,
            treatment_id:         pair.treatment_id,
            comparator_id:        pair.comparator_id,
            outcome_id:           pair.primary_outcome_id,
            point_estimate:       adjusted_estimate.point_estimate,
            ci_low:               adjusted_estimate.ci_low,
            ci_high:              adjusted_estimate.ci_high,
            estimator_agreement:  ensemble_estimate.estimator_agreement_score,
            disagreement_flag:    ensemble_estimate.disagreement_flag,
            cohort_summary:       cohort_summary,
            ood_flag:             ood_flag,
            evidence_level:       pair.evidence_level,
            formulary_status:     pair.formulary_status,
            guideline_references: pair.guideline_references,
            calibration_status:   pair.production_calibration_status
        })

    // Step 4G: persist scoring result.
    scoring_run_id = new UUID
    scoring_result = {
        patient_id:           patient_id,
        scoring_run_id:       scoring_run_id,
        request_context:      request_context,
        index_condition:      request_context.index_condition,
        eligible_pairs:       [pair.pair_id FOR pair in eligible_pairs],
        pair_results:         pair_results,
        scoring_status:       "completed",
        scoring_completed_at: current UTC timestamp
    }
    DynamoDB.PutItem("scoring-results", scoring_result)

    Kinesis.PutRecord(stream = "trx-events", record = {
        event_type:        "treatment_scoring_completed",
        patient_id:        patient_id,
        scoring_run_id:    scoring_run_id,
        eligible_pair_count: len(eligible_pairs),
        any_ood_flag:      any(p.ood_flag.is_ood FOR p in pair_results),
        any_disagreement: any(p.disagreement_flag FOR p in pair_results),
        timestamp:         current UTC timestamp
    })

    RETURN scoring_result

Step 5: Generate the clinician-facing comparison briefing with strict validator enforcement. The structured scoring result is rendered into a paragraph the clinician reads at the point of care. The LLM packages the comparison; the validator enforces strict no-recommendation language, explicit uncertainty, and required caveats. Skip the validator and an LLM that has been trained on the broader internet may quietly insert recommendation language ("the evidence supports prescribing X") that the clinician interprets as the system selecting a treatment.

FUNCTION generate_briefing(scoring_run_id):
    scoring = DynamoDB.GetItem("scoring-results", scoring_run_id)

    // Build the structured briefing context. Every fact in the
    // briefing must come from the scoring result; the LLM does not
    // introduce clinical claims not present in the structured data.
    briefing_context = {
        patient_summary:      build_clinical_summary(scoring.patient_id),
            // de-identified for the LLM call: condition, key labs,
            // current medications, key trajectory features
        index_condition:      scoring.index_condition,
        pair_results:         scoring.pair_results,
        request_context:      scoring.request_context,
        any_ood_flag:         any(p.ood_flag.is_ood FOR p in scoring.pair_results),
        any_disagreement:     any(p.disagreement_flag FOR p in scoring.pair_results),
        scoring_run_id:       scoring_run_id
    }

    briefing = Bedrock.InvokeModel(
        model_id = COMPARISON_BRIEFING_MODEL_ID,
        body     = build_briefing_prompt(briefing_context, BRIEFING_OUTPUT_SCHEMA)
    )
    briefing_parsed = parse_json(briefing.completion)
        // { headline, comparison_paragraph, per_treatment_summary,
        //   uncertainty_summary, caveats, suggested_clinician_review_points }

    // Validator. Strict enforcement of the no-recommendation rule
    // and the explicit-uncertainty rule. The validator is the
    // last line of defense before the briefing reaches the
    // clinician.
    validation_result = validate_briefing(briefing_parsed, observed_context = briefing_context)
        // Four-layer validator specification (shared pattern with
        // Recipes 4.5 through 4.7, extended here with
        // recommendation-language and uncertainty-completeness
        // layers specific to treatment response prediction):
        //
        // Layer 1 - Schema and length:
        //   Required fields present (headline, comparison_paragraph,
        //   per_treatment_summary, uncertainty_summary, caveats).
        //   No field exceeds 2000 characters. Every required caveat
        //   (observational-data, conditional-average, clinician-as-
        //   decision-maker) present in the caveats array.
        //
        // Layer 2 - Recommendation-language detection:
        //   Strict pattern matching against 11+ aggressive
        //   recommendation patterns:
        //     "should prescribe", "is the best choice",
        //     "recommended treatment", "we recommend",
        //     "the preferred option", "clearly superior",
        //     "patients should receive", "evidence supports
        //     prescribing", "optimal choice", "first-line
        //     recommendation", "treatment of choice"
        //   Plus contextual patterns that imply selection even
        //   without explicit recommendation verbs (e.g., "given
        //   this patient's profile, [treatment X]" without a
        //   comparative framing). Any match: regenerate with
        //   feedback on the first attempt, regenerate in strict
        //   mode on the second attempt, fall back to templated
        //   on the third.
        //
        // Layer 3 - Fact grounding:
        //   Every numeric value cited in the briefing text must
        //   trace byte-for-byte to a field in
        //   observed_context.pair_results. The validator parses
        //   all numeric tokens from the generated text and
        //   verifies each appears in the structured scoring data.
        //   Fabricated statistics, hallucinated cohort sizes, or
        //   invented confidence intervals trigger immediate
        //   regeneration without a second chance (the fact-grounding
        //   violation is more dangerous than a recommendation-
        //   language violation because it is harder for the
        //   clinician to catch).
        //
        // Layer 4 - Uncertainty completeness:
        //   Every cited point estimate must be accompanied by its
        //   confidence interval in the same sentence or adjacent
        //   clause. Every pair with an OOD warning (severity >= 0.50)
        //   must have the OOD warning surfaced explicitly in the
        //   briefing body. Every pair with a disagreement flag must
        //   have the disagreement surfaced. When any pair in the
        //   result set has been suppressed (severity >= 0.85), the
        //   briefing must include an "estimate suppressed" line for
        //   that pair rather than silently omitting it.
        //
        // Layer 5 - Required caveats:
        //   The caveats array must include: (a) observational-data
        //   limitations, (b) the conditional-average nature of the
        //   estimates (not individual guarantees), (c) the
        //   clinician's role as decision-maker, (d) patient
        //   preferences not modeled. Missing caveats trigger
        //   regeneration with the specific missing caveat listed
        //   in validator feedback.
        //
        // Failure-handling progression:
        //   Attempt 1: regenerate with validator feedback appended
        //     to the prompt (the LLM sees what failed and why).
        //   Attempt 2: regenerate in strict_mode (template-guided
        //     generation with field-by-field constraints).
        //   Attempt 3: fall back to render_templated_briefing(),
        //     which is deterministic and always passes validation.
        //     Emit briefing_validator_fallback event for monitoring.

    IF NOT validation_result.passed:
        IF validation_result.failure_count < MAX_REGENERATION_ATTEMPTS:
            // Regenerate with stricter guidance.
            briefing_context.validator_feedback = validation_result.feedback
            briefing = Bedrock.InvokeModel(
                model_id = COMPARISON_BRIEFING_MODEL_ID,
                body     = build_briefing_prompt(briefing_context,
                                                  BRIEFING_OUTPUT_SCHEMA,
                                                  strict_mode = true)
            )
            briefing_parsed = parse_json(briefing.completion)
            validation_result = validate_briefing(briefing_parsed,
                                                    observed_context = briefing_context)

        IF NOT validation_result.passed:
            // Fall back to a templated briefing that lists the
            // structured comparison without LLM narration. The
            // templated briefing is deterministic and always
            // passes validation; it is less readable but never
            // crosses into recommendation territory.
            briefing_parsed = render_templated_briefing(briefing_context)
            briefing_parsed.fallback_reason = validation_result.failure_summary

            Kinesis.PutRecord(stream = "trx-events", record = {
                event_type:        "briefing_validator_fallback",
                scoring_run_id:    scoring_run_id,
                reason:            validation_result.failure_summary,
                timestamp:         current UTC timestamp
            })

    DynamoDB.PutItem("briefings", {
        briefing_id:       new UUID,
        scoring_run_id:    scoring_run_id,
        patient_id:        scoring.patient_id,
        briefing_text:     briefing_parsed,
        briefing_context:  briefing_context,
        validator_status:  validation_result.passed,
        generated_at:      current UTC timestamp,
        ttl:               compute_ttl(scoring.request_context)
            // Briefings have a TTL: a briefing generated for a Tuesday
            // visit should not be presented unchanged at a Friday
            // visit. The clinician interface re-generates if the
            // briefing is stale.
    })

    RETURN briefing_parsed

Step 6: Capture the clinician's decision and the patient's subsequent outcome, and feed the matched pair back into surveillance and retraining. The decision record links the prediction at the time of decision to the actual treatment chosen and (later) the actual outcome. The matched prediction-outcome pair is the feedback that drives calibration drift detection, cohort-stratified performance monitoring, adverse-event surveillance, and retraining triggers. Skip this and you have a prediction system with no feedback loop, which is the single most common reason production ML systems quietly degrade over time.

FUNCTION record_decision(scoring_run_id, decision_payload):
    // decision_payload includes:
    //   - clinician_id (from the authenticated session)
    //   - chosen_treatment_id (or 'none' if no treatment selected)
    //   - clinician_rationale (free-text or structured)
    //   - patient_consent_recorded (boolean)
    //   - shared_decision_indicators (whether the patient summary
    //     was shared with the patient; whether the patient's
    //     stated preference influenced the decision)
    scoring = DynamoDB.GetItem("scoring-results", scoring_run_id)
    briefing = lookup_latest_briefing(scoring_run_id)

    // Identity-boundary checks. The artifact being mutated is a
    // clinical-decision audit record; misrouted decisions
    // contaminate the audit trail and break the Cures Act CDS
    // exemption argument.

    // Check 1: clinician has a treatment relationship to the patient.
    IF NOT validate_treatment_relationship(
            decision_payload.clinician_id, scoring.patient_id):
        CloudWatch.PutMetricData("TrxDecisionBoundary",
            "decision_authorization_violation", 1)
        RETURN { status: "rejected",
                 reason: "clinician_not_authorized_for_patient" }

    // Check 2: chosen treatment must be in the scoring run's
    // eligible pairs. The clinician cannot record a decision for
    // a treatment the model did not score.
    IF decision_payload.chosen_treatment_id != "none":
        scored_treatment_ids = extract_treatment_ids(scoring.pair_results)
        IF decision_payload.chosen_treatment_id NOT IN scored_treatment_ids:
            CloudWatch.PutMetricData("TrxDecisionBoundary",
                "decision_treatment_mismatch", 1)
            RETURN { status: "rejected",
                     reason: "chosen_treatment_not_in_scored_pairs" }

    // Check 3: scoring_run_id consistency when the payload carries
    // its own reference.
    IF decision_payload.scoring_run_id IS NOT NULL
       AND decision_payload.scoring_run_id != scoring_run_id:
        CloudWatch.PutMetricData("TrxDecisionBoundary",
            "decision_scoring_run_mismatch", 1)
        RETURN { status: "rejected",
                 reason: "scoring_run_id_mismatch" }

    decision = {
        decision_id:           new UUID,
        scoring_run_id:        scoring_run_id,
        patient_id:            scoring.patient_id,
        clinician_id:          decision_payload.clinician_id,
        chosen_treatment_id:   decision_payload.chosen_treatment_id,
        clinician_rationale:   decision_payload.clinician_rationale,
        patient_consent:       decision_payload.patient_consent_recorded,
        shared_decision:       decision_payload.shared_decision_indicators,
        // Frozen-at-decision-time view of the predictions: this is
        // what the clinician saw when deciding. Critical for audit
        // and for after-the-fact analysis of predictions versus
        // decisions.
        predictions_at_decision: scoring.pair_results,
        briefing_id_at_decision: briefing.briefing_id,
        decision_recorded_at:    current UTC timestamp
    }

    // Determine whether the decision agrees with the model's
    // best-effect estimate (for monitoring, not for judgment).
    decision.agrees_with_best_effect_estimate = compute_agreement(
        decision.chosen_treatment_id, scoring.pair_results)

    DynamoDB.PutItem("decision-records", decision)

    Kinesis.PutRecord(stream = "trx-events", record = {
        event_type:                  "treatment_decision_recorded",
        patient_id:                  scoring.patient_id,
        scoring_run_id:              scoring_run_id,
        decision_id:                 decision.decision_id,
        chosen_treatment_id:         decision.chosen_treatment_id,
        agrees_with_best_effect:     decision.agrees_with_best_effect_estimate,
        timestamp:                   current UTC timestamp
    })

FUNCTION match_outcome(decision_id, run_date):
    // Outcome matching runs in batch on a cadence aligned with each
    // pair's primary outcome timing (e.g., 90-day matching for the
    // 90-day A1c outcome). For each decision recorded N days ago,
    // pull the patient's actual outcome at that timing, match it
    // against the prediction at decision time, and persist the
    // prediction-outcome pair.
    decision = DynamoDB.GetItem("decision-records", decision_id)

    // Identity-boundary checks (mirroring record_decision):
    // no duplicate matching, timing-window consistency, no
    // superseded decision versions.
    existing_match = DynamoDB.Query("prediction-outcome-pairs",
        filter = "decision_id = :d AND outcome_status = :o",
        params = { :d = decision_id, :o = "observed" })
    IF len(existing_match) > 0:
        CloudWatch.PutMetricData("TrxOutcomeBoundary",
            "duplicate_outcome_match_rejected", 1)
        RETURN { status: "rejected", reason: "already_matched" }

    // chosen_treatment_id can correspond to multiple pairs in a
    // single scoring run (e.g., GLP-1 was scored against SGLT2,
    // sulfonylurea, and basal insulin in three separate pairs).
    // The methodologically-correct approach is to iterate over all
    // pairs for the chosen treatment and persist multiple
    // prediction-outcome rows per decision; the calibration analysis
    // accounts for the per-patient repeated-measures structure.
    eligible_pairs = lookup_all_pairs_for_treatment(
        decision.chosen_treatment_id, decision.predictions_at_decision)

    IF len(eligible_pairs) == 0:
        RETURN { status: "no_pair_found" }

    // Hoist cohort features outside the per-pair, per-outcome loop.
    // With multiple pairs and multiple outcomes per pair, redundant
    // lookups multiply; compute once per patient.
    cohort_features = lookup_cohort_features(decision.patient_id)

    // Iterate over all pairs the chosen treatment was scored against,
    // and for each pair iterate over all outcomes (primary, secondary,
    // and safety). Without this, surveillance only catches drift on
    // the primary outcome and misses cardiovascular, weight, and
    // adverse-event signals.
    FOR each pair in eligible_pairs:
        protocol = lookup_protocol(pair)

        // Validate run_date is consistent with the protocol's
        // outcome timing window for at least one outcome.
        IF NOT any_outcome_in_timing_window(protocol, decision.decision_recorded_at, run_date):
            CONTINUE

        FOR each outcome_def in protocol.outcomes:
            // Check timing window for this specific outcome.
            IF NOT outcome_in_timing_window(outcome_def, decision.decision_recorded_at, run_date):
                CONTINUE

            actual_outcome = compute_actual_outcome(
                patient_id   = decision.patient_id,
                outcome_def  = outcome_def,
                index_date   = decision.decision_recorded_at,
                as_of_date   = run_date)

            IF NOT actual_outcome.observed:
                DynamoDB.PutItem("prediction-outcome-pairs", {
                    pair_id:           new UUID,
                    decision_id:       decision_id,
                    patient_id:        decision.patient_id,
                    scoring_run_id:    decision.scoring_run_id,
                    chosen_pair_id:    pair.pair_id,
                    outcome_id:        outcome_def.outcome_id,
                    outcome_status:    "censored",
                    censor_reason:     actual_outcome.censor_reason,
                    recorded_at:       run_date
                })
                CONTINUE

            // Find the prediction at decision time for this pair.
            chosen_prediction = find_prediction_for_pair(
                decision.predictions_at_decision, pair.pair_id)

            // The predicted value is a CATE: E[Y(treatment) -
            // Y(comparator) | X], a treatment-effect *difference*.
            // The actual outcome is the single-arm observed value
            // Y(treatment_chosen). These are not directly comparable
            // because the counterfactual Y(comparator) is unobserved.
            // Persist the fields with unambiguous names; downstream
            // calibration uses IPTW-weighted aggregate CATE
            // re-estimation rather than naive predicted-vs-observed
            // slope.
            DynamoDB.PutItem("prediction-outcome-pairs", {
                pair_id:                    new UUID,
                decision_id:                decision_id,
                patient_id:                 decision.patient_id,
                scoring_run_id:             decision.scoring_run_id,
                chosen_treatment_id:        decision.chosen_treatment_id,
                chosen_pair_id:             pair.pair_id,
                outcome_id:                 outcome_def.outcome_id,
                predicted_treatment_effect: chosen_prediction.point_estimate,
                predicted_effect_ci:        [chosen_prediction.ci_low,
                                             chosen_prediction.ci_high],
                observed_single_arm_outcome: actual_outcome.value,
                outcome_status:             "observed",
                ood_flag_at_decision:       chosen_prediction.ood_flag,
                cohort_features:            cohort_features,
                recorded_at:                run_date
            })

            Kinesis.PutRecord(stream = "trx-events", record = {
                event_type:        "treatment_outcome_observed",
                patient_id:        decision.patient_id,
                decision_id:       decision_id,
                chosen_pair_id:    pair.pair_id,
                outcome_id:        outcome_def.outcome_id,
                predicted_effect:  chosen_prediction.point_estimate,
                observed_outcome:  actual_outcome.value,
                timestamp:         current UTC timestamp
            })

FUNCTION run_calibration_drift_detection(run_date):
    // Aggregate the prediction-outcome pairs accumulated since the
    // last surveillance run. For each treatment-comparator pair,
    // compute the calibration on observed outcomes. Compare to the
    // calibration computed at model promotion. Significant drift
    // triggers an alert that a human reviews.
    FOR each pair in CATALOG.list_production_pairs():
        recent_pairs = DynamoDB.Query(
            "prediction-outcome-pairs",
            filter = "chosen_pair_id = :p AND outcome_status = :o AND recorded_at >= :since",
            params = { :p = pair.pair_id,
                       :o = "observed",
                       :since = run_date - SURVEILLANCE_WINDOW_DAYS }
        )
        IF len(recent_pairs) < MIN_DRIFT_DETECTION_SAMPLE:
            CONTINUE

        current_calibration = compute_calibration_in_groups(recent_pairs)
        baseline_calibration = pair.production_calibration_status

        drift_signal = compare_calibrations(current_calibration, baseline_calibration)
            // { calibration_slope_delta, calibration_intercept_delta,
            //   per_cohort_drift, drift_severity }

        IF drift_signal.drift_severity >= DRIFT_ALERT_THRESHOLD:
            DynamoDB.PutItem("surveillance-alerts", {
                alert_id:           new UUID,
                alert_type:         "calibration_drift",
                treatment_pair_id:  pair.pair_id,
                drift_signal:       drift_signal,
                triggered_at:       run_date,
                review_status:      "pending"
            })

            Kinesis.PutRecord(stream = "trx-events", record = {
                event_type:        "prediction_calibration_alert",
                treatment_pair_id: pair.pair_id,
                drift_severity:    drift_signal.drift_severity,
                timestamp:         current UTC timestamp
            })

        // Cohort-stratified fairness drift. The per-cohort
        // calibration drift is the equity-relevant signal; even if
        // overall calibration is stable, cohort-specific drift may
        // be widening disparities.
        //
        // Surveillance threshold policy (defaults):
        //   DRIFT_ALERT_THRESHOLD = 0.15 (overall calibration slope
        //     or intercept change from baseline that triggers alert)
        //   COHORT_DRIFT_ALERT_THRESHOLD = 0.10 (tighter than overall
        //     because disparate impact can hide in aggregate stability;
        //     a cohort drifting at 0.10 while the whole population
        //     holds at 0.05 means the cohort is absorbing the
        //     degradation that the overall metric masks)
        //   MIN_DRIFT_DETECTION_SAMPLE = 500 overall, 100 per cohort
        //     (below these floors, the test is underpowered and the
        //     system reports "insufficient_sample" rather than a
        //     misleading number)
        //
        // Per-axis-per-pair overrides: the cross-functional review
        // committee can set tighter or looser thresholds per
        // fairness axis per pair. Overrides are stored in the
        // treatment catalog and logged as catalog change events.
        //
        // Chronic insufficient-sample framing: a cohort whose
        // enrollment is structurally low across multiple pairs
        // is one the system is systematically under-serving. The
        // surveillance dashboard surfaces "chronic insufficient
        // sample" as its own fairness signal: if a cohort never
        // reaches the MIN_DRIFT_DETECTION_SAMPLE floor across three
        // consecutive surveillance windows, an alert fires to the
        // equity lead with the explicit framing that the model
        // cannot be validated for that population.
        FOR cohort_axis, cohort_drift in drift_signal.per_cohort_drift:
            cohort_threshold = CATALOG.get_cohort_threshold(
                pair.pair_id, cohort_axis,
                default = COHORT_DRIFT_ALERT_THRESHOLD)
            IF cohort_drift.severity >= cohort_threshold:
                DynamoDB.PutItem("surveillance-alerts", {
                    alert_id:            new UUID,
                    alert_type:          "cohort_calibration_drift",
                    treatment_pair_id:   pair.pair_id,
                    cohort_axis:         cohort_axis,
                    drift_signal:        cohort_drift,
                    threshold_applied:   cohort_threshold,
                    triggered_at:        run_date,
                    review_status:       "pending"
                })

Expected Results

Sample scoring result:

{
  "patient_id": "pat-007842",
  "scoring_run_id": "score-7c3a9f2e04b1d8a6",
  "request_context": {
    "clinician_id": "clinician-0142",
    "index_condition": "type_2_diabetes_inadequately_controlled_on_metformin",
    "visit_context": "established_patient_followup",
    "formulary_preference": "tier_1_or_tier_2_only"
  },
  "eligible_pairs": ["t2d-glp1-vs-sglt2", "t2d-glp1-vs-sulfonylurea", "t2d-sglt2-vs-sulfonylurea", "t2d-glp1-vs-basal-insulin", "t2d-sglt2-vs-basal-insulin"],
  "pair_results": [
    {
      "treatment_pair_id": "t2d-glp1-vs-sglt2",
      "treatment_id": "glp1_receptor_agonist_class",
      "comparator_id": "sglt2_inhibitor_class",
      "outcome_id": "a1c_change_at_90_days",
      "point_estimate": -0.62,
      "ci_low": -0.94,
      "ci_high": -0.31,
      "estimator_agreement": 0.83,
      "disagreement_flag": false,
      "cohort_summary": {
        "total_size": 1284,
        "treated_size": 612,
        "comparator_size": 672,
        "demographic_match_score": 0.79,
        "condition_match_score": 0.84,
        "average_treated_outcome_a1c_change": -1.41,
        "average_comparator_outcome_a1c_change": -0.79,
        "training_data_recency": "2023-01-01_to_2025-12-31"
      },
      "ood_flag": { "is_ood": false, "severity": 0.18, "reasons": [] },
      "evidence_level": "RCT_supported_with_observational_extension",
      "formulary_status": "tier_2_with_prior_auth",
      "guideline_references": ["ada_standards_of_care_2026_section_9.4", "kdigo_2024_diabetes_guidelines"],
      "calibration_status": "production_calibrated_2026Q1",
      "interpretation_note": "GLP-1 estimated to produce 0.62 percentage points greater A1c reduction than SGLT2 at 90 days, in patients with this profile. CI: 0.31 to 0.94 pp. Estimator agreement strong; not OOD."
    },
    {
      "treatment_pair_id": "t2d-glp1-vs-sulfonylurea",
      "treatment_id": "glp1_receptor_agonist_class",
      "comparator_id": "sulfonylurea_class",
      "outcome_id": "a1c_change_at_90_days",
      "point_estimate": -0.34,
      "ci_low": -0.71,
      "ci_high": 0.03,
      "estimator_agreement": 0.71,
      "disagreement_flag": false,
      "cohort_summary": {
        "total_size": 891,
        "treated_size": 412,
        "comparator_size": 479,
        "demographic_match_score": 0.74,
        "condition_match_score": 0.82,
        "average_treated_outcome_a1c_change": -1.41,
        "average_comparator_outcome_a1c_change": -1.07,
        "training_data_recency": "2023-01-01_to_2025-12-31"
      },
      "ood_flag": { "is_ood": false, "severity": 0.22, "reasons": [] },
      "evidence_level": "RCT_supported_with_observational_extension",
      "formulary_status": "tier_2_with_prior_auth",
      "guideline_references": ["ada_standards_of_care_2026_section_9.4"],
      "calibration_status": "production_calibrated_2026Q1",
      "interpretation_note": "A1c-reduction estimate for GLP-1 versus sulfonylurea is 0.34 pp greater for GLP-1, but the CI crosses zero (0.71 pp better to 0.03 pp worse). The drug-class comparison on A1c alone may not be discriminating; secondary outcomes (weight, hypoglycemia) likely matter more for this patient."
    }
  ],
  "scoring_status": "completed",
  "scoring_completed_at": "2026-04-22T14:32:18Z"
}

Sample clinician-facing comparison briefing:

{
  "briefing_id": "brief-6d4e2a81c9f03b72",
  "scoring_run_id": "score-7c3a9f2e04b1d8a6",
  "patient_id": "pat-007842",
  "headline": "Comparison of GLP-1 versus SGLT2 versus sulfonylurea for second-line therapy. Estimates are model-derived from observational data and intended to inform, not replace, clinical judgment.",
  "comparison_paragraph": "For a patient with this profile (eGFR 64 declining, BMI 34, calcium score 240, albuminuria), the model estimates greater 90-day A1c reduction on GLP-1 than on SGLT2 by 0.62 percentage points (95 percent CI 0.31 to 0.94). The model estimates similar 90-day A1c reduction on GLP-1 versus sulfonylurea, with a wide confidence interval that crosses zero (point estimate 0.34 pp better on GLP-1; 95 percent CI 0.71 pp better to 0.03 pp worse). Cardiovascular and renal outcomes are not in the primary 90-day estimate; secondary outcomes from the cohort suggest weight reduction is substantially greater on GLP-1 and SGLT2 than on sulfonylurea. The cohort underlying the GLP-1 versus SGLT2 estimate has 1,284 patients with strong condition match; estimator agreement is strong; the patient is in-distribution.",
  "per_treatment_summary": {
    "glp1": "Estimated 90-day A1c reduction 1.41 pp, weight reduction 8 to 11 percent in 90 days. GI intolerance leading to discontinuation occurred in 14 percent of cohort patients in the first 90 days. Tier 2 formulary status with prior authorization required.",
    "sglt2": "Estimated 90-day A1c reduction 0.79 pp, weight reduction 2 to 4 percent in 90 days. Genitourinary infections occurred in 6 percent of cohort patients. Tier 1 formulary status. Cardiovascular and renal benefits established by RCT in patients with this risk profile.",
    "sulfonylurea": "Estimated 90-day A1c reduction 1.07 pp. Hypoglycemia events occurred in 9 percent of cohort patients in the first 90 days; weight gain averaged 1.8 kg over 90 days. Tier 1 formulary status."
  },
  "uncertainty_summary": "Estimates are derived from observational data with target trial emulation. Confidence intervals reflect statistical uncertainty and inter-estimator agreement. Sensitivity analyses indicate the GLP-1 versus SGLT2 conclusion is robust to moderate unmeasured confounding (E-value 1.84). The GLP-1 versus sulfonylurea A1c estimate is sensitive to stronger unmeasured confounding (E-value 1.21).",
  "caveats": [
    "Estimates are conditional-average effects within the matched cohort, not individual guarantees.",
    "Cardiovascular and renal benefits are evidenced by RCTs and are not fully captured in the 90-day A1c outcome.",
    "Patient preferences and barriers (cost, injectable acceptance, supply availability) are not in the model and should drive shared decision-making.",
    "Formulary status and prior-authorization requirements may affect actual access and adherence."
  ],
  "suggested_clinician_review_points": [
    "Weight management goal: weight is a strong secondary consideration in this patient's profile.",
    "Cardiovascular and renal indication: GLP-1 and SGLT2 both have indication-supported benefit; the choice between them is not driven by 90-day A1c alone.",
    "Patient willingness to use injectable: not in the model; ask the patient.",
    "Cost and coverage: formulary status indicated above; current copay should be confirmed."
  ],
  "validator_status": "passed",
  "generated_at": "2026-04-22T14:32:34Z"
}

Sample prediction-outcome pair (90 days post-decision):

{
  "pair_id": "po-8c2f4a91b7e3d605",
  "decision_id": "dec-a41e7c930bf28d64",
  "patient_id": "pat-007842",
  "scoring_run_id": "score-7c3a9f2e04b1d8a6",
  "chosen_treatment_id": "glp1_receptor_agonist_class",
  "chosen_pair_id": "t2d-glp1-vs-sglt2",
  "outcome_id": "a1c_change_at_90_days",
  "predicted_treatment_effect": -0.62,
  "predicted_effect_ci": [-0.94, -0.31],
  "observed_single_arm_outcome": -1.62,
  "outcome_status": "observed",
  "ood_flag_at_decision": { "is_ood": false, "severity": 0.18 },
  "cohort_features": {
    "language": "en",
    "sdoh_cohort": "moderate_food_security",
    "age_band": "55-64"
  },
  "recorded_at": "2026-07-21"
}

Performance benchmarks (illustrative, your mileage varies):

Where it struggles:

• Patients outside the support of the training data. The OOD flag catches the obvious cases (very high or very low propensity score, large extrapolation distance). The harder cases are patients who are technically in-distribution on each individual feature but in an unusual combination of features. The flag has false negatives, and the briefing's required uncertainty caveats are partial protection at best. Patients with rare combinations of comorbidities, atypical lab profiles, or recent significant changes in clinical state should be approached with extra clinical skepticism even when the OOD flag is negative.
• Treatments with rapid evolution in clinical use. GLP-1 receptor agonists are the canonical example: prescribing patterns have shifted dramatically over the last several years (cardiovascular indication, then weight management, then post-supply-constraint reallocation). The training cohort from any given period is not representative of the present prescribing distribution, and CATE estimates are anchored to outdated patterns. Frequent retraining helps; explicit period-specific cohort slicing helps; the underlying problem (treatment-environment shift faster than data accumulates) does not fully go away.
• Outcomes that take a long time to materialize. Cardiovascular and renal benefits read out in years, not months. The 90-day A1c outcome is fast feedback but does not capture the outcomes that drive the major clinical guideline recommendations. Multi-horizon models help; surrogate-outcome-to-true-outcome calibration helps. Be honest in the briefing about which outcomes are predicted and which are not.
• Treatments with strong heterogeneity that the covariates do not capture. Antidepressant response is the classic example: the response rate to any given SSRI in major depressive disorder is around 30 to 40 percent, and the patient features available in EHR data do not predict response well. The CATE estimates have wide confidence intervals because the underlying treatment-effect heterogeneity is poorly explained by observable features. Be honest: the model in this case may not discriminate meaningfully between options. The briefing should communicate that.
• Regulatory and formulary friction at the point of care. A model that recommends a treatment the patient cannot access (prior authorization denied, supply constrained, copay prohibitive) is not useful. The catalog's formulary status and supply constraints reduce the gap, but real-world access is more dynamic than catalog updates capture. The decision-support layer should integrate access checks that fail loudly rather than the model recommending a treatment the patient will leave the pharmacy without.
• Off-label use cases. Many real-world treatment decisions are partially off-label or in patient populations underrepresented in trials (older adults, pregnant patients, patients with multimorbidity). The training cohorts may include some off-label use, but the methodological assumptions of CATE estimation are weaker in the off-label setting. The model should suppress or heavily caveat predictions for off-label use; the catalog can flag treatment-condition pairs as on-label or off-label.
• Temporal confounding from concurrent care changes. A patient who started a GLP-1 also started seeing a dietitian and joined a walking group; the observed outcome reflects the bundle, not the drug alone. CATE estimators control for measured covariates, but not for concurrent unmeasured interventions. Sensitivity analyses bound the impact; perfect identification of drug-only effect from observational data is a fiction.
• Cohort fairness in the response model. If historical prescribing concentrated GLP-1 in certain demographic groups (because of access, insurance, clinician preferences, or patient preferences), the cohort underlying the model is demographically skewed, and the CATE estimates may not generalize cleanly to demographic groups underrepresented in the cohort. Cohort-stratified calibration and explicit fairness instrumentation are how you avoid the Obermeyer-style failure mode at scale.
• Clinician override patterns concentrated in specific cohorts. If clinicians override the model's predictions at substantially different rates for patients in different demographic groups, the override pattern itself is a fairness signal. The override-rate dashboard surfaces this; the policy response is harder.
• Adverse-event surveillance at low base rates. Severe adverse events from any individual drug are rare; detecting unexpected adverse-event rate increases requires either a large patient population or a long surveillance window or both. Single-institution surveillance is underpowered for many adverse-event signals; consortium and network-based surveillance (Sentinel, OHDSI) is the methodologically appropriate response.

Model retraining trigger and promotion path (A10). The SageMaker training-job trigger and model-promotion path for the propensity, outcome, and CATE-ensemble models follows the EventBridge-trigger plus SageMaker-Model-Registry-with-canary-run pattern from Recipes 4.4 through 4.7, with the additional governance gate from Step 3:

• EventBridge scheduled rule fires the weekly-retrain Step Functions state machine.
• Per treatment-comparator pair (ten to thirty pairs), the state machine starts a SageMaker Pipeline that trains the propensity model, the outcome model(s), and the CATE ensemble (two to three estimators).
• Each trained artifact registers in the SageMaker Model Registry under the pair's model group, with metadata (cohort version, training-cohort S3 path, training metrics, calibration-test results, fairness-test results).
• The pipeline's post-training step runs the calibration and fairness evaluations from Step 3.
• On "green" evaluation status: the pipeline creates a governance-review task and waits. On approval (human decision event routed through EventBridge), the pipeline promotes the new model package to the "Production" stage in the Model Registry and updates the real-time endpoint alias to point to the new artifact. A canary run (scoring a held-out patient cohort and comparing results to the prior model version) gates the alias swap; if the canary diverges beyond a threshold, the swap aborts and alerts fire.
• On "yellow": governance review is mandatory. On "red": the pipeline suspends the pair and alerts the clinical informatics team.
• With three or more model artifacts per pair times ten to thirty pairs, the model registry uses a hierarchical model-group structure: one group per pair, one package per training run, and the "Production" alias always points to the currently-approved package.

SMART on FHIR and CDS Hooks credential posture (N2). The clinician-facing scoring API is the highest-stakes integration in Chapter 4. The credential posture:

• SMART on FHIR launches: OAuth 2.0 with PKCE. JWT validation against the EHR's JWKS endpoint with audience and issuer pinning. Launch context includes requesting clinician, patient context, and encounter context. Tokens scoped to patient/*.read and user/*.read for the scoring request.
• CDS Hooks calls: mutual TLS or HMAC-signed bearer tokens with replay protection (timestamp plus nonce; reject requests with timestamps older than 5 minutes). The signing secret rotates on a 90-day schedule.
• OAuth client secrets and HMAC signing keys stored in AWS Secrets Manager with KMS envelope encryption. Automatic rotation via a Secrets Manager rotation Lambda on a 90-day schedule.
• Per-EHR-tenant TLS certificates managed via AWS Certificate Manager. Separate certificates per tenant; no wildcard sharing across tenants.
• Per-tenant audit logging: every SMART launch captures (launch context, requesting clinician, patient context, scoring API response metadata). Every CDS Hooks call captures (hook trigger, requesting system, patient context, response latency, validator status). All audit records flow to CloudTrail and the scoring-API-audit S3 bucket.

Patient consent flow and shared-decision capture (A6). The consent layer handles four distinct consent types:

• (a) Consent to the use of model-derived predictions in the patient's care. Captured at the point of care when the clinician first presents the briefing. Stored in the decision-records table alongside the clinician's treatment decision. Without this consent, the briefing is not surfaced (the clinician sees a "patient consent required" interstitial).
• (b) Consent to ongoing data use for model retraining. Captured separately (typically during onboarding or annual consent refresh). Stored in the patient-profile DynamoDB table with a versioned consent-status field. Patients who decline retraining consent have their prediction-outcome pairs excluded from the training cohort by a pre-training filter in the cohort-construction pipeline.
• (c) Capture of the patient's stated preferences and their influence on the chosen treatment. The decision-record schema includes a shared_decision_indicators field that records whether the patient summary was shared, whether the patient expressed a preference, and what that preference was. Preferences are structured (treatment preference, route preference, cost sensitivity, side-effect tolerance) rather than free-text.
• (d) Right to withdraw consent. A consent-withdrawal event triggers: exclusion of the patient's future data from retraining cohorts, suppression of model-derived predictions at future visits (the scoring orchestrator checks consent status before scoring), and retention of historical decision records for audit purposes (withdrawal does not retroactively delete audit artifacts; it stops new predictions).

Adverse-event surveillance as first-class pseudocode (A5). Integrated with calibration drift detection in Step 6:

FUNCTION run_adverse_event_surveillance(run_date):
    // Per treatment-comparator pair: define the adverse events of
    // interest at model promotion, establish expected rates, compute
    // observed rates, and fire alerts when observed exceeds expected.
    FOR each pair in CATALOG.list_production_pairs():
        ae_definitions = pair.adverse_event_definitions
            // Example for GLP-1:
            //   pancreatitis, severe_gi_event
            // Example for SGLT2:
            //   diabetic_ketoacidosis, fourniers_gangrene
            // Example for sulfonylurea:
            //   severe_hypoglycemia_hospitalization

        FOR each ae_def in ae_definitions:
            // Expected rate from training data and trial data,
            // established at model promotion.
            expected_rate = ae_def.expected_rate_per_million_patient_days

            // Observed rate in the surveillance window.
            exposure_days = compute_exposure_days(
                pair, ae_def, run_date, SURVEILLANCE_WINDOW_DAYS)
            IF exposure_days < ae_def.minimum_exposure_floor:
                CONTINUE  // underpowered; skip this AE for this window

            observed_events = count_adverse_events(
                pair, ae_def, run_date, SURVEILLANCE_WINDOW_DAYS)
            observed_rate = observed_events / (exposure_days / 1_000_000)

            // Statistical test: Poisson exact test at p < 0.01.
            p_value = poisson_exact_test(observed_events,
                expected_rate * exposure_days / 1_000_000)

            IF p_value < 0.01:
                // Fire alert. Include cohort-stratified breakdown.
                stratified_rates = compute_ae_rates_by_cohort(
                    pair, ae_def, run_date, SURVEILLANCE_WINDOW_DAYS)

                DynamoDB.PutItem("surveillance-alerts", {
                    alert_id:          new UUID,
                    alert_type:        "adverse_event_signal",
                    treatment_pair_id: pair.pair_id,
                    adverse_event:     ae_def.event_id,
                    expected_rate:     expected_rate,
                    observed_rate:     observed_rate,
                    observed_events:   observed_events,
                    exposure_days:     exposure_days,
                    p_value:           p_value,
                    stratified_rates:  stratified_rates,
                    triggered_at:      run_date,
                    review_status:     "pending"
                })
                // Sentinel and OHDSI are the consortium-scale path
                // for cross-institution signal confirmation.

Patient-facing summary validator (A7). Distinct from the clinician-facing validator; the patient-facing path has its own validate_patient_summary function:

• Layer 1: reading-level enforcement per the Recipe 4.2 pattern (Flesch-Kincaid grade level 6 to 8; sentences under 20 words average; no medical jargon without parenthetical lay explanation).
• Layer 2: prohibition of probabilistic point estimates rendered as percentages ("62 percent chance" is confusing to patients) and prohibition of "you will [outcome]" framing. Instead, require cohort-based phrasing: "among patients similar to you, treatment A was associated with [outcome description]."
• Layer 3: recommendation-language patterns (same as clinician validator) plus patient-specific extensions: prohibition of "your doctor recommends" or "this is the right treatment for you" (the system informs; the clinician and patient decide together).
• Layer 4: required content including shared-decision framing ("this information is meant to help you and your doctor discuss options together"), approved-claim-language compliance (no statements that would constitute off-label promotion), and a clear statement that the patient should ask their doctor about anything unclear.

Cohort-feature cache optimization (A8). The lookup_cohort_features(patient_id) call is hoisted out of per-pair loops throughout the pipeline:

• In Step 4 (score_patient): compute cohort features once before the per-pair scoring loop rather than inside it. With 5 to 10 eligible pairs per patient, the redundant Feature Store lookups multiply.
• In Step 6 (match_outcome): as implemented in the corrected pseudocode above, cohort features are computed once per patient before the per-pair, per-outcome iteration.
• In the monthly surveillance run (run_calibration_drift_detection): batch-fetch cohort features for all patients in the surveillance window before iterating per-pair. With thousands of patients per surveillance run and multiple pairs per patient, the redundant lookups are O(patients * pairs) without the cache versus O(patients) with it.

De-identification posture for treatment-response briefings (S4). The Privacy specification for Bedrock prompts and briefing text:

• Clinical features passed to the LLM use banded values: eGFR_band (e.g., "45-59"), BMI_band (e.g., "30-35"), A1c_band (e.g., "8.5-9.0"). Precise lab values are never passed in the prompt; the banding is sufficient for the briefing's clinical context.
• Demographic attributes (race, ethnicity, language, SDOH cohort) are excluded from prompts by default. Opt-in requires explicit pharmacy-and-therapeutics-committee approval documented in the treatment catalog, and is limited to cases where the attribute is clinically relevant (e.g., pharmacogenomic indications where ancestry is a proxy for allele frequency). Opt-in decisions are logged as catalog governance events.
• No patient_id or clinician_id in prompts. The prompt receives an opaque session token; the mapping from session token to patient and clinician lives only in the scoring orchestrator's memory and the audit log.
• Minimum-necessary disclosure remains the architectural posture even for HIPAA-eligible services. The briefing needs the clinical context to be useful; it does not need the patient's identity.
• Briefing text stored in the briefings DynamoDB table is PHI (it describes a specific patient's clinical situation and predicted treatment outcomes). Apply clinical-record-equivalent encryption and access controls.

DLQ coverage on all Lambda paths. Three DLQ patterns in this architecture:

• (a) Step Functions to Lambda pipeline: each Lambda task in the weekly-retrain, monthly-surveillance, and nightly-cohort state machines has a Catch clause pointing to an SQS failure queue keyed on (run_id, stage, treatment_pair_id). The Step Functions retry policy retries transient failures (throttles, timeouts) twice with exponential backoff; terminal failures (validation errors, data-quality failures) route directly to the DLQ without retry.
• (b) Kinesis to state-machine-worker Lambda: the event source mapping configures an OnFailure destination (SQS DLQ). Failed event batches land in the DLQ with the original event payload and failure metadata. A CloudWatch alarm fires when the DLQ depth exceeds zero; the operational team triages within the DLQ retention window.
• (c) SageMaker Real-Time inference failures on the point-of-care scoring path: the scoring orchestrator Lambda catches inference errors (endpoint throttling, model errors, timeout) and returns a degraded-state response: { scoring_status: "scoring_temporarily_unavailable", reason: "inference_service_error" }. The response is explicit, not a partial or stale prediction. The clinician interface renders "treatment comparison temporarily unavailable; clinical judgment without model input applies" rather than silently serving a cached or incomplete result. The scoring orchestrator emits a scoring_inference_failure metric that feeds a CloudWatch alarm.

Why This Isn't Production-Ready

The pseudocode and architecture above demonstrate the pattern. A production deployment needs to close several gaps that are intentionally out of scope for a recipe.

Treatment-catalog curation as an ongoing program. Like the program registry in 4.7, the treatment catalog is a living artifact. New drugs, new indications, formulary changes, evolving guideline recommendations, supply constraints, post-marketing safety signals; all of these land as catalog updates. Plan for at least 0.5 to 1.0 FTE of pharmacy and therapeutics, clinical informatics, and health economics time on catalog maintenance ongoing, plus a structured change-management process with parallel evaluation against the prior catalog version when significant changes ship.

Causal-inference rigor at production scale. The methodology described in The Technology section is not a checklist; it is a discipline that requires a dedicated methodologist or a methodologist-trained data scientist in the loop. Plan for the staffing: at minimum a senior data scientist with formal causal-inference training, a clinical informaticist who understands the outcome definitions and target trials, and a biostatistician who can evaluate the sensitivity analyses. The choice of cohort, the outcome definitions, the sensitivity analyses, the calibration tests, and the fairness tests are not template work. They are design work that has to be done well at every catalog entry. The pattern that fails is the data science team shipping CATE estimators on hand-engineered cohorts without methodological review; the resulting estimates look authoritative and are systematically biased.

Regulatory pathway determination and documentation. The model risk tier per treatment-comparator pair determines whether the pair is in scope for FDA SaMD regulation, the 21st Century Cures Act CDS exemption, or another regulatory framework. The decision is fact-specific: the form of the recommendation (suggestive paragraph versus ranked list versus single recommendation), the underlying evidence (RCT-backed versus observational), and the clinician's ability to review the basis (which the briefing's structured evidence is intended to support) all matter. Plan for regulatory legal review at scoping, a predetermined change control plan if SaMD applies, postmarket surveillance documentation, and a complaint-handling process. Retrofitting regulatory compliance is dramatically more expensive than building it in.

EHR integration and clinician workflow design. The recipe assumes a SMART on FHIR app or CDS Hooks endpoint as the integration point. The clinician workflow design (when does the briefing surface, how does the clinician dismiss it, how does the chosen-treatment field get back to the system, how does the rationale get captured) is a substantial UX project. Many clinical decision support systems fail at this layer rather than at the modeling layer; the recommendation is technically correct, presented at the wrong moment, in a way that interrupts the clinician's flow, and gets dismissed without consideration. Plan for a dedicated clinical-informatics-led UX effort, with iterative deployment to a small clinician cohort, clinician feedback loops, and willingness to redesign the surface before broad rollout.

Patient consent and shared decision-making. Treatment decisions in healthcare are shared decisions between clinician and patient. The briefing supports the clinician; a patient-facing version supports the patient. Whether and how the patient-facing version is shared, whether the patient's preferences are captured back into the decision record, and whether the patient consents to having their data used for ongoing model improvement are all design decisions that have legal, ethical, and operational implications. Integrate with the institution's existing shared decision-making programs; do not invent a parallel patient-engagement layer.

Cross-recipe orchestration with Recipes 4.5, 4.6, 4.7, 4.9, and 4.10. A patient who gets a treatment-response prediction at the visit may also be a candidate for medication-adherence intervention (4.5), care-gap closure (4.6), care-management enrollment (4.7), personalized care plan generation (4.9), or dynamic treatment regime recommendation (4.10). The integration points need to be explicit: the chosen treatment from 4.8 feeds the care plan in 4.9; the predicted adherence from 4.5 feeds the 4.8 prediction's caveat about adherence assumptions; the care-management enrollment in 4.7 may include monitoring for the chosen treatment's outcomes. Document the cross-recipe data flow; design for the integration rather than bolting it on.

Model retirement and sunset. Some models will, over time, lose calibration faster than retraining can recover, become structurally biased in ways that fairness instrumentation cannot fully correct, or be superseded by a regulatory-cleared alternative. The catalog needs a sunset path: a treatment-comparator pair can be retired from production with explicit rationale, and the retirement triggers downstream cleanup (suppress the briefings, archive the decision records with the retired-model annotation, communicate to the clinical-informatics team). The pattern that fails is models that quietly degrade in production because nobody is responsible for retiring them; the result is clinicians who learn to ignore the briefing, which is the worst of both worlds (the system is taking up screen real estate without changing decisions).

Adverse-event surveillance at network scale. Single-institution surveillance is underpowered for low-base-rate adverse events. The methodologically appropriate response is consortium-based surveillance (Sentinel, OHDSI) where treatment-response surveillance signals are pooled across multiple institutions with privacy-preserving methods. Building the consortium integration is a large external partnership that takes time; the recipe assumes single-institution operation, but the production version benefits substantially from consortium membership.

Privacy in scoring results and decision records. The scoring-results table joins (patient_id, predicted treatment outcomes, similar-patient cohort summaries) and is highly inferential. The decision-records table joins (patient_id, chosen treatment, predictions at decision time, clinician rationale) and is closer to clinical-record audit content than typical analytics audit. Apply tighter controls than for engagement data: narrower IAM read scopes, optional separate-table partitioning by sensitivity tier, additional CloudTrail data event capture, and a documented minimum-necessary access policy. Briefings stored in DynamoDB are PHI and treatment-recommendation-relevant; treat them with the same encryption, IAM, and audit posture as clinical notes.

Idempotency and retry semantics. Same pattern as 4.4 through 4.7. Each stage's outputs are addressed by deterministic keys (cohort run, scoring run id, decision id) and writes are conditional, so a Step Functions retry that re-attempts a completed step is a no-op rather than a duplicate. The Step Functions Catch should distinguish retryable infrastructure failures from terminal logic failures and route terminal failures to the DLQ.

Cost-aware model serving. Real-time CATE inference per pair times ten to thirty pairs at thousands of scoring requests per day adds up. SageMaker multi-model endpoints amortize across pairs; SageMaker Asynchronous Inference handles bursty workloads; SageMaker Serverless Inference is appropriate for low-volume pairs. The architecture above assumes a single endpoint per pair for clarity; production deployments should rationalize the serving topology against actual usage patterns and cost.

Variations and Extensions

Multi-outcome treatment scoring. The pseudocode focuses on a primary outcome per pair. A more sophisticated version produces per-(patient, treatment) estimates across a vector of outcomes (A1c, weight, hypoglycemia, cardiovascular events, total cost) with a cross-outcome trade-off framework. The clinician-facing view becomes a multi-dimensional comparison rather than a single-number ranking. Implementing this responsibly requires explicit weight elicitation (often patient-specific) and a UX that does not collapse the trade-offs to a single recommendation.

Subgroup-conditional CATE. Standard CATE estimators estimate effects conditional on the full covariate vector. In some clinical applications, the meaningful question is the effect within a clinically defined subgroup (patients with eGFR less than 45, patients with prior MI, patients above age 75). Subgroup-conditional CATE is a special case but worth implementing as a first-class feature: the catalog includes subgroup definitions, and the briefing presents subgroup-specific estimates when the patient falls in a defined subgroup. Be mindful of multiple-testing inflation when many subgroups are evaluated.

Federated learning across institutions. Single-institution training cohorts are often underpowered for low-prevalence treatment-condition pairs. Federated learning across multiple institutions (with privacy-preserving methods like homomorphic encryption or differential privacy) produces pooled models without sharing patient-level data. OHDSI's OMOP-CDM and PCORnet's data networks provide harmonized data substrates that make federated CATE estimation tractable. Plan for federation as a multi-institution partnership, not a single-vendor project.

Adaptive trial integration. Some treatment decisions can be informed by ongoing adaptive clinical trials (platform trials, basket trials, master protocols). When a patient is eligible for an active trial and the trial design supports adaptive randomization with response-prediction inputs, the recipe's CATE output can be one input to the trial's adaptive randomization algorithm. This integration sits squarely in research-clinical boundary territory and requires IRB-approved protocols.

Patient-reported outcome integration. Standard CATE estimation uses clinically observed outcomes (lab values, encounter codes). Patient-reported outcomes (PROMs, symptom scales, quality-of-life measures) are increasingly captured in EHRs and patient portals. Adding PROMs as additional outcomes per pair captures the patient experience that clinical metrics miss. Be mindful of PROM completion bias; the patients who complete PROMs are not a random sample of the cohort.

Pharmacogenomic integration. Some treatments have well-evidenced pharmacogenomic predictors of response (CYP2C19 for clopidogrel, HLA for carbamazepine, CYP2D6 for some antidepressants). When pharmacogenomic data is available for the index patient, it becomes a high-leverage covariate. The catalog can flag treatment-pair pairs with pharmacogenomic relevance and the prediction pipeline can prioritize PGx-derived features. PGx data has its own privacy and consent considerations beyond standard EHR data.

Cost-effectiveness extensions. A model that predicts only clinical effects misses the cost-effectiveness dimension that drives many real-world treatment decisions. An extended pipeline computes per-patient cost-of-treatment (drug cost, monitoring cost, downstream cost based on predicted outcomes) alongside clinical effect, and the briefing surfaces a cost-versus-benefit comparison. Cost-effectiveness analysis is methodologically distinct (incremental cost-effectiveness ratios, willingness-to-pay thresholds) and benefits from health economics expertise on the team.

Sequencing and trial-and-error optimization. Some treatment decisions are inherently sequential: try drug A; if response is inadequate, try drug B; if not, try drug C. The single-shot CATE framework does not capture the sequencing optimization. Recipe 4.10 (Dynamic Treatment Regime Recommendation) extends the framework to sequential decisions with reinforcement-learning-style estimation. Recipe 4.8 is the building block for 4.10; teams targeting sequential optimization should design the catalog and the data infrastructure with the 4.10 extension in mind from the start.

Real-world evidence pipeline integration. The catalog and the evaluation infrastructure described here are the substrate for real-world evidence generation. Per-treatment-comparator cohort effects, with the methodological rigor of target trial emulation, are the kind of evidence that supports regulatory submissions, label expansions, and post-marketing safety analyses. A research collaboration with a regulatory or RWE partner extends the value of the system substantially beyond decision support.

Negative-control and falsification analyses. A robust CATE pipeline includes negative-control outcomes (outcomes that should not be affected by the treatment; if the estimator finds an effect on a negative-control outcome, the estimator is misspecified or there is unmeasured confounding) and falsification tests (effects on treatment-naive periods, effects on outcomes that occurred before treatment). These are research-grade discipline but increasingly expected in production-grade real-world evidence pipelines.

Time-varying treatment effects. The single-snapshot CATE assumes a treatment effect at a single point in time. For chronic conditions, the treatment effect varies with disease stage, comorbidity progression, and treatment duration. Time-varying CATE estimation (g-methods, marginal structural models) handles this more rigorously than the static framework. Crosses methodologically into Recipe 4.10 territory.

Additional Resources

AWS Documentation:

• Amazon SageMaker Developer Guide
• Amazon SageMaker Pipelines
• Amazon SageMaker Model Registry
• Amazon SageMaker Real-Time Inference
• Amazon SageMaker Multi-Model Endpoints
• Amazon SageMaker Batch Transform
• Amazon SageMaker Feature Store
• AWS Step Functions Developer Guide
• Amazon Bedrock User Guide
• Amazon API Gateway Developer Guide
• Amazon Cognito Developer Guide
• AWS HealthLake Developer Guide
• Amazon QuickSight User Guide
• AWS HIPAA Eligible Services
• Architecting for HIPAA on AWS (Whitepaper)

AWS Sample Repos:

• amazon-sagemaker-examples: Reference notebooks for SageMaker Pipelines, Model Registry, Real-Time Inference, and Batch Transform applicable to per-pair model lifecycles
• amazon-sagemaker-feature-store-end-to-end-workshop: End-to-end Feature Store usage applicable to the per-patient and per-treatment-history feature pipelines
• amazon-bedrock-workshop: Hands-on labs covering structured-output prompting that informs clinician-facing comparison briefings and patient-facing summaries

AWS Solutions and Blogs:

• AWS Solutions Library (filter AI/ML and Healthcare): browse for healthcare ML and decision-support reference architectures
• AWS Machine Learning Blog: search "causal inference," "uplift modeling," and "SageMaker Pipelines" for relevant deep-dives
• AWS for Industries Blog (Healthcare and Life Sciences): search "clinical decision support," "real-world evidence," and "value-based care" for end-to-end customer architectures

External References (Methodology and Causal Inference):

• EconML: Microsoft Research's library for heterogeneous treatment effect estimation; the methodological foundation for the CATE ensemble in this recipe
• DoWhy: Causal-inference framework for structuring assumptions and running sensitivity analyses
• causalml: Uber's causal inference library, complementary to EconML, with strong meta-learner implementations
• grf: The R generalized random forests package implementing causal forests; the canonical implementation of the Athey-Wager method
• bcf: Bayesian Causal Forests, the BART-based CATE estimator
• Hernán and Robins, Causal Inference: What If: the standard textbook for causal inference in epidemiology and observational research; the target trial emulation framework is the methodological backbone of this recipe
• VanderWeele and Ding, Sensitivity Analysis in Observational Research: Introducing the E-value: the E-value sensitivity-analysis approach used in Step 3
• Yadlowsky et al., Calibration of CATE estimators: contemporary work on calibration of treatment-effect predictions
• Obermeyer et al. 2019, Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations: the canonical cautionary tale for fairness failures in healthcare AI; required reading

External References (Regulatory and Clinical):

• FDA Software as a Medical Device (SaMD) framework
• FDA Clinical Decision Support Software guidance
• Good Machine Learning Practice for Medical Device Development (FDA, Health Canada, MHRA)
• OHDSI (Observational Health Data Sciences and Informatics): community methods, the OMOP common data model, and consortium-scale real-world evidence generation
• PCORnet: the National Patient-Centered Clinical Research Network for distributed observational research and treatment-effect estimation
• Sentinel Initiative: FDA's national electronic system for monitoring the safety of medical products in the post-marketing setting

Estimated Implementation Time

← Main Recipe 4.8 · Python Example · Chapter Preface