Recipe 4.8: Treatment Response Prediction ⭐⭐⭐⭐
Complexity: Complex · Phase: Research-to-Production · Estimated Cost: ~$0.02-0.10 per-patient treatment-comparison decision (depends on per-treatment CATE model serving, similar-cohort retrieval, and clinician-facing rationale generation)
The Problem
Marcus is 58. He has type 2 diabetes diagnosed eight years ago, an A1c that has crept from 7.1 to 8.7 over the last fourteen months despite metformin twice a day, a BMI of 34, an eGFR of 64 (slowly declining; he was 78 three years ago), no diagnosed cardiovascular disease but a calcium score of 240 from a screening CT he got last year, and a moderately elevated urinary albumin-to-creatinine ratio that his nephrologist has been "watching" for the better part of two years. His most recent visit, his primary care physician told him it was time to add a second medication, talked through some options, and ordered a lab follow-up in six weeks. Marcus left the office unclear about which medication he'd actually be starting. He's not alone. Neither, frankly, is his physician.
The decision in front of Marcus's PCP is not a small one. Adding a medication for a patient with this profile is a five-way fork. There is metformin plus a sulfonylurea (cheap, effective at lowering A1c, hypoglycemia risk, weight gain). There is metformin plus a DPP-4 inhibitor (modest A1c reduction, weight neutral, generally well tolerated, expensive without good preferred-formulary status on his insurance). There is metformin plus a GLP-1 receptor agonist (substantial A1c reduction, weight loss of 10 to 15 percent in many patients, cardiovascular and renal benefit in patients like Marcus, but injectable for the most-effective formulations, GI side effects, expensive, supply is sometimes constrained). There is metformin plus an SGLT2 inhibitor (moderate A1c reduction, weight loss, cardioprotective and renoprotective benefit at his eGFR, genitourinary infection risk, ketoacidosis risk in specific situations, expensive but well tolerated). And there is metformin plus basal insulin (most-effective A1c reduction at higher A1c levels, hypoglycemia risk, weight gain, requires the patient to inject and self-monitor glucose, accessible and inexpensive in generic formulations).
Five options. The clinical guidelines say "consider GLP-1 or SGLT2 in patients with cardiovascular or renal indications." Marcus has cardiovascular indications (calcium score, family history) and emerging renal indications (declining eGFR, albuminuria). The guidelines do not say "this specific patient should get this specific drug." They say "consider." The PCP is the one who has to consider. And the PCP is not, in any honest accounting, going to spend forty minutes drilling into the trial data, the patient's specific phenotype, the cost-coverage table for this insurance plan, the supply situation for the preferred GLP-1, and the patient's likelihood of adhering to an injectable, in the seven-minute slot allotted to this part of the visit.
What ends up happening, much of the time, is that the PCP picks one of the options based on a combination of factors: their own clinical experience with similar patients, the most recent manufacturer rep visit, what's on the local formulary, what they think the patient will be willing and able to take, what they remember from a recent CME, and, sometimes, what's just easiest to prescribe in the EHR. The patient leaves with a prescription. Six weeks later they come back. The A1c has moved, or it hasn't. The patient has tolerated the medication, or they haven't. The PCP makes the next adjustment. The whole thing iterates over months and years. Most patients, eventually, get to a stable regimen. Some don't. Marcus is somewhere in the population we hope for and not the population we worry about, but the path he takes through that fork has real consequences for his next ten years: cardiovascular risk reduction or not, renal protection or not, weight that goes up or down, hypoglycemia risk, total medication cost, whether he stops taking his medication entirely because the side effects were too much.
And here is the thing that should have been bothering us the entire time: there are thousands of patients exactly like Marcus in this plan's panel. The plan, the practice, and the EHR vendor all have data on what happened to those patients. Some got the GLP-1 and dropped six points of A1c. Some got the GLP-1 and stopped it after three weeks because of nausea. Some got the SGLT2 and saw a beautiful eGFR-stabilization curve. Some got the SGLT2 and got a yeast infection that they really didn't appreciate. Some got the sulfonylurea and never came back to discuss the late-night hypoglycemic episode that scared their spouse. The data exists. The data is messy (the patients who got GLP-1 are systematically different from the patients who got sulfonylurea, because the prescribing decision wasn't random), but the data exists. And nobody, in the moment Marcus is sitting in front of his PCP, is consulting that data in a structured way that produces an estimate of which medication is most likely to work for this patient.
Treatment response prediction is the practice of producing that estimate. Not a guideline-level "consider GLP-1 or SGLT2." A patient-level "for Marcus, given his eGFR trajectory, his weight, his calcium score, his medication-adherence pattern, his insurance coverage, and his stated preferences, the predicted three-month A1c reduction on a GLP-1 is 1.4 percentage points (95 percent CI 0.8 to 1.9), the predicted three-month A1c reduction on an SGLT2 is 0.7 percentage points (95 percent CI 0.4 to 1.1), the predicted weight change on a GLP-1 is minus 8.4 percent (95 percent CI minus 5.1 to minus 11.7), the predicted GI-intolerance discontinuation rate is 18 percent in the first 60 days, and the patient's predicted adherence on injectable is materially lower than on oral if cost-coverage is borderline." That paragraph is what the PCP wants on the screen during the visit. That paragraph is not the same as a guideline. That paragraph is a probabilistic, individualized estimate of treatment effect, and producing it well is one of the hardest open problems in healthcare ML.
The reason it's hard is not that we lack the data, although the data is harder to use than it looks. The reason it's hard is that the question being asked is fundamentally a causal question, not a correlational one. We are not asking "what is the probability that Marcus's A1c will be 7.0 in three months?" We are asking "what is the difference between Marcus's three-month A1c if he is prescribed GLP-1 versus if he is prescribed SGLT2?" That difference, the individualized treatment effect, is, for any given patient, fundamentally unobservable: Marcus will get one drug, not all five, and the others are counterfactuals we never see. Estimating the unobserved is the whole problem. Doing it from observational data, where the patients who got drug A differ systematically from the patients who got drug B (because clinicians chose, not because randomization chose), is even more of a problem. Doing it in a way that doesn't replicate the historical patterns of who-got-prescribed-what (which, in the United States, correlates uncomfortably with race, income, geography, and insurance type) is more of a problem still.
And then there is the wrinkle that makes treatment response prediction qualitatively different from every other recipe in this chapter: the consumer of the output is a clinician, in a treatment-decision moment, and the recommendation has direct clinical impact. The earlier recipes in this chapter (4.1 through 4.7) recommend channels, content, programs, and care-management resources. The downstream actor is operations or care management, the decision is reversible, and the stakes are real but bounded. Recipe 4.8 sits one floor higher. The downstream actor is a prescribing clinician, the decision affects a real prescription that the patient will take or not take, and the regulatory framing is different: in many implementations, treatment response prediction tools fall within the FDA's definition of Software as a Medical Device (SaMD), specifically clinical decision support that does not meet the criteria for the Cures Act non-device exemption when the clinician cannot independently review the basis of the recommendation.
This recipe, then, is a recipe with a very specific posture: the system informs the clinician with individualized treatment-effect estimates, with honest uncertainty quantification, with the basis of the estimate (the similar patients, the features driving the estimate, the data quality) made transparent enough that the clinician can review it. The system does not select the treatment. The system does not present a single ranked answer that a busy clinician will rubber-stamp without inspection. The system presents a structured comparison, with confidence intervals, with the heterogeneity of the underlying cohort, and with the explicit caveat that the estimate is from observational data and is bounded by what the data can tell us. The clinician decides. The patient consents. The system tracks the outcome and feeds it back into the next iteration of the model.
We are going to build the architecture for that. The scaffolding is similar to Recipe 4.7 in some respects (per-treatment models, feature pipelines, batch and on-demand serving, validator-protected LLM packaging, equity instrumentation), but the heart of this recipe is causal-inference modeling done with appropriate methodological discipline, presented to a clinician at the point of care, with regulatory and ethical guardrails that the earlier recipes did not require. The hard parts are not the AWS services. The hard parts are the methodology and the governance. The recipe takes both seriously.
Let's get into how you build it.
The Technology: Causal Inference, Heterogeneous Treatment Effects, and the "Similar Patient" Trap
What Treatment Response Prediction Actually Asks
The question is: for this specific patient, what is the difference in expected outcome between treatment A and treatment B (or between treatment A and no treatment, or between treatment A and standard care)? The output is an estimated individualized treatment effect (ITE), or, more honestly, a conditional average treatment effect (CATE) given the patient's covariates. The conditional average treatment effect is the average outcome difference among the subpopulation of patients with the same covariates as the index patient. The "individualized" framing is aspirational; the conditional-average framing is what the math can actually deliver, and it depends on the conditioning set being rich enough that within-subpopulation variation is small.
This is fundamentally a counterfactual question. For any given patient, we observe one outcome (the one corresponding to the treatment they actually received) and we never observe the others. That asymmetry is the fundamental problem of causal inference. Donald Rubin's framework, the Neyman-Rubin potential outcomes model, formalizes it: each patient has a vector of potential outcomes Y(0), Y(1), Y(2), ... one per possible treatment, and we observe only the one corresponding to the treatment actually administered. The CATE is the expected difference between these potential outcomes within the subpopulation defined by patient covariates.
We need to estimate that difference from observational data, where the treatment was not assigned at random. Patients who got GLP-1 differ from patients who got SGLT2 in ways that are visible in the data (eGFR, BMI, prior cardiovascular events) and in ways that may not be visible (clinician preferences, regional prescribing patterns, patient willingness to use injectables, ability to pay copay). The visible differences are measured confounders; controlling for them is statistically tractable. The invisible differences are unmeasured confounders; controlling for them is, in general, mathematically impossible without additional structure.
So the technical work is a sequence of disciplined moves: identify what causal quantity we want to estimate, identify the assumptions under which the data can identify it, build models that respect those assumptions, quantify uncertainty (both statistical from sample size and structural from the assumptions themselves), and present the result with the assumptions explicit so the clinician can decide whether to trust the estimate for the patient in front of them.
The "Similar Patient" Methodology and Its Failure Modes
The intuitive framing of treatment response prediction is "find patients similar to this one, see what worked for them, predict accordingly." This framing is approximately right and dangerously wrong at the same time. It is approximately right because the underlying mathematics of CATE estimation is, in many estimators, doing a sophisticated form of weighted-similar-patient averaging. It is dangerously wrong because the intuitive form (k-nearest-neighbor matching on a hand-picked feature set) has systematic failure modes that experienced practitioners have learned to avoid:
Curse of dimensionality. "Similar" in high-dimensional feature space (hundreds of clinical and demographic features) means "no two patients are actually similar." The nearest neighbors in 200-dimensional space are far away. Distance becomes meaningless without strong feature selection, dimensionality reduction (PCA, autoencoders, learned embeddings), or methods that handle high dimensionality intrinsically (random forests, gradient boosting, deep learning).
Confounding by indication. The patients who received GLP-1 differ systematically from the patients who received SGLT2 in ways that drove the prescribing decision. If the model finds "similar patients to Marcus who got GLP-1," those patients had specific reasons they got GLP-1 that Marcus may or may not share. Treating their outcomes as Marcus's expected outcome conflates the drug effect with the selection effect.
Hidden subgroups within the matched cohort. A "similar patient" cohort can contain heterogeneous subgroups that average out to a misleading mean. Five matched patients had a 2-point A1c drop on GLP-1; what the average hides is that two of them dropped 4 points and three of them dropped 0.5 points because they discontinued. The average is not Marcus's likely outcome; it is a poorly-summarized distribution over very different outcomes.
Treatment effect heterogeneity that the matching misses. Two patients with the same covariates can have very different responses to the same drug because of factors not in the covariates (genetics, microbiome, behavioral, environmental). The CATE estimate has structural uncertainty that no amount of within-covariate matching can eliminate. The honest output communicates that uncertainty rather than reporting the conditional mean as if it were a point prediction.
Selection bias in the source data. The patients in the historical data are not a random sample of the eligible population. They are the patients who showed up for visits, agreed to start medications, filled prescriptions, and stayed in the system long enough to have an observed outcome. If those filters correlate with response (and they do, almost always), the response predictions inherit the bias. A model trained on patients who stayed on GLP-1 long enough to have measured A1c will systematically over-estimate GLP-1 effectiveness, because the patients who stopped early (because of side effects, cost, or other reasons) are missing from the outcome data.
Calibration drift over time. Treatment effects in observational data are anchored to the prescribing patterns and patient mix of the historical period. If GLP-1 prescribing shifted from "cardiovascular indication, treatment-resistant" to "weight management, treatment-naive" over the last three years, the patients in the training data are not representative of the patients the model will be asked to predict for going forward. This is concept drift with a vengeance, and it requires explicit recalibration.
The serious literature on treatment effect estimation has developed methods that address these failure modes head-on. Hand-rolled k-nearest-neighbor on a hand-picked feature set is a research demonstration, not a production approach. The right approach uses one or more of the methods below.
Methods for Heterogeneous Treatment Effect Estimation
Several method families have matured to production-grade and are appropriate for treatment response prediction:
Meta-learners. A family of approaches that combine outcome models (predicting Y given features X and treatment T) with treatment-assignment models (predicting T given X) to estimate the CATE. The S-learner uses a single outcome model with treatment as a feature; the T-learner trains separate outcome models per treatment arm; the X-learner combines T-learner predictions with weighting to improve performance under treatment-arm imbalance; the DR-learner (doubly robust) combines outcome and treatment models in a way that is consistent if either is correctly specified; the R-learner (named after Robinson) uses orthogonalization to reduce sensitivity to misspecification of the nuisance models. The Microsoft Research EconML library implements all of these. The choice among them is empirical; performance varies by data characteristics, and benchmarking on held-out data with known treatment effects (or randomized subsets) is essential.
Causal forests. Athey and Wager's adaptation of random forests for CATE estimation. Each tree splits to maximize within-leaf homogeneity of treatment effect rather than within-leaf homogeneity of outcome. The forest aggregates per-leaf treatment effect estimates, with honest sample-splitting to prevent overfitting. The grf R package and EconML's CausalForestDML implement this method. Causal forests handle high-dimensional features and complex non-linear treatment effect heterogeneity natively, with built-in confidence interval estimation.
Bayesian additive regression trees (BART). A Bayesian sum-of-trees model with regularization priors. BART naturally produces posterior distributions over predicted outcomes, which translate directly into uncertainty estimates over treatment effects. The bartCause and BCF (Bayesian Causal Forests) packages are mature implementations. BART is computationally heavier than gradient boosting but the Bayesian posterior is operationally useful for the uncertainty quantification clinical decision support requires.
Deep learning for HTE. TARNet, Dragonnet, CFRNet, and related architectures use shared-and-separate-head neural networks to estimate counterfactual outcomes. They handle high-dimensional inputs (including unstructured text, images) at the cost of more demanding model selection and validation. For tabular EHR data with hundreds to low thousands of features, gradient-boosted methods often match or beat deep learning; deep learning's edge appears when imaging or text features are integrated into the prediction.
Target trial emulation. Hernán and Robins's framework for using observational data to estimate treatment effects as if from a hypothetical randomized trial. The method specifies the hypothetical trial protocol (eligibility, treatment strategies, outcomes, follow-up) and then constructs the analytic dataset from observational data to match that protocol as closely as possible. Target trial emulation has become the methodological gold standard for observational treatment-effect studies and produces estimates with much better correspondence to randomized-trial benchmarks than ad-hoc observational analyses. It is methodologically rigorous, time-consuming, and worth the investment for any treatment-effect estimate that will inform clinical decisions.
Targeted Maximum Likelihood Estimation (TMLE). A doubly-robust estimation framework that produces semi-parametrically efficient estimates of average treatment effects with good finite-sample properties. The R tmle and Python zEpid packages implement it. TMLE is well-suited to the population-level effect estimation that anchors the per-patient CATE estimates.
Inverse probability of treatment weighting (IPTW). Weight observations by the inverse of the estimated treatment probability given covariates, then estimate effects on the weighted pseudo-population. Simpler than the doubly-robust methods, more sensitive to propensity model misspecification. Often used as a complement to other methods rather than as the primary estimator.
A production system does not pick one method and call it done. The defensible pattern is:
- Use multiple estimators (typically at least two from different method families).
- Compare estimates and investigate when they disagree (disagreement is a signal of model misspecification, unmeasured confounding, or both).
- Bound the result with sensitivity analysis (how strong would unmeasured confounding need to be to change the conclusion?).
- Validate, where possible, against randomized-trial benchmarks for known patient subgroups.
- Report estimates with explicit uncertainty, not as point predictions.
Uncertainty Quantification: Why It Is Non-Negotiable Here
Most ML systems hide uncertainty. A classification model predicts a class; a regression model predicts a number. The uncertainty is somewhere in the model's confidence score, but the operational interface is a point output. Treatment response prediction cannot work that way. The clinician needs to know whether the predicted A1c reduction of 1.4 percentage points has a 95 percent confidence interval of 0.8 to 1.9 (a useful, actionable estimate) or a 95 percent confidence interval of minus 0.5 to 3.3 (a wide-enough uncertainty that the estimate may not meaningfully discriminate between this drug and a comparator).
There are several sources of uncertainty in CATE estimation, and a production system distinguishes them:
Sampling uncertainty. The sample size of similar patients is finite. Confidence intervals from bootstrap or analytic estimators (causal forest's built-in CI, Bayesian posterior intervals, sandwich variance estimators for IPTW) capture this.
Model uncertainty. Different correctly-specified estimators may produce different point estimates because they make different bias-variance tradeoffs. Reporting estimates from multiple methods quantifies this. When the methods agree, the CATE is robust; when they disagree, the disagreement is the uncertainty.
Unmeasured confounding. The structural assumption underlying any observational CATE estimate is unconfoundedness (treatment is independent of potential outcomes given measured covariates). This is not an empirically testable assumption. Sensitivity analyses (E-value, Rosenbaum bounds, simulation-based perturbation) bound how much unmeasured confounding could change the conclusion.
Distributional shift. The patient in front of the clinician may not be in the support of the training data. Out-of-distribution flags (low density of similar patients in the training data, propensity scores near 0 or 1, large extrapolation distances) indicate the estimate is being extrapolated rather than interpolated. Estimates flagged as extrapolation should be presented with explicit warnings or suppressed entirely.
Outcome definition. Different operational definitions of the outcome (three-month A1c reduction, six-month A1c reduction, A1c-below-7-at-six-months, time to A1c-below-7) yield different estimates. The reported uncertainty should make the outcome definition explicit; the same drug can look better or worse depending on which outcome the model targets.
Where the Field Has Moved
A few years' worth of progress that matters for production implementations:
- Target trial emulation has become the methodological gold standard for observational treatment-effect studies, with publications in top medical journals demonstrating close correspondence to randomized-trial benchmarks for several drug classes. The Hernán-Robins textbook and the widely-cited diabetes-treatment emulation papers have made the method accessible to applied researchers.
- Doubly-robust meta-learners and causal forests have matured into production-grade libraries (EconML, grf, DoWhy, causalml) with stable APIs and reasonable defaults. The methodological barrier to entry has dropped substantially in the last three to five years.
- Calibration of treatment effect predictions has emerged as a research focus. Work on calibration of CATE estimators (Kuusisto et al., Yadlowsky et al.) is producing methods for ensuring that the predicted treatment effects match observed effects in subgroups, in the same spirit as calibration of probabilistic classifiers.
- FDA's framework for AI/ML-based Software as a Medical Device has evolved through the predetermined change control plan guidance, the Good Machine Learning Practice principles (a joint FDA-Health Canada-MHRA effort), and ongoing regulatory science. Treatment response prediction tools that meet the SaMD definition are in scope.
- Federated and consortium approaches (OHDSI, PCORnet, Sentinel) are producing pooled treatment-effect estimates across multiple healthcare systems with privacy-preserving methods. These pooled estimates address the small-sample problem at single institutions and are increasingly being used as priors or anchors for institution-specific models.
- Phenotyping, particularly through deep representation learning, has improved. Patient phenotyping pipelines that turn raw EHR data into clinically meaningful clusters or embeddings (often using foundation models trained on EHR sequences) make the "similar patient" matching far more meaningful than feature-engineered approaches. The patient embedding becomes the conditioning variable for CATE estimation.
Where LLMs Fit (and Don't)
The same pattern as Recipes 4.5 through 4.7, with treatment-response-specific notes:
- Causal estimation, similar-patient matching, uncertainty quantification, treatment ranking. Not the LLM's job. Deterministic statistical models trained on validated cohorts.
- Clinician-facing treatment-comparison briefings. Yes. A structured-output prompt takes the per-treatment CATE estimates with confidence intervals, the cohort characteristics, the data quality flags, and the patient's clinical context, and produces a paragraph the clinician reads. The briefing surfaces the comparison, the magnitude, the uncertainty, and the caveats. The clinician interprets and decides.
- Patient-facing decision-support summaries (when the clinician chooses to share). Yes, with the same validator pattern as prior recipes. The patient version uses lay-language equivalents (probabilities translated into "patients like you," confidence intervals translated into "we are more confident about A than B"), with reading-level matched to the patient and approved-claim language enforced.
- Free-form clinical reasoning about which drug to choose. No. The LLM does not pick; it packages. The line is the same line as in 4.7. Treatment-response prediction is the highest-stakes recipe in this chapter, so the line is even more important.
- Evidence synthesis and citation. Yes, when the system surfaces guideline references, trial evidence, and consensus statements alongside the model's prediction. The LLM packages a structured evidence summary; the underlying citations come from a curated knowledge source (formulary, UpToDate or similar via licensed integration, society guidelines), not from the LLM's own knowledge.
General Architecture Pattern
The pipeline has six logical components: a treatment-catalog component that maintains the structured representation of treatment options and their comparators, a feature-pipeline component that produces patient features and outcome labels from source data, a causal-modeling component that trains and serves per-treatment-comparator CATE estimators with uncertainty, a cohort-retrieval component that surfaces the similar-patient evidence underlying each estimate, a clinician-facing decision-support component that packages the estimates into a treatment-comparison view at the point of care, and a feedback-and-evaluation component that captures actual outcomes and drives retraining, calibration monitoring, and surveillance.
┌───────── TREATMENT CATALOG (governance-controlled) ───────────┐
│ │
│ [Pharmacy / Therapeutics committee] [Clinical informatics] │
│ [Health economics and outcomes research] [Compliance] │
│ │ │ │ │
│ └──────────┬────────────┴────────┬─────────┘ │
│ ▼ ▼ │
│ [Treatment spec: treatment_id, comparator_id(s), │
│ eligibility predicates, outcome definitions, │
│ target_patient_phenotype, evidence_level, │
│ guideline_references, formulary_status, │
│ supply_constraints, version, effective_dates, │
│ model_risk_tier, cleared_for_decision_support_use] │
│ │ │
│ ▼ │
│ [Persist to treatment-catalog store; versioned; │
│ governance approval required for new entries │
│ and tier promotions] │
│ │
└────────────────────────────────────────────────────────────────┘
┌───────── FEATURE PIPELINE AND COHORT CONSTRUCTION (batch) ────┐
│ │
│ [EHR / FHIR] [Claims] [Pharmacy] [Lab] [Vitals] │
│ [SDOH] [PROMs] [Patient registries] │
│ │ │
│ ▼ │
│ [Phenotype computation: condition flags, │
│ severity scores, comorbidity profile, │
│ disease trajectory features] │
│ │ │
│ ▼ │
│ [Outcome label construction per treatment │
│ and outcome definition: timed clinical │
│ response, persistence, discontinuation, │
│ adverse event, downstream utilization] │
│ │ │
│ ▼ │
│ [Cohort construction: index date, │
│ washout window, eligibility filtering, │
│ treatment exposure assignment, censoring] │
│ │ │
│ ▼ │
│ [Persist patient features to feature store; │
│ persist cohorts to evaluation lake] │
│ │
└────────────────────────────────────────────────────────────────┘
┌───────── CAUSAL MODELING (offline, scheduled retrain) ────────┐
│ │
│ [Cohorts] [Patient features] [Outcome labels] │
│ [Treatment catalog] │
│ │ │ │ │
│ └──────────┬─────┴────────┬─────────┘ │
│ ▼ ▼ │
│ [Stage A: target trial emulation specification │
│ per treatment-comparator pair] │
│ │ │
│ ▼ │
│ [Stage B: propensity score model │
│ (treatment assignment given covariates)] │
│ │ │
│ ▼ │
│ [Stage C: outcome model │
│ (outcome given covariates and treatment)] │
│ │ │
│ ▼ │
│ [Stage D: CATE estimator ensemble │
│ (causal forest, DR-learner, BART or │
│ equivalent; per treatment-comparator pair)] │
│ │ │
│ ▼ │
│ [Stage E: uncertainty quantification │
│ (sampling CI, model-agreement CI, │
│ sensitivity-analysis bounds, OOD flag)] │
│ │ │
│ ▼ │
│ [Stage F: calibration assessment │
│ per cohort and overall; calibration plot │
│ per treatment-comparator pair] │
│ │ │
│ ▼ │
│ [Stage G: governance gate │
│ (medical-director sign-off, model registry │
│ promotion, change-control plan adherence)] │
│ │ │
│ ▼ │
│ [Persist model artifacts to versioned registry; │
│ archive training cohort for reproducibility] │
│ │
└────────────────────────────────────────────────────────────────┘
┌───────── COHORT RETRIEVAL AND SCORING (on-demand or batch) ───┐
│ │
│ [Index patient features] [Treatment catalog] │
│ [Production CATE models] [Cohort index] │
│ │ │
│ ▼ │
│ [Determine eligible treatments and │
│ comparators for this patient] │
│ │ │
│ ▼ │
│ [Per-treatment-comparator pair: invoke CATE │
│ model, retrieve similar-patient cohort, │
│ compute uncertainty, compute OOD flag] │
│ │ │
│ ▼ │
│ [Persist scoring result; emit │
│ scoring-event record for audit] │
│ │
└────────────────────────────────────────────────────────────────┘
┌───────── CLINICIAN-FACING DECISION SUPPORT ───────────────────┐
│ │
│ [Scoring result] [Patient summary] [Guideline references] │
│ │ │
│ ▼ │
│ [Structured comparison view │
│ (per-treatment estimate, CI, cohort size, │
│ cohort match quality, outcome definition, │
│ evidence level, formulary status)] │
│ │ │
│ ▼ │
│ [LLM-generated comparison briefing │
│ (validated, no recommendation language, │
│ explicit uncertainty, explicit caveats)] │
│ │ │
│ ▼ │
│ [Clinician review at point of care; │
│ clinician records decision and rationale; │
│ optional patient-facing summary generation] │
│ │
└────────────────────────────────────────────────────────────────┘
┌───────── FEEDBACK, OUTCOMES, SURVEILLANCE ────────────────────┐
│ │
│ [Clinician decision] [Patient outcomes over time] │
│ [Adverse events] [Post-deployment events] │
│ │ │
│ ▼ │
│ [Match prediction to actual outcome; │
│ compute prediction error per estimator, │
│ per cohort, per treatment] │
│ │ │
│ ▼ │
│ [Calibration drift detection; │
│ cohort-stratified performance monitoring; │
│ adverse-event pattern surveillance] │
│ │ │
│ ▼ │
│ [Trigger retraining, model recalibration, │
│ or model retirement; report to governance] │
│ │
└────────────────────────────────────────────────────────────────┘
The treatment catalog is governance, not engineering. A treatment is more than a drug name. It is a structured spec that includes: the treatment identifier (typically RxNorm for drugs, CPT for procedures), the comparators against which the model will estimate effects (this is the most important and most-overlooked design decision), the eligibility predicates that define who is in the analysis cohort, the outcome definitions (primary and secondary, with explicit timing), the target patient phenotype the model is most useful for, the evidence level (randomized-trial backed, observational only, mixed), the formulary status, the supply constraints if any, and a model-risk tier that reflects how high-stakes the prediction is. Pharmacy and Therapeutics committee, clinical informatics, health economics and outcomes research, and compliance jointly own the catalog. New treatments and tier promotions require sign-off.
Feature pipeline and cohort construction is the unsexy work that determines whether the rest works. Phenotype computation turns raw EHR data into clinically meaningful flags (active heart failure, NYHA class, eGFR trajectory). Outcome labels are constructed from longitudinal data with explicit timing (three-month A1c, twelve-month total cost, time-to-discontinuation, time-to-hospitalization). Cohort construction implements the target trial emulation: index date is when the treatment decision was made, washout window excludes patients with relevant exposures in a defined prior period, eligibility filtering applies the treatment catalog's predicates, exposure assignment determines which treatment arm each patient is in, censoring rules handle dropouts and competing events. The output is a patient-features dataset and a cohorts dataset that the modeling stage consumes.
Causal modeling is the methodologically heavy stage. A target trial protocol is specified per treatment-comparator pair. A propensity score model estimates the probability of receiving each treatment given covariates; this is the foundation for the doubly-robust methods and for the IPTW alternative. An outcome model estimates the outcome given covariates and treatment; this is the foundation for the meta-learner methods. The CATE estimator ensemble runs at least two estimators from different method families on the same cohort; disagreement among estimators is a signal of model misspecification or unmeasured confounding. Uncertainty quantification combines sampling uncertainty (from the model's confidence intervals), model uncertainty (from estimator disagreement), and structural uncertainty (from sensitivity analyses). Calibration assessment checks that predicted treatment effects match observed effects in held-out cohorts and in subgroups. The governance gate is a human review of the model's performance, fairness, and calibration before promotion to production; this is not optional and should not be a rubber stamp.
Cohort retrieval and scoring is the on-demand or batch path. For an index patient, the system determines eligible treatments and comparators from the catalog, invokes the CATE model for each treatment-comparator pair, retrieves the similar-patient cohort underlying the estimate, computes uncertainty, and flags out-of-distribution cases where the patient is not well-represented in the training data. The output is a structured scoring result: per-treatment-comparator pair, the CATE estimate, the confidence interval, the cohort size and match quality, the OOD flag, and the calibration status.
Clinician-facing decision support is the part the clinician sees. The structured scoring result is rendered as a comparison view: per treatment, the predicted outcome change, the confidence interval, the cohort size and demographic match, the outcome definition (so the clinician knows what is being predicted), the evidence level, and the formulary status. An LLM-generated briefing provides a paragraph synthesis. The validator enforces strict rules: no recommendation language ("the patient should be prescribed X"), no overstatement of certainty, explicit acknowledgment of unmeasured confounding, and required caveats about the analysis assumptions. The clinician reviews and decides; the system records the decision and the clinician's rationale for audit.
Feedback, outcomes, and surveillance is what turns this from a research demo into a production system. Each prediction is matched to the patient's actual subsequent outcome (where observable). Prediction errors are computed per estimator, per cohort, per treatment. Calibration drift detection flags when predictions diverge from outcomes over time. Cohort-stratified performance monitoring flags when the model's accuracy differs across protected populations. Adverse-event surveillance flags when patients who received the recommended treatment had unexpected adverse events at higher-than-expected rates. The output is trigger signals for retraining, recalibration, or model retirement. Post-deployment surveillance is a regulatory expectation for SaMD-class tools and an ethical expectation regardless of regulatory classification.
Equity instrumentation is non-negotiable. Calibration parity across cohorts (the model is equally accurate for each demographic subgroup). Estimate parity at clinical equipoise (when the model's prediction is in a clinically narrow band, it should not systematically point one direction for one cohort and another direction for another). Adverse-event parity (recommended treatments do not produce disproportionate adverse events in any cohort). Each axis is monitored, with thresholds that trigger committee review when crossed. The Obermeyer-style failure mode is the canonical concern; for treatment recommendations specifically, biased predictions can directly cause inferior care, so the instrumentation has to be built in from the beginning.
Regulatory posture is set early. The treatment-catalog model-risk tier determines the regulatory pathway. Tier-1 (low risk, well-evidenced, advisory only, clinician fully reviews basis) may qualify for the 21st Century Cures Act exemption from FDA SaMD regulation. Tier-2 and above are likely SaMD and require a regulatory plan: predetermined change control plan, Good Machine Learning Practice adherence, postmarket surveillance, complaint handling, and quality system documentation. Decide the tier early; retrofitting regulatory compliance onto a system not designed for it is expensive.
The AWS build lives in a companion page. This recipe covers the problem, the underlying technology, and the vendor-agnostic architecture. For the AWS services, architecture diagram, prerequisites, and the step-by-step pseudocode walkthrough, see the Architecture and Implementation companion. The Python example is linked from there.
The Honest Take
Treatment response prediction is the recipe in this chapter where the gap between "the model produces a number" and "the number reliably informs a better decision" is widest. Every plan, every health system, and every research group has, at some point, trained a regression model on observational treatment data, predicted outcomes by treatment arm, computed differences, and called it treatment response prediction. The number is a number. The number, in most cases, conflates the drug effect with the selection effect with the confounding-by-indication effect with the survivorship-bias-in-the-cohort effect. The architecture in this recipe is largely about the discipline that distinguishes a defensible CATE estimate from that confused number. Most of that discipline is not AWS-specific. Most of it is methodological choices made well, governance applied seriously, and uncertainty communicated honestly. The cloud infrastructure is comparatively easy.
The trap most specific to this domain is the difference between average treatment effect and individualized treatment effect. The average treatment effect for GLP-1 versus SGLT2 in a population is a well-defined quantity that can be reasonably estimated from observational data with appropriate methods. The individualized treatment effect for Marcus is, in a strong technical sense, unobservable: Marcus will take one drug, and the counterfactual outcomes on the other drugs are forever invisible. What we estimate is the conditional average treatment effect given Marcus's covariates. The "individualized" label is aspirational; the conditional-average framing is what the math delivers. The CATE is useful, often very useful, and it is not the same thing as a personalized prediction. The briefing has to communicate that distinction. A briefing that says "for you, GLP-1 will lower A1c by 1.4 percentage points" is overstating what the model knows. A briefing that says "for patients similar to you, the average A1c reduction on GLP-1 was 1.4 percentage points greater than on SGLT2, with this confidence interval and these caveats" is the truth, as best the system can tell. The difference looks small. It is everything.
A trap I keep seeing fresh teams fall into: treating the CATE estimator as a black-box predictor and skipping the cohort overlap diagnostic. Propensity overlap is a precondition for credible CATE estimation: if the treated and comparator cohorts have substantially non-overlapping covariate distributions, the estimator is extrapolating across the distribution gap, and the resulting estimates are dominated by modeling assumptions rather than data. The overlap check in the propensity-model training stage is short. It is also the difference between an honest output and a confidently wrong one. Make it a hard gate, not a warning.
Another trap, related: trusting estimator agreement as the sole signal of robustness. Estimators from the same method family (different XGBoost hyperparameter settings, different causal forest random seeds) will agree because they are making correlated assumptions. Estimator agreement across method families (causal forest plus DR-learner plus BART) is more meaningful but still does not address shared blind spots like unmeasured confounding. Sensitivity analyses are the structural-uncertainty check that estimator agreement does not provide. The combination of agreement, calibration, and sensitivity analysis is the closest the methodology gets to honest-confidence. None of them alone is enough.
The thing that surprises people coming from generic ML backgrounds is how much of the work is methodological rather than engineering. The propensity model, the outcome model, the CATE ensemble, the calibration tests, the sensitivity analyses, the fairness tests; each requires a methodological choice that has consequences. The default choices in EconML are reasonable starting points; they are not durable production choices. Investing in causal-inference depth on the team (a methodologist or a methodologist-trained data scientist) is the highest-leverage staffing decision you will make. The ML engineering is comparatively easy; the causal inference is hard, and the gap between methodologically-light and methodologically-rigorous CATE estimates is the gap between a system that informs better decisions and a system that confidently produces noise.
The thing about LLMs in this recipe: they are useful for packaging the comparison, and they are dangerous if not constrained. The validator's no-recommendation-language rule is non-negotiable. A clinician reading a briefing that subtly recommends a treatment is a clinician acting on a recommendation that the underlying model is not designed to make. The validator's pattern matching for recommendation phrasing should be aggressive, the regeneration loop should be tight, and the templated fallback should be deterministic and always-passing. The fallback is less readable; that is acceptable. A less readable templated comparison view that is faithful to the data is better than an LLM paragraph that is more readable but selects a treatment.
The thing I would do differently the second time: invest in the surveillance pipeline before broadening the catalog. The pattern many teams follow is to launch with a well-evidenced, well-calibrated treatment-comparator pair, see initial success, and expand the catalog quickly to ten or twenty pairs. The expanded catalog produces more recommendations, but the surveillance infrastructure is undersized for the increased volume. Calibration drift detection lags; cohort-stratified fairness monitoring is partial; adverse-event surveillance is underpowered. The system looks like it is delivering value, and the lagging surveillance is producing the appearance of stability rather than confirmed stability. The discipline is to expand the catalog only as fast as the surveillance infrastructure can support, not as fast as the modeling team can train new models. Treat the catalog growth rate as a function of the surveillance capacity, not as a function of demand.
The thing about clinician overrides: override patterns are diagnostic information about the model, not a problem to be reduced. A clinician who looks at the briefing and chooses a different treatment is doing exactly what the system is designed to support: independent clinical judgment informed by the prediction. A high override rate is not a model failure; it may indicate a model that is producing useful additional context that the clinician then incorporates with their broader clinical knowledge. The override-rate dashboard is a tool for understanding where the model is most and least informative, not a tool for pressuring clinicians toward agreement. The pattern that fails is treating override rate as a metric to minimize; that pressure produces clinicians who rubber-stamp the recommendation, which defeats the entire purpose.
The thing about fairness: cohort-stratified calibration is the floor, not the ceiling. A model that is equally well-calibrated across cohorts is a necessary condition; it is not sufficient. Even a perfectly-calibrated-across-cohorts model can produce systematically different recommendations to different cohorts because the cohorts have systematically different covariates that drive the underlying clinical effect. That can be appropriate (different patients have different optimal treatments). It can also be inappropriate (covariates that correlate with race or socioeconomic status may be driving recommendations in ways that reflect historical access disparities rather than clinical reality). Distinguishing the two requires deeper analysis than a calibration plot. Plan for fairness analysis as a methodological discipline, not a metric.
A trap worth flagging: confusing accuracy with usefulness. A model that predicts a 0.5 percentage point A1c difference between two treatments with a confidence interval of plus-or-minus 0.4 percentage points is statistically significant and clinically marginal. A clinician glancing at the briefing may conclude "the difference is small enough that other factors dominate," which is exactly the right conclusion. The model is not failing; the model is correctly indicating that the comparison is not strongly discriminating. The briefing should explicitly support that conclusion when it applies. A common failure mode is a briefing that emphasizes the statistical significance and downplays the clinical-effect-size implication; the clinician then chooses based on the headline rather than the nuance. The validator's uncertainty-completeness rule is partial protection; the briefing-design discipline of stating clinical-effect-size context alongside statistical significance is the substantive fix.
Another trap: the seductive simplicity of patient embeddings. Foundation-model patient embeddings are increasingly available and produce a single vector representation of a patient that can be used as the conditioning variable for CATE estimation. The embeddings are real progress and the methodological convenience is genuine. The trap is treating the embedding as fully sufficient and discarding the careful feature-engineering and outcome-construction work that the recipe describes. The embedding compresses the patient's clinical state, which is useful for similarity-based retrieval and for nonparametric estimators; it does not substitute for the explicit definition of confounders, the explicit specification of the target trial, or the explicit construction of outcome labels. Use the embedding as a useful additional input, not as a replacement for the methodological discipline.
Last point, because it is specific to this use case: treatment response prediction tools, even when they are technically advisory, change clinical practice. A clinician with the briefing in front of them is a different decision-maker than a clinician without it. The change can be in the direction the system designers intended (better-informed decisions, better outcomes for the patient population). The change can also be in directions the system designers did not intend: clinicians anchoring on the model's point estimate at the expense of clinical context; patients perceiving the recommendation as authoritative when the clinician shares it; downstream second-opinion processes deferring to the model's view. None of those are model failures in a narrow sense. All of them are system failures in the broader sense that the system is changing practice in ways that were not part of the design. Build the system as if it will change practice, because it will. Document the intended changes. Monitor for the unintended changes. Be willing to ship the system back to a more limited scope if the unintended changes outweigh the intended ones. The hardest decision in this work is not whether to ship the model. It is whether to keep it shipped after watching what it actually does.
Related Recipes
- Recipe 4.4 (Wellness Program Recommendations): The uplift pattern from 4.4 is methodologically related (CATE estimation), with simpler outcomes and lower clinical stakes. Practitioners often build 4.4 as a methodological warmup for 4.8.
- Recipe 4.5 (Medication Adherence Intervention Targeting): Predicted adherence from 4.5 feeds the 4.8 prediction's caveat about adherence assumptions; a treatment response prediction that assumes full adherence overestimates effect for patients predicted to have low adherence.
- Recipe 4.6 (Care Gap Prioritization): The multi-pathway orchestration pattern from 4.6 informs the multi-treatment-pair scoring in 4.8.
- Recipe 4.7 (Care Management Program Enrollment): The methodological foundation (per-pair causal estimation, calibration, fairness, governance) is shared. 4.7 applies it to program enrollment with capacity constraints; 4.8 applies it to treatment selection with regulatory considerations.
- Recipe 4.9 (Personalized Care Plan Generation): The CATE estimates from 4.8 are key inputs to the broader care plan in 4.9. Care plans synthesize across multiple treatment decisions, lifestyle interventions, and care-coordination steps; the per-treatment-pair predictions are the building blocks.
- Recipe 4.10 (Dynamic Treatment Regime Recommendation): Extends 4.8 to sequential decision-making with reinforcement-learning-style estimation. 4.8 is the static counterpart; 4.10 is the sequential extension.
- Recipe 2.x (LLM / Generative AI): The clinician-facing briefing and the patient-facing summary use LLM techniques developed across Chapter 2; the validator pattern from 2.5 (After-Visit Summary) applies directly here.
- Recipe 7.x (Predictive Analytics / Risk Scoring): Risk-stratification scores from Chapter 7 feed the patient-feature pipeline. Some of the same modeling discipline (calibration, fairness, validation) applies, with the additional causal-inference layer that 4.8 requires.
- Recipe 12.x (Time Series Analysis / Forecasting): Disease-trajectory forecasting from Chapter 12 produces features that feed CATE estimation; the longitudinal patterns that predict outcome trajectories are also predictors of treatment response.
- Recipe 13.x (Knowledge Graphs): The treatment catalog, with relationships between treatments (substitutable, sequential, complementary) and links to guideline references and trial evidence, is naturally modeled as a knowledge graph at higher sophistication levels.
- Recipe 15.x (Reinforcement Learning): Sequential treatment decisions with long-horizon outcomes are a reinforcement-learning problem at the most sophisticated level. 4.10 sits at the boundary; full RL formulations cross into Chapter 15.
Tags
personalization · treatment-response-prediction · causal-inference · heterogeneous-treatment-effects · cate-estimation · target-trial-emulation · propensity-score · meta-learners · causal-forest · bart · doubly-robust · sensitivity-analysis · calibration · equity · cohort-analysis · fda-samd · clinical-decision-support · smart-on-fhir · cds-hooks · bedrock · sagemaker · feature-store · dynamodb · step-functions · lambda · healthlake · complex · research-to-production · hipaa
← Recipe 4.7: Care Management Program Enrollment · Chapter 4 Preface · Next: Recipe 4.9 - Personalized Care Plan Generation →