Recipe 7.8: Disease Progression Modeling

Complexity: Complex · Phase: Research/Production · Estimated Cost: ~$2,500-$8,000/month (model training + inference at scale)

The Problem

A nephrologist is looking at a patient with Stage 3a chronic kidney disease. The eGFR is 52. It was 58 a year ago, 63 two years before that. The question that matters most to this patient, to their family, and to the care team is: how fast is this going to get worse? Will they need dialysis in two years? Five? Ten? Should we escalate treatment now, or is this a slow decline that can be managed conservatively for another decade?

Right now, the answer is mostly clinical intuition. The nephrologist has seen hundreds of CKD patients. They know the general trajectory. But "general trajectory" is not the same as "this patient's trajectory." Two patients with identical eGFR values today can have wildly different outcomes depending on their diabetes control, blood pressure management, medication adherence, genetic factors, and a dozen other variables that interact in ways no human can reliably compute across a multi-year horizon.

This isn't unique to kidney disease. Diabetes progresses from controlled to uncontrolled to complications. Heart failure moves through stages. COPD declines. Multiple sclerosis relapses and remits. Parkinson's advances. In every case, the clinical question is the same: given everything we know about this patient right now, what does their future look like?

The stakes are enormous. Intervene too early and you subject patients to aggressive treatments they didn't need yet (with side effects, costs, and quality-of-life impacts). Intervene too late and you've missed the window where treatment could have slowed progression. The sweet spot is knowing, with reasonable confidence, where a patient is headed so you can time interventions appropriately.

Disease progression modeling is the ML approach to this problem. It takes longitudinal patient data (labs, vitals, medications, diagnoses, procedures over time) and learns the patterns that predict how a disease will evolve for a specific individual. Not population averages. Individual trajectories.

It's also one of the hardest problems in healthcare ML, and the reasons why are worth understanding before you build anything.

The Technology: Modeling How Diseases Evolve

Why This Is Fundamentally Hard

Disease progression modeling sits at the intersection of several difficult ML problems, and the combination is what makes it genuinely complex.

Long time horizons. You're predicting months to years into the future, not hours or days. Every additional month of prediction horizon compounds uncertainty. A model that's well-calibrated at 6 months might be useless at 36 months. The further out you predict, the more external factors (new medications, lifestyle changes, comorbidities) can alter the trajectory.

Treatment effects confound observation. Here's the fundamental paradox: you're trying to predict what will happen to a patient, but the treatments they receive change what happens. A patient whose CKD was progressing rapidly might have been put on an ACE inhibitor that slowed the decline. If you train naively on observational data, you'll learn that "patients who got ACE inhibitors progressed slowly," which conflates the treatment effect with the natural disease course. Separating what the disease would have done from what treatment made it do is a causal inference problem, and it's genuinely hard.

Irregular observation intervals. Patients don't show up for labs on a regular schedule. A stable patient might have labs every 6 months. A deteriorating patient might have labs every 2 weeks. The observation frequency itself is informative (more frequent visits often signal clinical concern), and your model needs to handle irregular time series gracefully. Standard time series models that expect evenly-spaced observations don't work here without significant adaptation.

Competing risks and multi-state transitions. A CKD patient doesn't just progress linearly from Stage 3 to Stage 5. They might stabilize. They might improve temporarily. They might develop a cardiovascular event that changes everything. They might die from something unrelated. Disease progression is really a multi-state model with transitions between states, and some transitions are absorbing (you don't come back from dialysis initiation). Modeling this correctly requires thinking about competing risks, not just a single outcome.

Censoring. Many patients in your training data haven't reached the endpoint yet. A patient with 3 years of CKD data who hasn't progressed to Stage 4 isn't a "non-progressor." They might progress next year. This is right-censoring, and it's the same problem that survival analysis was invented to handle. Ignoring it (treating censored patients as non-events) biases your model toward optimism.

Measurement bias in your primary biomarker. If you're modeling CKD progression, eGFR is almost certainly your central feature. But eGFR itself carries a history of measurement bias. Until 2021, the standard CKD-EPI equation included a race coefficient that systematically overestimated kidney function in Black patients, potentially delaying referrals and transplant evaluations. The NKF/ASN Task Force recommended adopting the 2021 race-free CKD-EPI creatinine equation, and most labs have transitioned. But your training data likely spans years where the race-adjusted formula was standard. If you train on historical eGFR values without accounting for which equation generated them, you're encoding that bias into your model's learned trajectories. The fix: identify which eGFR equation was used for each historical measurement (check lab metadata), recalculate where possible using the 2021 race-free equation, and evaluate your trained model's performance stratified by race, sex, and age group. Disparate accuracy across subgroups isn't just a fairness concern; it's a signal that your model learned from biased input data.

The Modeling Approaches

There are several families of models used for disease progression, each with different strengths:

Joint longitudinal-survival models. These simultaneously model the trajectory of a biomarker (like eGFR over time) and the time to an event (like dialysis initiation). The biomarker trajectory informs the event risk, and the event risk accounts for the fact that some trajectories are cut short. This is the classical statistical approach, well-understood and interpretable, but it struggles with high-dimensional feature spaces and complex nonlinear relationships.

Hidden Markov models (HMMs) and multi-state models. These represent disease as a sequence of discrete states with probabilistic transitions. A patient is "in" a state (e.g., CKD Stage 3a) and has some probability of transitioning to adjacent states (Stage 3b, or back to Stage 2, or to death) at each time step. The "hidden" part means the true disease state might not be directly observable; you infer it from noisy measurements. These are elegant for diseases with clear staging but struggle when progression is continuous rather than discrete.

Recurrent neural networks and temporal models. LSTMs, GRUs, and transformer-based architectures can learn complex temporal patterns from sequential patient data. They handle irregular time intervals (with appropriate encoding), capture nonlinear relationships, and can incorporate high-dimensional feature sets. The tradeoff: they're less interpretable, require more data, and can overfit on small cohorts. They also don't naturally handle censoring without custom loss functions.

Gaussian process models. These provide a probabilistic framework for modeling continuous trajectories with uncertainty quantification built in. They handle irregular observations naturally and produce confidence intervals that widen as you predict further into the future (which is honest). They're computationally expensive for large datasets but excellent for individual patient trajectory modeling.

Survival analysis extensions. Cox proportional hazards models, accelerated failure time models, and their modern deep learning extensions (DeepSurv, Deep Recurrent Survival Analysis) handle censoring natively and can model time-to-event outcomes. They're the right foundation when your primary question is "when will this patient reach a specific milestone?"

In practice, production systems often combine approaches: a temporal model for trajectory prediction paired with a survival model for milestone timing, with uncertainty quantification layered on top.

Handling Treatment Effects

This deserves its own section because it's where most naive implementations fail.

If you train a model on observational data without accounting for treatment, you'll learn correlations that don't reflect causal disease progression. Patients who received aggressive treatment will appear to have better outcomes, but that's the treatment working, not the disease being milder.

There are several approaches to this:

Marginal structural models use inverse probability of treatment weighting to create a pseudo-population where treatment assignment is independent of confounders. This lets you estimate what would have happened without treatment.

G-computation models the outcome under different treatment scenarios explicitly, allowing you to predict progression under the patient's current treatment plan versus alternatives.

Causal forests and heterogeneous treatment effect estimation learn how treatment effects vary across patient subgroups, which is useful for identifying who benefits most from escalation.

Simpler approaches include conditioning on treatment as a feature (acknowledging that your predictions are "given current treatment continues") or building separate models for treated and untreated populations. These are less rigorous but more practical for a first implementation.

The honest answer: perfectly separating disease progression from treatment effects requires randomized trial data or very careful causal inference methodology. Most production systems take the pragmatic approach of conditioning on current treatment and being transparent about that assumption.

Uncertainty Quantification

This is non-negotiable for clinical use. A point prediction of "eGFR will be 38 in two years" is less useful (and potentially dangerous) than "eGFR will likely be between 32 and 44 in two years, with 80% confidence." Clinicians need to understand the range of possible futures, not just the most likely one.

Approaches include:

Prediction intervals from Gaussian processes or Bayesian models
Monte Carlo dropout in neural networks (approximate Bayesian inference)
Quantile regression (predicting the 10th, 50th, and 90th percentile outcomes)
Ensemble disagreement (training multiple models and measuring how much they disagree)

The uncertainty should grow with prediction horizon. If your model is equally confident about next month and next year, something is wrong.

General Architecture Pattern

[Longitudinal Data Assembly] → [Feature Engineering] → [Model Training] → [Individual Prediction] → [Clinical Integration]

Longitudinal Data Assembly. Gather the patient's full temporal record: labs over time, vitals over time, medications (start/stop dates, dosages), diagnoses (onset dates), procedures, and relevant social/demographic factors. This is not a single snapshot; it's a time-indexed sequence. The assembly step must handle data from multiple source systems (EHR, claims, labs, pharmacy) and align them on a common timeline.

Feature Engineering. Transform raw temporal data into model-ready features. This includes: rate of change (is eGFR declining, and how fast?), variability (is blood pressure stable or swinging?), treatment history encoding (what medications, for how long, at what doses?), comorbidity burden (how many other conditions are active?), and time-since-last-observation (how stale is our information?). For deep learning approaches, you might feed raw sequences directly, but even then, engineered features often improve performance.

Model Training. Train on a retrospective cohort with sufficient follow-up. The training data must include patients who progressed and patients who didn't (or who were censored). Handle class imbalance (most patients progress slowly), censoring (many patients haven't reached the endpoint), and treatment confounding (patients who progressed may have received different treatments). Validate on a held-out temporal cohort (train on patients from 2015-2020, validate on 2021-2023) to avoid data leakage.

Individual Prediction. Given a new patient's history up to today, generate a predicted trajectory with uncertainty bounds. This should include: predicted biomarker values at future time points, probability of reaching specific milestones (e.g., Stage 4, dialysis) within specific time windows, and confidence intervals that honestly reflect uncertainty.

Clinical Integration. Surface predictions where clinicians make decisions. This means integration with the EHR workflow, not a standalone dashboard that nobody checks. Include explanations of what's driving the prediction (which factors are accelerating or decelerating progression) and clear communication of uncertainty. Provide actionable thresholds: "if progression continues at this rate, the patient will reach Stage 4 within 18 months, suggesting nephrology referral now."

The AWS build lives in a companion page. This recipe covers the problem, the underlying technology, and the vendor-agnostic architecture. For the AWS services, architecture diagram, prerequisites, and the step-by-step pseudocode walkthrough, see the Architecture and Implementation companion. The Python example is linked from there.

The Honest Take

Disease progression modeling is one of those problems where the concept is intuitive, the clinical value is obvious, and the implementation is humbling. Here's what I've learned:

The data problem is bigger than the model problem. You'll spend 70% of your time assembling clean longitudinal data and 30% on the actual modeling. Patient records are fragmented across systems, labs are coded inconsistently, medication histories have gaps, and "lost to follow-up" might mean the patient moved, died, or switched providers. Getting a clean training cohort with reliable outcomes is the hard part.

Treatment confounding will haunt you. Your first model will look great on validation metrics and then a nephrologist will point out that it's basically predicting "patients who got aggressive treatment did better," which is obvious and not useful. Accounting for treatment effects properly requires either causal inference expertise or very careful framing of what your model actually predicts ("progression given current treatment continues").

Clinicians will ask questions your model can't answer. "What if we add this medication?" "What if the patient loses 20 pounds?" These are counterfactual questions, and a standard predictive model doesn't answer them. You need causal models or simulation-based approaches for "what if" scenarios, and those are a significant step up in complexity.

Calibration matters more than discrimination. A model with a C-index of 0.75 that's well-calibrated (when it says 60% risk, 60% of patients actually progress) is more clinically useful than a model with a C-index of 0.80 that's poorly calibrated. Clinicians make decisions based on the probability values, not the ranking.

The uncertainty bounds are the product, not the point estimate. I cannot stress this enough. A clinician who sees "42% probability of progression" will treat it as a fact. A clinician who sees "somewhere between 25% and 60% probability" will appropriately factor in their own clinical judgment. Wide uncertainty bounds are honest, not a failure.

Model drift is real and faster than you'd expect. Treatment guidelines change. New medications become available. Coding practices shift. A model trained on 2018-2022 data will start degrading by 2024 as the population and treatment landscape evolve. Plan for quarterly retraining from day one.

Recipe 7.5 (30-Day Readmission Risk): Shorter-horizon prediction using similar longitudinal features but different outcome definition and clinical integration pattern
Recipe 7.6 (Rising Risk Identification): Complementary approach focused on rate-of-change detection rather than absolute trajectory prediction
Recipe 12.8 (Disease Progression Trajectory Modeling): Time series perspective on the same problem, focusing on temporal modeling techniques
Recipe 6.4 (Disease Severity Stratification): Provides the staging framework that progression models predict transitions between
Recipe 4.8 (Treatment Response Prediction): Related problem of predicting how a patient will respond to a specific intervention