Recipe 15.4: Sepsis Treatment Optimization

Complexity: Medium · Phase: Research/Pilot · Estimated Cost: ~$2,000-$8,000/month (training infrastructure)

The Problem

Sepsis is one of the leading causes of death in US hospitals. Roughly 1.7 million adults develop sepsis in the US each year, and somewhere between 250,000 and 350,000 of them die. The mortality rate varies wildly depending on how quickly treatment starts and how well it's managed in those critical first hours. The treatment decisions are sequential, interdependent, and time-sensitive. A clinician managing a septic patient in the ICU is making dozens of decisions per hour. How much IV fluid? Which vasopressor, and at what dose? When to start antibiotics, and which ones? When to escalate, when to hold steady, when to back off.

There's no single "right answer" for sepsis management. The Surviving Sepsis Campaign guidelines provide a framework, but within that framework there's enormous variation in practice. Two equally skilled intensivists will manage the same patient differently. Some of that variation is justified (patient-specific factors), and some of it isn't (habit, training bias, cognitive load at 3 AM). Studies have shown that adherence to sepsis bundles varies from 30% to 70% across institutions, and that variation correlates with mortality differences.

The question that's been driving a decade of research: can we learn, from the thousands of sepsis cases already treated, a treatment policy that would have produced better outcomes than what clinicians actually did? Not replacing clinicians. Augmenting them. Identifying patterns in the data that suggest "patients like this one tend to do better when you give more fluid earlier" or "backing off vasopressors sooner in this patient profile reduces organ damage."

This is a reinforcement learning problem. The patient's physiological state evolves over time. Each treatment decision changes that trajectory. The goal is to learn a policy (a mapping from patient states to treatment actions) that maximizes some measure of patient outcome. And it's one of the most studied RL applications in healthcare, which means we know a lot about both the promise and the pitfalls.

The Technology: Reinforcement Learning for Sequential Medical Decisions

What Is Reinforcement Learning?

Reinforcement learning (RL) is a framework for learning optimal decision-making in sequential settings. Unlike supervised learning (where you have labeled examples of correct answers), RL learns from the consequences of actions. An agent observes a state, takes an action, receives a reward signal, transitions to a new state, and repeats. Over many episodes, the agent learns which actions in which states lead to the best cumulative reward.

The core components:

State (s): A representation of the current situation. In sepsis, this is the patient's physiological status at a given time point: vital signs, lab values, fluid balance, current medications, time since admission.
Action (a): The decision to be made. In sepsis, this is typically discretized into treatment choices: fluid volume, vasopressor dose, antibiotic selection.
Reward (r): A signal indicating how good the outcome was. In sepsis, this is usually tied to survival, organ function scores, or ICU length of stay.
Policy (π): The learned mapping from states to actions. This is what we're trying to optimize.
Value function (V or Q): An estimate of the expected cumulative future reward from a given state (or state-action pair). The policy is derived from this.

What we're optimizing - find the policy π that maximizes expected cumulative reward. The math - π* = argmax E[Σ γ^t * r_t], where γ discounts future rewards (we care about long-term survival, but prefer getting there sooner).

Offline RL: Learning from Historical Data

The critical constraint in healthcare is that you cannot explore freely. In a video game, an RL agent can try random actions and learn from failures. In an ICU, you cannot randomly withhold fluids from a septic patient to see what happens. This means we must use offline reinforcement learning (also called batch RL). The agent learns entirely from historical data: records of what clinicians actually did, what happened to the patient, and what the outcome was.

Offline RL uses a dataset of trajectories, sequences of (state, action, reward, next_state) tuples collected under some historical behavior policy (whatever the clinicians actually did). The goal is to learn a policy that would perform better than the historical behavior, without ever actually deploying that policy on real patients during training.

This introduces a fundamental challenge called distribution shift. The agent is learning about actions it never actually took. If the historical data shows that clinicians always gave high-dose vasopressors to patients with MAP below 65, the agent has no data about what would have happened if they hadn't. Estimating the value of untaken actions from observational data is statistically treacherous.

The main offline RL algorithms used in sepsis research:

Fitted Q-Iteration (FQI). Iteratively estimates the Q-function (expected reward for each state-action pair) using regression. Simple, well-understood, but can diverge if the function approximator is too expressive.

Conservative Q-Learning (CQL). Adds a penalty for overestimating the value of actions that are underrepresented in the data. This addresses the distribution shift problem by being pessimistic about unfamiliar actions. If the data doesn't show what happens when you give zero fluids to a hypotensive patient, CQL won't optimistically assume it works out.

Batch-Constrained Q-Learning (BCQ). Restricts the learned policy to only select actions that are similar to what was observed in the data. If clinicians never gave a particular drug combination, BCQ won't recommend it, even if the Q-function suggests it might be good.

Decision Transformer. A more recent approach that frames RL as a sequence modeling problem. Instead of learning value functions, it learns to predict actions conditioned on desired outcomes. "Given that I want the patient to survive, what sequence of actions should I take from this state?"

The MDP Formulation for Sepsis

The standard formulation (following the influential work by Komorowski et al., 2018) discretizes the problem:

State space. Patient features at each time step (typically every 4 hours): vital signs (heart rate, blood pressure, temperature, respiratory rate, SpO2), lab values (lactate, creatinine, bilirubin, platelets, WBC, pH, PaO2/FiO2), fluid balance (cumulative input/output), SOFA score components, demographics, and time since ICU admission. These are often clustered into discrete states (e.g., 750 states using k-means clustering on the feature vectors) or used as continuous features with neural network function approximators.

Action space. Typically discretized into a grid of (IV fluid volume, vasopressor dose) combinations. A common choice is 5 levels of IV fluids × 5 levels of vasopressors = 25 discrete actions. Some formulations add antibiotic timing as a third dimension.

Reward. This is where it gets philosophically interesting. Common choices:

Terminal reward only: +15 for survival at 90 days, -15 for death. Simple but sparse.
Intermediate rewards: SOFA score changes (improvement = positive reward, deterioration = negative). Provides denser signal but introduces assumptions about what "better" means at each step.
Composite: terminal survival reward plus intermediate lactate clearance or MAP maintenance bonuses.

Transition dynamics. Learned implicitly from the data. The agent doesn't model how the patient's physiology evolves; it just observes the empirical transitions in the historical dataset.

Discount factor (γ). Typically 0.99 for 4-hour time steps, reflecting that we care about long-term survival but slightly prefer getting there sooner.

Why This Is Hard (Beyond the Obvious)

Confounding. Clinicians don't treat randomly. Sicker patients get more aggressive treatment. If you naively learn from observational data, you might conclude that vasopressors cause death (because patients who received high-dose vasopressors died more often). This is confounding, not causation. Addressing it requires careful state representation (include enough variables to capture the patient's true severity) or explicit causal inference methods.

Partial observability. The state representation never captures everything the clinician knew. The nurse noticed the patient looked "off." The attending had a gut feeling based on 20 years of experience. The family mentioned the patient had been declining for days before admission. None of this is in the structured EHR data. The RL agent is making decisions based on an incomplete picture.

Non-stationarity. Treatment protocols change over time. A policy learned from 2015-2018 data may not be optimal for 2026 patients, because the standard of care has shifted, new drugs are available, and patient populations have changed.

Evaluation is the hardest part. How do you know if your learned policy is actually better than what clinicians did? You can't deploy it and see (not without extensive safety validation). You're stuck with off-policy evaluation (OPE): estimating how well the policy would have performed using historical data. OPE methods (importance sampling, doubly robust estimators, fitted Q-evaluation) all have significant variance and bias issues, especially when the learned policy differs substantially from the behavior policy.

General Architecture Pattern

EHR Data

Cohort Selection & Preprocessing

State / Action / Reward Construction

Offline RL Training

Off-Policy Evaluation

Clinical Validation

Decision-Support Interface

Cohort selection. Identify sepsis patients from historical EHR data using clinical criteria (Sepsis-3 definitions: suspected infection plus organ dysfunction). Exclude patients with DNR/comfort-care-only orders, those who died within the first hour (no opportunity for treatment optimization), and those with incomplete data.

Preprocessing. Handle missing values (forward-fill for vitals, imputation for labs), align to regular time steps (4-hour windows), normalize features, and construct the trajectory format.

State construction. Extract and engineer features at each time step. Decide between discrete state spaces (clustering) and continuous representations (neural networks). Include enough clinical context to mitigate confounding.

Action discretization. Map continuous treatment variables (mL of fluid, mcg/kg/min of vasopressor) into discrete bins. The granularity tradeoff: more bins give finer control but require more data to learn reliably.

Reward engineering. Define what "good" means. This is a clinical and ethical decision, not just a technical one. Involve clinicians in reward design.

Training. Apply offline RL algorithm (FQI, CQL, BCQ, or Decision Transformer) to learn the optimal policy from the historical trajectories.

Evaluation. Use off-policy evaluation to estimate the learned policy's performance. Compare against the clinician behavior policy and against simple baselines (always give max fluids, always follow guidelines exactly). Use multiple OPE methods and report confidence intervals.

Clinical validation. Before any deployment: review learned policies with domain experts. Do the recommendations make clinical sense? Are there obvious failure modes? Would a clinician trust this recommendation in context?

The AWS build lives in a companion page. This recipe covers the problem, the underlying technology, and the vendor-agnostic architecture. For the AWS services, architecture diagram, prerequisites, and the step-by-step pseudocode walkthrough, see the Architecture and Implementation companion. The Python example is linked from there.

The Honest Take

Let me be direct about where this stands in 2026: sepsis RL is one of the most published topics in healthcare AI, and it is still not routinely used in clinical practice. The research is compelling. The Komorowski et al. (2018) paper in Nature Medicine showed that patients whose clinicians happened to agree with the RL policy had lower mortality. But "happened to agree" is not the same as "was caused by." The gap between a promising retrospective analysis and a deployed clinical tool is enormous.

The off-policy evaluation problem is the fundamental bottleneck. Every OPE method has known failure modes. Importance sampling has high variance. Fitted Q-evaluation can be biased. You're trying to answer a causal question ("would this policy have saved more lives?") with observational data, and that's inherently limited. The confidence intervals on your estimated policy value will be wide enough to drive a truck through.

The clinician agreement rate is both encouraging and concerning. Encouraging because it means the policy isn't recommending wildly different things from expert practice. Concerning because the disagreement is where the value supposedly lives, and it's also where the uncertainty is highest.

If you're building this, start with the data pipeline and evaluation infrastructure, not the RL algorithm. The algorithm is the easy part. Getting clean trajectories from messy EHR data, defining a clinically meaningful reward function, and building trustworthy evaluation are where you'll spend 90% of your time.

Recipe 15.1 (Alert Threshold Optimization): A simpler RL application that demonstrates the core concepts (state, action, reward) in a lower-stakes setting. Good starting point before tackling treatment optimization.
Recipe 15.3 (Clinical Trial Adaptive Randomization): Another sequential decision problem in healthcare, but with a cleaner reward signal and established regulatory framework.
Recipe 15.5 (Ventilator Weaning Protocols): A closely related ICU RL application with similar architecture but narrower action space and clearer success criteria.
Recipe 7.9 (Mortality Risk Scoring, ICU): The predictive model that could feed into the state representation for this recipe. Risk scores are features, not policies.
Recipe 3.7 (Patient Deterioration Early Warning): Upstream detection that identifies which patients need the treatment optimization system's attention.