Chapter 15: Sequential Decision-Making & Reinforcement Learning

Teaching Machines to Make Sequential Decisions

Most of the ML in this book follows a familiar pattern: observe data, learn a mapping, predict an outcome. A patient's lab results go in, a risk score comes out. An image goes in, a classification comes out. The model makes one decision, and that decision doesn't change what happens next.

Reinforcement learning is different. Fundamentally different. And that difference is exactly why it's both the most exciting and the most terrifying application of AI in healthcare.

Here's the core idea: instead of learning a single prediction, an RL agent learns a policy, a strategy for making sequences of decisions over time where each decision changes the environment and affects what decisions are available next. The agent doesn't just predict what will happen; it decides what to do, observes the consequences, and adjusts its strategy accordingly.

If that sounds like what clinicians do every day, you're paying attention.

Why Sequential Decisions Are a Different Beast

Think about managing a sepsis patient in the ICU. The physician doesn't make one decision. They make dozens: when to start antibiotics, which antibiotics, how aggressively to push fluids, whether to add vasopressors, when to escalate, when to de-escalate. Each decision changes the patient's state. The right next action depends on what you did before and how the patient responded. You can't evaluate any single decision in isolation because the outcome depends on the entire sequence.

This is a Markov Decision Process (MDP), whether the clinician thinks of it that way or not. There's a state (patient vitals, labs, current treatments), actions (clinical interventions), transitions (how the patient responds), and rewards (did the patient get better?). The goal is to find the policy that maximizes long-term outcomes, not just the immediate next step.

Traditional supervised learning can't handle this well. You could train a model to predict "given this patient state, what did the best clinicians do?" but that's imitation learning, not optimization. It copies existing behavior rather than discovering potentially better strategies. And it can't reason about the downstream consequences of actions. If giving more fluids now means you can avoid vasopressors later (which have their own side effects), a supervised model won't discover that trade-off. An RL agent, at least in theory, can.

The Key Paradigms (And Why They Matter for Healthcare)

Online vs. Offline RL

This distinction is everything in healthcare.

Online RL learns by interacting with the environment directly. The agent takes actions, observes outcomes, and updates its policy in real time. This is how AlphaGo learned to play Go: by playing millions of games against itself. It's also completely unacceptable for most clinical applications. You cannot explore randomly with real patients. "Let's try a suboptimal treatment to see what happens" is not an ethical experimental design.

Offline RL (also called batch RL) learns entirely from historical data. You have a dataset of past patient trajectories: states, actions taken, outcomes observed. The agent learns a policy from this fixed dataset without ever interacting with a live patient. This is the only viable path for most healthcare RL applications, and it comes with its own set of hard problems.

The biggest challenge with offline RL is distribution shift. The agent learns from data generated by some historical policy (whatever clinicians actually did). If the learned policy recommends actions that are very different from what was historically observed, you have no data to evaluate whether those novel actions would actually work. The agent might learn "never give vasopressors" because in the training data, patients who received vasopressors had worse outcomes. But that's confounded: those patients received vasopressors because they were sicker. Offline RL must handle this counterfactual reasoning carefully, or it learns dangerous nonsense.

Model-Based vs. Model-Free

Model-free RL learns a policy directly from experience without building an explicit model of how the environment works. Q-learning and policy gradient methods fall here. The agent learns "in state X, action A leads to good outcomes" without understanding why. This is simpler to implement but requires enormous amounts of data and can be brittle when the environment changes.

Model-based RL first learns a model of the environment (how states transition given actions) and then uses that model to plan. In healthcare terms: first learn a model of patient physiology (if I give insulin, blood glucose drops by approximately this much over this timeframe), then use that model to find optimal treatment sequences. This is more data-efficient and more interpretable, but the model can be wrong, and planning with a wrong model gives you confidently bad decisions.

For healthcare, model-based approaches are often preferred because they're more interpretable (clinicians can inspect the learned dynamics model) and more data-efficient (you don't need millions of patient trajectories). But they require domain expertise to validate the learned model against known physiology.

Multi-Armed Bandits: The Simpler Cousin

Not every sequential decision problem needs full RL. Multi-armed bandits handle the simpler case where actions don't change the underlying state. "Which notification channel works best for this patient?" is a bandit problem: the patient's preferences don't fundamentally change based on which channel you tried last time. Bandits are easier to implement, easier to validate, and easier to deploy safely. Several recipes in this chapter start here because it's the right tool for many real healthcare optimization problems.

The Safety Problem (This Is Where It Gets Hard)

Here's the thing that keeps RL researchers up at night when they think about healthcare applications: exploration is dangerous.

In a video game, the agent can die a thousand times while learning. In a Go match, losing is just losing. In healthcare, exploration means trying things on patients. Even in offline RL (where you're not directly experimenting), the learned policy might recommend actions that are outside the safe operating envelope. A policy that says "give 10x the normal insulin dose" might technically optimize some reward function, but it would kill the patient.

Healthcare RL needs constrained optimization. The policy must satisfy safety constraints at all times, not just on average. This means:

Hard constraints: Actions that are never acceptable regardless of predicted benefit. Dose limits. Contraindicated drug combinations. Physiological boundaries that must not be crossed.

Soft constraints: Preferences that should be respected unless there's strong evidence to deviate. Clinical guidelines. Standard-of-care protocols. Institutional policies.

Conservative policies: When uncertain, do what the historical data supports. Only deviate from observed clinical practice when the evidence for improvement is strong. This is the principle behind Conservative Q-Learning (CQL) and similar algorithms designed specifically for offline RL in high-stakes domains.

Clinician-in-the-loop: For the foreseeable future, RL policies in healthcare are decision support, not autonomous agents. The system recommends; the clinician decides. This isn't just a regulatory requirement (though it is that too). It's a recognition that our models are not yet trustworthy enough to act independently on patients.

The Reward Problem

Defining what "good" means is surprisingly hard in healthcare.

In games, the reward is clear: win or lose, score points, maximize a number. In healthcare, what's the reward signal? Patient survival? That's too binary and too delayed. Reduction in symptoms? Hard to measure continuously. Discharge from the ICU? That conflates "got better" with "gave up and transferred to comfort care."

Most healthcare RL work uses composite reward functions that combine multiple clinical indicators: vital sign stability, lab value normalization, reduction in organ dysfunction scores, absence of adverse events. But the weighting of these components is a clinical judgment call, and different weightings produce different optimal policies. A policy optimized for short-term vital sign stability might make different recommendations than one optimized for 90-day mortality.

This isn't a technical problem with a technical solution. It's a values problem that requires clinical input. Every recipe in this chapter is explicit about what reward function is being optimized and what trade-offs that implies.

The Regulatory Landscape

Let's be direct: the FDA pathway for RL-based clinical decision support is unclear. The existing frameworks for Software as a Medical Device (SaMD) were designed for static models that produce the same output given the same input. An RL policy that adapts over time, or that recommends different actions for similar-looking patients based on their treatment history, doesn't fit neatly into existing regulatory categories.

The simpler use cases in this chapter (alert threshold optimization, notification timing) likely fall under clinical decision support exemptions because they're not making treatment decisions. The complex use cases (sepsis management, chemotherapy dosing) would almost certainly require FDA clearance, and the pathway for that clearance is still being defined.

This is not a reason to avoid the technology. It's a reason to understand where each use case falls on the regulatory spectrum and to build accordingly. The recipes are ordered partly by regulatory complexity: the early ones are deployable today, the later ones are research-stage with a path toward clinical use.

What This Chapter Covers

The recipes progress from immediately practical to research-frontier:

Simple (deployable now): Alert threshold optimization and notification timing. These use bandit algorithms or simple RL to tune operational parameters. The "patient" isn't directly affected by exploration because the actions are low-stakes (adjusting a threshold, choosing a notification time). Clear reward signals, fast feedback loops, minimal regulatory burden.

Medium (requires careful validation): Sepsis treatment optimization, ventilator weaning, glucose control. These are well-studied problems with published research. Offline RL on historical EHR data. Deployment as clinical decision support with physician override. Significant validation required before any clinical use.

Complex (research-stage): Chronic disease management, chemotherapy dosing, radiation therapy adaptation. Long time horizons, sparse rewards, high stakes. These represent where the field is heading, not where it is today. Included because understanding the architecture and challenges prepares you for when these become feasible.

A Note on Honesty

I want to be upfront about something: reinforcement learning in healthcare is mostly research. The gap between "published a paper showing our offline RL policy would have improved outcomes on a retrospective dataset" and "deployed a system that actually helps patients in real time" is enormous. It involves regulatory approval, clinical validation, workflow integration, clinician trust, and ongoing monitoring.

The simpler recipes in this chapter (threshold tuning, notification optimization) are genuinely deployable today. They use well-understood algorithms, have clear safety boundaries, and don't require regulatory approval. Start there.

The complex recipes are included because the architecture patterns, safety frameworks, and evaluation methodologies are valuable to understand even if full deployment is years away. When you're ready to build toward treatment optimization, you'll want to have thought through these problems in advance.

Let's start with something concrete. Recipe 15.1 takes a problem every health system has (alert fatigue) and shows how a simple RL formulation can learn better thresholds than any static rule.

→ Recipe 15.1 — Alert Threshold Optimization