Recipe 15.4 Architecture and Implementation: Sepsis Treatment Optimization

Companion to Recipe 15.4: Sepsis Treatment Optimization. This page covers the AWS architecture, services, prerequisites, and pseudocode. For the problem framing and the conceptual approach, start with the main recipe.

The AWS Implementation

Why These Services

Amazon SageMaker for model training and experimentation. Sepsis RL training involves iterating on state representations, reward functions, and algorithm hyperparameters. SageMaker provides managed notebook instances for exploration, training jobs with configurable compute (GPU instances for neural network-based approaches), and experiment tracking to compare policy variants. The managed infrastructure means you're not maintaining GPU clusters between training runs.

Amazon S3 for data lake storage. The preprocessed EHR trajectories, trained model artifacts, evaluation results, and audit logs all live in S3. Versioning tracks which dataset version produced which policy. Lifecycle policies manage the retention of intermediate artifacts.

AWS Glue for ETL and cohort construction. Extracting sepsis cohorts from raw EHR data involves complex SQL-like transformations: joining diagnosis tables with lab results, vital signs, and medication administration records; applying Sepsis-3 criteria; constructing time-aligned trajectories. Glue handles this at scale without provisioning Spark clusters manually.

Amazon DynamoDB for policy serving (discrete state space path). If you discretize the continuous state space into clusters (typically ~750 k-means clusters over the physiological feature space), the resulting policy is a lookup table: cluster ID maps to action ID. DynamoDB provides single-digit-millisecond lookups for this table. This path trades off representational fidelity for interpretability and serving simplicity. The tradeoff: discretization loses information between cluster boundaries, but the policy is easy to inspect, audit, and explain.

Amazon SageMaker Endpoints for real-time inference (continuous state space path). This recipe's code uses continuous states with a neural Q-network served as a SageMaker real-time endpoint. The clinician-facing system sends the raw patient state vector, the endpoint runs it through the neural network and returns the recommended action with confidence information. This path preserves full state information but makes the policy harder to interpret. Choose one approach: discrete (DynamoDB) for interpretability, continuous (SageMaker endpoint) for expressiveness. The rest of this recipe assumes the continuous/neural network path.

AWS Step Functions for pipeline orchestration. The full pipeline (data extraction, preprocessing, training, evaluation, model registration) is a multi-step workflow with dependencies. Step Functions coordinates the sequence, handles retries, and provides visibility into pipeline state.

Amazon CloudWatch for monitoring and alerting. Track model inference latency, prediction distribution drift (are we seeing patient states outside the training distribution?), and system health. Alert on anomalies that might indicate the model is being queried with out-of-distribution inputs.

Architecture Diagram

flowchart TD
    A[EHR Data Source\nFHIR / HL7 / Database] -->|Extract| B[AWS Glue\nCohort Selection & ETL]
    B -->|Trajectories| C[S3 Data Lake\nProcessed Trajectories]
    C -->|Training Data| D[SageMaker Training Job\nOffline RL Algorithm]
    D -->|Model Artifact| E[S3 Model Registry]
    E -->|Evaluate| F[SageMaker Processing Job\nOff-Policy Evaluation]
    F -->|Metrics| G[CloudWatch Metrics\nPolicy Performance]
    E -->|Deploy| H[SageMaker Endpoint\nPolicy Inference]
    
    I[Clinical System\nEHR / CDSS] -->|Patient State| H
    H -->|Recommended Action\n+ Confidence| I
    
    J[Step Functions\nPipeline Orchestration] --> B
    J --> D
    J --> F

    style C fill:#f9f,stroke:#333
    style D fill:#ff9,stroke:#333
    style H fill:#9ff,stroke:#333

Prerequisites

| | Glue ETL role: glue:StartJobRun, s3:GetObject on arn:aws:s3:::sepsis-rl-trajectories/raw/*, s3:PutObject on arn:aws:s3:::sepsis-rl-trajectories/processed/*, kms:Decrypt/kms:GenerateDataKey for the data KMS key | | | SageMaker Training role: s3:GetObject on arn:aws:s3:::sepsis-rl-trajectories/processed/*, s3:PutObject on arn:aws:s3:::sepsis-rl-trajectories/models/*, sagemaker:CreateTrainingJob, sagemaker:AddTags, kms:Decrypt/kms:GenerateDataKey for training volume KMS key | | | SageMaker Inference role: s3:GetObject on arn:aws:s3:::sepsis-rl-trajectories/models/* (read model artifact only), sagemaker:InvokeEndpoint, dynamodb:PutItem on the audit table, cloudwatch:PutMetricData. No access to raw patient data. | | | Step Functions orchestration role: states:StartExecution, glue:StartJobRun, sagemaker:CreateTrainingJob, sagemaker:CreateProcessingJob. Cannot read S3 data directly; delegates to the stage-specific roles. |

| BAA | AWS BAA signed (required: patient physiological data is PHI) | | De-identification | Training data should be de-identified per HIPAA Safe Harbor (remove 18 identifiers) or used under a Limited Data Set with a Data Use Agreement. Replace patient IDs with pseudonymous identifiers before trajectory construction. The trained model artifact (neural network weights) is not itself PHI because it does not contain individually identifiable information, but the trajectory dataset absolutely is. | | Encryption | S3: SSE-KMS for all trajectory data and model artifacts; DynamoDB: encryption at rest; SageMaker: KMS-encrypted training volumes and endpoints; all API calls over TLS | | VPC | Production: SageMaker in VPC with VPC endpoints for S3 (gateway, free), DynamoDB (gateway, free), CloudWatch Logs (interface), SageMaker API (interface), SageMaker Runtime (interface), and KMS (interface). Without SageMaker API and Runtime endpoints, private subnet deployment requires a NAT Gateway (which becomes an egress point for PHI) or fails entirely. Interface endpoints cost ~$7.20/month per endpoint per AZ. Budget 4-5 interface endpoints across 2 AZs: ~$60-70/month for network isolation. No public internet access for training jobs processing PHI. |

| CloudTrail | Enabled: log all SageMaker, S3, and Glue API calls for audit trail | | Data Requirements | Minimum 10,000-20,000 sepsis episodes with complete trajectory data (vitals q4h, labs, medication administration records). MIMIC-III/IV for research; institutional EHR data for production. | | IRB Approval | Required before accessing patient data for model development. This is research, not routine operations. | | Cost Estimate | Training: $50-200 per training run (ml.p3.2xlarge, 2-8 hours). Inference endpoint: $0.10/hour (ml.m5.large). Storage: ~$5/month per 100GB of trajectory data. Total monthly: $2,000-8,000 depending on experimentation intensity. |

Ingredients

Code

Walkthrough

Step 1: Cohort extraction and trajectory construction. The first step pulls sepsis patients from the EHR and constructs the trajectory dataset that the RL algorithm will learn from.

This involves identifying patients meeting Sepsis-3 criteria (suspected infection plus acute organ dysfunction, operationalized as a SOFA score increase of 2 or more points), extracting their time-series data at regular intervals, and formatting it into (state, action, reward, next_state) tuples. The time alignment matters: vitals might be recorded every 15 minutes, labs every 6 hours, and medications at irregular intervals. Everything gets aligned to a consistent time grid (typically 4-hour windows). Skip this step or do it sloppily, and your RL agent learns from noise rather than signal.

FUNCTION build_sepsis_trajectories(ehr_database, start_date, end_date):
    // Step 1a: Identify sepsis patients using Sepsis-3 criteria.
    // Sepsis-3 = suspected infection + organ dysfunction (SOFA increase >= 2).
    // We look for patients with both an infection marker (blood culture ordered,
    // antibiotics started) and a SOFA score jump within a 24-hour window.
    sepsis_cohort = query ehr_database:
        SELECT patients WHERE:
            - suspected_infection (antibiotics started OR blood culture ordered)
            - SOFA_score_increase >= 2 within 24 hours of infection suspicion
            - ICU admission between start_date and end_date
            - NOT (comfort_care_only OR DNR_within_first_hour)
            - survived_at_least_4_hours_after_sepsis_onset

    trajectories = empty list

    FOR each patient in sepsis_cohort:
        // Step 1b: Extract raw time-series data for this patient.
        vitals = get_vitals(patient)        // HR, MAP, temp, RR, SpO2 (q15min-q1h)
        labs   = get_labs(patient)          // lactate, creatinine, bilirubin, WBC, platelets, pH
        meds   = get_medications(patient)   // IV fluids (mL), vasopressors (mcg/kg/min), antibiotics
        outputs = get_outputs(patient)      // urine output, drain output

        // Step 1c: Align to 4-hour time windows.
        // For vitals: take the mean within each window.
        // For labs: forward-fill (use most recent available value).
        // For medications: sum total dose administered within the window.
        time_steps = align_to_grid(
            vitals, labs, meds, outputs,
            window_size = 4 hours,
            start = sepsis_onset_time(patient),
            end = ICU_discharge_or_death(patient)
        )

        // Step 1d: Construct state, action, reward tuples for each time step.
        patient_trajectory = empty list
        FOR t = 0 to length(time_steps) - 1:
            state_t = construct_state(time_steps[t])
            // State includes: vitals, labs, fluid balance, SOFA components,
            // hours since admission, demographics

            action_t = discretize_action(
                iv_fluid_volume = time_steps[t].total_iv_fluid_mL,
                vasopressor_dose = time_steps[t].max_vasopressor_dose
            )
            // Discretize into 5x5 grid: 5 fluid levels x 5 vasopressor levels = 25 actions

            reward_t = compute_reward(time_steps[t], time_steps[t+1], patient_outcome)
            // Intermediate: SOFA improvement/deterioration
            // Terminal: +15 survival at 90 days, -15 death

            next_state = construct_state(time_steps[t+1]) IF t < last step ELSE terminal

            append (state_t, action_t, reward_t, next_state) to patient_trajectory

        append patient_trajectory to trajectories

    RETURN trajectories

Step 2: State representation and action discretization. The raw features need to be transformed into a representation the RL algorithm can work with. For tabular methods (FQI with discrete states), this means clustering the continuous feature vectors into a manageable number of discrete states. For neural network methods, it means normalization and potentially dimensionality reduction. The action discretization maps continuous treatment doses into bins. The choice of bin boundaries matters: you want clinically meaningful thresholds, not arbitrary quantiles. A vasopressor dose of 0 is qualitatively different from 0.01 mcg/kg/min, which is different from 0.3 mcg/kg/min. The bins should reflect these clinical breakpoints.

// Action discretization: 5 levels each for fluids and vasopressors.
// These thresholds are based on clinical practice patterns, not arbitrary quantiles.

FLUID_BINS = [0, 250, 500, 1000, 2000]  // mL per 4-hour window
// 0: no fluids this window
// 250: minimal maintenance
// 500: moderate resuscitation
// 1000: aggressive resuscitation
// 2000+: very aggressive (bolus territory)

VASOPRESSOR_BINS = [0, 0.08, 0.16, 0.28, 0.45]  // norepinephrine equivalent mcg/kg/min
// 0: no vasopressors
// 0.08: low-dose support
// 0.16: moderate support
// 0.28: high-dose support
// 0.45+: maximal support

FUNCTION discretize_action(iv_fluid_mL, vasopressor_dose):
    fluid_level = bin_index(iv_fluid_mL, FLUID_BINS)        // 0-4
    vaso_level  = bin_index(vasopressor_dose, VASOPRESSOR_BINS)  // 0-4
    action_id   = fluid_level * 5 + vaso_level               // 0-24 (25 total actions)
    RETURN action_id

FUNCTION construct_state(time_step):
    // Build a feature vector capturing the patient's current physiological status.
    // Each feature is normalized to zero mean, unit variance based on training set statistics.
    state_features = [
        normalize(time_step.heart_rate),
        normalize(time_step.mean_arterial_pressure),
        normalize(time_step.temperature),
        normalize(time_step.respiratory_rate),
        normalize(time_step.spo2),
        normalize(time_step.lactate),
        normalize(time_step.creatinine),
        normalize(time_step.bilirubin),
        normalize(time_step.platelet_count),
        normalize(time_step.wbc),
        normalize(time_step.ph),
        normalize(time_step.pao2_fio2_ratio),
        normalize(time_step.urine_output_4h),
        normalize(time_step.cumulative_fluid_balance),
        normalize(time_step.gcs_score),
        normalize(time_step.sofa_score),
        normalize(time_step.hours_since_admission),
        one_hot(time_step.age_bucket),          // age group
        one_hot(time_step.infection_source)     // lung, abdominal, urinary, other
    ]
    RETURN state_features

Step 3: Offline RL training with safety constraints. This is where the policy is actually learned. We use Conservative Q-Learning (CQL) because it explicitly addresses the distribution shift problem by penalizing overestimation of Q-values for actions that are rare in the training data. This is critical in healthcare: if clinicians rarely gave zero fluids to hypotensive patients (because that would be dangerous), we don't want the algorithm to optimistically assume that action would work well. CQL stays conservative, preferring actions that are well-supported by the data. The safety constraint layer adds hard limits: the policy cannot recommend actions that violate clinical safety boundaries regardless of what the Q-function says.

FUNCTION train_sepsis_policy(trajectories, config):
    // Initialize the Q-network: maps (state, action) -> expected cumulative reward.
    // Two networks for stability (standard in deep RL: target network lags behind).
    q_network        = initialize_neural_network(input_dim = state_dim + action_dim, output_dim = 1)
    target_q_network = copy(q_network)

    // CQL-specific: alpha controls how conservative the policy is.
    // Higher alpha = more conservative = stays closer to clinician behavior.
    // Lower alpha = more willing to deviate from observed behavior.
    // Start conservative and tune based on off-policy evaluation results.
    cql_alpha = config.cql_alpha  // typically 1.0 to 5.0

    replay_buffer = load_all_trajectories(trajectories)

    FOR iteration = 1 to config.num_iterations:
        batch = sample_batch(replay_buffer, size = config.batch_size)

        FOR each (state, action, reward, next_state) in batch:
            // Standard Q-learning target: reward + discounted future value
            IF next_state is terminal:
                target = reward
            ELSE:
                // Use target network to estimate future value (stability trick)
                best_next_action = argmax over actions: target_q_network(next_state, a)
                target = reward + config.gamma * target_q_network(next_state, best_next_action)

            // CQL penalty: push down Q-values for all actions, push up for observed action.
            // This makes the agent pessimistic about actions it hasn't seen work.
            cql_penalty = cql_alpha * (
                log_sum_exp(q_network(state, all_actions))  // push down average Q
                - q_network(state, action)                   // push up observed action's Q
            )

            loss = (q_network(state, action) - target)^2 + cql_penalty

        update q_network to minimize total loss
        // Periodically sync target network (soft update or hard copy every N steps)
        IF iteration % config.target_update_freq == 0:
            soft_update(target_q_network, q_network, tau = 0.005)

    // Extract the learned policy: for each state, pick the action with highest Q-value,
    // subject to safety constraints.
    FUNCTION learned_policy(state):
        q_values = [q_network(state, a) for a in all_actions]
        // Apply safety constraints before selecting action
        safe_actions = apply_safety_constraints(state, all_actions)
        best_action = argmax q_values[a] for a in safe_actions
        RETURN best_action

    RETURN learned_policy, q_network

Step 4: Safety constraint layer. This is non-negotiable. The learned policy must respect hard clinical boundaries regardless of what the Q-function suggests. These constraints encode domain knowledge that should never be violated: minimum vasopressor support for severely hypotensive patients, maximum fluid volumes to prevent overload, and contraindicated actions based on patient-specific factors. Think of this as a guardrail around the RL policy. The policy suggests; the safety layer vetoes dangerous suggestions.

FUNCTION apply_safety_constraints(state, candidate_actions):
    // Hard constraints that override the learned policy.
    // These represent clinical boundaries that should never be crossed.
    safe_actions = copy(candidate_actions)

    // Constraint 1: If MAP < 55, do not recommend zero vasopressors.
    // Rationale: MAP below 55 is associated with acute organ injury.
    // The policy should not suggest removing hemodynamic support at this level.
    IF state.mean_arterial_pressure < 55:
        remove actions where vasopressor_level == 0 from safe_actions

    // Constraint 2: If cumulative fluid balance > 6000 mL positive,
    // do not recommend highest fluid level.
    // Rationale: Fluid overload is independently associated with mortality in sepsis.
    // Beyond a certain point, more fluid causes harm (pulmonary edema, abdominal compartment).
    IF state.cumulative_fluid_balance > 6000:
        remove actions where fluid_level == 4 from safe_actions

    // Constraint 3: If lactate is rising AND vasopressors are already high,
    // do not recommend reducing vasopressors.
    // Rationale: Rising lactate with high vasopressor need suggests worsening shock.
    // Reducing support here is clinically inappropriate.
    IF state.lactate_trend == "rising" AND state.current_vasopressor_level >= 3:
        remove actions where vasopressor_level < state.current_vasopressor_level from safe_actions

    // Constraint 4: Never recommend an action that would result in
    // zero treatment (no fluids AND no vasopressors) for a patient with SOFA >= 6.
    // Rationale: SOFA >= 6 indicates significant organ dysfunction requiring active management.
    IF state.sofa_score >= 6:
        remove action_id == 0 from safe_actions  // action 0 = no fluids, no vasopressors

    // If all actions are constrained out (shouldn't happen with well-designed constraints),
    // fall back to the clinician's most recent action (maintain current treatment).
    IF safe_actions is empty:
        safe_actions = [action matching current treatment levels]

    RETURN safe_actions

Step 4b: Safety constraint monitoring. Every time a safety constraint fires, that's signal. If constraints are triggering on a large fraction of recommendations, something is wrong: the policy may be drifting, the patient population may be shifting, or the constraint thresholds need recalibration. Publish constraint trigger rates to CloudWatch and alarm when they exceed acceptable thresholds.

FUNCTION monitor_constraint_triggers(recommendation_batch, window_hours = 24):
    // Track how often each constraint fires over a rolling window.
    // If any constraint fires on >20% of recommendations in 24 hours,
    // alert clinical informatics for review.

    constraint_counts = {
        "map_vasopressor_floor": 0,
        "fluid_overload_cap": 0,
        "lactate_rising_hold": 0,
        "sofa_zero_treatment_block": 0,
        "any_constraint": 0,
    }
    total_recommendations = length(recommendation_batch)

    FOR each recommendation in recommendation_batch:
        IF recommendation.constraints_triggered is not empty:
            constraint_counts["any_constraint"] += 1
            FOR each constraint_name in recommendation.constraints_triggered:
                constraint_counts[constraint_name] += 1

    // Publish per-constraint trigger rates to CloudWatch as custom metrics.
    FOR each (constraint_name, count) in constraint_counts:
        trigger_rate = count / total_recommendations
        publish_metric(
            namespace = "SepsisRL/SafetyConstraints",
            metric_name = constraint_name + "_trigger_rate",
            value = trigger_rate,
            unit = "None"
        )

    // CloudWatch Alarm: any_constraint_trigger_rate > 0.20 for 1 consecutive period
    // Action: notify clinical informatics team via SNS
    // This threshold means the policy is disagreeing with safety boundaries
    // on more than 1 in 5 recommendations, which warrants human review.

Step 5: Off-policy evaluation. Before anyone even thinks about showing this to a clinician, you need to estimate how well the learned policy would have performed compared to what actually happened. This is off-policy evaluation (OPE), and it's the hardest part of the entire pipeline. You're asking: "If we had followed this policy instead of what the clinicians did, would patients have done better?" You can't know for certain without actually deploying it (which you won't do without extensive validation). OPE gives you an estimate with uncertainty bounds. Use multiple methods and be honest about the limitations.

FUNCTION evaluate_policy_offline(learned_policy, test_trajectories, behavior_policy):
    // Method 1: Weighted Importance Sampling (WIS)
    // Idea: reweight historical outcomes by the probability ratio of the learned policy
    // vs. the behavior policy. If the learned policy would have taken the same actions
    // the clinician took, the weight is ~1. If it would have done something different,
    // the trajectory gets down-weighted (we can't trust its outcome as evidence for our policy).
    wis_estimates = empty list
    FOR each trajectory in test_trajectories:
        weight = 1.0
        FOR each (state, action, reward) in trajectory:
            pi_prob = learned_policy.probability(action | state)
            behavior_prob = behavior_policy.probability(action | state)
            weight = weight * (pi_prob / behavior_prob)
            // Clip weights to prevent extreme values from dominating
            weight = clip(weight, 0.001, 100.0)
        wis_estimates.append(weight * trajectory.total_reward)

    wis_value = mean(wis_estimates) / mean(weights)  // self-normalized

    // Method 2: Fitted Q-Evaluation (FQE)
    // Train a separate Q-function to evaluate the learned policy (not to optimize it).
    // This avoids the high variance of importance sampling.
    fqe_q = train_q_function_for_policy(learned_policy, test_trajectories)
    fqe_value = mean(fqe_q(initial_state, learned_policy(initial_state))
                     for initial_state in test_trajectories)

    // Method 3: Compare action agreement with clinicians
    // Not a value estimate, but a sanity check: how often does the learned policy
    // agree with what clinicians actually did?
    agreement_rate = count(learned_policy(s) == clinician_action(s)) / total_decisions

    // Report all estimates with confidence intervals
    RETURN {
        "wis_estimated_value": wis_value,
        "wis_95_ci": bootstrap_confidence_interval(wis_estimates),
        "fqe_estimated_value": fqe_value,
        "clinician_agreement_rate": agreement_rate,
        "clinician_actual_value": mean(trajectory.total_reward for trajectory in test_trajectories)
    }

Step 5b: Distribution shift detection. The policy was trained on historical data. At inference time, you will encounter patient states the training data never covered. Blindly serving recommendations for out-of-distribution (OOD) states is dangerous. You need a mechanism to detect when a patient's state is far from anything the model trained on, suppress low-confidence recommendations, and alert operators when OOD rates climb.

The approach: during training, compute the mean and covariance of the state feature vectors across all training transitions. At inference time, compute the Mahalanobis distance between the incoming state and the training distribution. States with high Mahalanobis distance are OOD. The threshold can be calibrated on the training set (e.g., 99th percentile of training distances).

FUNCTION compute_training_distribution(all_training_states):
    // Compute multivariate statistics from training data.
    // These are stored alongside the model artifact for use at inference time.
    mean_vector = mean(all_training_states, axis=0)        // shape: (state_dim,)
    covariance_matrix = covariance(all_training_states)    // shape: (state_dim, state_dim)
    // Regularize covariance to ensure invertibility (add small diagonal).
    covariance_matrix += 1e-6 * identity_matrix(state_dim)
    inverse_covariance = invert(covariance_matrix)

    // Compute Mahalanobis distances for all training states to set threshold.
    training_distances = []
    FOR each state in all_training_states:
        diff = state - mean_vector
        distance = sqrt(diff^T @ inverse_covariance @ diff)
        training_distances.append(distance)

    // Use 99th percentile as OOD threshold.
    ood_threshold = percentile(training_distances, 99)

    RETURN {mean_vector, inverse_covariance, ood_threshold}

FUNCTION check_distribution_shift(state, distribution_stats):
    diff = state - distribution_stats.mean_vector
    mahalanobis_distance = sqrt(diff^T @ distribution_stats.inverse_covariance @ diff)

    is_ood = (mahalanobis_distance > distribution_stats.ood_threshold)

    // Publish OOD metrics to CloudWatch.
    publish_metric("SepsisRL/DistributionShift", "MahalanobisDistance", mahalanobis_distance)
    publish_metric("SepsisRL/DistributionShift", "OODFlag", 1 if is_ood else 0)

    IF is_ood:
        // Suppress recommendation confidence. Don't remove it entirely
        // (clinician may still find it informative), but flag clearly.
        RETURN {
            "ood_detected": true,
            "mahalanobis_distance": mahalanobis_distance,
            "suppress_confidence": true,
            "warning": "Patient state is outside the training distribution. Recommendation confidence is low."
        }
    ELSE:
        RETURN {"ood_detected": false, "mahalanobis_distance": mahalanobis_distance}

// CloudWatch Alarm: track rolling OOD percentage.
// If >5% of recommendations in a 24-hour window are OOD, alert the team.
// Rising OOD rates indicate population drift or a data pipeline issue.

Step 6: Clinical decision support interface. If (and only if) the policy passes off-policy evaluation, clinical review, and institutional approval, it can be deployed as a decision support tool. Not an autonomous agent. A recommendation system that shows clinicians what the policy suggests, along with confidence information and the reasoning (which state features are driving the recommendation). The clinician always makes the final decision. The system is advisory only.

FUNCTION serve_recommendation(patient_state, policy_endpoint):
    // Package the current patient state into the format the model expects.
    state_vector = construct_state(patient_state)

    // Query the policy endpoint for a recommendation.
    response = call policy_endpoint with state_vector

    recommendation = {
        "recommended_action": response.action,
        "fluid_recommendation": decode_fluid_level(response.action),
        "vasopressor_recommendation": decode_vasopressor_level(response.action),
        "confidence": response.q_value_margin,  // gap between best and second-best action
        "agreement_with_current": (response.action == current_treatment_action),
        "safety_constraints_applied": response.constraints_triggered,
        "top_3_actions": response.top_actions_with_q_values,
        // Explainability: which state features most influenced this recommendation
        "key_drivers": response.feature_importance_top_5,
        "disclaimer": "Advisory only. Clinical judgment supersedes all recommendations."
    }

    // Log every recommendation for audit and outcome tracking.
    log_to_audit_trail(patient_state, recommendation, timestamp)

    // Audit storage must be tamper-evident. Options:
    // 1. S3 Object Lock (compliance mode): prevents deletion or overwrite for a
    //    configurable retention period. Even the root account cannot delete objects
    //    during the retention window.
    // 2. CloudWatch Logs with a resource policy preventing DeleteLogGroup/DeleteLogStream.
    // 3. Separate audit account: write-only cross-account access (the production account
    //    can write audit records but cannot read or delete them).
    // Compliance mode Object Lock is the simplest path. Set retention to match your
    // institutional record retention policy (typically 7-10 years for clinical data).

    RETURN recommendation

Curious how this looks in Python? The pseudocode above covers the concepts. If you'd like to see sample Python code that demonstrates these patterns using boto3, check out the Python Example. It walks through each step with inline comments and notes on what you'd need to change for a real deployment.

Expected Results

Sample recommendation output:

{
  "patient_id": "ANON-ICU-4821",
  "timestamp": "2026-03-15T08:00:00Z",
  "current_state": {
    "map": 62,
    "heart_rate": 112,
    "lactate": 4.2,
    "sofa_score": 8,
    "hours_since_admission": 12,
    "cumulative_fluid_balance_mL": 3200,
    "current_vasopressor": "norepinephrine 0.12 mcg/kg/min"
  },
  "recommendation": {
    "fluid_level": 2,
    "fluid_interpretation": "500 mL IV fluid bolus over next 4 hours",
    "vasopressor_level": 2,
    "vasopressor_interpretation": "Maintain current norepinephrine dose (0.08-0.16 range)",
    "confidence": 0.73,
    "agreement_with_current_treatment": true,
    "safety_constraints_triggered": [],
    "key_drivers": [
      "Lactate 4.2 (elevated, trending down from 5.1)",
      "MAP 62 (borderline, responding to current support)",
      "Fluid balance +3200 mL (moderate, room for additional resuscitation)"
    ]
  },
  "disclaimer": "Advisory only. Clinical judgment supersedes all recommendations."
}

Performance benchmarks (from off-policy evaluation on held-out test set):

Where it struggles:

• Patients with rare comorbidity combinations not well-represented in training data
• Very early sepsis (first 4 hours) where the state representation is sparse
• Patients whose trajectory diverges significantly from anything in the training set
• Cases where the "right" action depends on information not captured in structured EHR data (clinical gestalt, family preferences, goals of care)

Why This Isn't Production-Ready

The pseudocode and architecture above demonstrate the pattern. Deploying this as a clinical decision support tool requires addressing several gaps that are intentionally outside the scope of a cookbook recipe:

Regulatory pathway. An RL-based treatment recommendation system likely falls under FDA oversight as a Clinical Decision Support (CDS) tool. The regulatory pathway depends on whether it meets the criteria for exemption under 21st Century Cures Act Section 3060. If it provides specific treatment recommendations (which it does), it likely requires FDA clearance or approval. This is a multi-year process.

Prospective validation. Off-policy evaluation is necessary but not sufficient. Before clinical deployment, you need a prospective observational study: run the system in shadow mode (generate recommendations but don't show them to clinicians), then retrospectively compare its recommendations against actual outcomes. This requires IRB approval, institutional buy-in, and months of data collection.

Model updating and drift. Treatment patterns change. New drugs become available. Patient populations shift. The model needs a retraining pipeline with drift detection. But retraining an RL policy on new data and validating it is not a simple model refresh; it requires re-running the full evaluation pipeline.

Model versioning and rollback. Use SageMaker Model Registry with approval workflows to track every policy version. Before promoting a new policy, run OPE comparing new vs. currently-deployed policy on the same held-out data. Deploy using a canary pattern: route 5-10% of recommendations through the new policy while monitoring safety constraint trigger rates and clinician override rates. If either metric degrades beyond a threshold (e.g., constraint triggers increase by >5 percentage points, or override rate increases by >10 percentage points), automatically roll back to the previous approved model version. The rollback should be automated via CloudWatch Alarms triggering a Step Functions state that swaps the endpoint back to the previous model artifact.

Liability and accountability. If a clinician follows the system's recommendation and the patient has a bad outcome, who is responsible? This is an unsolved legal and ethical question. Most institutions will require that the system be clearly advisory and that clinicians document their independent clinical reasoning.

Variations and Extensions

Multi-agent formulation. Instead of a single policy for all sepsis patients, train separate policies for different sepsis subtypes (pulmonary, abdominal, urinary source) or severity strata. Patients with different infection sources may have different optimal treatment trajectories. This requires larger datasets but can improve policy quality for each subgroup.

Incorporating antibiotic timing. The standard formulation focuses on fluids and vasopressors. Extending the action space to include antibiotic timing decisions (early broad-spectrum vs. waiting for cultures) adds clinical relevance but increases the action space complexity. Each hour of antibiotic delay in sepsis is associated with increased mortality, making this a high-value extension.

Reward function experimentation. The reward function encodes your clinical priorities, and small changes produce meaningfully different policies. Parameterize the reward weights (survival bonus, SOFA improvement coefficient, lactate clearance bonus) and train multiple policies in parallel across a grid of reward configurations. Use SageMaker Experiments to track which reward parameters produced which policy, then compare all candidate policies using the OPE pipeline. This reveals how sensitive the policy is to reward design choices. If two very different reward functions produce similar policies, that's reassuring. If small reward changes flip the policy's behavior, the reward function needs more careful calibration with clinical input.

Explainable policy with counterfactual reasoning. Instead of just recommending an action, show the clinician: "If you give 500 mL more fluid, the model estimates MAP will improve by 5 mmHg within 2 hours. If you increase vasopressors instead, the estimated MAP improvement is 8 mmHg but with higher risk of arrhythmia." This requires learning a transition model in addition to the policy, but dramatically improves clinical trust and utility.

Additional Resources

AWS Documentation:

• Amazon SageMaker Training Jobs
• Amazon SageMaker Real-Time Inference
• AWS Glue ETL Jobs
• AWS Step Functions
• AWS HIPAA Eligible Services
• Architecting for HIPAA on AWS
• Amazon SageMaker Pricing

Research References:

• Komorowski, M., Celi, L.A., Badawi, O., Gordon, A.C., & Faisal, A.A. (2018). The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24(11), 1716-1720.
• MIMIC-IV Clinical Database: Freely available critical care dataset (requires credentialed access) commonly used for sepsis RL research.
• Raghu, A., Komorowski, M., et al. (2017). Deep Reinforcement Learning for Sepsis Treatment. NeurIPS Workshop on ML for Health.
• Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.

AWS Sample Repos:

• amazon-sagemaker-examples: SageMaker training and deployment patterns including custom algorithm containers
• aws-healthcare-lifescience-ai-ml: Healthcare-specific ML examples on AWS including clinical data processing patterns

AWS Solutions and Blogs:

• Machine Learning for Healthcare on AWS: Overview of AWS ML services for healthcare use cases
• AWS for Health: Healthcare-specific compliance and architecture guidance

Estimated Implementation Time

← Main Recipe 15.4 · Python Example · Chapter Preface