Recipe 14.9 Architecture and Implementation: Chemotherapy Scheduling

Companion to Recipe 14.9: Chemotherapy Scheduling. This page covers the AWS architecture, services, prerequisites, and implementation guidance. For the problem framing, conceptual approach, and pseudocode walkthrough, start with the main recipe.

The AWS Implementation

Why These Services

Amazon SageMaker for model training and hosting. The duration prediction models (how long will this patient's infusion actually take, given their regimen, cycle number, and history?) are ML models that need training infrastructure and a hosting endpoint. SageMaker provides both, with the ability to retrain on new data as your center accumulates history.

AWS Lambda for the scheduling engine orchestration. The batch scheduling job (build tomorrow's schedule) runs once nightly. The real-time adjustment engine fires on events (patient arrival, delay notification, cancellation). Both are event-driven, short-lived compute tasks. Lambda's pay-per-invocation model fits perfectly.

AWS Step Functions for the multi-step scheduling workflow. The full scheduling pipeline (pull orders, predict durations, run optimizer, validate constraints, publish schedule, notify pharmacy) has multiple steps with error handling and retry logic. Step Functions provides the orchestration with built-in state management and visibility.

Amazon DynamoDB for schedule state. Nurses check the schedule board constantly. The patient portal refreshes every 30 seconds. Pharmacy needs instant visibility into timing changes. You need single-digit-millisecond reads on a data structure that updates maybe 50 times per day but gets read thousands of times. DynamoDB's read performance and conditional writes (preventing race conditions when multiple adjustments happen simultaneously) fit this access pattern exactly.

Amazon EventBridge for event routing. Schedule changes, patient arrivals, pharmacy completions, and disruption notifications all flow as events. EventBridge routes these to the appropriate handlers (Lambda functions) based on event type, enabling loose coupling between the scheduling engine and its consumers.

Amazon S3 for optimization artifacts. Solver logs, schedule history, model training data, and audit trails all land in S3. This supports both compliance (you need to explain why a patient was scheduled when they were) and continuous improvement (analyzing historical schedules to tune the optimizer). Solver logs contain PHI (patient IDs, regimens, scheduling decisions), so store them in a dedicated bucket with S3 Object Lock for compliance retention (minimum 6 years per HIPAA), lifecycle policies transitioning to Glacier after 90 days, and bucket policies restricting access to the scheduling service role and authorized administrators.

AWS HealthLake or Amazon RDS for clinical data. Treatment protocols, drug stability profiles, and patient treatment histories live in a clinical data store. HealthLake if you're working with FHIR resources; RDS (PostgreSQL) if you need relational queries across protocol definitions and scheduling rules.

Note on API access patterns: the staff dashboard and pharmacy system are internal consumers (hospital network only), while the patient portal is internet-facing. Consider separate API Gateway stages: a private API with IAM authentication for internal consumers, and a public API with WAF, rate limiting, and Cognito/OIDC authentication for the patient portal.

Architecture Diagram

flowchart TD
    A[Oncology EHR\nTreatment Orders] -->|FHIR/HL7| B[EventBridge\nOrder Events]
    B --> C[Lambda\nOrder Processor]
    C --> D[DynamoDB\nScheduling Queue]
    
    E[Step Functions\nBatch Scheduler] -->|Nightly| F[Lambda\nDuration Predictor]
    F -->|SageMaker Endpoint| G[ML Model\nInfusion Duration]
    E --> H[Lambda\nConstraint Optimizer]
    H -->|OR-Tools / Custom| I[Optimization Engine]
    I --> J[DynamoDB\nPublished Schedule]
    
    J --> K[API Gateway\nSchedule API]
    K --> L[Staff Dashboard]
    K --> M[Patient Portal]
    K --> N[Pharmacy System]
    
    L -->|Overrides| K
    K -->|Re-solve / Lock / Reassign| P[Lambda\nReal-Time Adjuster]
    
    O[EventBridge\nDisruption Events] --> P
    P --> J
    P --> N
    
    Q[S3\nSchedule History] --> E
    G --> Q

    style I fill:#ff9,stroke:#333
    style J fill:#9ff,stroke:#333
    style G fill:#f9f,stroke:#333

Human Override Mechanism

The optimizer produces a recommendation, not a commandment. Clinical staff must be able to override any scheduling decision. The staff dashboard provides four override actions:

Drag-and-drop reassignment. A charge nurse sees that Patient A is anxious about being near the window. She drags that patient's card from Chair 12 to Chair 5. The system validates the move (no overlap, nursing capacity still within bounds) and publishes the updated schedule. If the move creates a constraint violation, the dashboard shows the conflict and offers to re-solve the affected subset.

Assignment locking. Pharmacy has already prepped for Patient B in Chair 7 at 09:00. The charge nurse locks that assignment so subsequent re-solves cannot move it. Locked assignments become hard constraints in any future optimization pass.

Ad-hoc constraint addition. A patient calls ahead requesting to not be seated near another patient (family dispute, emotional response to seeing a specific regimen). The charge nurse adds a "not same section" constraint between the two patients and triggers a partial re-solve for the affected time window.

Re-solve requests. After several morning cancellations, the schedule has gaps. The charge nurse hits "re-optimize afternoon" to pack the remaining patients more efficiently. The re-solve treats locked assignments and already-started infusions as fixed, optimizing only the movable patients.

Every override writes an audit record: staff ID, timestamp, action type, before state, after state, and free-text reason. These records serve compliance (who changed the schedule and why?) and model improvement (if nurses always move first-dose patients to the chairs near the nursing station, that's a missing constraint the optimizer should learn).

Prerequisites

Ingredients

Failover and Graceful Degradation

The optimization layer enhances existing scheduling. It does not replace it. If the optimizer goes down, patients still need to be scheduled, pharmacy still needs to prep, and nurses still need assignments. Design for degradation from day one.

Batch Optimizer Failure

The Step Functions workflow that builds tomorrow's schedule runs at 2 AM. If it fails (solver timeout, infeasible constraints, Lambda error, SageMaker endpoint unavailable), the system has four hours to recover before staff arrive.

Retry policy. Step Functions retries the batch job twice with 15-minute backoff. If the solver hits the time limit without finding a feasible solution, relax soft constraints (drop preference satisfaction, allow slight nursing overload) and retry. Log the relaxation for review.

Fallback to template-based schedule. If the batch optimizer has not produced a published schedule by 6 AM, the system falls back to the template-based schedule. Every center already has one: it's the "default" schedule pattern that rotating patients get assigned to before optimization exists. The system copies last week's same-day template (e.g., last Monday's schedule as a template for this Monday), slots in today's patients by protocol duration, and publishes to DynamoDB with a "TEMPLATE_FALLBACK" status flag. Staff see a yellow banner: "Today's schedule was generated from the template. Manual adjustments may be needed."

Alert chain. CloudWatch alarm fires if the batch job has not written a schedule to DynamoDB by 5:30 AM. The alarm triggers SNS notification to the on-call engineer and the charge nurse's pager. Include the Step Functions execution URL in the alert so the engineer can diagnose immediately.

Real-Time Adjuster Timeout

The real-time adjuster fires on disruption events (patient late, cancellation, extended duration). It must respond quickly because staff are waiting.

Timeout threshold. If the real-time adjuster Lambda does not return a valid adjustment within 5 seconds, the invocation is considered failed. The function's timeout is set to 10 seconds (giving 5 seconds for the solver and 5 seconds for validation and persistence), but the caller treats anything over 5 seconds as a timeout.

Fallback to human queue. On timeout, the disruption event routes to the charge nurse's manual resolution queue in the staff dashboard. The event appears with a red flag: "Automatic resolution timed out. Manual reassignment needed." The staff dashboard still shows the current schedule (pre-disruption state) so the nurse has full context.

Consecutive failure detection. If 3 or more consecutive real-time adjustments exceed the 5-second threshold, CloudWatch fires an alarm. This pattern usually indicates either degraded SageMaker endpoint latency, DynamoDB throttling, or a schedule state that's become too fragmented for quick re-optimization. The alert tells the engineer to check endpoint health and consider a full re-solve of the remaining day.

Dashboard Availability

The staff dashboard must display the current schedule regardless of optimizer availability. It reads directly from DynamoDB (the published schedule table), not from the optimization engine. Even if every Lambda function is down, the dashboard shows the last-published schedule. The architecture separates "schedule generation" (can fail) from "schedule display" (must always work).

If the DynamoDB table itself becomes unreachable (region-level issue), the dashboard falls back to a cached copy in the browser's local storage, updated on every successful fetch. The cache shows a staleness indicator: "Last updated 3 minutes ago" in green, or "Last updated 45 minutes ago, connection issues" in amber.

Variations and Extensions

Multi-Site Scheduling

For health systems with multiple infusion centers, extend the optimization to include site assignment. A patient might be willing to drive 10 extra minutes to a less-busy center if it means a 2-hour-earlier appointment. The problem becomes a two-stage optimization: assign patients to sites, then schedule within each site. The site assignment can account for travel time, center specialization (some centers handle specific regimens better), and system-wide load balancing.

Predictive No-Show Management

Integrate a no-show prediction model (see Recipe 7.1) to identify patients likely to miss their appointment. For high-risk no-shows, consider strategic overbooking: schedule a waitlist patient into the same slot with a conditional confirmation. If the primary patient shows, the waitlist patient gets rescheduled. If they don't, the chair isn't wasted. This requires careful communication with patients and a robust waitlist management system.

Treatment Sequencing Optimization

Some patients receive multiple treatments across different departments (radiation in the morning, chemo in the afternoon, lab work before both). Extend the optimizer to coordinate across departments, minimizing total time the patient spends in the facility. This is a job-shop scheduling variant where each patient is a "job" with operations on different "machines" (departments).

Additional Resources

AWS Documentation

• Amazon SageMaker Developer Guide - Model training and endpoint hosting for duration prediction
• AWS Step Functions Developer Guide - Workflow orchestration for the scheduling pipeline
• Amazon DynamoDB Developer Guide - Low-latency schedule state storage
• Amazon EventBridge User Guide - Event-driven architecture for disruption handling
• AWS Lambda Developer Guide - Serverless compute for scheduling functions
• HIPAA Eligible Services Reference - Verify all services used are HIPAA eligible

Optimization Libraries and Solvers

• Google OR-Tools - Open-source CP-SAT solver, excellent for scheduling problems
• HiGHS - Open-source MIP solver, good for linear formulations
• OR-Tools can be packaged as a Lambda layer (~50MB) or deployed in a container image for Lambda or ECS. The CP-SAT solver is the recommended entry point for scheduling problems at infusion center scale.

Healthcare Scheduling Research

• Hahn-Goldberg et al. (2014), "Dynamic optimization of chemotherapy outpatient scheduling with uncertainty," Health Care Management Science
• Turkcan et al. (2012), "Chemotherapy operations planning and scheduling," IIE Transactions on Healthcare Systems Engineering
• Oncology Nursing Society (ONS) publishes infusion center staffing and operational guidelines relevant to nursing ratio constraints

Estimated Implementation Time

Tags: optimization, scheduling, chemotherapy, infusion-center, constraint-programming, resource-allocation, pharmacy-coordination, nursing-workload, operations-research

← Recipe 14.8: Ambulance Routing and Dispatch | Chapter 14 Index | Recipe 14.10: Health System Network Design →

← Main Recipe 14.9 · Python Example · Chapter Preface