Chapter 12: Forecasting & Time-Series Analysis

Teaching Machines to See the Future (Sort Of)

Healthcare generates time-stamped data the way a fire hydrant generates water pressure. Vital signs every few seconds in the ICU. Lab results drawn weekly, monthly, or whenever the patient remembers their appointment. Appointment volumes by the hour. Supply consumption by the shift. Revenue by the day. Census counts that change with every admission and discharge. Every one of these data points has a timestamp, and every one of them is part of a sequence that tells a story about what happened, what's happening now, and (if you're careful and a little bit lucky) what's about to happen next.

Time series analysis is the discipline of extracting signal from these sequences. Forecasting is the audacious extension of that: using historical patterns to predict future values. And healthcare is simultaneously one of the best and worst domains for this work. Best because the data volumes are enormous and the patterns are real (hospitals really do get busier on Mondays, flu season really does follow a curve, kidney function really does decline in a predictable-ish trajectory). Worst because the stakes are high, the data is messy, interventions change trajectories, and the consequences of being wrong range from "we overstaffed by two nurses" to "we missed early sepsis."

Here's the thing that makes time series in healthcare genuinely interesting as an engineering problem: the data doesn't behave like textbook time series data. Textbook examples assume regular sampling intervals, stationary distributions, and exogenous variables that stay put. Healthcare gives you irregular sampling (labs drawn "as needed"), non-stationary distributions (a hospital that just merged with another system has fundamentally different patterns than it did last year), and interventions that deliberately change the trajectory you're trying to forecast. A patient starts a new medication, and their lab values shift. A hospital opens a new wing, and census patterns reorganize. A pandemic hits, and every historical pattern becomes irrelevant overnight.

This chapter works through these challenges from simple to complex, starting with forecasting problems where being slightly wrong is merely inconvenient and ending with problems where real-time accuracy has clinical implications.

What Time Series Analysis Actually Is

At its core, time series analysis is about understanding the structure of sequential, time-stamped observations. You're looking for components that repeat and components that evolve:

Trend: The long-term direction. Is ED volume growing 3% per year? Is this patient's A1C slowly climbing? Trend tells you where the baseline is heading.

Seasonality: Patterns that repeat at fixed intervals. Monday mornings are busy. January is flu season. Q4 has more elective surgeries (patients have met their deductible). Seasonality is the most exploitable pattern in forecasting because it's predictable by definition.

Cyclicality: Patterns that repeat but not at fixed intervals. Economic downturns affect elective procedure volumes. Pandemic waves come in surges that aren't calendar-aligned. Cyclicality is harder to forecast than seasonality because you can't just look at "what happened 12 months ago."

Noise: The random variation that no model can predict. A multi-car accident floods the ED. A snowstorm keeps patients home. Noise is irreducible, and understanding where your signal ends and noise begins is one of the most important skills in time series work.

Interventions and changepoints: This is where healthcare gets weird. In most forecasting domains, you assume the data-generating process is stable. In healthcare, you deliberately change it. A new protocol reduces length of stay. A marketing campaign increases appointment bookings. A staffing change alters discharge timing. Your model needs to either detect these changepoints or be told about them, and most classical time series methods aren't great at either.

The Spectrum of Forecasting Difficulty

Not all forecasting problems are created equal. The difficulty depends on several factors that interact in ways that aren't always obvious:

Easy: Aggregate, Operational, Tolerant of Error

Appointment volume forecasting (Recipe 12.1) is about as friendly as time series gets. You have years of historical data. The patterns are seasonal and trend-driven. Being off by 10% means you slightly overstaffed or understaffed, and tomorrow is a fresh chance to get it right. The forecast horizon is short (days to weeks), and you're predicting aggregate counts, not individual outcomes.

Medium: Patient-Level, Clinical Context, Irregular Sampling

Lab result trend analysis (Recipe 12.4) sits in the middle. You're tracking individual patients over time, which means smaller sample sizes and more noise per series. Labs aren't drawn at regular intervals (every 3 months if the patient is compliant, every 3 days if they're inpatient, sporadically if they're not engaged in care). You need to separate "this value is high because the patient is genuinely worsening" from "this value is high because they were dehydrated when the blood was drawn." Clinical context matters, and your model doesn't have most of it.

Hard: Real-Time, High-Stakes, Multi-Variate, Streaming

Vital sign trajectory monitoring (Recipe 12.7) and physiological waveform analysis (Recipe 12.10) are at the complex end. You're processing continuous streams of data (sometimes at 250+ samples per second for ECG). Latency matters because you're trying to detect deterioration before it becomes obvious to the clinical team. False positives trigger alert fatigue, which is a genuine patient safety problem. And the data comes from devices that introduce their own artifacts (patient moved, lead fell off, nurse was adjusting the sensor).

Why Healthcare Time Series Is Different

If you've done time series work in finance, retail, or energy, you'll find healthcare familiar in structure but alien in the details. Here's what catches people off guard:

Irregular Sampling Is the Norm

In retail forecasting, you get a daily sales number. Every day. No gaps. In healthcare, a patient might have labs drawn on January 5, March 22, July 1, and then not again for two years. The intervals aren't just irregular, they're informative: the reason for the gap (patient was healthy, patient was non-compliant, patient switched providers, patient died) changes the interpretation of the surrounding values. This is sometimes called "informative missingness," and it's a research problem that doesn't have clean solutions yet.

Interventions Change the Future You're Predicting

When you forecast retail demand, you assume the customer's purchasing behavior isn't being actively manipulated by your forecast. In healthcare, the whole point is often to intervene. You predict a patient is declining so that a clinician can change their treatment. If the intervention works, your forecast was "wrong" in the best possible way. This creates a fundamental challenge: your historical data includes interventions that altered outcomes, and your future predictions need to account for interventions that haven't happened yet.

The Consequences Are Asymmetric

Understaffing an ED has different consequences than overstaffing it. Missing early sepsis is not the same as a false alert. In most healthcare forecasting, the cost of a false negative (missed event) dramatically exceeds the cost of a false positive (unnecessary action). This asymmetry should inform your choice of loss function, threshold, and evaluation metrics. Accuracy is rarely the right metric; sensitivity at a fixed specificity, or cost-weighted error, is closer to what matters.

Stationarity Is a Fiction

Healthcare systems change constantly. Mergers and acquisitions reshape patient populations. New providers join and leave. Protocols change. Pandemics happen. Reimbursement rules shift, which changes what procedures get scheduled. The assumption that "the future will look like the past" is always a simplification, but in healthcare it's a particularly aggressive one. Your models need to either be retrained frequently or be inherently adaptive.

Regulatory and Privacy Constraints Shape Architecture

Time series data in healthcare is almost always PHI. Vital signs, lab results, census counts (when combined with dates), and appointment records all fall under HIPAA. This means your forecasting infrastructure needs encryption at rest and in transit, audit logging, access controls, and BAA-covered compute environments. You can't just spin up a Jupyter notebook on a public cloud instance and start training on patient data. The architecture decisions in this chapter all assume HIPAA compliance as a baseline requirement.

The Methods, Briefly

The recipes in this chapter use a range of techniques. Here's a quick orientation so you know what you're walking into:

Classical statistical methods (ARIMA, ETS, Prophet): These remain excellent for operational forecasting where patterns are seasonal and data is plentiful. Don't let anyone tell you that deep learning has replaced ARIMA for predicting next week's appointment volume. It hasn't. The simple methods are often more interpretable, faster to train, and just as accurate for well-behaved series.

Machine learning approaches (gradient boosted trees, random forests): When you have rich feature sets (day of week, weather, events, historical patterns as engineered features), ML methods that treat forecasting as a regression problem can be surprisingly effective. They handle non-linearities and feature interactions well. The downside is they don't naturally model temporal dependencies, so you're encoding time structure through feature engineering.

Deep learning (LSTMs, Transformers, temporal CNNs): These shine when you have lots of related series (thousands of patients, hundreds of supply items) and want to learn shared patterns across them. They're also the go-to for streaming physiological data where the temporal structure is complex and the features are learned rather than engineered. The cost is interpretability and data requirements.

Probabilistic forecasting: Several recipes emphasize predicting distributions rather than point estimates. When you forecast "the ED will see 42 patients tomorrow," that's less useful than "the ED will see between 35 and 50 patients tomorrow with 90% probability." Uncertainty quantification is critical in healthcare because decisions depend on worst-case scenarios, not averages.

HIPAA and PHI Considerations

Every recipe in this chapter handles data that is or could be PHI. The key compliance patterns you'll see repeated:

• All compute runs in BAA-covered environments
• Data at rest is encrypted with KMS-managed keys
• Data in transit uses TLS 1.2+
• Access is logged via CloudTrail (or equivalent audit mechanism)
• Models are trained on de-identified or limited data sets where possible
• Real-time predictions use VPC-isolated endpoints
• Patient-level forecasts are treated as clinical data and stored accordingly

The operational forecasting recipes (appointment volume, supply inventory, census) may seem lower-risk because they're aggregate numbers, but be careful. If you can determine that "Patient X had an appointment on Tuesday" from the training data, your aggregate model just became a PHI processing system. The line between aggregate and individual is thinner than it looks.

How This Chapter Is Organized

We start with problems where the data is clean, the patterns are strong, and being wrong is merely inconvenient. We end with problems where the data is streaming, the signal is subtle, and being wrong has immediate clinical implications.

The progression isn't just about technical complexity. It's also about consequence. The early recipes optimize operations. The later recipes inform clinical decisions. That distinction matters for how you architect, validate, and deploy these systems.

A Note on Evaluation

One thing that trips people up in healthcare forecasting: your standard evaluation metrics might be lying to you. Mean Absolute Error looks great when your model is accurate 95% of the time but catastrophically wrong during the 5% that matters most (the COVID surge, the ice storm, the unexpected sepsis). Always evaluate your models on the tails, not just the average. If your ED forecasting model is perfect on normal Tuesdays but useless during mass casualty events, it's failing at exactly the moment you need it most.

Every recipe includes honest discussion of what "good performance" actually means in context, because in healthcare, "the model is 93% accurate" is never the end of the conversation.

Let's start with the friendliest problem in the chapter: predicting how many patients will show up next week.

Next: Recipe 12.1: Appointment Volume Forecasting

Further Reading

• Forecasting: Principles and Practice (Hyndman & Athanasopoulos) - the gold standard open textbook on time series forecasting methods
• Prophet: Forecasting at Scale (Meta/Facebook Research) - widely used library for business time series with strong seasonality
• GluonTS: Probabilistic Time Series Modeling - deep learning toolkit for time series that supports uncertainty estimation