Chapter 6: Clustering & Patient Segmentation
Finding the Patients Who Look Like Each Other
Every healthcare organization has the same dirty secret: they treat their patient population as if it were one homogeneous group. Same outreach letters. Same care management protocols. Same risk thresholds. Same intervention timing. And then they wonder why their diabetes management program works brilliantly for some patients and completely fails for others.
The answer is obvious once you say it out loud: because "patients with diabetes" is not one group. It's dozens of groups. The 72-year-old with well-controlled Type 2 on metformin who sees her PCP every quarter is not the same patient as the 34-year-old with brittle Type 1 who cycles through the ED every few months. They share a diagnosis code. They share almost nothing else. Treating them identically is not just inefficient; it's a failure of imagination.
Cohort analysis and clustering is the discipline of finding meaningful groups within your patient population. Not the groups you already know about (age brackets, diagnosis codes, payer types) but the groups that emerge from the data itself. The ones that nobody drew on a whiteboard because nobody knew they existed until an algorithm found the pattern.
This is one of those areas where the technology is genuinely mature, the algorithms are well-understood, and the hard part is entirely about healthcare context. The math is the easy part. Deciding what "similar" means for patients? That's where it gets interesting.
What Clustering Actually Does
At its core, clustering is unsupervised learning. You hand the algorithm a bunch of data points (patients, providers, encounters, whatever) described by a set of features, and you ask: "find me groups of things that are more similar to each other than they are to things in other groups." No labels. No training examples. No "here's what a high-risk patient looks like." Just: find the structure.
This is fundamentally different from classification, where you already know the categories and you're training a model to assign new items to existing buckets. Clustering discovers the buckets. That's both its power and its danger. Power because it can find patterns humans never thought to look for. Danger because it can also find patterns that are statistically real but clinically meaningless.
The simplest mental model: imagine plotting your patients on a scatter plot where the X axis is "number of ED visits per year" and the Y axis is "number of chronic conditions." You'd probably see clumps. Some patients cluster in the low-low corner (healthy, rarely use services). Some cluster in the high-high corner (complex, frequent utilizers). Some are in the high-visits-but-low-conditions zone (maybe behavioral health, maybe social determinants driving ED use). Those clumps are clusters. The algorithm finds them without you having to define the boundaries in advance.
Now scale that from two dimensions to two hundred. Age, diagnoses, medications, lab values, utilization patterns, social determinants, geographic factors, payer information, provider relationships. The human brain can't visualize 200-dimensional space, but clustering algorithms navigate it just fine. That's where the real value lives: in the high-dimensional patterns that are invisible to human intuition.
The Algorithmic Landscape
You don't need to be a mathematician to build useful clustering systems, but you do need to understand the trade-offs between approaches. Here's the landscape at a practical level.
K-Means and Its Variants
The workhorse. You tell it how many clusters you want (k), it iterates until it finds k groups that minimize within-group distance. Fast, scalable, easy to explain to stakeholders. The catch: you have to pick k in advance, it assumes roughly spherical clusters, and it's sensitive to outliers. For healthcare, k-means works great when you have a reasonable hypothesis about how many segments exist (say, 4-6 utilization tiers) and your features are relatively well-behaved numerically.
K-means++ improves initialization. Mini-batch k-means handles large datasets. K-medoids (PAM) is more robust to outliers because it uses actual data points as cluster centers rather than computed means.
Hierarchical Clustering
Builds a tree (dendrogram) of nested clusters, either bottom-up (agglomerative, starting with each point as its own cluster and merging) or top-down (divisive, starting with one cluster and splitting). The beauty is you don't have to pick k in advance; you can cut the tree at whatever level makes clinical sense. The cost: it doesn't scale well to large populations. Fine for a few thousand patients, painful for a few hundred thousand.
Useful when you want to explore the natural hierarchy of patient similarity. "These 50 patients are very similar. They're part of a broader group of 200 who share some characteristics. That group of 200 is part of a larger segment of 1,000." That nested structure is often clinically meaningful.
DBSCAN and Density-Based Methods
Instead of assuming clusters are spherical blobs, density-based methods find regions of high density separated by regions of low density. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is the classic. It has two huge advantages for healthcare: it automatically determines the number of clusters, and it explicitly identifies outliers (points that don't belong to any cluster). In healthcare, those outliers are often the most interesting patients.
The downside: it struggles with clusters of varying density, and parameter tuning (epsilon, min_points) requires domain knowledge. HDBSCAN improves on this substantially and is often the better choice for real-world healthcare data.
Gaussian Mixture Models
GMMs assume your data is generated by a mixture of Gaussian distributions. Each cluster is a Gaussian with its own mean and covariance. Unlike k-means, GMMs give you soft assignments: "this patient is 70% likely to belong to cluster A and 30% likely to belong to cluster B." That probabilistic membership is genuinely useful in healthcare, where patients often don't fit neatly into one box.
Dimensionality Reduction + Clustering
When you have hundreds of features (which is common in healthcare; think of all the diagnosis codes, lab values, and medications a patient might have), clustering directly in that high-dimensional space often produces poor results. The "curse of dimensionality" means that distance metrics become less meaningful as dimensions increase. The standard approach: reduce dimensions first (PCA, UMAP, t-SNE, autoencoders), then cluster in the reduced space. This two-step pipeline is the most common pattern you'll see in production healthcare clustering systems.
Why Healthcare Makes This Hard
Clustering algorithms are well-understood. The math has been stable for decades. So why isn't every health system running sophisticated patient segmentation? Because healthcare data is uniquely hostile to clustering in ways that don't show up in textbook examples.
The Feature Engineering Problem
What makes two patients "similar"? This sounds like a simple question until you try to answer it. Similar in what sense? Clinically? Financially? Behaviorally? Demographically? The choice of features determines what clusters you find, and that choice is a clinical decision, not a technical one.
Include diagnosis codes and you'll find disease-based clusters. Include utilization patterns and you'll find behavioral clusters. Include social determinants and you'll find vulnerability clusters. Include all of them and you'll find... something. Whether that something is clinically actionable depends entirely on whether you chose the right features for the right question.
The Mixed Data Type Problem
Healthcare data is a mess of types. Continuous (lab values, age, BMI), categorical (diagnosis codes, payer type, race), ordinal (pain scale, functional status), binary (smoker/non-smoker), temporal (sequence of encounters), and text (clinical notes). Most clustering algorithms expect a single distance metric across all features. Computing "distance" between a hemoglobin A1c value and an ICD-10 code requires careful thought about encoding and normalization.
The Missing Data Problem
Healthcare data is incomplete by nature. Patients who don't get labs don't have lab values. Patients who don't fill out social determinant screenings don't have SDOH data. The missingness itself is informative (a patient with no labs in two years is telling you something) but most clustering algorithms can't handle missing values natively. Imputation strategies matter enormously and can introduce bias.
The Temporal Problem
Patients change over time. A patient who was a "healthy, low-utilizer" three years ago might be a "complex, high-utilizer" today after a cancer diagnosis. Static clustering (one snapshot in time) misses trajectories. But temporal clustering is substantially harder and less well-tooled.
The Validation Problem
This is the big one. In supervised learning, you have ground truth: the model either predicted correctly or it didn't. In clustering, there's no ground truth. How do you know your clusters are "good"? Internal metrics (silhouette score, Davies-Bouldin index) tell you about mathematical separation but not clinical meaning. External validation requires clinical experts to review the clusters and say "yes, these groups make sense and are actionable." That's expensive and subjective.
The Equity Dimension
I want to flag something that doesn't get enough attention in technical discussions of patient clustering: these algorithms can encode and amplify existing disparities.
If your training data reflects a system where Black patients receive fewer referrals to specialists, your utilization-based clusters will show Black patients as "lower utilizers" when the reality is "under-served." If your features include zip code (even indirectly through geographic clustering), you're encoding structural racism into your segments. If your "high-risk" cluster disproportionately contains patients from marginalized communities, is that because they're genuinely higher risk, or because the system has failed them in ways that create risk?
Every recipe in this chapter includes a section on bias considerations. Not because it's fashionable, but because clustering without equity analysis is genuinely dangerous in healthcare. You can build a technically excellent segmentation model that systematically disadvantages vulnerable populations if you're not careful.
How This Chapter Progresses
The ten recipes in this chapter move from straightforward, well-bounded clustering problems to genuinely complex research-grade challenges.
Recipes 6.1-6.3 start with clustering problems where the features are clear, the interpretation is straightforward, and the results drive operational (not clinical) decisions. Geographic clustering, utilization segmentation, and financial risk grouping. These are the "quick wins" where you can demonstrate value without needing deep clinical validation.
Recipes 6.4-6.5 introduce clinical complexity. Disease severity stratification requires clinical domain knowledge to select features and validate results. Provider practice pattern analysis adds political sensitivity (providers don't love being clustered and compared). These recipes teach you how to work with clinical stakeholders and handle the human side of clustering.
Recipes 6.6-6.7 tackle similarity-based problems where the goal isn't just "find groups" but "find patients like this specific patient." Patient similarity for care planning and clinical trial matching require careful feature selection, validated similarity metrics, and real-time inference. The architecture patterns shift from batch analytics to interactive systems.
Recipes 6.8-6.10 are the complex, research-adjacent problems. Disease subtype discovery, social determinant phenotyping, and multi-morbidity pattern discovery all involve unsupervised learning on high-dimensional clinical data with no ground truth. These recipes teach validation strategies, clinical collaboration patterns, and how to handle the uncertainty inherent in discovering new knowledge from data.
What You'll Learn
By the end of this chapter, you'll understand:
- How to choose the right clustering algorithm for your specific healthcare problem
- Feature engineering strategies for mixed healthcare data types
- Validation approaches when there's no ground truth (which is most of the time)
- How to present clustering results to clinical stakeholders who don't speak "silhouette score"
- Architecture patterns for both batch segmentation and real-time similarity search
- Where clustering creates genuine clinical value versus where it's just an interesting academic exercise
- How to audit your clusters for equity and bias before they drive decisions
The technology here is mature. The algorithms work. The hard part, and the part these recipes focus on, is making clustering useful in a healthcare context where "interesting pattern" isn't good enough. The clusters have to drive action. They have to be interpretable by clinicians. They have to be fair. And they have to update as patients change.
Let's start with the simplest version of the problem and build from there.