Chapter 9: Medical Imaging & Computer Vision

Teaching Machines to See What Doctors See

Radiology has a dirty secret: the bottleneck isn't the imaging equipment. MRI machines, CT scanners, digital X-ray systems. These are engineering marvels that produce exquisitely detailed images in seconds. The bottleneck is the human sitting in a dark room, scrolling through hundreds of slices, looking for the one subtle finding that changes a patient's treatment plan. And that human is exhausted, overworked, and reading more studies per day than the profession was designed for.

Here's what fascinates me about computer vision in healthcare: we're not trying to replace the radiologist's judgment. We're trying to give them superhuman triage. Imagine a system that pre-reads every chest X-ray in the queue and says "these three have something that looks like a pneumothorax, read them first." The radiologist still makes the call. But the patient with a collapsed lung doesn't wait four hours behind routine follow-ups.

That's the simple version. The complex version involves pathology slides with billions of pixels, surgical video streams analyzed in real time, and multi-modal imaging fusion that combines PET, CT, and MRI into a unified interpretation. This chapter covers the full spectrum.

What Computer Vision Actually Does With Medical Images

At its core, computer vision is pattern recognition on pixel grids. You feed the model an image (or a volume of images, in the case of CT/MRI), and it learns to identify structures, anomalies, or measurements that correlate with clinical findings. The fundamental operations are:

Classification: Is this image normal or abnormal? Does this lesion look benign or malignant? This is the simplest task. One image in, one label out.

Detection: Where in this image is the finding? Draw a bounding box around the nodule, the fracture, the hemorrhage. Harder than classification because the model needs spatial awareness, not just a global judgment.

Segmentation: Outline the exact boundary of the structure. Trace the tumor margin pixel by pixel. Measure the volume of the ventricle. This is the most demanding task because it requires per-pixel classification across the entire image.

Registration: Align two images taken at different times or from different modalities so you can compare them directly. Did the tumor grow? Is the PET hotspot in the same location as the MRI lesion?

Each of these operations maps to different clinical workflows, and the recipes in this chapter progress through them roughly in order of complexity.

Why Medical Imaging Is Uniquely Hard for AI

If you've worked with computer vision in other domains (autonomous vehicles, manufacturing inspection, retail), you might think medical imaging is just another application. It's not. Several properties make it genuinely distinct:

The Images Are Enormous

A standard photograph is maybe 12 megapixels. A chest X-ray is similar. But a whole-slide pathology image? That's 100,000 x 100,000 pixels. A billion pixels in a single image. You can't feed that into a neural network. You have to tile it, process patches, and reassemble results. The engineering challenge of handling gigapixel images at scale is non-trivial, and it shapes the entire architecture of pathology AI systems.

CT and MRI volumes add a third dimension. A chest CT might be 512 x 512 x 300 slices. That's 78 million voxels per study. Processing these efficiently requires specialized architectures (3D convolutions, or clever 2.5D approaches that process adjacent slices together).

Subtle Findings Matter

In autonomous driving, the objects you need to detect are large, obvious, and high-contrast. A pedestrian is clearly different from a road surface. In medical imaging, the finding you're looking for might be a 3mm nodule in a field of similar-looking tissue. The difference between a benign calcification and an early malignancy might be a subtle textural pattern that even experienced radiologists disagree about.

This means you need models with extremely high sensitivity (you cannot miss a cancer) while maintaining reasonable specificity (you cannot flag everything as suspicious, or the radiologist drowns in false positives). The operating point on the ROC curve matters enormously, and it's a clinical decision, not just a technical one.

Ground Truth Is Expensive and Ambiguous

Training a computer vision model requires labeled data. In consumer applications, you can crowdsource labels cheaply. In medical imaging, your labels need to come from board-certified specialists, and those specialists are expensive and busy. Worse, they often disagree. Inter-reader variability in radiology is well-documented: two radiologists looking at the same image will disagree on findings a meaningful percentage of the time.

This creates a philosophical problem: if your ground truth is noisy, what does "accuracy" even mean? Many medical AI systems are validated against consensus reads (multiple radiologists agreeing) rather than single-reader labels, which adds cost and complexity to the data pipeline.

Bias Is a Patient Safety Issue

If your training data comes predominantly from one demographic (which it often does, because academic medical centers that produce research datasets serve specific populations), your model may perform differently on patients from other demographics. A dermatology AI trained mostly on light skin may miss melanoma on dark skin. A chest X-ray model trained on adult images may fail on pediatric anatomy.

In healthcare, this isn't just a fairness concern. It's a patient safety concern. A model that works well on 80% of your population but fails on the other 20% is dangerous, because the failures are invisible unless you specifically test for them.

Regulatory Requirements Are Real

Unlike most software, medical imaging AI that makes or supports clinical decisions is regulated by the FDA (in the US) and equivalent bodies internationally. The regulatory pathway depends on the intended use:

Triage/prioritization (flagging studies for faster read): Generally lower regulatory burden, often 510(k) pathway
Computer-aided detection (highlighting potential findings): Moderate burden, requires clinical validation
Autonomous diagnosis (making a clinical determination without physician review): Highest burden, requires extensive clinical trials

The recipes in this chapter are explicit about where each use case falls on this spectrum. Some (image quality assessment, photo verification) don't trigger FDA oversight at all. Others (diabetic retinopathy screening, radiology triage) are squarely in regulated territory.

The Technology Stack

Medical imaging AI builds on the same deep learning foundations as general computer vision, but with domain-specific adaptations:

Convolutional Neural Networks (CNNs) remain the workhorse for most medical imaging tasks. Architectures like ResNet, DenseNet, and EfficientNet (originally developed for ImageNet classification) transfer surprisingly well to medical images with appropriate fine-tuning. The key insight is that low-level visual features (edges, textures, gradients) are universal; it's the high-level interpretation that's domain-specific.

U-Net and its variants dominate segmentation tasks. The encoder-decoder architecture with skip connections was literally invented for medical image segmentation (biomedical cell segmentation, specifically), and it remains the go-to architecture for outlining structures in medical images. If you're segmenting anything in healthcare imaging, you're probably starting with some flavor of U-Net.

Vision Transformers (ViT) are increasingly competitive with CNNs for medical imaging, particularly for tasks that benefit from global context (understanding the relationship between distant parts of an image). They're especially promising for pathology, where the relevant context might span large regions of a slide.

Transfer learning is essential because medical imaging datasets are small relative to general computer vision datasets. ImageNet has 14 million images. A typical medical imaging research dataset might have 10,000 to 100,000 images. Pre-training on natural images and fine-tuning on medical data is standard practice, and it works remarkably well despite the visual domain gap.

DICOM is the universal standard for medical image storage and transmission. Every recipe in this chapter deals with DICOM at some level. It's not just an image format; it's a container that includes patient demographics, acquisition parameters, and study metadata. Understanding DICOM is table stakes for medical imaging AI.

The Classic Failure Modes

Before we build anything, let's talk about where medical imaging AI breaks:

Distribution shift: The model was trained on images from GE scanners and deployed on Siemens scanners. Different manufacturers produce subtly different image characteristics, and models are more sensitive to these differences than you'd expect.
Prevalence mismatch: The model was trained on a dataset with 50% positive cases (because that's how you build a balanced training set) but deployed in a population where the prevalence is 2%. Your positive predictive value craters.
Edge-case blindness: The model has never seen a chest X-ray with a pacemaker, or a patient with situs inversus, or an image taken portable in the ICU with all the lines and tubes. Novel presentations break models that seemed robust on clean test sets.
Automation bias: Clinicians start trusting the AI output and stop looking carefully at cases the AI marks as normal. This is a human factors problem, not a technical one, but it's real and it's dangerous.
Calibration drift: The model's confidence scores become unreliable over time as the patient population or imaging equipment changes. A model that was well-calibrated at deployment may become overconfident or underconfident after six months.

Every recipe includes strategies for monitoring and mitigating these failure modes. The pattern is consistent: never deploy without ongoing performance monitoring, stratified by relevant subgroups.

How the Recipes Progress

The ten recipes in this chapter are ordered by complexity along several axes: regulatory burden, clinical risk, technical difficulty, and integration complexity.

Recipes 9.1-9.2 (Simple) start with non-diagnostic applications. Image quality assessment and patient photo verification don't require clinical interpretation, don't trigger FDA oversight, and use well-established computer vision techniques. These are great starting points because you get real operational value with manageable risk.

Recipes 9.3-9.5 (Simple to Medium) introduce clinical measurement and triage. Wound measurement, dermatology triage, and chest X-ray prioritization start touching clinical workflows but remain in the "decision support" category rather than autonomous diagnosis. The regulatory considerations become real here, but the technical approaches are well-validated.

Recipe 9.6 (Medium-Complex) covers diabetic retinopathy screening, which is notable because it's one of the few areas where autonomous AI diagnosis (without physician review) has received FDA clearance. This recipe explores what it takes to reach that bar.

Recipes 9.7-9.8 (Complex) tackle multi-modality radiology triage and pathology slide analysis. These involve massive data volumes, complex integration with existing clinical systems (PACS, LIS), and multiple finding types per study. The engineering challenges are substantial.

Recipes 9.9-9.10 (Complex) push into research-adjacent territory: surgical video analysis and multi-modal imaging fusion. These represent the frontier of what's possible, with real-time processing requirements and integration challenges that are still being solved.

A Note on Build vs. Buy

More than any other chapter in this book, computer vision in healthcare has a robust commercial ecosystem. Companies like Aidoc, Viz.ai, Paige, and dozens of others offer FDA-cleared algorithms for specific clinical use cases. The "build" recipes in this chapter are educational. They teach you how these systems work, what the architecture looks like, and where the hard problems are. For production clinical deployment, you'll likely integrate a commercial solution (or build on top of one) rather than training your own model from scratch.

That said, there are plenty of non-clinical computer vision applications (image quality, identity verification, wound measurement, operational analytics) where building your own is entirely reasonable. The recipes make clear which category each use case falls into.

Let's start with the simplest pattern: making sure the image is good enough to use before anyone tries to interpret it.

→ Recipe 9.1: Image Quality Assessment