A pretrained hand keypoint model that puts all 21 landmarks in the right place on a benchmark clip looks finished. It is not. The demo tells you the model can localize a hand when the hand is fully visible, evenly lit, and roughly facing the camera — which is exactly the condition an in-cabin monitoring system spends the least time in. The moment a hand crosses out of frame, grips the wheel at an angle, or moves fast enough to blur, the interesting behaviour begins, and none of it shows up in the number you validated on.

That gap is the whole subject here. Hand keypoint detection is a well-understood perception task with mature tooling. What is less well understood — and what routinely bites teams shipping it into a cabin — is that a keypoint estimator has its own degradation surface, and the aggregate landmark accuracy you read off a hand-pose benchmark tells you almost nothing about how that surface behaves under the conditions that actually matter.

How does hand keypoint detection work in practice?

At its core, hand keypoint detection takes an image (or a cropped hand region) and predicts the 2D — sometimes 3D — positions of a fixed set of anatomical landmarks. The common convention, popularized by MediaPipe Hands and mirrored across most open datasets, is 21 landmarks: one at the wrist and four along each finger (the metacarpophalangeal, proximal, distal joints, and the fingertip). The model’s job is to place each of those 21 points on the pixel where that joint actually is.

In practice, a keypoint detector is rarely the end of the pipeline. It is an estimator feeding downstream logic — gesture classification, hand-on-wheel detection, reach-and-grab intent, distraction scoring. That downstream logic consumes the landmark coordinates and, if it is well built, the per-landmark confidence. The distinction matters: a keypoint model does not output “the hand.” It outputs a set of point estimates, each with an implied or explicit uncertainty, and the system on the other side has to decide what to do when those estimates are shaky.

Treating the detector as a component with a contract — coordinates plus confidence, valid within a stated operating envelope — is the shift from a demo mindset to a production mindset. It is the same discipline the broader autonomous vehicle perception failure modes that survive benchmarks demand of every estimator in the stack, narrowed here to a single hand-pose head.

What outputs does a keypoint detector actually produce?

Two things, and the second is the one teams underuse.

The first is the landmark coordinates — for 21 landmarks, that’s 21 (x, y) pairs, or (x, y, z) if the model regresses relative depth. These are usually normalized to the input crop, which is why calibration and crop geometry feed straight into interpretation; a landmark at normalized (0.5, 0.5) means nothing until you know the crop it lives in.

The second is a per-landmark confidence or visibility score. In heatmap-based models this falls out naturally as the peak value of each landmark’s response map; in coordinate-regression models it is a separately learned scalar. This score is the model’s own signal that a given point is uncertain — for example, because the joint is occluded or off-frame. A pipeline that reads coordinates but discards confidence throws away the one output that lets it distinguish a well-localized landmark from a confident guess. That is the difference between a system that degrades gracefully and one that fails silently.

Heatmap regression versus direct coordinate regression: what trade-offs?

The two dominant architectural families answer the same question — where is each landmark — in structurally different ways, and the choice carries consequences that show up under exactly the hard conditions a cabin imposes.

Axis	Heatmap regression	Direct coordinate regression
Output	Per-landmark 2D response map, argmax → coordinate	Direct (x, y) values from a regression head
Spatial precision	High; sub-pixel via soft-argmax	Lower; bounded by head capacity
Confidence signal	Natural (peak value, map sharpness)	Learned separately, less interpretable
Occluded / off-frame landmarks	Flat / diffuse map signals uncertainty	Head still emits a plausible-looking point
Compute & memory	Higher (map resolution scales cost)	Lower; friendlier to embedded targets
Failure texture	Uncertainty is visible in the map	Failure is quiet — a wrong point looks like a right one

The trade-off that matters most in a cabin is the last row. A heatmap model whose landmark is occluded tends to produce a flat, low-peak, or multi-modal map — the uncertainty is legible if you look at it. A coordinate-regression head, by contrast, is trained to always emit a point, so when a fingertip leaves the frame it will happily place the landmark somewhere plausible with no built-in tell. Neither is wrong; the point is that the architecture shapes how the model fails, and that shape is a property you have to test for rather than assume. This is analogous to the estimator-behaviour distinctions we draw in body pose estimation for automotive perception, where the full-body pose head faces the same occlusion-vs-confidence tension at a larger scale.

Why does strong aggregate accuracy still emit confident-but-wrong keypoints?

Because aggregate metrics average over a frame distribution that is dominated by easy frames. A hand-pose benchmark reports a single Percentage of Correct Keypoints (PCK) or mean localization error over its test split, and that split is mostly clean, front-facing, well-lit hands. A model can score extremely well on that average while being systematically wrong on the small fraction of hard frames — and in a cabin, the hard frames are the ones the safety story cares about.

The mechanism is worth naming precisely: a keypoint model has a long-tail failure distribution that a mean metric smooths out. Push the input toward partial occlusion, motion blur, or off-axis geometry and both the localization error and the false-confidence rate climb — but because those inputs are rare in the benchmark, they barely move the headline number. Validating on aggregate landmark accuracy is measuring the model where it is strong and looking away from where it is weak.

This is the same reasoning we apply when writing a robustness specification for an automotive perception model: the specification exists precisely to force per-scenario evaluation instead of a single blended score. A keypoint head belongs under that same specification discipline.

How does the model behave under occlusion, motion blur, and off-axis geometry?

Each degradation class stresses the estimator differently, and each produces a characteristic failure signature you can measure — if you build the test slices to expose it.

Partial occlusion. When a hand grips the wheel or another object hides fingers, the affected landmarks lose their visual evidence. A well-behaved model drops confidence on the occluded points; a poorly behaved one hallucinates them from the learned hand prior. The metric that catches this is the false-confidence rate on occluded landmarks — the fraction of occluded points the model reports as high-confidence yet mislocalizes. In our experience across in-cabin perception work, this metric moves independently of aggregate PCK, which is why it earns its own slice.

Motion blur. A fast reach or gesture smears the hand across pixels. Localization error rises because the joint is no longer a sharp point, and the effect is worst on the fast-moving fingertips. Blur-augmented evaluation frames — synthetically blurred or captured at representative shutter speeds — expose how error scales with motion magnitude. (This is an observed pattern from perception validation, not a published benchmark rate; the exact degradation curve is model- and sensor-specific.)

Off-axis camera geometry. Cabin cameras rarely see the hand head-on. Steep viewing angles foreshorten the hand and push it toward the edge of the lens where distortion is highest, so the landmark distribution the model sees at inference differs from its training distribution. Off-axis test slices — hands at the actual mounting angles and frame positions the deployed camera produces — reveal a systematic bias the front-facing benchmark never touches. The link between mounting geometry and estimator behaviour is why how extrinsic calibration feeds camera pose in perception is a companion concern, not a separate one.

What scenario classes should an in-cabin test set include?

Use this as a coverage checklist. Each row is a slice the test set should exercise explicitly, with its own pass/fail evidence rather than folding into an aggregate.

• Full occlusion transitions — hand entering and leaving frame, gripping the wheel, holding objects. Target metric: false-confidence rate on occluded/absent landmarks.
• Motion blur band — reaches and gestures at representative speeds and shutter settings. Target metric: per-landmark localization error (pixel / PCK) as a function of motion magnitude.
• Low-light and IR — night driving, backlit windows, IR-illuminated capture. Target metric: localization error and confidence collapse under low SNR.
• Off-axis geometry — hands at the real mounting angle and at frame edges where lens distortion peaks. Target metric: systematic localization bias vs. front-facing baseline.
• Downstream-response probes — inject uncertain or missing keypoints and observe whether the consuming logic degrades gracefully or misfires. Target metric: reduction in downstream misfires attributable to catching keypoint-blind regressions before sign-off.

The last row is the one that closes the loop. A keypoint model does not exist to score well in isolation; it exists to feed something. Testing how the downstream logic reacts when keypoints are uncertain or missing is what turns “the estimator degrades” into “the system degrades safely” — the property a release reviewer actually needs to see.

FAQ

What’s worth understanding about hand keypoint detection first?

A hand keypoint detector predicts the 2D or 3D positions of a fixed set of anatomical landmarks — conventionally 21, covering the wrist and each finger’s joints — from an image or cropped hand region. In practice it is an estimator feeding downstream logic like gesture or hand-on-wheel detection, so it outputs point estimates with uncertainty rather than a finished decision. Treating it as a component with a contract (coordinates plus confidence, valid within a stated envelope) is the shift from demo to production.

What model outputs does a keypoint detector produce — landmark coordinates, confidence scores, and how are they represented?

It produces landmark coordinates — 21 (x, y) or (x, y, z) values, usually normalized to the input crop — and a per-landmark confidence or visibility score. In heatmap models the confidence falls out as the peak value of each landmark’s response map; in coordinate-regression models it is a separately learned scalar. A pipeline that reads coordinates but discards confidence loses the one signal that distinguishes a well-localized point from a confident guess.

What are the main architectural approaches to hand keypoint detection, and what trade-offs do they carry?

The two families are heatmap regression, which predicts a per-landmark response map and takes its argmax, and direct coordinate regression, which emits (x, y) values from a regression head. Heatmaps give higher spatial precision and a natural, legible uncertainty signal but cost more compute; coordinate regression is lighter and embedded-friendly but tends to emit a plausible-looking point even when the landmark is occluded. The architecture shapes how the model fails, which is a property to test for rather than assume.

What keypoint-specific failure modes does a perception robustness audit need to test for?

Partial occlusion (landmarks losing visual evidence and being hallucinated from the hand prior), motion blur (localization error rising, worst on fast fingertips), low light, and off-axis camera geometry (foreshortening and edge distortion shifting the input distribution). Each produces a characteristic signature — most usefully the false-confidence rate on occluded landmarks — that moves independently of aggregate PCK. The audit needs per-scenario slices to expose them.

Why can a model with strong aggregate landmark accuracy still emit confident-but-wrong keypoints on hard frames?

Because aggregate metrics average over a frame distribution dominated by easy, front-facing, well-lit hands, and a keypoint model’s failures live in the long tail. A model can score well on mean PCK while being systematically wrong on the small fraction of occluded, blurred, or off-axis frames — the exact frames a cabin safety story cares about. Validating on aggregate accuracy measures the model where it is strong and looks away from where it is weak.

What scenario classes should an in-cabin test set include to exercise keypoint robustness before release?

Full occlusion transitions (entering/leaving frame, gripping the wheel), a motion-blur band at representative speeds, low-light and IR conditions, off-axis geometry at the real mounting angle and frame edges, and downstream-response probes that inject uncertain or missing keypoints. Each slice should carry its own pass/fail evidence rather than folding into an aggregate. The downstream probe is what turns “the estimator degrades” into “the system degrades safely.”

Where this leaves a team about to ship

The pretrained model on the demo clip is not the risk. The risk is the inference you draw from it — that clean-frame accuracy generalizes to the cabin’s occlusion, blur, and geometry distribution. It does not, and the only way to know how far it falls short is to build the degradation slices and read the per-scenario numbers instead of the blended one.

Those slices are not a research exercise; they are edge-class coverage for the validation evidence a release reviewer signs against. The reliability-audit methodology we apply before sign-off treats keypoint occlusion, motion blur, and off-axis geometry as an explicit slice of the robustness audit — the same discipline that governs the rest of the computer vision perception stack and, at the system level, the way in-cabin sensing shapes the perception evidence it needs. If you want the harness that turns those slices into evidence a reviewer accepts, our Production AI Monitoring Harness is scoped to exactly this: exercising a perception estimator’s degradation surface and producing the validation record that catches keypoint-blind regressions before release.