Detection counts are not availability lift. A shelf-execution pipeline that flags more stock-outs this month than last month has told you something about the model’s sensitivity and about the shelf’s traffic, not about whether the shelf got better. Yet the first report that reaches finance and merchandising after go-live is very often exactly that: a rising flag volume, presented as recovered availability.
The two numbers can move in opposite directions. Detection counts measure how much the model sees; availability lift measures how much the shelf changed. A pipeline can raise the first while the second stays flat, if the restock workflow never closes the loop between a flag and a person with a case of product in their hands.
So the measurement design has to be fixed before deployment, not reconstructed afterwards from whatever the dashboard happens to log. Below is the method we use, and the order it has to be done in.
What does measuring on-shelf availability lift actually require?
Four things, and skipping any one of them makes the result indefensible in an internal review:
- A written on-shelf availability definition — what counts as available, at what facing depth, agreed with merchandising before any camera is pointed at a shelf.
- A fixed SKU cohort and store cohort — the same set of items and fixtures measured before and after, so the denominator does not drift.
- A baseline from manual shelf audits on that same cohort, taken at a cadence you can repeat after go-live.
- A counterfactual — held-back control stores, or a staggered rollout — so seasonality and promotional cycles are not credited to the model.
Everything else in this article is detail hanging off those four.
Define the metric before the pipeline exists
On-shelf availability sounds self-defining until three people in the room describe it differently. Is a SKU available if one dusty facing survives at the back of the shelf? If it is present but in the wrong bay? If it is present, correct, but unfaced and invisible from the aisle?
Write the rule down. In practice a workable definition names: the fixture and bay scope, the minimum facings required to count as available, whether misplaced-but-present counts as available (usually no — that is a planogram issue with an availability consequence), and the observation window. The distinction between a system stock-out and an on-shelf stock-out matters here, and our parent overview of shelf-execution AI for planogram compliance and on-shelf availability develops why the ledger view and the shelf view diverge in the first place.
The SKU cohort should be fixed and stated. Fast-moving lines behave nothing like slow tail items, and a cohort quietly reweighted towards fast movers between baseline and measurement will manufacture a lift that has nothing to do with the pipeline. Lock the list, and report any forced changes — a delisted SKU, a packaging switch — as an exclusion with a reason.
Baseline against audit ground truth, not against the model
The baseline is a manual shelf audit: a human walking the fixed cohort and recording availability against the written definition. That audit is also the ongoing reference standard for detection quality after go-live, which is why it should not be treated as a one-off pre-launch chore. Periodic audits continue for the life of the deployment.
Detection precision and recall are reported against periodic manual audits, separately from the shelf-outcome metrics, so model quality is never substituted for shelf outcome. This separation is the single most useful discipline in the whole method. It gives you two independent readings — is the model still seeing correctly, and is the shelf actually better — and when they disagree you learn something. Rising flag volume with flat audited availability points at the workflow. Flat flag volume with falling audited availability points at the model, and usually at one of the quiet degradation classes: an illumination change in one aisle, a supplier artwork redesign, a new stacked-facing arrangement that occludes the label. Those failure modes are worth reading in detail, because the ways shelf-execution pipelines degrade under lighting and packaging change are precisely the ones that produce no dashboard alarm at all.
The measurement frame
| Metric | What it measures | Source | Evidence class |
|---|---|---|---|
| On-shelf availability rate, before vs after | Shelf outcome on a fixed SKU cohort | Manual shelf audit on the cohort | Operational measurement, per-deployment |
| Planogram compliance rate | Whether the shelf matches the intended layout | Audit, model output reconciled to audit | Operational measurement, per-deployment |
| Median time-to-restock from detection | Workflow responsiveness, not model quality | Task system timestamps vs pre-deployment store-round interval | Operational measurement, per-deployment |
| Detection precision / recall | Model quality only | Periodic manual audit as ground truth | Operational measurement, per-deployment |
| Avoided hardware procurement | One-off cost line, where existing cameras and mobile devices were reused | Procurement record | One-off cost, stated beside the recurring gain |
Two rules about this table. First, no row substitutes for another — a good precision figure is not an availability result, and a fast time-to-restock is not a lift if availability did not move. Second, the numbers are per-deployment operational measurements on a named cohort, not portable industry benchmarks; a lift observed in one estate says very little about the next one until the same cohort discipline is repeated there.
Time-to-restock reads the workflow, not the model
Median time from out-of-stock detection to confirmed restock is the metric that tells you whether the loop closes. Compare it to the pre-deployment baseline that actually applied: the store-round interval. If staff walked the aisle every four hours, the expected pre-deployment detection gap was roughly half that, and any post-deployment median above it means detections are arriving and going nowhere.
That is a workflow finding, and it is fixed in the task design rather than the model — we cover the escalation path in the article on turning a detected planogram break into an owned task. For measurement purposes the point is narrower: report time-to-restock separately, and never fold it into an availability claim.
Keeping seasonality out of the result
Retail demand moves on its own. Promotional weeks, weather, a competitor’s price change, the back-to-school peak — all of them shift availability without any help from a model. A before-and-after comparison across a period containing any of these events cannot distinguish the pipeline’s contribution from the calendar’s.
Two workable counterfactuals:
- Held-back control stores. Match on format, footfall band, and cohort mix. Measure the same audits on the same cadence in both arms. The lift is the difference in differences, not the treated arm’s raw change.
- Staggered rollout. Bring stores live in waves, so each wave’s pre-launch period overlaps another wave’s post-launch period. This is usually easier to sell operationally than holding stores back indefinitely, and it survives the objection that control stores are being deliberately under-served.
Either way, the audit cadence must be identical across arms. Auditing treated stores more often than controls will bias the comparison, and it happens more than you would expect, because treated stores are the interesting ones.
Sampling cadence sets the ceiling on precision
There is a hard constraint that sits underneath all of this. You cannot measure availability lift at a finer time resolution than the pipeline samples the shelf. If a fixture’s shelf state is captured every ninety minutes, an availability duration metric has roughly a ninety-minute granularity and a time-to-restock median has the same floor of uncertainty.
Capture cadence is bounded by throughput — how many fixtures the inference stack can process per interval on the available hardware. In our experience this is where a measurement plan and an infrastructure plan have to be reconciled early: throughput and capture-cadence findings determine how often a shelf state can be sampled, which bounds how tightly availability lift can be measured. Where cadence turns out to be the binding constraint, the honest move is to widen the reported time resolution rather than to imply a precision the sampling does not support.
The same discipline shows up outside retail. Measuring a production computer-vision system against audited ground truth rather than against its own detection counts is how industrial deployments are hardened too — different domain, identical logic. Our broader computer vision engineering practice and our retail work both start measurement design from the ground-truth side.
Frequently Asked Questions
What does measuring on-shelf availability lift from shelf-execution AI mean in practice? Measure Shelf Availability Lift is a frequent topic. It means reporting how much the shelf changed, on a fixed SKU and store cohort, against a manual-audit baseline and a counterfactual arm — not how many stock-outs the model flagged. The detection stream is an input to the measurement, never the result itself.
How do you define the on-shelf availability metric and the SKU cohort before deployment so the baseline is comparable? Write the availability rule down before any capture begins: fixture and bay scope, minimum facings that count as available, how misplaced-but-present is treated, and the observation window. Then freeze the SKU list and the store list, and report any forced exclusion with its reason rather than silently reweighting the cohort.
What baseline do you measure against — manual shelf audits, control stores, or a staggered rollout? All three do different jobs. Manual shelf audits give the ground-truth level before go-live and remain the reference standard afterwards; held-back control stores or a staggered rollout supply the counterfactual that keeps calendar effects out of the lift figure.
How do you separate detection performance from shelf outcome? Report precision and recall against periodic manual audits in their own section, and keep availability rate, planogram compliance, and time-to-restock in a separate set. When the two sets disagree — flags rising while audited availability is flat — that disagreement is the finding.
How do you measure time-to-restock from out-of-stock detection, and what does it show? Take the median interval from detection to confirmed restock in the task system, and compare it against the pre-deployment store-round interval rather than against zero. It is a reading on the workflow: a median no better than the old round interval means detections are arriving and not being acted on.
How do you keep seasonality and promotional cycles from being credited to the shelf-execution pipeline? Use a control arm or a staggered rollout and compute the lift as a difference in differences, with the same audit cadence in every arm. A single before-and-after window that contains a promotion or a seasonal peak cannot separate the model’s contribution from the calendar’s.
What measurement signals indicate detection has degraded from lighting changes or packaging redesigns? Flag volume that stays steady or falls while audited availability worsens, or a recall drop against audit in one aisle or one supplier’s SKUs while the rest of the estate holds. Because degradation of this kind rarely reduces dashboard activity, the periodic audit is the only reliable early signal.
The measurement question that stays open longest is not statistical. It is whether the organisation is willing to keep paying for manual audits after the pipeline is live — because the moment the audits stop, the only remaining reference for shelf truth is the model being evaluated, and the lift figure quietly becomes unfalsifiable. How long should that audit cadence run before you trust the pipeline enough to thin it?
Putting Measure Shelf Availability Lift to work
None of this is theoretical. Measure Shelf Availability Lift rewards teams that measure first and argue later — start with the smallest instrumented slice and let the numbers settle the design.
