Retail shelf monitoring with computer vision detects three kinds of shelf event: empty facings (out-of-stocks), products sitting in the wrong place against the planogram, and, by vendor account, price-tag problems. It reports what was physically on the shelf when the image was taken, and that is why it can expose inventory records that are wrong. Its blind spots are just as specific: occluded, overlapping or backward-facing products, near-identical SKUs, packaging the model has never seen, and the odd “gap” that turns out not to be a gap.
Shelf monitoring is one of the operational use cases where retail vision earns its money, not one of the customer-experience showpieces. The financial case belongs in what ROI retail computer vision delivers. The question here is narrower: what do the cameras actually see?
What retail shelf monitoring detects, event by event
One public account of a deployed system is a study of a planogram compliance system built for more than 7,000 7-Eleven stores in Taiwan. Its pipeline detects shelf rows, detects products in each row, classifies each product, then lines the result up against the digital planogram. Planogram verification compares a “realogram” (the photographed shelf) with the reference layout and gives every product a status such as correct, incorrect position, wrong shelf or out-of-stock. To do the alignment, the authors adapted the Needleman–Wunsch algorithm, originally built for comparing DNA sequences, using dynamic programming to find correctly placed, missing, misplaced and wrong-shelf products.
Out-of-stocks need their own rule. When the space between two neighbouring detected products in a row is wider than a threshold, the system counts an item as missing. So an empty facing is not detected as an object: the system infers it from the gaps between products it did detect.
The reported accuracy is high. Shelf detection reached 99.23% precision and 98.93% recall, and product detection reached 94.61% precision and 93.02% recall, with YOLOv8 beating the transformer-based Deformable DETR on both tasks. The models were tested on a fourth store kept out of training altogether, so the figures reflect performance in a completely new environment. Classification covered 471 product categories, built from 99,135 product images. Any product whose top classification confidence falls below a threshold is labelled “unknown” rather than forced into a class.
A smaller academic gap-detection pipeline reports much lower figures on a different task. Trained on SKU-110K and fine-tuned on 370 store images, it reached 0.6844 precision, 0.7115 recall and 0.7354 [email protected] with YOLOv12 when scoring predicted gap regions against ground truth. The authors argue that precision matters more than recall for out-of-stock alerts, because every false positive sends a staff member to check a shelf that was fine. Simbe Robotics describes the same design choice for its commercial system, deliberately accepting some missed detections to avoid false ones.
Don’t put those two sets of figures side by side as a league table. One measures whether products were found, the other whether predicted gap regions overlapped the labelled gaps, on datasets of very different sizes. Before you read any accuracy claim for shelf monitoring, ask what unit was counted: products, gaps, labels or whole shelves. The accuracy limits of shelf detection and SKU classification go deeper into why the unit matters.
Simbe says its Tally robot flags pricing errors and promotion errors, reads shelf labels with 99% identification accuracy, and detects plugs (the wrong product in a slot), spreads (a product taking more facings than assigned) and missing price tags. These are Simbe’s own published figures. None of the research sources here measures price-tag reading on its own, so in our view price and promotion checks are the capabilities a pilot should test hardest.
Shelf events at a glance
| Shelf event | How vision detects it | Best evidence available | Known weak spot |
|---|---|---|---|
| Out-of-stock (empty facing) | Gap between detected products wider than a threshold | 7-Eleven deployment; academic gap pipeline at 0.6844 precision | False gaps from spurious detections |
| Planogram break (wrong position or shelf) | Product classification plus alignment against the reference planogram | 7-Eleven: 94.61% product detection precision | Occluded or backward-facing items |
| Plugs and spreads | Comparing detected facings with assigned facings | Vendor claim only (Simbe) | Look-alike variants |
| Price-tag error or missing tag | Identifying shelf labels in the image | Vendor claim only (Simbe: 99% label identification) | Tags cut off in partial captures |
| Low stock before empty | Depth cameras estimating empty volume in virtual boxes | Vendor claim only (Simbe) | Relies on 3D depth cameras |
Fixed cameras, phone photos or roving robots: what changes with the capture method
Shelf images come from three sources: a field rep’s phone, a fixed camera, or an in-store robot, and the same recognition step can run on any of them. What changes from one to the next is how often the shelf gets looked at, how complete each view is, and who has to do something for the image to exist.
The authors note that verification systems usually assume a whole shelf fits in one image, which narrow aisles, tight spaces and deep shelves often make impossible. Their answer was multi-image stitching into a “virtual shelf”, so that several overlapping photos are treated as one shelf. Partial captures matter because missing fragments can drop price tags, facings or whole SKUs from the analysis. A phone-based system is only as frequent as the people taking the photos, so it audits the shelf rather than monitoring it.
Fixed cameras give continuous, frequent shelf visibility. Pretius, a practitioner source, puts most shelf-visibility systems at 5 to 60 second refresh intervals. The same source lists what gets in the way: trolleys, customers, morning sun glare and fogged cooler doors, handled with multi-frame smoothing and redundant cameras. The trade-off is coverage. Every camera sees a fixed patch of shelf from a fixed angle, so the number of cameras grows with the area you want to cover, and the angles are whatever the mounting allows. Which existing cameras can be reused and what has to be bought is covered in shelf monitoring hardware: what to reuse vs what to procure.
Roving robots sit in the middle. Simbe says Tally drives the aisles on its own, photographs every shelf section, uses depth sensors and 360° fisheye cameras to reach recessed areas, and can also scan RFID. The counterweight is Walmart. CNBC, citing The Wall Street Journal, reported that Walmart had ended its contract with Bossa Nova Robotics after finding that people could do about the same work, with about 500 robots in its more than 4,700 stores when the contract ended. One retailer’s decision does not settle the question. It does show that a robot has to beat the cost of staff walking the same aisles, not just beat a manual audit for accuracy.
Cameras versus RFID tags and sensor-equipped smart shelves
Cameras and RFID answer different questions. RFID gives a direct measurement of how much stock there is, through readers and tags, without anyone counting by hand. RFID research reports a clear improvement in inventory accuracy, with fewer out-of-stocks and better ordering as the result. RFID also enables quick, accurate cycle counting in place of once- or twice-yearly inventories, and records exactly what was stolen, when and from where.
RFID measures stock quantity, not the shelf the shopper sees. A pack pushed to the back of a facing or shelved in the wrong place is physically present but invisible to the shopper. Cost is the other line: Kang and Gershwin (MIT Auto-ID, 2004) point to the high up-front cost of readers and tags, and warn that the technology was not guaranteed to give an exact count.
We do not put numbers on sensor-equipped smart shelves (weight or presence sensing per lane). Our view is that they suit a small number of high-value lanes where a per-lane signal is worth fitting hardware to the shelf, while cameras suit broad coverage where the arrangement itself matters. Some vendors combine the approaches. Simbe, for example, describes mobile robots, fixed cameras and RFID integration in one system.
| Question you need answered | Camera monitoring | Item-level RFID |
|---|---|---|
| Is the facing empty? | Yes, from the image | Not directly; it measures stock quantity |
| Is the product in the right planogram slot? | Yes | Not measured; RFID reports quantity |
| Is the shelf label identified? | Yes, by vendor claim (Simbe: 99% label identification) | Not stated |
| How many units exist in the store? | Only what is visible | Yes, by direct measurement |
| What hardware is needed? | Phones, fixed cameras or robots | RFID readers and tags |
Can shelf monitoring catch phantom inventory?
Yes: shelf images give the independent reading that the inventory record cannot give itself. Phantom inventory is stock that the inventory or point-of-sale system records as available while the product is actually missing from the shelf. Record inaccuracy has been measured at scale. Kang and Gershwin (MIT Auto-ID, 2004) cite a study of close to 370,000 SKUs at one retailer in which more than 65% of inventory records did not match physical stock at the store-SKU level, and 20% were off by six or more items. At another retailer, average perfect record accuracy across stores was only 51%, and the best store matched just 70% to 75% of its records.
Small errors grow. In Kang and Gershwin’s simulation, stock loss of just 1% of average demand left enough error in the record to lose 17% of demand to out-of-stocks. The record stays above the reorder point, so replenishment freezes while the shelf sits empty.
The record cannot find its own error, because phantom inventory is by definition a failure of that record. Shelf images provide the independent reading: when a photo shows an empty facing for a SKU the system lists as available, the mismatch becomes a time-stamped, location-specific fact. One practitioner source describes doing this by correlating repeated zero-facing events with ERP stock levels.
The word “repeated” matters. A single empty facing next to a positive record may come from backroom stock never moved to the shelf, misplacement or shrink. Kang and Gershwin make a related warning about resetting records to zero after zero sales: for slow-moving products, zero sales does not mean zero stock, and a wrong reset leads to overstocking. We would treat a vision-detected mismatch as a prompt for a check (backroom, misplacement or record adjustment) until it repeats across captures, and only then let it correct the record. How those corrections then feed replenishment is the subject of automated ordering from shelf data. The shrink behind many phantom records, and how to test vendor claims about it, is covered in what computer vision can and cannot detect in retail shrinkage.
Where shelf-recognition AI still fails
Occluded, overlapping and backward-facing products
The 7-Eleven authors say plainly that their system does not solve occlusion, backward-facing products or overlapping merchandise. Their dataset is mostly standard shelf images with packaging facing forward, so the model was never trained or evaluated on those cases. When conditions stray far from the training data, the system outputs “Unknown” rather than guessing. They report that in real stores these situations are relatively rare, that staff can handle them without much burden, and that the deployment has saved substantial manual inspection time. That is an honest design: an unknown sent to a person adds little staff burden, by the authors’ account, while a confident wrong status corrupts the compliance report.
Look-alike SKUs and dense shelves
Retail pushes recognition into territory most benchmarks never test. The CVPR 2020 Retail Vision Workshop listed detection in densely packed scenes, class imbalance, new classes over time and “ultrafine-grained” classes that are virtually indistinguishable by appearance as the field’s core problems. SKU-110K, released by Trax Retail researchers at CVPR 2019, captures the density side: 11,743 shelf images with over 1.7 million annotated products, roughly 147 per image, often identical-looking and side by side. The SKU-110K dense-shelf benchmark is single-class: it tests finding products, not telling them apart.
A 2026 study of open-source vision-language models on grocery retrieval found high Recall@5 (94.5%) but a 17.5% drop at Recall@1. In plain terms, the embeddings suit coarse retrieval but are unstable for fine-grained ranking between SKUs. How retrieval-based recognition deals with that is covered under embedding-based retrieval for retail vision. Commercial systems add non-visual cues, and Simbe says it uses OCR, colour and size analysis, and barcodes to separate variants such as original and fat-free versions of the same pack.
Packaging redesigns and new products
New products can appear on shelves before anyone has labelled them. The 7-Eleven team reports 98.33% few-shot accuracy and 98.39% zero-shot top-1 accuracy for its classifier on unseen categories. Those are results on that team’s own dataset, and nothing in the evidence measures packaging redesigns directly.
Gaps that are not gaps
Gap logic inherits detection errors. In the academic gap pipeline, a false product detection on a non-product label near the top of the unit caused the vertical-extension rule to flag a large gap that did not exist. Uneven lighting, non-frontal viewpoints and cluttered arrangements distort the image in the first place.
In-store compute
Real-time inference is a stated requirement of the 7-Eleven system. Its authors report that NVIDIA TensorRT acceleration improved response speed by up to 1.5× and kept throughput stable as load approached saturation. How heavy a model a store can run at the needed cadence is a sizing question for the shelf monitoring hardware decision.
None of these failures means the task is intractable. Our position is that they are scale failures: thousands of imbalanced SKU classes, look-alikes that more data alone does not fix, constrained edge hardware and a steady stream of unknown items compound in a way small pilots never expose, and that takes engineering rather than a model swap. That argument is made in full in why off-the-shelf retail vision fails at scale, and it is the work our AI engineering for retail practice exists to do.
Which shelf monitoring alerts should a retailer act on first?
In our own analysis of retail vision, the use cases in production with measurable financial impact are loss prevention, shelf monitoring for planogram compliance, and traffic-to-conversion analytics, not frictionless checkout or virtual try-on. Within shelf monitoring, our judgement is that out-of-stock and phantom-inventory alerts pay back first, price-tag and promotion checks next, and planogram compliance reporting on its own last. Alerts come first because each leads to a cheap, immediate action (restock, backroom check, record fix) whose effect shows up in sales. In Kang and Gershwin’s simulation, lost sales from shoplifting, through record error and late replenishment, can be ten to twenty times the stolen units.
Price-tag and promotion checks should wait until their accuracy has been checked in your own stores, since the evidence for them so far is Simbe’s own. Compliance reporting comes last because a compliance score that nobody acts on saves nothing. How to model those returns before committing capital is covered in what ROI retail computer vision delivers.
The question a pilot should settle is not “how accurate is the model?” but “which shelf events, at what precision, trigger an action someone in the store will actually take?” Precision on gap alerts decides how often staff are sent to check shelves that were fine.
