Panoptic segmentation assigns every pixel in an image exactly one semantic label and an instance id, so the output is a single map of the whole scene with no overlaps. Semantic segmentation gives every pixel a class label but does not separate individual objects, and instance segmentation detects and segments each object while ignoring the uncountable background and letting masks overlap. So panoptic segmentation is not just a more advanced version of the image segmentation methods a team already runs. It is a different output contract, and the right choice depends on what the deployment has to report.
How Is Panoptic Segmentation Different from Instance and Semantic Segmentation?
The difference is easiest to see in what each task allows a single pixel to carry. A semantic model treats segmentation as per-pixel classification: an FCN-style network independently predicts a category for every pixel. An instance model works by mask classification, predicting a set of binary masks, each with one class label. Detection-based instance methods output a ranked list of proposals, and because each proposal is processed independently, the proposals are allowed to overlap, so one pixel can belong to several instances at once.
Panoptic segmentation forbids that. Each pixel gets one semantic label and one instance id, so overlaps are impossible by construction. Kirillov and colleagues, who defined the task, require no confidence scores for panoptic segments, so humans and machines must generate the same type of annotation. The format is a strict generalisation of semantic segmentation: if the ground truth specifies no instances, the two task formats are identical, even though their metrics differ.
The original paper is firm that panoptic segmentation is not a multitask problem. Running semantic and instance heads as separate tasks allows their outputs to be independent and inconsistent, whereas panoptic segmentation requires one coherent segmentation of the scene. That coherence requirement is the whole point, and it is also what makes the task harder to engineer than either of its parents.
The Three Tasks Side by Side
| Semantic segmentation | Instance segmentation | Panoptic segmentation | |
|---|---|---|---|
| What each pixel carries | One class label | Membership in zero, one or several object masks | One class label and one instance id, id ignored for stuff |
| Overlaps allowed | No, one label per pixel | Yes | No, by construction |
| Classes covered | Things and stuff | Things only | Things and stuff |
| Confidence scores required | No | Yes | No |
| Standard metric | mIoU | AP | PQ |
What Do “things” and “stuff” Mean, and Why Do They Define the Panoptic Task?
“Things” are countable objects and “stuff” is amorphous background regions; panoptic segmentation unifies instance segmentation of things with semantic segmentation of stuff. Semantic segmentation partitions the whole image regardless of whether a class is countable, while instance segmentation only handles thing classes but must tell instances apart. Because stuff is not countable, all pixels of a stuff category are treated as one instance, which is why the panoptic format ignores the instance id for stuff labels.
The split is set by each dataset’s definition. COCO defines 80 thing and 53 stuff categories, and Cityscapes defines 8 things and 11 stuff. Panoptic and semantic segmentation are evaluated on the union of both groups, while instance segmentation is evaluated only on things.
For an inspection project, deciding which classes are things is the first real design decision, and it should follow the operational question rather than visual intuition. A scratch that the line counts and rejects at a threshold is a thing. Corrosion measured as a fraction of surface area is stuff, even if it appears as separate patches, because nobody downstream cares how many patches there are.
How Is Panoptic Quality (PQ) Calculated, and How Does It Relate to mIoU and Mask AP?
Panoptic quality (PQ) is computed in two steps: match predicted segments to ground-truth segments, then score the matches. A predicted segment and a ground-truth segment can match only if their intersection over union (IoU) is strictly greater than 0.5, and together with the non-overlap rule this guarantees that each ground-truth segment matches at most one prediction.
For each class c, matched pairs are true positives (TP), unmatched predictions are false positives (FP) and unmatched ground-truth segments are false negatives (FN), and the per-class score is:
PQ_c = Σ IoU(s, ŝ) over TP / ( |TP| + ½|FP| + ½|FN| ).
Multiplying and dividing by |TP| splits this into two interpretable factors: segmentation quality, SQ = Σ IoU / |TP| (the average IoU of matched segments), and recognition quality, RQ = |TP| / ( |TP| + ½|FP| + ½|FN| ), an F1-style detection score, so PQ = SQ × RQ. The original paper uses this decomposition to interpret results, computes SQ and RQ for every class, stuff and things alike, and stresses that PQ is not a combination of a semantic metric and an instance metric. On the 0.5 threshold, the authors found that PQ changes relatively little between IoU 0.25 and 0.5, that many low-IoU matches are false, and so kept 0.5 as the default.
Mask AP works differently. Average precision needs a confidence score on every segment to build a precision/recall curve, which is natural for detection but meaningless for semantic output, so AP cannot score semantic segmentation. Kirillov and colleagues found that AP and PQ track closely and expect AP gains in a detector to improve PQ, while mIoU is a different case. It counts correctly predicted pixels, whereas PQ operates at the level of segments, so on Cityscapes the PQ gap between methods matched the IoU gap, while on ADE20k the PQ gap was much larger.
Where PQ Misleads
PQ treats every instance equally regardless of size: the DeeperLab authors point out that a 10×10-pixel instance counts as much as a 1000×1000-pixel one. That makes PQ sensitive to small false-positive regions, and heuristics such as deleting small regions can raise the score, as the open-source evaluation code itself notes. Porzi et al. (CVPR 2019) argue the opposite bias for stuff: since all pixels of a stuff class form one big segment, the 0.5 IoU rule over-penalises stuff errors, and they propose keeping PQ for things while scoring stuff differently.
The Seamless Scene Segmentation results report PQ for things and PQ for stuff separately. A single aggregate PQ hides whether the model is failing at counting defects or at covering surfaces, and those are different failures with different fixes.
COCO Panoptic Format and Tooling
The COCO panoptic format, published in the cocodataset panopticapi repository, stores all segments of an image in a single PNG rather than as independent segments, which keeps the non-overlap property by construction. The same repository ships a script that heuristically merges semantic and instance predictions into a panoptic prediction.
Can Instance and Semantic Models Be Combined Post-Hoc Instead of Training a Panoptic Model, and What Breaks?
Yes, separate models can be merged, and the task’s own authors did exactly that for their baseline, while calling the heuristic “simple and likely suboptimal”. The merge runs in a fixed order:
- Sort predicted instances by confidence and drop low-scoring ones.
- Walk the instances from most confident down, removing pixels already claimed by an earlier segment; keep the remainder only if a sufficient fraction survives, otherwise discard the whole instance.
- Overlay the semantic map, resolving any thing/stuff conflict in favour of the thing label and its instance id.
- Remove stuff regions labelled “other” or below an area threshold.
What breaks is consistency and efficiency. Porzi et al. note that separate models carry significant computational overhead, offer no way to enforce labelling consistency between them, and spend capacity modelling redundant information. In the first panoptic challenges, every competitive entry still used separate instance and semantic networks with no shared computation.
Post-hoc merging is a reasonable prototype, but not a production design until PQ for stuff is measured before and after the merge.
Which Model Architectures Handle Panoptic Segmentation, and How Do They Differ from Mask R-CNN?
Mask R-CNN is a detection-based instance model, and such models output ranked, overlapping proposals, exactly what panoptic segmentation forbids. Its trade-offs in deployment are covered in our piece on Mask R-CNN in production. The panoptic architectures differ mainly in where they resolve thing/stuff conflicts.
- Panoptic FPN starts from Mask R-CNN with FPN and adds a lightweight semantic branch for dense per-pixel prediction. Its two outputs can still overlap, so it relies on the same NMS-like post-processing described above.
- UPSNet puts a deformable-convolution semantic head and a Mask R-CNN-style instance head on one backbone, then adds a parameter-free panoptic head that resolves conflicts by pixel-wise classification, including an extra “unknown” class.
- Panoptic-DeepLab is bottom-up: a standard semantic branch plus a class-agnostic instance branch that regresses instance centres.
- DeeperLab produces per-pixel semantic and instance predictions in a single fully convolutional pass and fuses them with a fast algorithm.
- Mask classification predicts N masks with N labels and assigns categories or instances as the semantics of each segment.
Mask2Former’s paper reports 57.8 PQ on COCO panoptic val2017, 50.1 AP on COCO val2017 instance segmentation and 57.7 mIoU on ADE20K (multi-scale inference). A model trained only on panoptic annotations can be used for instance and semantic segmentation, although the authors found it performs slightly worse than the same model trained specifically for each task. Universality of the architecture does not remove the need to train for the task you will be scored on.
What Throughput and Latency Cost Does Panoptic Segmentation Add Compared with a Single-Task Model?
What the published results establish is the comparison that matters more often. Panoptic FPN matched the accuracy of two separately trained FPNs with roughly half the compute, because the backbone dominates and is shared. Against Mask R-CNN alone, Panoptic FPN’s addition is a lightweight semantic branch.
The Panoptic-DeepLab authors report that, with a MobileNetV3 backbone, it runs at 15.8 frames per second on a single 1025×2049 image while scoring 54.1 PQ on the Cityscapes test set, against 65.5 PQ for its strongest configuration. That is an accuracy-for-latency trade, not a free lunch.
Where merge and overlap-resolution logic runs outside the network, end-to-end latency has to be measured from image in to panoptic PNG out, not from the forward pass.
How Much More Expensive Is Panoptic Annotation than Semantic or Instance Labelling?
Of the label types Du et al. compare, panoptic annotation is the most expensive. Du et al. (AAAI 2021), working on MS-COCO, give a panoptic label as 1104 seconds per image, against 7 seconds for image-level labels and 32 seconds for bounding boxes. Normalising the panoptic cost to 1, they put image labels at 0.006, bounding boxes at 0.03 and semantic segmentation maps for the stuff classes at 0.68, following earlier work by Caesar et al.. Fully supervised panoptic training needs per-pixel background classes and foreground instance labels together.
Du et al. give no instance-only cost; on their figures, stuff-only semantic maps cost about two thirds of a full panoptic label on COCO, and boxes cost a small fraction. Those ratios come from COCO scenes and will not transfer exactly to images of a single part on a conveyor.
The cost argument cuts both ways. A panoptic map contains the semantic map, because the format is a strict generalisation of it, and Mask2Former trained only on panoptic annotations can be used for instance and semantic tasks. Annotating semantically and discovering later that defect counts are needed means adding instance labels to every image afterwards; annotating panoptically from the start costs more up front but keeps both options open.
When Does an Industrial Inspection Deployment Actually Need Panoptic Output Rather than Semantic Masks?
An inspection system needs panoptic output only when the decision requires both counted objects and full coverage of the remaining surface in one consistent map. The test is the question the line has to answer:
| The decision asks | Task that answers it |
|---|---|
| What fraction of this surface is corroded, stained or uncoated? | Semantic segmentation; instance identity is wasted effort |
| How many discrete defects are on this part, and how big is each? | Instance segmentation at minimum; semantic masks cannot separate touching defects |
| How many defects, and what share of every other surface region, with no pixel counted twice? | Panoptic segmentation |
| Is there any defect at all? | Possibly detection or classification, not segmentation |
The third row is narrower than it looks. Overlapping instance masks double-count area where defects touch, which matters if per-defect area feeds an accept/reject rule; a panoptic map gives each pixel to exactly one defect or surface class, so areas sum correctly. If the downstream rule only counts defects, the instance segmentation approaches used in manufacturing inspection are the fit to look at, and the counting logic itself is covered in object counting in production.
Choosing the wrong formulation produces a model that scores well on its own metric and still cannot answer the operational question. Problems with open-source solutions, such as detection and tracking, are often treated as solved, but real conditions and scale change the picture. Matching the segmentation task to the operational decision is part of TechnoLynx’s computer vision R&D work.
Pinning the Contract Before the Model
The order of decisions matters more than the model choice. Fix which classes are things and which are stuff from the decision the line makes, fix the metric that follows from it (mIoU, mask AP, or PQ split into things and stuff), and only then pick between Panoptic FPN, a bottom-up design or a universal model. The wider pattern of models validated against the wrong contract is the subject of why computer vision in production breaks off-the-shelf assumptions.