Adding Material and Texture Discrimination to a Real-Time GPU Detector

Choose the cheapest change that removes the confusion, and judge every candidate by latency measured on the target GPU and host, not by accuracy on a paper benchmark. For a YOLO object detection pipeline, the order we would try is: retrain the existing single-stage detector with hard negatives, then fuse a texture head onto the shared backbone, and only after that add a segmentation stage or a new imaging modality such as near-infrared or polarised light. The latency headroom for that extra stage can be found by restructuring the pipeline around the model rather than by tuning the model itself.

TechnoLynx starts this work by measuring each option on the customer’s own GPU: it prototypes the retrained detector and the extended variants side by side on the target hardware, then optimises whichever one fits the budget, as part of its computer vision R&D work.

If you have not yet settled the detector family, the guide to matching YOLO-family detectors to the right pipeline decision covers that choice. This page assumes the detector is already running and asks what to add to it.

Four ways to add material discrimination, compared

Option	What changes	Added inference cost	When it fits	Main risk
Retrain the single-stage detector with hard negatives (e.g. frost-negative examples)	Training data and labels only	None from the architecture	The budget is tight and the material confusion is modest	May not generalise if the texture varies with product type or conditions
Fused texture head on the shared backbone	One extra head reading existing feature maps	The head only; measure it	You need a learned texture cue but have little headroom	Fine-tuning can shift the original detection accuracy
Detection plus segmentation stage	A mask branch, or a second network on candidate regions	Lower with a shared backbone, higher with a separate model such as Mask R-CNN	Mask boundaries matter, or objects overlap heavily	Latency and engineering complexity
Extra imaging modality (NIR, polarised light)	Cameras, lighting, calibration, input channels	Depends on the added channels and preprocessing	The materials are genuinely indistinguishable in RGB	Hardware change and recalibration, not just software

Retrain the single-stage detector, or move to detection plus segmentation?

Retrain first. When a single-stage detector confuses frost with a real contaminant, we would treat it as short of hard negatives before treating it as short of architecture, because retraining with frost-negative examples adds nothing to inference time. Move to a detector-plus-segmentation pipeline only when that retrain still leaves texture confusion the line cannot tolerate, and only if the added stage fits inside the cycle time.

TechnoLynx frames this as a latency-versus-accuracy trade-off under the line-speed constraint: it prototypes and benchmarks both architectures (for example a YOLOv8 or YOLO-NAS detector alone against YOLO plus a U-Net or Mask R-CNN stage) on the target GPU, then validates precision and recall per class, frost against contaminant, next to the measured latency. Where a new camera is adopted, the same engagement supports integrating it and runs the optimised pipeline on the plant’s GPU hardware.

The single-stage baseline is cheap for a structural reason. YOLO, as Redmon et al. introduced it in 2015, predicts bounding boxes and class probabilities from the full image in one evaluation of a single network. One-stage detectors buy that speed at the cost of some accuracy against two-stage designs. Retraining keeps the same network and its speed, and asks it to learn a finer distinction. The risk is generalisation: if frost texture changes with product type or condensation conditions, a retrain that looks clean on one product can fail on the next.

Segmentation cost depends on whether the backbone is shared

A segmentation stage is not one price. Mask R-CNN, as He et al. described it in 2017, extends Faster R-CNN with a mask branch in parallel with box recognition, adds only a small overhead to Faster R-CNN, and runs at 5 fps. In a benchmark of detection methods on radio-astronomical images, Mask R-CNN was listed at 198.0 GFLOPs against 43.7 GFLOPs for YOLOv7. Bolting a full two-stage segmenter behind a YOLO detector means paying for a second heavy network.

Sharing the backbone changes the arithmetic. S4Net puts a bounding-box detector and a segmentation branch on the same base model, and its authors report 40.0 fps at 320×320 on a GTX 1080 Ti, rising to 90.9 fps with a MobileNet base. Valeo’s 2019 joint network shares a small ResNet10-like encoder between a YOLO v2-style detection decoder and an FCN8 segmentation decoder, reaches 30 fps at 1280×384 on a low-power embedded SoC, and reports the same accuracy as separate networks. The same authors note that the multi-task segmentation results came out slightly lower than single-task ones, in exchange for a large gain in memory and compute efficiency. YOLACT, a fully convolutional real-time instance segmenter, reports 29.8 mAP on MS COCO at 33.5 fps on a single Titan Xp.

These figures come from different GPUs, resolutions and years, so they show the ordering of costs, not the number your line will see. S4Net’s authors found the same pressure on proposals: beyond 20 proposals the accuracy gain flattened while runtime rose sharply, so they used 20 at inference. Cap the candidate count deliberately, because the second stage’s runtime rises with the number of proposals.

Which camera separates ice from plastic when RGB cannot?

The sources support polarised light for ice in anti-icing fluid films, and near-infrared hyperspectral imaging for plastic and rubber foreign bodies in grain. Grishaev et al. (IWAIS 2022) report that visible-spectrum methods for detecting ice err by more than 5% against experienced technicians because ice contrast is low, and that ice birefringence revealed by polarised light raises that contrast, with the crossed p-s polariser and analyser state giving the better image. The same paper notes that infrared cameras are expensive and lower in resolution than ordinary ones. Gowen and O’Donnell describe pushbroom NIR hyperspectral imaging at 950–1700 nm as particularly well suited to identifying plastic and rubber foreign bodies, because those materials show characteristic absorbances in the near infrared, and point out that metal detectors cannot find glass or plastic contamination.

Which one fits depends on what the contaminants actually are. Frost against a metal or glass fragment is a different separation problem from frost against a plastic film. The sources do not settle broader multispectral setups, so we would not pick one on general grounds.

Checking the added stage against a 1.5 m/s line budget

The latency budget is a distance divided by a speed. At 1.5 m/s the belt moves 15 cm every 100 ms, so the budget is whatever stretch of belt lies between the camera’s view and the reject point, divided by 1.5 m/s. That window has to hold host-to-device transfer, pre- and post-processing and every model stage, not just the forward pass.

Worked procedure: measuring the added stage on the target machine

Build both pipeline variants as TensorRT engines and run trtexec on each, comparing median latency and throughput rather than the minimum.
Rerun with --includeDataTransfers, since trtexec excludes host-to-device and device-to-host copies by default.
Profile per layer with trtexec --dumpProfile to find the expensive layers, then use Nsight Systems to see why they are expensive at kernel and stream level.
Control clocks and watch for thermal throttling: NVIDIA’s TensorRT documentation warns that two runs of the same engine on the same GPU can differ by tens of percent.
Time the stages inside the application, with CUDA events for GPU-only work and wall-clock timing for the end-to-end figure, because pre- and post-processing belong in the system number.
Compare reported Enqueue Time with GPU Compute Time on the deployment system; small workloads tend to be enqueue-bound, and launch overhead varies by CPU, driver and operating system.

The enqueue check matters more than it looks. In a public GitHub benchmark of detection and segmentation models across eight GPUs, the single-image latency of a nano YOLO model was 3.6 ms on an RTX 5090 host against 16.3 ms on an RTX 4090 host, with the worst host 4.9 times slower at launching kernels than the best. Its author attributes this to small networks launching many tiny kernels that the CPU, driver and Python must issue. A texture head or segmentation branch adds exactly that kind of small kernel, so measure on the production host, not a development workstation.

If the target device itself is still open, matching embedded GPU choice to detection-tracking latency budgets covers that decision. LynxBenchAI’s published results show how candidate devices run the public PyTorch stack that custom models are built on, which makes them broadly indicative for shortlisting hardware; they do not run or measure your pipeline, so the on-device run above still decides.

Restructure the data path before tuning the model

An arXiv study of a two-stage fabric-defect cascade on an NVIDIA Jetson Nano with TensorRT FP16 found non-inference time of 53.63 ms per image in the sequential baseline, more than the 47.77 ms spent inside TensorRT. Its parallel pipeline reached 13.45 fps against 9.86 fps for sequential YOLO-only, and 91% of that 1.36× gain came from overlapping JPEG decode with inference, not from the cascade. The authors’ recommended order is to pipeline the ingest path first (worth about 25 ms per image there), then recalibrate the stage-one screener (up to about 19 ms of inference), and only then revisit the architecture. NVIDIA’s TensorRT guidance offers a related fix where PCIe transfer is the bottleneck: send JPEG-compressed images and decode them on the GPU instead of raw pixels.

That evidence comes from a Jetson Nano running from disk. Pipelining the data path is available to any configuration, though. Before you trim a texture head to fit, check whether decode and transfer are eating the budget, because in that study pipelining the data path freed more time than the cascade did.

Will depth fusion or instance segmentation tell crushed plastic from cardboard?

Depth separates objects; it does not reliably tell you what they are made of. For overlapping, flattened items under a tight pick-cycle budget, keep the single-stage detector, add a lightweight material-classification head where plastic and cardboard get confused, and move to full instance segmentation only when mask boundaries matter for the grasp.

Clutter is a real problem for detectors on this kind of material. The SortWaste dataset, collected at a material recovery facility, shows precision and [email protected] falling from the full test set to the high-clutter subset across the detectors its authors tested, including Faster R-CNN, RetinaNet and YOLOv11. Its authors report detection degrading steadily once their ClutterScore rises above 0.4, and Mixed Plastic as the hardest class because of its visual heterogeneity. The ZeroWaste dataset authors describe the same challenge: highly deformable, often translucent objects in cluttered scenes with little of the context present in human-centric datasets.

Instance segmentation earns its cost on overlap. In the authors’ COCO ablation, CenterMask’s Local Shape branch brought about 10 AP and separated densely overlapping objects with coarse masks, and with a DLA-34 backbone the model reaches 32.5 mAP at 25.2 fps. BCNet, a two-stage occlusion-aware mask head, scores 31.71 AP on the heavily overlapping COCO-OCC split against 30.32 for Mask Scoring R-CNN. Both gains concern where one object ends and the next begins. That helps a gripper, but neither result addresses whether a flattened item is plastic or board.

A survey of camouflaged object detection notes that texture can separate an object from its surroundings when colour is similar. Bell et al. (CVPR 2015) treat material recognition as a task of its own, reaching 85.2% mean class accuracy on patches and 73.1% for per-pixel prediction on the MINC database, and stress that training with surrounding context is crucial in real-world use. That supports a dedicated material head over hoping a better box detector learns the distinction on its own.

Use RGB-D fusion to resolve spatial overlap and stacking, not material confusion. Adding a material head after detection is a separate accuracy-and-latency trade-off from improving the detector backbone, and each needs its own measurement against the pick-cycle budget on the target hardware.

Yes. A head that reads the existing ResNet feature maps is structurally the cheapest learned option, because the shared layers run once per frame. Its actual latency delta on the edge GPU still has to be measured against the 150 ms budget before you commit, since it depends on the hardware, precision and implementation, not on the architecture diagram. The Valeo authors put the reasoning simply: early network layers are largely task-independent, so work shared between two networks frees compute for each.

TechnoLynx scopes this by splitting the 150 ms budget into what the current detection already spends and the headroom left, shortlisting a fused head, classical-feature fusion and a hybrid descriptor against that headroom, and building and benchmarking each on the target edge GPU with the detector’s own precision settings before integrating the chosen one and monitoring it in production.

Learned texture branches on an existing backbone

Published designs add texture branches on top of CNN backbones, including pretrained ResNets. MuLTER builds on ImageNet-pretrained ResNet18 or ResNet50 layers and adds learnable encoding modules at each level, fusing low- and high-level features from Res1 to Res4 of ResNet50. Its authors describe the result as differentiable end to end, extensible to other CNNs, and compact, with a 512-dimensional representation. Deep-TEN places an encoding layer on top of the convolutional layers and learns the dictionary and encoding with the classifier, giving an orderless representation suited to material and texture recognition. FV-CNN shows why sharing matters for speed: its convolutional features are computed once for the whole image and pooled differently per region, rather than re-running the CNN for each one. On material recognition in clutter it improved accuracy from 46.5% to 58.1% over fully connected CNN features.

Classical LBP or GLCM features fused with the CNN

The DNT method fuses histograms of local binary patterns with patch-pooled CNN features encoded by an LSTM before the classification layer. A survey of camouflaged object detection notes that texture extraction is costly yet effective, and that features learned by deep networks generally work better than hand-crafted colour, texture and gradient features. Decide where an LBP or GLCM stage runs, and include that path in the end-to-end timing.

Transformer and hybrid texture descriptors

Next-ViT’s authors treat latency on specific hardware, measured under TensorRT and CoreML, as the direct efficiency measure, and report that their hybrid captures richer texture information than ResNet. The same paper cites work finding that Transformer blocks can erode high-frequency information such as local texture, and reports that LayerNorm and GELU add significant TensorRT latency for negligible gain over BatchNorm and ReLU. A published latency for any of these models will not carry over to your edge GPU, input size and precision. The TensorRT per-layer profile shows each added layer’s share of end-to-end latency directly, so the head’s real cost can be read off the deployed engine.

When RGB does not carry the signal

We would treat diagnosis, not architecture, as the hard part of this decision. If a retrain with good hard negatives fixes the confusion, the problem was data. If a fused texture head fixes it, the signal was in the pixels and the detector simply was not looking at it. If neither does, the camera may not be capturing the difference at all, and no model stage will recover what the sensor never recorded; that is the point to price a polarised or NIR camera instead of a bigger network. Settle which of these three you are facing on a few hundred labelled frames from the real line before you spend latency budget on any of them.