Whether a CPU Halcon pipeline holds 2 packs per second can only be settled by measuring your own line. The deciding number is the per-pack cycle time, which is 500 ms at 2 packs per second. A system-level figure covers pre-processing and post-processing as well as the network. So measure first. If the CPU falls short, try batching or pipelining across cavities and model-side optimisation next, and add an edge GPU only when its measured gain is worth what it costs.

TechnoLynx runs this decision as an Inference Cost-Cut Pack engagement. The first step is a per-cavity and per-pack latency profile of your current Halcon pipeline, checked against the 2 packs per second target and your acceptable false-reject rate. From there TechnoLynx works through batching, model optimisation and, only if the numbers call for it, a port to edge GPU.

Can a CNN cavity segmentation model keep 2 packs per second on a CPU Halcon pipeline?

One published number comes from MVTec’s 2018 Halcon release announcement. It said that CPU inference of a pretrained CNN, optimised for Intel x86, ran in about two milliseconds and matched a mid-range GPU. That is a vendor claim from 2018 about the networks of that release, not a measurement of a segmentation network on blister cavities. It tells you MVTec built Halcon to run CNN inference on CPUs. It tells you nothing about your cycle time.

The useful question is where the time goes. The network is not always the biggest cost. A 2018 study of CNNs for mobile computer vision found that, for a single image, loading the model and setting up the network took longer than inference. A 2025 case study of video analytics pipelines concluded that bottlenecks came from scheduling and data flow as much as from neural network inference. In that study, steps such as connected component analysis and bounding-box extraction stayed CPU-bound even after detection had been accelerated.

Worked example: turning 2 packs per second into a budget

Two packs per second means one pack has to leave inspection every 500 ms. Assume, for illustration only, ten cavities per pack and a strictly serial pipeline. Acquisition, pre-processing, inference, post-processing and the decision for all ten cavities then share those 500 ms. That works out to 50 ms per cavity for everything.

Now assume instead, again for illustration, that acquisition, inference and post-processing run as three overlapped stages. Each stage then gets the full 500 ms per pack, so inference alone gets 50 ms per cavity. With three stages, each pack’s verdict arrives up to three cycles after acquisition instead of one. That is acceptable only if the reject station sits far enough downstream. Throughput and decision deadline are two separate constraints, and a pipelined design trades one against the other.

How do you measure per-cavity and per-pack latency against the 2 packs/second budget?

Measure the whole cycle with a wall clock, broken down by stage. Measuring the network alone is not enough. NVIDIA’s TensorRT benchmarking documentation separates the inference-only number from the system-level number, and notes that pre- and post-processing belong in the system figure. Wall-clock timing is what puts inference in context within the larger system. A procedure that holds up on an inspection line:

1. Fix the constraints before timing anything. Write down the 500 ms per-pack cycle, the latest moment the reject actuator can receive a verdict, and the false-reject rate you can live with.
2. Time every stage separately for each pack: image acquisition, cropping or pre-processing per cavity, the apply_dl_model call, mask post-processing and the decision.
3. Exclude start-up and warm-up. A 2026 benchmark of industrial vision runtimes threw away the first 10 of 1,000 runs because initialisation made the first iterations much slower. In production, load the model once, outside the cycle.
4. Record the distribution, not just the mean. TensorRT’s documentation uses median latency and throughput as its comparison metrics. A line also misses packs on its slow cycles, so track the tail (p99 and maximum) against the 500 ms ceiling.
5. Measure under real load. Run the CNN alongside the other Halcon operators that share the CPU. On a GPU, NVIDIA warns that two runs of the same engine can differ by tens of percent depending on clock behaviour and thermal throttling.
6. Measure false rejects with the same configuration. Every change in batch size, precision or runtime gets re-scored on a held-out set of good and defective cavities, not just re-timed.

This profile is the baseline that every later step is compared against. Without it, each escalation is a guess.

Does batching or pipelining the cavities of one pack cut per-pack time enough to avoid new hardware?

Sometimes, and only a measurement on your own setup shows how much. Batching runs the CNN on several images at once, which reuses convolution weights more efficiently. It raises throughput but makes each batch take longer than a single image. A blister pack suits batching well: Halcon’s apply_dl_model accepts a batch with any number of samples, so all the cavities of one pack can go through in a single call.

The best batch size is hard to estimate without measuring. The 2018 CNN characterisation study found that the latency-throughput response is not linear in batch size. Adding one image sometimes improved throughput at almost no latency cost, and sometimes made both worse, so the optimum is hard to estimate without measurement. It describes one possible strategy: set an upper bound for latency and increase the batch size until that bound is reached. On CPU-only devices, a 2021 edge study found that larger batches improved throughput only until the device’s CPU and memory limits were reached.

Where batching hurts

Batching adds waiting time. The DeepRT scheduler paper (2021) splits batched latency into execution time plus the queuing time that early inputs spend waiting for late ones. Its authors stress that this added latency must not cause missed deadlines. If all cavities come from one image of the pack, the queuing cost is close to zero. If several cameras trigger at different moments, the early cavities wait, and that wait counts against the cycle.

Batching and parallel workers do not simply add up. In the 2025 video analytics case study, batched inference cut a stage from 36 to 23 seconds, but adding multiprocessing brought it to 25 seconds because larger batches caused memory contention. On GPUs, DeepRT found that batching beats running several model instances concurrently, both for throughput and for request execution time. Pipelining, meaning overlapping acquisition and post-processing with inference, attacks a different cost: it keeps the CPU-bound stages off the critical path rather than speeding up the network. OpenVINO’s documentation describes running many inference requests in parallel to use the hardware better, at a latency cost that does not grow linearly with the number of parallel requests.

Which model-side optimisations reduce CNN latency before you escalate hardware?

Two levers come before buying anything: the runtime the model executes on, and the numeric precision it runs in. Both depend on the hardware and the model.

Runtime choice. The 2026 industrial benchmark found OpenVINO the fastest option on Intel CPUs for every model it tested. For YOLOv8 it cut inference time by 30–60% compared with PyTorch on three machines. The same study found that the best runtime depended on the model family as well as the hardware, and it timed network inference only. These percentages therefore describe its models, not your segmentation network. Inside Halcon, MVTec’s AI² interface lets you switch to different accelerator back ends without changing application code.

A vendor back end is not automatically faster than a standard one; the same code runs about the same. What a vendor back end offers is access to lower-level features, and that access only pays off if someone commits real effort to it, which is not always worth doing. The latency profile tells you whether it is worth it for your line.

Precision. Halcon’s optimize_dl_model_for_inference operator optimises a model for a target device and bases its calibration on samples you provide. MVTec says the samples should be representative, preferably from the training split, and that 10–20 per class is enough for most applications. NVIDIA’s 2020 study of int8 quantisation kept accuracy within 1% of the floating-point baseline, even for networks known to be hard to quantise. Leaving the few most sensitive layers in floating point recovers accuracy at some cost in speed.

Quantisation does not always make a model faster. A study of post-training quantisation on CPUs and edge devices found that it could speed inference up, slow it down or leave it unchanged, depending on hardware, model and framework. In that study, MobileNetV2’s depthwise convolutions took about five times longer after quantisation, so overall latency went up. What matters on the line is whether borderline cavities cross your reject threshold, so re-score false rejects after every precision change.

Is inference better on CPU or GPU for a small segmentation CNN on an inspection line?

In the published studies, GPUs delivered more throughput than CPUs. How much more depends heavily on which CPU they are compared against. A 2021 study of edge devices measured GPU and EdgeTPU inference at 10–63× the throughput of CPU inference. The CPUs in that comparison were a Raspberry Pi 4 and an Odroid board. A 2020 study put edge accelerators (a VPU, a TPU and two Jetson GPUs) at between one half and 3.5× the performance of an x86 server processor.

A GPU also brings host-side overheads of its own. For models made of many small kernels, the CPU time spent launching each CUDA kernel can exceed the GPU work itself, which matters for a small CNN run one cavity at a time. Host-to-device and device-to-host copies over PCIe can become the bottleneck too. The CPU stays in the loop: in the 2018 characterisation study, a weaker CPU slowed GPU-powered runs by 10–30%. Post-processing that has no GPU implementation can end up limiting the whole model.

On GPUs, batching has proved very effective. In the 2020 edge accelerator study, a batch of 16 raised throughput 13.94× on a Jetson Nano GPU, with smaller gains beyond 16. In the video analytics case study, batches up to 16 cut inference time per frame from about 6.6 ms to 2 ms, and larger batches got slower again because of memory pressure. Both results came from other models on other hardware, so treat them as evidence that a measured batch-size sweep is worthwhile, not as the speed-up to expect.

When is edge GPU inference justified over staying on the Halcon CPU toolkit?

An edge GPU is justified when three things hold: batching and model optimisation have been measured and still leave a gap, the profile shows the shortfall is in the network rather than in acquisition or post-processing, and the false-reject rate holds at the required speed. If post-processing is the bottleneck, a GPU for the CNN does not fix it. Profiling at kernel level, as in our piece on diagnosing GPU bottlenecks with kernel-level profiling, is how you tell whether a GPU port is actually being used or is waiting on the host.

On NVIDIA hardware, the 2026 industrial benchmark found TensorRT the fastest runtime for its CNN models, UNet included. That benchmark timed network inference only, on its own YOLOv8 and UNet models. It therefore marks TensorRT as a candidate to measure, not a result for your network. To compare candidate devices before buying, LynxBenchAI publishes results for how each device runs the public PyTorch stack that custom models are built on, which makes them broadly indicative for a custom segmentation CNN. LynxBenchAI does not run your Halcon pipeline. The final number still comes from your own model on the shortlisted device.

Sizing for a future line-speed increase, or for several lines sharing GPUs, is a separate capacity-planning question.

How do you weigh a measured throughput gain against the added hardware cost?

Treat each step as something you buy only when the previous step’s measurement shows a remaining gap. The comparison that matters is “meets 500 ms per pack, with margin in the tail and the false-reject rate intact” against the full cost of the change. Cost per inference is the wrong comparison here.

Step	What it can buy	What it costs	Move on when
Profile the current CPU pipeline	Shows whether the network, pre-processing or post-processing dominates	Engineering time only	Always the first step
Batch the cavities of a pack; pipeline the stages	Higher throughput, stages overlapped	More latency per batch, queuing if cameras are not synchronised, larger tail spread	Tail latency still exceeds the cycle after a measured batch-size sweep
Change the runtime or precision on the same CPU	Lower inference time on suited models	Validation effort; quantisation may not speed up your model	Measured gain is too small, or false rejects rise
Port to an edge GPU	Large throughput gains for the network when it is batched	Hardware, integration, a new runtime to support, CPU and PCIe overheads that remain	Gap persists and the profile shows the network is the bottleneck

If batching or a runtime change closes the gap with margin, no GPU hardware or integration work is needed. If they close it with no margin, a GPU can be bought as headroom, and that decision should be made knowingly.

Who can help us get a Halcon cavity inspection line to 2 packs per second?

TechnoLynx profiles your existing Halcon pipeline per cavity and per pack, then tells you whether CPU batching, model optimisation or an edge GPU closes the gap, and at what cost. The work starts by setting the 2 packs per second target and your false-reject ceiling as hard constraints and measuring a baseline on the current CPU toolkit. If needed, TechnoLynx then designs batching and pipelining across cavities and cameras. It ports and optimises the model for an edge GPU with suitable precision and batch size, and validates throughput, latency and false rejects before deployment. The engagement is TechnoLynx’s Inference Cost-Cut Pack, drawing on its computer vision R&D work.

This is engineering help with inference speed and deployment. It is not regulatory sign-off for a validated packaging line, and it does not replace your vision integrator’s work on optics, lighting or reject mechanics. If your team already has someone comfortable with profiling tools and Halcon’s deep learning operators, the measurement procedure above is a fair place to start in-house. MVTec’s own support is the natural route for questions specific to Halcon operators. Whoever you choose, ask them for a per-stage latency profile of your line before they recommend any hardware.

FAQ

What does edge inference mean?

Edge inference means running a trained model on hardware at or near where the data is captured, rather than in a data centre. It is often preferred for industrial vision when low latency, data security or limited connectivity matter.