Synthetic data for object detection reduces the number of real labelled images a detector needs. It does not remove them. In published comparisons, detectors trained only on rendered or randomised images lose accuracy on real cameras. In one fruit-detection study, hybrid synthetic-and-real training came close to real-only accuracy with fewer real samples. How many real labels you still need has to be measured, with a learning-curve study against validation data that looks like deployment. It cannot be assumed.
TechnoLynx runs this as a scoped piece of engineering. With the client, TechnoLynx sets the recall target and the confidence it has to hold at, then designs the learning-curve experiment and the synthetic generation strategy. TechnoLynx builds the synthetic pipeline and the annotation taxonomy and QA, and measures recall across real-data volumes before handing over a collection and annotation plan. All of this sits within TechnoLynx’s Computer Vision R&D.
Can synthetic data replace real labelled images for an object detector?
In the fruit and pallet studies it could not. A fruit-detection study found that real-only models scored highest, above 0.97 [email protected]. Synthetic-only models did significantly worse, and hybrid training recovered most of the gap while needing fewer real samples. The authors concluded that synthetic data alone cannot fully replace real-world data in their application, but can substantially reduce the need for real annotations when used in a hybrid strategy. A pallet-detection study using 160 real training images reached the same verdict: synthetic data alone was insufficient, but adding it reduced the real data needed.
Some results point the other way, and they deserve a fair hearing. Johnson-Roberson et al. trained a vehicle detector only on game-engine images and beat the same architecture trained on human-annotated Cityscapes data, when both were tested on KITTI. Hinterstoisser et al. trained detectors for 64 retail objects on fully synthetic data that outperformed models trained with real data on a challenging evaluation dataset. A 2025 manufacturing study reached mAP@50 between 94.1% and 99.5% across four use cases with synthetic-only training.
Look at what those wins needed. The KITTI comparison was scored on all 7,481 real images in the KITTI training set. The manufacturing study warns that a synthetic validation set may not reliably pick the best model for real test data, and it still saw weaker results on visually similar and simply shaped objects. Even where synthetic training carries the model, real labelled images are how you find out whether it worked. Synthetic-only training did best here on CAD-modelled manufacturing parts, while the fruit study found its benefits limited for natural objects with complex textures.
How much can synthetic pretraining or domain randomisation cut the real data a detector needs?
The published reductions are large but depend on the class, so treat them as a range, not a budget. A 2022 study on mixed synthetic and real training for driving scenes found that mixed datasets with 5% to 20% real images reduced the need for real data the most without losing detection performance. In its best case (10% real images combined with Synscapes), only about 30% of the original real data was needed, a 70% cut. The same study found a wide gap between classes: cars still needed 64% of the real data, while people needed only 20%. The authors trace this to representation. Cars were already plentiful in the real set, while the underrepresented person class gained far more from synthetic instances.
Dwibedi et al. found that cut-and-paste synthetic data plus just 10% of the real data beat a model trained on all the real data in a cross-domain setting. A balanced cut-and-paste method with 10% real images scored 53.3% against 52.5% for 100% real images on GMU Kitchen, which its authors read as similar performance. For semantic segmentation, Richter et al. showed that game data plus one third of the CamVid training set outperformed the full CamVid set. In Unity’s PeopleSansPeople study the gains were steepest when real data was scarcest: keypoint AP of 44.43 with synthetic pretraining against 6.40 from 641 real images alone. With abundant real data the margin narrowed to 63.47 against 62.00.
The saving depends on how well your real set already covers each class and how many synthetic images you add. Synthetic gains flatten past a point, and in the fruit study adding more synthetic data did not consistently help.
How do you run a learning-curve study to find how much real labelled data you need?
A learning curve plots generalisation performance against training-set size. You use it to predict what more data would buy you. Hestness et al. found that generalisation error follows a power law in training-set size across several domains, image classification among them. They also argue that predictable curves can guide decisions on whether and how to grow a dataset. For a detector whose recall you have to defend, we would run the study in this order:
- Freeze a real, deployment-representative validation set first. Match the sensor, geography, lighting and object mix the model will meet. Do not rely on synthetic validation images: in one study they did not reliably pick the best model for real test data.
- Label a modest real seed set and train on nested subsets of it. The PeopleSansPeople benchmark, for example, used subsets of 1%, 10%, 50% and 100% of its real training set.
- Measure recall per class, with interval estimates, at every subset size. Boyd et al. found that some common confidence-interval procedures for precision-recall area are invalid, and they recommend two simple intervals that hold under a range of assumptions. Per-class reporting matters because the real-data saving differed sharply between cars and people in one study.
- Repeat each subset with synthetic pretraining. The horizontal gap between the real-only curve and the synthetic-pretrained curve, at your target recall, is the labelling you save.
- Work out which region of the curve you are in before extrapolating. Hestness et al. describe a small-data region where models barely beat guessing, then a power-law region where each new sample improves generalisation. Exponents in real applications settled between about -0.07 and -0.35, which prior theory does not explain.
- Extrapolate cautiously and collect in rounds. Mahmood et al. (NVIDIA, 2022) found that a good power-law fit does not translate directly into an accurate estimate of the data needed for a target. A tuned correction factor and collection over multiple rounds improved the estimates significantly.
- Stop where added labels no longer narrow the recall interval meaningfully, then switch to targeted collection. At that point, we would label the detector’s failure cases rather than label uniformly.
Hestness et al. expect a non-zero lower bound on error in most real applications, and name mislabelled training or validation samples as one cause. So errors in your real labels raise the floor of the curve itself.
We would plan rare configurations separately, setting coverage targets for them and collecting for them directly.
What annotation taxonomy and QA do new classes like cones, drums and arrow boards need?
We would write a taxonomy for new or rare classes, not bolt a few labels onto an existing schema. The ROADWork dataset defines 15 work-zone object categories. These include cones, tubular markers, fences, barriers and drums, temporary traffic control signs, message boards and arrow boards, plus workers, work vehicles, police officers and police vehicles. The same dataset shows large geographic variation: police vehicle designs and the look of barriers, vertical panels and tubular markers differ between cities. A taxonomy built in one region can therefore split or merge classes wrongly in another.
ROADWork’s workflow also used automatic pre-annotation. The team bootstrapped boxes with the Detic open-vocabulary detector, discarded its category predictions because they contained many classification errors, and kept manual verification in the loop. The pallet study dropped boxes with less than 25% visibility and ran manual checks for annotation accuracy, duplicates and corrupt files.
When QA runs is a design choice in its own right. A 2026 position paper argues that catching errors before annotation starts costs a fraction of finding them after review cycles finish. Its authors present this as a hypothesis that controlled experiments have yet to test. We hold the same view: an ambiguous class definition is cheaper to fix in the guideline than through relabelling. Near-perfect accuracy is almost never a plausible target, so the workflow needs fault tolerance and an appetite for human review. Our piece on building defensible ground truth through active learning and human-in-the-loop labelling covers the review pipeline itself.
Four ways to generate synthetic training images, and what each one varies
This table summarises the published evidence:
| Method | How it works | What it varies | Evidence and limits |
|---|---|---|---|
| Cut-and-paste | Object instances are cut from real images and pasted onto other backgrounds; patch-level realism is enough training signal for current detectors | Object placement and background | Foreground and background gaps are unbalanced; diversifying foregrounds with GAN seeds and simplifying backgrounds raised accuracy from 67.2% to 78.1% |
| Rendered scenes (game engines, simulators) | Photorealistic engines render scenes with automatic labels | Whatever the scene model exposes | A single simulated image carries less variation than a real one, so far more are needed |
| Domain randomisation | Simulator parameters such as lighting, pose and textures are randomised in non-realistic ways so the network learns the object’s essential features | Objects, background and distractors, camera, illumination, post-processing such as noise and blur | Structured randomisation, which places objects by context-aware distributions, outperformed plain randomisation on KITTI car detection |
| Generative models | Text-to-image models such as Stable Diffusion generate instances, or CLIP filters crawled images, to feed copy-paste at scale | Object appearance without 3D assets | GAN-generated datasets trailed classical methods in one study, though they avoid building high-fidelity 3D assets |
One finding cuts across all four. Increasing variation often helps synthetic-to-real transfer more than increasing visual realism does. The pallet study found a trade-off as well: half-realistic synthetic data helped in-distribution accuracy, while randomised data helped out-of-distribution robustness.
Should you pretrain on synthetic data and fine-tune on real data, or train on a mix?
When real data is limited, we would default to pretraining on synthetic data and then fine-tuning on real. Nowruzi et al. found that fine-tuning a synthetic-trained model on limited real data beat mixed training. They also note that the large synthetic training run can then be done once, separately from the cheaper fine-tuning step. Li et al.’s “bridged transfer” (synthetic first, then real) was more accurate than mixing. NVIDIA’s domain-randomisation work found that fine-tuning on real data produced better results than real data alone. In the manufacturing study, synthetic pretraining plus fine-tuning on a single real image reached 97.5% mAP@50.
The evidence is not unanimous. The 2022 driving study saw no significant difference between pretraining with fine-tuning and mixed training. In the pallet study, bridged transfer generally performed best, but some gains fell within run-to-run variability. Our reading is that fine-tuning wins on workflow even where accuracy ties: you can retrain the real stage as new labels arrive without regenerating or rebalancing the synthetic set. One mechanism may explain why a small real set goes so far. Detectors trained on real and synthetic data differ mainly in their later, high-level layers, so real data may only need to adjust how features are combined rather than relearn low-level features. Our guide to fine-tuning pretrained models on small labelled samples covers layer freezing, learning rates and overfitting at that stage.
Why detectors trained on synthetic images lose accuracy on real cameras
The synthetic-to-real gap comes from differences between the two data distributions: object appearance, environmental conditions, sensor characteristics, background complexity and scene composition. It is narrowed by modelling the sensor, domain randomisation and adding a small real sample. TechnoLynx builds synthetic data pipelines as part of Computer Vision R&D. Without a real validation sample, the size of the gap cannot be measured at all. A 2026 review of domain generalisation argues that some real-world variability is inherently unpredictable, so fully closing the gap may not be possible.
Does the renderer need to model the camera’s sensor and lens, or only look realistic?
It needs the sensor. Nowruzi et al. summarise the literature this way: realistic sensor distortion and environment models affect final performance more than photorealism does. They attribute CARLA’s poor results to its less accurate environment and camera models. Carlson et al. built a physically based pipeline that varies chromatic aberration, blur, exposure, noise and colour cast in synthetic images, and found that it reduced the domain gap for object detection in urban driving scenes. A follow-up learned to transfer those sensor effects from a real dataset onto synthetic data. Not every effect pays off equally. Hinterstoisser et al. found that blurring rendered objects gave a large boost, while noise and random light colour added little, with the effect depending on the camera.
Calibration narrows the gap without erasing it. The DiffPhysCam simulator models defocus blur, exposure and noise and calibrates them against a real camera through physical experiments. Even so, its authors report a noticeable remaining gap between simulation and real-world results. Sensor and lens simulation also adds rendering cost and engineering work.
A jersey-number study found synthetic-only models did not generalise to real-world data, while synthetic pretraining then real fine-tuning beat an 82% baseline. If your renderer does not yet model noise, lens distortion and motion blur, we would add those before scaling up generation. Once a small real labelled sample exists, we would fine-tune on it rather than keep widening synthetic diversity.
Who can build a synthetic data pipeline and size the real labelling for our detector?
TechnoLynx builds the synthetic data pipeline and runs the learning-curve study that tells you how much real labelling your detector still needs. The work covers framing recall targets, designing the randomisation and sensor-modelling strategy, setting up the annotation taxonomy and QA, and validating recall across real-data volumes. It is delivered as Computer Vision R&D, and the first step is defining the recall target and the deployment-representative validation set together. Building in-house makes sense when your team already has rendering, training and evaluation skills and only needs time. Bringing in a specialist makes sense when the gap between synthetic and real results is unexplained, or when you need a defensible labelling budget quickly. When choosing a partner, ask how they would measure the gap on your real data before generating anything.
The engagement does not suit everyone. If what you need is a large volume of labelled images against a taxonomy you already trust, an annotation service is the right route. If real data is plentiful and the problem is well defined, a straightforward real-data training run may be all you need.
Budget the validation set before the synthetic pipeline
Synthetic data cuts real labelling in one place only: the training set. The deployment-representative validation set, with enough real examples per class to put a tight interval on recall, costs the same whether you render a million images or none. It is also the only thing that turns “synthetic helped” into a number you can defend. Every saving above depends on it.
