We know of no published synthetic-to-real ratio that holds across orchard perception datasets. Even the people who study it say that how much real data you need to stabilise transfer is still an open question in many applications. A 2022 study that blended the Cityscapes driving dataset with the Synscapes synthetic set estimated, from a power-law fit, that real shares between 5% and 20% cut the need for real data the most without losing detection performance. Separately, a 2019 study and a 2022 study reported that fine-tuning on real images beat mixing them with synthetic data. Treat those numbers as starting points. Our position is that the minimum real fraction for your detector has to be measured on your own held-out orchard rows, one obstacle class at a time.
Trusting synthetic orchard data takes a staged validation, not one check
Synthetic orchard scenes earn trust, in our view, only through a sequence of checks. First comes a cheap feasibility gate before any large training run. Then a quantitative sim-to-real gap metric, an audit of which randomisation settings matter, a synthetic-versus-real mix ablation sized to the data-collection budget, and finally a go/no-go package for the safety board. Each stage answers a different question, so passing one tells you little about the next.
TechnoLynx works through these stages jointly with the orchard robotics team. The first step is to agree the feasibility gate, the gap metric and the ablation design together. After that, TechnoLynx implements the metrics, runs the scene-generation and detector-training pipelines on the target compute, checks the measured gap against held-out real orchard data, and packages the results for sign-off.
Which check answers which decision
| Decision in front of you | Check | What it gives you | What it cannot give you |
|---|---|---|---|
| Is it worth starting a large training run? | Training-free dataset metrics, plus a small evaluation on annotated real images | A ranking of candidate synthetic pools by likely downstream mAP | A measured gap or a field accuracy figure |
| How far is the simulator from real orchards? | Domain-classifier separability, feature-space distance, cross-validation transfer, per-class error comparison | A number you can track for distribution gap and overfitting risk | Quirks that the randomisation space never modelled |
| Should we spend on the renderer or on field data? | Leave-one-out randomisation ablation scored on held-out real clips | Which settings move real accuracy | Whether the remaining gap is a sensor limit |
| How much real footage should we fund? | Mix ablation across real fractions, per class | The point where adding more data stops helping | Transfer to new varieties, cameras or sites |
| Is the blended recipe acceptable? | Per-class precision and recall deltas against a fixed real-only reference set | A pass or fail per obstacle class | A threshold that applies everywhere |
| Can we expand the field trial? | Safety case that combines the measured gap, hazard analysis and field tests | An argued residual-risk statement | Proof of safety in conditions nobody tested |
A quick feasibility gate before committing to a big training run
Normally the only way to learn what a synthetic set is worth is to train a detector on it. That is slow, and it needs dense annotation. Training-free metrics avoid training a detector on every candidate. A 2026 study argues that global image-set statistics such as FID, KID and MMD are the wrong tool for choosing detection data, because they ignore what each image contains: object count, box scale and class mix. Its composition-stratified alternative, Conditional-Composition Domain Match (CCDM), ranked 19 candidate training sets on COCO and VisDrone-DET in close agreement with YOLOv8 mAP (Spearman ρ of 0.97 and 0.96). Each candidate took about 12.7 minutes to assess. Training YOLOv8m on one H200 took an average of 1.3 hours per candidate, so the screen was roughly six times faster. The authors also tested the realistic case where real reference images exist but have no annotations, and estimated the per-image metadata with an off-the-shelf detector. That case matters, since, as the authors note, collecting images is cheap while annotating them is expensive.
A 2025 metric, SDQM, combines pixel, spatial, frequency and feature-space comparisons, and its authors report a correlation of r = 0.87 with the mAP50 of detectors trained on the synthetic data. The other check is the direct one: train a model and score it on annotated real images, as a 2022 domain-adaptation study did with mean IoU.
These are all proxies. They rank candidate synthetic sets against each other. The CCDM results come from COCO and VisDrone-DET, not orchard canopy imagery. Use the gate to stop bad pools early, not to approve good ones.
Putting a number on the sim-to-real gap
None of our sources names a standard sim-to-real gap metric for agricultural vision. The sources we rely on use a small set of measures. The first is a domain classifier. SDQM’s Dataset Separability component trains a network to label embeddings as real or synthetic and records how easily it succeeds. The authors warn that a complex enough classifier can separate almost any two sets, so you have to read its accuracy alongside the classifier’s complexity. In a 2021 study on segmentation tests, a learned discriminator separated real and synthetic inputs easily, and it found the more realistic synthetic set harder to separate. That same study showed that IoU alone cannot tell you whether a model behaves the same way on real and synthetic data.
The second measure is feature-space distance, and the details matter here. In the CCDM study, stratifying by class broke moment-matching metrics on small per-stratum samples: FID’s correlation with mAP fell from +0.898 to −0.104, while kernel metrics such as MMD stayed robust. Among SDQM’s components, V-Information, β-Recall, Dataset Separability and Pixel Intensity Match tracked detector performance most closely, and Label Overlap tracked it least.
The third is a cross-validation transfer matrix. One framework trains the same detector on the synthetic set and on several real benchmarks, scores each model on the other sets’ test splits, and normalises the results into a cross-performance matrix. On vehicle detection against KITTI and BDD100K, Virtual KITTI scored 0.49 for simulation quality and 0.45 for transfer quality.
Synthetic datasets can perform very well on their own test set and badly on any other. In the SDQM experiments, YOLOv11n reached mAP50 between 0.31 and 0.79 on synthetic training data but only 0.06 to 0.40 on real evaluation data. Two checks expose the same problem. One compares misclassifications class by class across real and synthetic sets, to see whether both sets produce the same failure modes. The other scores the model on a real test set with different backgrounds, as the fruit study did.
For orchards, one available reference is the NREC Agricultural Person-Detection Dataset. It contains labelled stereo video from orange and apple orchards, with metadata for occlusion by vegetation, pose and distance, so you can evaluate each of those effects separately. Its authors showed that methods which succeed on urban pedestrian data do not transfer directly to orchards. This matches a position we hold more generally: detection that works in demos breaks at the edge cases of real deployments, even though off-the-shelf models remain reasonable for prototypes and controlled trials.
Any simulation used for training data simplifies the real world, with assumptions behind its models. Coverage audits and worst-bucket checks only probe the dimensions somebody thought to model. Corner cases and rare events are too scarce in many real-world datasets for extensive testing.
Which randomisation settings earn their compute
A 2018 leave-one-out ablation on KITTI car detection tested individual randomisation settings. With full randomisation the detector reached 73.7 AP. Training under fixed lighting dropped it to 67.6, while switching off only the brightness and contrast augmentation barely moved it (73.6). Removing random textures from the target object cut AP to 69.0, and turning off standard augmentation gave 72.0.
A 2022 study reached a similar conclusion on lighting. Varying the lighting generalised far better than varying object poses: randomised poses under real lighting reached only 23.77 AP. Its authors recommend modelling high-level features such as shape and pose accurately, and randomising low-level ones such as colour, lighting and texture, which are hard to render faithfully.
More randomisation is not always better. In a 2025 pallet-detection study, half-realistic synthetic data helped in-distribution accuracy and did little for out-of-distribution robustness. Heavily randomised data did the reverse. For an orchard renderer, we would run the leave-one-out ablation on illumination through the canopy, bark and leaf texture, and vegetation occlusion first, score every arm on the same held-out real clips, and drop the settings that do not move the result.
Deciding how much more real footage to fund
The funding question is really about which footage to collect. The 2019 study’s rules of thumb hold that samples from the test environment contribute more to performance, and that realistic sensor-distortion and environment models matter more than photorealism. In the 2026 fruit-detection study, doubling real training images from 50 to 100 did not improve performance under domain shift, which the authors read as overfitting to the training distribution. Our reading is that new orchards, varieties and seasons buy more than extra hours at the trial site.
A mix ablation assumes the detector and sensor stack are not the bottleneck. Sensor placement and lidar-camera fusion for orchard geometry are a separate design problem, and more labelled clips will not fix them.
What the field-trial approval board needs to see
The standards context is ISO 18497-2:2024, which sets design principles for obstacle protection systems on partially automated, semi-autonomous and autonomous agricultural machinery, and ISO 18497-4:2024, which sets verification methods and validation principles for the same machinery. Both were published in 2024 and will replace ISO 18497:2018.
For the ML component, AMLAS (Assurance of Machine Learning in Autonomous Systems, published in 2021) gives safety case patterns that connect system-level hazard analysis to ML safety requirements. Its argument has to state the trade-offs, assumptions and uncertainties behind the verification results. Validating synthetic data under AMLAS means identifying where the simulation diverges from the real world. Where class balance cannot be achieved, the imbalance is recorded and justified for the operational context.
The ARRNet agricultural robot test procedure (September 2025) counts success as detection in at least 95% of trials, reaction within 0.5 s, at least 300 mm of clearance, stop accuracy within ±50 mm and no collisions, with each trial repeated at least five times. That procedure covers platforms under 2 m² operating in semi-structured settings such as polytunnels and drilled fields. A larger orchard machine therefore borrows its structure rather than its exact thresholds.
For the residual-risk statement, the method we draw on comes from rail, not agriculture. An assessment of obstacle detection for GoA 4 freight trains partitions obstacle images into equivalence classes. It sizes verification runs using a variant of the coupon collector’s problem and estimates the residual probability of a systematic misclassification. We judge that the statistical method carries over. The freight-rail hazard rates and redundancy architecture do not. An honest orchard statement gives the measured per-class miss rate on the reference set, the gap metric, and the conditions not yet covered, in plain words. TechnoLynx supplies that engineering evidence in the sign-off package. The approval decision stays with the board and the machine’s safety owner. A package is only as credible as the gap metric and safety-case method underneath it.
Is there a standard synthetic-to-real ratio for orchard perception datasets?
No published ratio holds across datasets. The minimum real fraction therefore has to be found per dataset, with a mix ablation on held-out orchard rows. The reported best points differ by dataset and domain. It is not a constant you can lift from someone else’s paper.
TechnoLynx does this measurement as part of Computer Vision R&D, which builds and trains detectors for production conditions. For a synthetic-blend orchard detector, we would start with a per-class mix ablation against a frozen set of held-out rows.
Ratios reported in driving, warehouse and fruit studies
The 2022 Cityscapes and Synscapes study modelled detection performance against training-set size with a power law. It estimated that mixing could cut the need for real data by up to 70%. In its best case, a 10% real share, only around 30% of the original real data was needed. The per-class result matters more than the headline. Cars still needed at least 64% of the real data, while people needed only 20%. The reason was that Cityscapes already contains many cars, while people are underrepresented, so synthetic people added far more. We judge that the same logic applies in an orchard to rare obstacles such as people or fallen ladders, compared with ever-present trunks. This is why the ablation, and the acceptance bars below, have to be per class.
A 2019 study tested real shares of 10%, 5% and 2.5%. Each step up from 2.5% improved performance. With two synthetic sources mixed in, person detection at 10% real surpassed training on the full real dataset, while car detection did not.
In the 2025 pallet study, adding synthetic data to a 10% real share boosted both in-distribution and out-of-distribution results, although some gains fell within run-to-run variability. The 2026 Isaac Sim fruit study covers fruit detection with limited real data. Real-only models scored highest, above 0.97 [email protected], and hybrids came close with fewer real images. Its authors add that synthetic data helps less for natural objects with complex textures, such as fruit.
Fine-tuning on real data versus mixing it in
The comparison deserves its own ablation arm, because the strategy changed the results in the studies below. In the 2022 study, mixing improved results up to a five-to-one ratio and peaked at 80.13 AP. Fine-tuning a synthetic-trained model on only a small amount of real data reached 82.05 AP, and 83.7 at one-to-one. The 2019 study found fine-tuning better than mixing and more stable across runs. The 2025 pallet study found bridged transfer (fine-tune on synthetic first, then on real) generally ahead of mixing, with the same in-distribution versus out-of-distribution trade-off between half-realistic and random data. The minimum real fraction under fine-tuning can differ from the one under mixing, so report both.
Running the mix ablation without dozens of retrains
Three choices keep the run count down. First, fine-tuning lets the large synthetic training run happen once, with separate, much shorter fine-tunes on each real fraction. Second, the power-law form, 1 − mAP50 = 10^β · N^γ, is a linear fit on a log-log plot of real-data size against performance. Fit that curve per class. Third, screen synthetic pools with a training-free metric before they reach the ablation at all.
A defensible mAP delta from a small real dataset depends on handling variance. A 2021 analysis of ML benchmarks found that data sampling, parameter initialisation and hyperparameter choice all change results markedly. It also found that randomising more of those sources across runs gets closer to the ideal estimate, at 51 times lower compute. We would repeat each ratio with different seeds and resampled training subsets, and report the delta with its spread rather than as one number.
Synthetic volume also saturates. Performance in the 2022 study peaked at 20k synthetic images. The 2018 study saw saturation after about 10K images with pretrained weights and about 50K without. In the fruit study, which compared 1,000 and 5,000 synthetic images, more synthetic data alone brought no substantial gain. Treat the point where extra synthetic data stops lifting held-out mAP as your practical minimum. Expect it to move as the real set grows or diversifies, and audit the accuracy of real labels first.
Should the acceptance bar differ per obstacle class?
Yes. Set the bar per obstacle class, as precision and recall deltas against a fixed real-only reference set, and tighten it where a miss costs more. LynxBenchAI covers one adjacent piece of this. Its published hardware results measure AI performance across accelerators, so they inform which edge device should carry the detector once the bars are agreed. It does not run or score your detector.
A single aggregate figure hides the classes that matter. Per-class error analysis is needed beyond global metrics, and precision and recall for people sit well below cars in the 2019 study. AMLAS gives an example of a perception model whose safety requirements are minimum accuracies per class. Five candidate models each pass, but each is strongest on a different class, so choosing between them is an explicit trade-off, and that trade-off may change between rural and urban contexts. Asymmetric miss costs are a reason for different thresholds. A rice-and-weed detection study set asymmetric thresholds by error cost, because a missed weed is recoverable while a missed rice plant means herbicide on the crop. It is a single study, but we judge that its logic transfers. A missed person or a low branch at head height deserves a tighter tolerance than a missed trunk. The numeric thresholds should come from your own hazard analysis, which determines how ML failures contribute to system hazards.
Error measures fluctuate more for small objects, and variance from sampling and initialisation changes results markedly. If a class has too few held-out examples to resolve the bar, collect more for that class before arguing about the threshold. If the bar is stated on confidence scores, those scores need checking too. We cover validating and calibrating model confidence before relying on it separately.
As for who signs off and against what, keep the reference set fixed across every ablation run, as the fruit study kept two fixed real test sets. AMLAS asks that verification be independent of development, with verification data hidden from the developers. Sign-off on the bar belongs to whoever is accountable for field safety, not to the team whose recipe is being tested.
Who can help us validate a synthetic-blend obstacle detector before the next field trial?
TechnoLynx helps teams building autonomous orchard robots and machines find out how much real footage their synthetic-blend detector needs, and builds the evidence to show it. In practice that means designing the per-class mix ablation and gap metrics, running generation and training on your compute, and packaging the results for your safety board. If you are weighing in-house work against outside help, the deciding question is whether your team already has a frozen reference set and the capacity to run repeated ablations; if it does, a review may be enough. When choosing a partner, ask for the ablation design and the variance treatment before any results. This work runs under Computer Vision R&D, starting with a joint scoping of the validation stages.
The engagement does not suit teams that want certification. TechnoLynx provides engineering evidence, not regulatory sign-off. For structured field testing, ARRNet’s published procedure tests detection, response timing, safe stopping and collision avoidance. A team that already has independent verification in place may only need help with the gap metric.
FAQ
What does “sim to real” mean?
Sim to real means training or testing a model on simulated data and then using it on real-world data. The difference between the two, called the domain gap, usually lowers performance. Synthetic data comes with its ground truth labels for free, which makes it cost-efficient to produce.
What is the meaning of mean average precision (mAP)?
Mean average precision is the area under the precision-recall curve for each class, averaged across classes, at a stated IoU threshold such as 0.5 for mAP50.
