Dataset versioning keeps model evaluation trustworthy only when it comes with a split policy. Every dataset version sits behind one governed registry (DVC-style pointers and hashes, or lakeFS-style branches, on the storage you already have). Metadata is enforced before data enters the shared pool. Training, validation and test sets are assigned by the unit that carries dependence, such as the orchard, the store or the capture log, and never image by image. When near-identical images from one video sequence end up on both sides of a split, the test gives an over-optimistic performance assessment. A model can pick up subject-specific acquisition patterns instead of the features you care about.
The same discipline applies to the numbers around the model. Release thresholds and archive versions need an owner and an audit trail, or every release reopens the argument. In vision work, the data, the processes that produce more of it and realistic accuracy expectations come before model selection. This is infrastructure work as much as data science, part of the wider job of building, serving and monitoring models covered under AI infrastructure and production SaaS.
How much do synthetic orchard scenes actually move mAP on an orchard we never trained on?
You measure the marginal contribution of synthetic orchard scenes with a controlled ablation. Train real-only, synthetic-only and a small, fixed set of mixed-ratio models, then evaluate every configuration on one real orchard that no training mix has touched. The defensible answer is the mAP difference between configurations on that holdout, with a confidence interval computed at the right level of grouping. A delta from ad hoc retraining is not.
TechnoLynx designs and runs this kind of synthetic-versus-real ablation through its Computer Vision R&D work. The first step is fixing the ablation grid and the unseen-orchard holdout protocol before any model is trained. After that come the pipeline, mAP with statistical confidence, an independence check on the holdout, and documentation that lets the next synthetic batch be tested the same way.
Published synthetic-data studies run comparable controlled comparisons. A sim-to-real fruit detection study trained on two synthetic set sizes (1,000 and 5,000 images) and two real subsets (50 and 100 images). It evaluated all trained models on two fixed real-world test sets that were never used for training or hyperparameter tuning and stayed constant across experiments. A fire-segmentation study replaced fixed fractions of real data with synthetic images. It used paired t-tests to judge whether each change was statistically significant. A 2026 AgriEngineering benchmark across three agricultural tasks, including apple detection in orchards, reported that combining synthetic data with only 10 real images recovered 75–98% of the gap between training on those 10 images alone and training on the full real set. That figure belongs to its datasets and its synthetic generator. It tells you nothing about yours until your own holdout says so.
Worked example: an ablation grid fixed in advance
Assume one real orchard is set aside as the holdout before any training, and that the synthetic batch and real seasons are each a versioned dataset. The grid below is the whole experiment. You do not add configurations later because one result looked interesting.
| Configuration | Training data | Evaluated on | What it answers |
|---|---|---|---|
| Real-only | Real seasons from training orchards | Unseen holdout orchard | Baseline generalisation |
| Synthetic-only | Synthetic batch only | Unseen holdout orchard | How far simulation alone transfers |
| Mixed, low ratio | Real plus a small synthetic share | Unseen holdout orchard | Whether a little synthetic data helps |
| Mixed, high ratio | Real plus a large synthetic share | Unseen holdout orchard | Whether more synthetic keeps helping or starts to hurt |
You avoid “retraining dozens of times” by fixing this grid up front and repeating each cell over a few seeds, rather than tweaking mixes after the fact. A leakage-aware medical imaging benchmark used this pattern. It trained each deep model under three random seeds with an identical protocol and reported mean ± standard deviation.
Is a small real dataset enough for a statistically defensible mAP delta?
Often it is not, and the honest report says so. The same benchmark resampled its 28 external test patients rather than its 1,867 images, because resampling individual images in grouped data underestimates uncertainty. The patient-level 95% intervals came out wide enough that differences between models were not resolved. For orchards, the equivalent is to bootstrap over rows, capture logs or trees, not over images, and to accept wide intervals when the holdout is small.
Two further limits apply. A holdout from the same acquisition pipeline as training, as in that benchmark, leaves robustness to cross-site shift untested. And the result describes the current generation pipeline only, so the generator’s randomisation ranges and seed have to be recorded and fixed, as the sim-to-real fruit detection study did for reproducibility. Checking whether the synthetic scenes are close enough to real orchards in the first place is a separate sim-to-real gap question and needs its own evidence.
Should season 1 and 2 orchard data be split by orchard, by date or at random?
If the claim is “this model works on an orchard it has never seen”, split by orchard and keep the third orchard strictly held out. Use date-based splits only when you are testing temporal drift. Use random sampling only for in-distribution validation, never to support a held-out generalisation claim. Random splits place images from the same orchard on both sides, so validation is not independent of training.
TechnoLynx sets up this split policy and the versioning scheme around it as part of Computer Vision R&D engagements. The work starts by writing down the dataset governance requirements against the generalisation goal. The orchard-based split and the version tracking come next, built into the ablation pipeline, along with a check that the third-orchard holdout stays unseen across every run.
What leakage from image-level splits does to the score
Roberts et al. showed that when hold-out data is not independent of training data, models look more reliable than they are. They also showed that a randomly selected hold-out does not give independent validation when a dependence structure is present. A leukaemia-detection benchmark measured the size of the effect directly. Random image-level splitting inflated AUROC by about 0.04 compared with subject-disjoint splitting, even for conservative frozen-feature classifiers. The same paper set prior studies reporting accuracies and F1-scores as high as 97–98% under image-level splits against leakage-free evaluations that converged on roughly 0.86–0.90 F1.
Two boundaries matter. The authors ran their leakage ablation only on the frozen-feature classifiers, not on end-to-end networks. Their external test set also came from the same acquisition pipeline as training, so robustness to cross-site shift remained untested. That second point is exactly why an unseen orchard is the right holdout. Separating subjects or trees inside one site does not tell you how the model behaves at another site. The problem is not confined to one field: Kapoor and Narayanan (Patterns, 2023) found leakage in 17 fields, affecting 294 papers, sometimes with wildly overoptimistic conclusions.
An orchard dataset that got the split right
An orchard person-detection dataset is a useful worked precedent. Its authors kept each set of capture logs intact within one split and targeted a training:validation:test ratio of roughly 2:1:1. They grouped log sets by the clothing of the person filmed and spread variety across the splits. They also used different regions of the grove in different collections, so the same section never appeared in more than one log set. Night-time logs that were too few to build three representative splits were left out of the benchmark rather than forced in.
Time-ordered data needs its own rule. scikit-learn’s documentation warns that shuffling time-ordered samples tests the model on data artificially close to training and inflates the validation score. Use a time-aware splitter such as TimeSeriesSplit when drift is the question.
A dataset versioning rule that makes an ablation reproducible
The MLCommons Croissant 1.0 specification recommends MAJOR.MINOR.PATCH semantic versioning for datasets, together with file checksums. Its rule maps cleanly onto split governance:
| Bump | When | Example in an orchard project |
|---|---|---|
| PATCH | Data unchanged, only serialisation or packaging differs | Re-encoding images to a different format |
| MINOR | Existing data still retrievable as it was; additions only | A new synthetic batch added as a new split or record set |
| MAJOR | Existing records edited, removed or moved across splits | Reassigning an orchard from training to test |
Croissant strongly recommends recording checksums for every file in a versioned dataset, so downloads can be checked against the declared definition. With a tool like DVC, the Git commit holding the training code is tied to a specific hash of the data version, and dvc pull restores exactly that data. A versioning record should also capture how a version was made, meaning the script and parameters that produced it from its parent. For synthetic batches, that means the generator settings and seed.
Holding out separate validation and test sets reduces the samples available for learning the model.
How do we get every region into one dataset versioning registry instead of separate folders?
Build one governed catalogue that sits on top of the regional pipelines rather than replacing them. Each region keeps its storage and capture workflow, and a versioning layer records every submission. A validation gate rejects anything that lacks the required metadata.
TechnoLynx does this work by auditing each region’s shelf-image pipeline and storage for compatibility with a versioning tool first. It then designs the ingestion metadata schema, integrates a tool such as DVC with the existing storage, adds the schema gates and stands up the central catalogue that indexes every region’s contributions.
DVC’s documentation describes a data registry as a DVC project dedicated to versioning datasets. The repository holds the metadata and full change history while the data stays in one or more remotes, acting as “data management middleware between ML projects and cloud storage”. Datasets enter the registry with dvc add, small .dvc files stand in for the data in an ordinary Git workflow, and the data is pushed to remote storage that others can reach. Releases can be tagged with semantic versions and lifecycle stages saved as Git tags. Those remotes can be cloud buckets or network-attached storage; DVC calls the latter “local remotes”. For a fuller comparison of when DVC or lakeFS fits, see our guide to data version control tools for production.
Can a versioning tool sit on top of the workflows regions already use?
If a region’s current tools can talk to a lineage and versioning layer without major rework, bolt the layer on and treat the catalogue as the index above it. Migrate a region to a new pipeline only where integration is not feasible, and keep the shared catalogue as the common layer either way. The metadata side also works without restructuring. The Croissant authors note that their format describes data as it already exists, and that supporting it required adding metadata, not changing data. Where storage is object-store centred, lakeFS puts a Git interface on top of S3 or on-premise storage, with branches, commits and promotion to production.
The integration has to be tested against each region’s real submissions, not a clean sample.
Which metadata fields must a dataset carry before it enters the shared pool?
A minimum set, assembled from published metadata frameworks and adapted to multi-region image data:
| Field group | What it records | Basis |
|---|---|---|
| Version | Who created it, when, commit message, semantic version | DVC guide (datos.gob.es); Data Cards |
| Schema | Field names, types and constraints, so structure can be validated automatically | Data Package via datos.gob.es |
| Provenance | Origin: capturing device, operator, date | DVC guide |
| Lineage | Script and parameters that produced it from its parent | DVC guide |
| Split | Which split each record belongs to, with per-split statistics | Croissant; Data Cards |
| Integrity | Checksum per file | Croissant 1.0 |
| Domain fields | Store or site ID, region, store format, camera or sensor type, planogram reference, consent and compliance flags | Our recommendation for retail shelf imagery |
The Data Cards authors (2022) recommend automating split-statistics fields for accuracy, rather than leaving them to be typed by hand.
Validation gates that regions cannot bypass
A gate is only a gate if non-compliant data physically cannot reach the training pool. lakeFS implements this with Actions. A YAML policy lists required commit fields and a required metadata file, and checks values against patterns or allowed lists. Data that fails cannot be merged to production. At Google, Breck et al. described a data validator that checks each batch against a schema and flags disagreements as anomalies. They also recommended treating the schema as a production asset, reviewed and versioned alongside source code. Automatic verification is preferable where possible, because it is not affected by human factors and can be rerun on every iteration of the dataset.
The registry then becomes a single point of governance. Someone has to be resourced to review schema changes and rejected submissions.
What split rule stops the same store leaking across training and validation?
Assign whole stores, not images, to training or validation, and stratify the assignment by region and store format. Record each store’s assignment once in the governed catalogue, so every regional pipeline reads the same answer instead of deciding locally. Treat images of one store as a dependent group, the structure that leaks across an image-level split.
scikit-learn’s documentation states the principle. When samples come in dependent groups, every sample in a validation fold must come from groups absent from the paired training fold. GroupKFold guarantees a group never appears in both sets. StratifiedGroupKFold additionally tries to preserve class distribution while keeping each group within one split. A 2025 e-commerce search competition team followed the grouped-then-stratified order: they grouped records by query, then stratified those groups by language and label distribution across folds. For store data, the group is the store ID and the stratification key combines region and format. An ultrasound image–text study applied the same logic to clinical cases. It split at case level so no frame from one case crossed splits, and stratified its 6:2:2 split over the multi-source corpus.
| Split decision | Default | Change it when |
|---|---|---|
| Unit of assignment | Store (or site, orchard, case) | Images within one store are genuinely independent |
| Stratification | Region × store format | Cells become too sparse; merge formats or stratify on region only |
| Ratios | 70/15/15 for under 1 million samples; 90/5/5 above, per AWS | Group count, not image count, is small |
| Duplicates | Deduplicate before splitting | Never skip: duplicates across subsets can look like generalisation |
AWS Prescriptive Guidance recommends stratified split algorithms for small or highly imbalanced datasets. It also warns that duplicate samples can leave subsets with samples in common even after a split. When several dependence axes exist at once, the R package splitGraph turns metadata into a dependency graph. It treats the collection site as a first-class leakage axis and reports, for the constraint you chose, which leakage paths it actually severs.
Stopping a regional team validating on stores used elsewhere
Centralised assignment is the control. If each region keeps its own legacy split file, the policy exists on paper only. A practical mechanism is a store-to-split table in the catalogue that the training and evaluation loaders must query, with the evaluation job refusing to run when a store appears in both. Two cautions apply. Keep stratification dimensions few when each cell holds only a few stores. And store-level splitting reduces training diversity when a region has only a handful of stores per format.
Who signs off when QA and sales disagree on a release threshold?
A named role, either a QA lead or a small change board, should hold final sign-off. That authority is written into policy and backed by management mandate, not merely documented. Published banking guidance offers a useful parallel. The Federal Reserve, OCC and FDIC’s revised Guidance on Model Risk Management says model risk management benefits from clear roles and well-defined accountability, including for conflicts of interest between development and validation lines. It also says sound governance names the individuals responsible for each lifecycle activity. That guidance is written for banks, so it transfers here by analogy rather than as a requirement.
Versioning thresholds so they are not re-litigated every release
Treat each threshold and SLA figure as a versioned artefact. It should have a value, the evaluation dataset version and hash it was tested against, the model version, an approver and a date. The SLA document should quote only thresholds that exist in that versioned record, so sales cannot promise a number QA does not test against. Reviews then happen on a fixed cadence, and a disagreement mid-release is logged as a change request instead of reopening the threshold. Even a manual changelog kept alongside versioned data gives a basic level of traceability.
Which archive version is official for a given model version?
Treat the version hash a model’s training run recorded as the official archive for that model. A formal change log tying each run to an exact archive version and hash is what makes old backtests rerunnable. A reprocessed calibration changes existing records, which under Croissant’s rule is a MAJOR version. It should therefore arrive as a new version or branch that the named owner promotes, never as an overwrite of the history.
On retention, the datos.gob.es DVC guide suggests keeping versions used to train production models indefinitely or as the law requires, preserving milestone versions, and keeping recent versions in hot storage for a set window. It suggests unmerged experimental versions can be deleted after a grace period. It also describes moving old, unlabelled versions to cold storage on a schedule. One trap is easy to miss: DVC’s documentation notes that cloud lifecycle policies may delete object versions, after which DVC cannot recover them. Retention rules and bucket lifecycle rules have to be written together. Assembling these records into an audit-ready evidence pack is a further step beyond the governance itself.
Where should the authoritative copy of each regional dataset live?
Make whatever the governed catalogue points to the authoritative copy. Make the catalogue a pointer-based index over the existing cloud bucket and regional NAS shares, and move data physically only where access, compliance or performance demands it.
TechnoLynx approaches this by auditing where data and model versions actually sit across buckets and NAS shares first. It then designs the registry schema, builds the catalogue as an index over current storage, and plans the phased retirement of redundant copies, with governance that stops data scattering again.
DVC supports indexing data in place. DVC’s import-url --no-download saves tracking metadata without copying the data locally. With --version-aware on a supported cloud provider, DVC tracks the provider’s own version IDs and assumes the versions remain at the source. Cloud versioning support covers Amazon S3, Azure Blob Storage and Google Cloud Storage, each with versioning enabled on the bucket or container. Native bucket versioning keeps old objects but leaves the link between code and data version to be managed by hand or by platform tools. NAS shares can serve as DVC “local remotes”. The model side needs the same treatment: dataset versions are only useful if each model version records which one it was trained on, a pairing covered in our piece on versioned model registries and matching lineage depth to the problem.
An index-only approach leaves data physically scattered. Make the catalogue the only sanctioned path into training.
Who can set up dataset versioning and leakage-proof splits for our vision pipeline?
TechnoLynx builds the registry, metadata gates, group-level split policy and ablation pipeline for teams whose computer vision evaluation has to hold up under scrutiny, and does it on the storage and tools the regions already use. Whether to do this in-house or bring in a specialist depends on whether you have engineers who can own the gates and catalogue after launch. A partner earns its place by leaving the policy, code and documentation with your team, not by running them indefinitely. When choosing one, ask to see a split policy, a versioning rule and an ablation report from the first weeks, before any modelling. The engagement runs under Computer Vision R&D, starting with an audit of current storage, splits and evaluation results.
Outside help pays off mainly when multiple regions, synthetic data and contested thresholds meet in one programme.
FAQ
What percentage of data is typically allocated to the training set?
AWS Prescriptive Guidance gives 70% training, 15% validation and 15% test as common for datasets under about 1 million samples, and 90/5/5 for larger ones. With grouped data, apply the ratios to groups such as stores or orchards, not to images.
What are the popular version control systems for data?
Git versions code and small metadata files. Tools built for data, such as DVC, Pachyderm and Delta Lake, manage large binary data through metadata and pointers into object storage. lakeFS adds Git-style branches and commits directly on object storage.
