A GenAI prototype that scored well on a curated demo set and then degraded the week it met real traffic is not a modelling failure. It is a data-quality-blindness failure — someone accepted the data without ever making that acceptance an explicit decision. A data-centric approach is the discipline that turns that silent assumption into an auditable one, before the prototyping budget commits rather than after a launch stalls.

The instinct, when a prototype underperforms in production, is to reach for the model. Try a larger base model, rewrite the prompts, add a re-ranking step, turn a few sampling knobs. Sometimes that helps at the margin. But when the gap between demo accuracy and production accuracy is large and stubborn, the model was rarely the constraint. The data was — specifically, the difference between the tidy sample the prototype was built on and the messy live distribution it now has to serve.

What does a data-centric approach mean in practice?

The cleanest way to describe a data-centric approach is by what it holds fixed. In a model-centric workflow you hold the dataset roughly constant and iterate on the model: architecture, hyperparameters, prompt structure, decoding strategy. In a data-centric workflow you hold the model roughly constant — pick a reasonable base and stop fiddling — and iterate on the data instead. You treat coverage, labelling consistency, edge-case representation, and drift as the things you systematically improve.

This is not a philosophical preference. It reflects where the leverage actually sits in most applied GenAI work. A base model like a well-chosen open-weight LLM or a hosted frontier model is already competent at the general task; what determines whether it works for your problem is whether the data it sees at inference — and the data you evaluated it against — actually resembles production. Our experience across delivery engagements is that once a competent base model is in place, further model tuning yields diminishing returns while data work keeps paying off (an observed pattern across projects, not a benchmarked ratio).

In practice, adopting the approach looks like four habits:

• Curate against the production distribution, not the demo. The evaluation set should be sampled to mirror the real traffic mix — including the boring high-volume cases and the awkward long tail — rather than the handful of clean examples that made the demo look good.
• Measure labelling agreement, not just label existence. Two annotators disagreeing 20% of the time is a signal that the task itself is under-specified, and no model will resolve an ambiguity the humans could not.
• Enumerate edge cases explicitly. The cases that dominate real traffic are usually the ones nobody put in the sample: mixed-language input, malformed records, adversarial phrasing, empty fields.
• Watch for drift between sample and live. The curated set is a snapshot; production is a moving distribution. The gap between them is the failure surface.

For a fuller treatment of the underlying concept, our explainer on the data-centric approach to AI and what it means in practice for feasibility walks through the same distinction from the concept side. This article is about what it changes operationally.

Why does a prototype that passes on curated data fail in production?

The divergence point is specific and predictable: it is the moment a GenAI prototype meets production data it was never exercised against. The curated demo set did its job of proving the idea is plausible — and in doing so it quietly hid three things.

First, the noise. Real inputs arrive with typos, truncation, encoding artefacts, and formatting the sample never contained. Second, the long tail. A demo of fifty representative queries can look flawless while the thousandth query — an unusual product code, a rare document layout, a dialect the model wasn’t tuned for — falls outside anything the prototype saw. Third, the ambiguous cases, where even a careful human would hesitate; these dominate real traffic in ways curated sets systematically underweight because whoever assembled the sample naturally reached for clear examples.

A model-centric response to this failure treats the symptom. Swapping in a bigger model can lift the average slightly, but it does nothing about the fact that the evaluation never contained the cases that break. You are optimising blind. A data-centric response reconstructs the evaluation set to include the noise, the long tail, and the ambiguity — and only then asks whether the model is good enough. Very often the answer, once the data is honest, is that the model was fine and the sample was lying. We treat this pattern at more length in the data-centric approach and why GenAI fails on production data, which is the companion piece to this one on the production-failure side.

Data-centric vs model-centric: a comparison

The two approaches are not mutually exclusive — mature teams do both — but they answer different questions and the order matters. When a prototype-to-production gap opens, the data-centric question comes first.

Dimension	Model-centric	Data-centric
What you hold fixed	The dataset	The model
What you iterate on	Architecture, prompts, decoding, hyperparameters	Coverage, labelling, edge cases, drift
Primary failure it addresses	Model under-capacity or poor tuning	Evaluation set that misrepresents production
Diagnostic question	“Is the model good enough?”	“Does my data look like production?”
Where leverage sits after a competent base model	Diminishing	Sustained
Cost of ignoring it	Over-spend on tuning that can’t fix blind spots	Launch stalls when live traffic exposes the gap

The failure mode this table guards against is spending weeks on model tuning to close a gap that model tuning structurally cannot close, because the evaluation set never contained the cases that fail. If you are choosing tooling to support the data side — versioned datasets, reproducible splits — our guide to data version control tools for production GenAI, and when to use DVC or LakeFS covers the mechanics of keeping data changes as auditable as code changes.

What does a data readiness audit check before prototyping begins?

The practical form a data-centric approach takes at the start of a project is a data readiness audit — the step that catches data-quality blindness before a curated-data success gets mistaken for production readiness. It produces a data readiness state: a documented answer to whether the data can support the intended GenAI capability at all.

Use the checklist below as a gate. If you cannot answer these before prototyping spend commits, the honest position is that you don’t yet know whether the project is feasible — which is itself a valuable finding.

Data readiness checklist

• Representative coverage — Does the evaluation sample reflect the production distribution, including the high-volume mundane cases and the long tail? Can you point to how it was sampled, or was it assembled by convenience?
• Labelling agreement — For any supervised or evaluated task, what is the inter-annotator agreement rate? Low agreement means the task is under-specified and no model will rescue it.
• Edge-case inventory — Is there a written list of the awkward inputs (malformed, multilingual, adversarial, empty) and a documented decision on how each is handled?
• Drift exposure — How different is the curated set from a fresh pull of live data today? Is there a mechanism to re-measure that gap over time?
• Provenance and versioning — Can you reproduce exactly which data produced a given result, and roll back a data change the way you’d roll back code?
• Acceptance decision — Is there a named owner who has explicitly signed off that this data is fit for the intended use — or has the data simply been assumed adequate by default?

That last item is the point of the whole exercise. Data-quality blindness is not usually a technical oversight; it is an accountability gap. Someone accepted the data, but no one decided to. A data-centric approach makes that acceptance an explicit, auditable decision with a name attached. Related tooling — the data labelling and annotation services that sit behind the data-quality gate in GenAI failure — is where the labelling-agreement half of this audit gets its numbers.

Which data-quality signals matter most, and how do you measure them?

Four signals carry most of the weight for GenAI, and each has a concrete measurement rather than a vibe.

Coverage is measured by comparing the distribution of your evaluation set against a fresh sample of production inputs across the dimensions that matter — input length, language, source system, category. Large divergences flag blind spots. Labelling consistency is measured by inter-annotator agreement (Cohen’s or Fleiss’ kappa for categorical tasks); persistently low agreement is a task-definition problem, not a staffing one. Edge-case representation is measured by whether your enumerated edge cases actually appear in the evaluation set in proportions that trigger failures before launch, not after. Drift is measured by re-running the coverage comparison on a schedule and watching the gap between the curated snapshot and live data widen or hold.

None of these require exotic infrastructure. They require the decision to look. When teams instrument this, the same pipelines that support production monitoring — the discipline covered in our guide to how to monitor ML models in production — extend naturally backward into pre-launch data readiness. The measurement machinery is the same; only the timing differs.

How does this fit into a GenAI feasibility assessment?

A data-centric approach is not a standalone activity bolted onto a project. It is the substance of the feasibility question. A GenAI feasibility assessment that skips data readiness is measuring the wrong thing — it confirms the model can do the task in principle while leaving unanswered whether your data lets it do the task in production. This is exactly the discipline our broader [generative AI practice](generative AI) brings to the start of an engagement, and it connects to the wider project-failure methodology in our [R&D consulting practice](our service packs), where data-centric delivery is one facet of a general AI-delivery discipline rather than a GenAI-only trick.

Establishing a data readiness state before prototyping spend commits is what separates a feasibility assessment that saves money from one that merely delays the bill. The prototype-to-production gap — accuracy on a curated set minus accuracy on live data — is the number the whole exercise exists to shrink, and it shrinks most cheaply before a single line of production integration is written.

FAQ

How does a data-centric approach work?

A data-centric approach holds the model roughly constant and iterates on the data — coverage, labelling consistency, edge-case representation, and drift — rather than iterating on architecture, prompts, and hyperparameters. In practice it means curating your evaluation set against the real production distribution, measuring labelling agreement, enumerating edge cases explicitly, and watching for drift between the sample and live traffic. The leverage sits in the data once a competent base model is in place.

Why does a GenAI prototype that succeeds on curated data fail on production data, and how does a data-centric approach prevent that?

The curated demo set hides three things production exposes: input noise, the long tail of rare cases, and ambiguous inputs that even humans hesitate on. A model-centric fix treats the symptom by swapping models, but it cannot fix cases the evaluation never contained. A data-centric approach reconstructs the evaluation set to include the noise, the long tail, and the ambiguity before launch, so the gap is measured and closed rather than discovered in production.

What is the difference between a data-centric and a model-centric approach for generative AI projects?

Model-centric work holds the dataset fixed and iterates on the model; data-centric work holds the model fixed and iterates on the data. They answer different questions — “is the model good enough?” versus “does my data look like production?” — and the order matters. When a prototype-to-production gap opens, the data-centric question comes first, because model tuning structurally cannot close a gap caused by an evaluation set that never contained the failing cases.

What does a data readiness audit check before prototyping begins?

It checks representative coverage of the production distribution, labelling agreement rates, a documented edge-case inventory, drift exposure between the curated set and live data, and data provenance and versioning. Above all it checks for a named acceptance decision — whether someone has explicitly signed off that the data is fit for the intended use, rather than assuming it by default. If those questions cannot be answered before spend commits, feasibility is genuinely unknown.

Which data-quality signals — coverage, labelling consistency, edge-case representation, drift — matter most for GenAI, and how do you measure them?

Coverage is measured by comparing the evaluation set’s distribution against a fresh production sample across length, language, source, and category. Labelling consistency is measured by inter-annotator agreement such as Cohen’s or Fleiss’ kappa. Edge-case representation is measured by whether enumerated awkward inputs actually appear in the evaluation set. Drift is measured by re-running the coverage comparison on a schedule and watching the gap. None require exotic tooling — only the decision to look.

Who is accountable when data-quality blindness stalls a GenAI project, and how does making data acceptance an explicit decision change that?

Data-quality blindness is usually an accountability gap rather than a technical oversight: someone accepted the data, but no one decided to. Making data acceptance an explicit, auditable decision attaches a named owner to the sign-off that the data is fit for the intended use. That converts a silent assumption into a reviewable decision, which is what lets a stalled launch be traced back to a specific, correctable point rather than a diffuse “the model didn’t work.”

How does a data-centric approach fit into a GenAI feasibility assessment before budget commits?

Data readiness is the substance of the feasibility question, not an add-on. An assessment that skips it confirms the model can do the task in principle while leaving unanswered whether your data lets it do the task in production. Establishing a documented data readiness state before prototyping spend commits is what separates a feasibility assessment that saves money from one that merely delays the bill.

The uncomfortable finding a good data readiness audit sometimes returns is that the data is not ready and the project should wait. That is a cheaper answer to receive at the assessment stage than after a launch has already stalled — and it is the failure class, data-quality blindness, that a feasibility assessment exists to catch before curated-data success is mistaken for production readiness.