A use case that looks automatable on paper can collapse into the speculative bucket the moment you price the annotation it actually needs. Data labeling is not a downstream chore you schedule after the build decision — it is an input to the feasibility decision itself. We see this pattern regularly. A buyer arrives confident: “We have the data.” What they usually mean is that raw material exists somewhere — support tickets, scanned forms, product images, chat transcripts. What they rarely know is whether that material can be labeled reliably, consistently, and at the volume a model needs. Those three properties, not the raw volume, determine whether the project is buildable. What data labeling and annotation services actually do Labeling turns raw data into supervised signal. A support ticket becomes a ticket tagged with an intent. An X-ray becomes a scan with a bounding box around a lesion. A pair of model outputs becomes a ranked preference. The service — whether delivered by an in-house team, an external vendor, or a model-assisted pipeline — is the machinery that produces that signal at a defined quality bar. The word “services” hides most of the difficulty. The mechanical part (drawing boxes, applying tags) is cheap and easy to demonstrate. The hard part is the definition work: writing an annotation guideline precise enough that two competent people, shown the same item, produce the same label. That agreement — not throughput — is what makes labeled data usable as a training or evaluation target. This is why we treat annotation readiness as a data-readiness signal inside a feasibility assessment. It sits alongside the questions covered in our data-centric approach to AI for feasibility: does the data represent the production distribution, and can it be labeled to a defensible standard? A “yes” to volume with a “no” to consistent labeling is a project that fails after the budget is spent, not before. What kinds of labeling exist, and which does your use case need? Different GenAI use cases consume different annotation shapes. Picking the wrong one — or underestimating the hardest one you need — is how a scoped project quietly balloons. Labeling type What it captures Typical use case Relative cost per item Classification / tagging One label per item from a fixed set Ticket routing, intent detection, content moderation Low Span / token labeling Marked substrings inside text Named-entity extraction, PII redaction, clause tagging Medium Bounding boxes / segmentation Regions inside an image Defect detection, medical imaging, object localisation Medium–high Ranking / pairwise preference Ordering of candidate outputs RLHF, reward-model training, response tuning High (needs judgment) Free-form reference answers Written gold outputs Instruction tuning, evaluation sets Highest (bespoke, expert) The cost column is directional, not a benchmarked rate — it reflects the pattern we see across annotation engagements, where judgment-heavy tasks like preference ranking and reference-answer writing dominate the budget even when the item count is small. A team that assumes all labeling is “just tagging” will price a classification task and then discover the model actually needs preference data, which carries a different cost structure entirely. Preference and ranking data deserve special caution. Unlike a bounding box, there is often no ground truth — only informed judgment — so the quality of the labeling guideline and the calibration of annotators matters more than raw volume. If you are heading toward reinforcement-learning-from-feedback territory, it helps to understand where that fits relative to other approaches; our note on contextual bandit algorithms versus generative models walks through when feedback-driven methods earn their complexity. How does labeling quality decide whether a use case is feasible? The single number that most often flips a feasibility verdict is inter-annotator agreement: the rate at which independent annotators, given the same items and the same guideline, produce the same label. Report it with a chance-corrected statistic (Cohen’s kappa, Krippendorff’s alpha) rather than raw percent agreement, because on skewed label distributions raw agreement can look high while the real signal is close to noise. Here is the mechanism that makes this a feasibility gate. A model cannot learn a distinction that human labelers cannot agree on. If two experts disagree on whether a chest X-ray shows a finding 30% of the time, the ceiling on model accuracy is fixed by that disagreement — no architecture, no amount of GPU time, no fine-tuning recovers signal that was never consistently defined. The model faithfully learns the confusion in the labels. So a use case with roughly 95% agreement (chance-corrected) on a labeled set of a few thousand items is defensibly automatable — the task is well-defined and a model has a clear target. A use case where agreement sits in the 60–70% range is not “harder to build”; it is a different classification. It belongs in the speculative or research bucket until the labeling guideline is refined enough to lift agreement, or until the task is redefined into something labelers can agree on. This is an observed pattern from annotation work, not a fixed threshold; the exact number that constitutes “enough” depends on error tolerance and how the model’s mistakes are absorbed downstream. This is the same failure that our colleagues describe from the other direction in why GenAI fails on production data: the model is rarely the bottleneck. The label definition is. How much labeled data is enough, and how do you estimate cost per item? There is no universal number, but there is a defensible way to reach one. Rather than asking “how much data do we need,” ask “what is the smallest labeled set that lets us measure whether the model clears our accuracy bar?” That reframes an open-ended budget into a bounded experiment. A worked feasibility estimate Assume a support-ticket classification use case with 12 intent classes. Work the numbers before committing: Evaluation set first. You need enough labeled items per class to detect a meaningful accuracy difference — on the order of a few hundred per class for a reliable estimate. For 12 classes, that is roughly 3,000–5,000 items just to measure. Agreement pilot. Double-label a subset (say 300 items) with two annotators to compute inter-annotator agreement before scaling. If agreement is low, stop and fix the guideline — do not buy 50,000 labels against a definition nobody agrees on. Cost per item. Estimate annotator minutes per item, multiply by loaded labeling cost, and add a review pass. A 30-second classification and a 10-minute expert radiology annotation differ by more than an order of magnitude per item; the same headline “we need 5,000 labels” hides wildly different budgets. Training volume. Only after the evaluation set and agreement check do you scale to training volume, and for many GenAI use cases few-shot or retrieval approaches reduce the labeled-training requirement substantially. The point of the sequence is that the agreement pilot and evaluation set are cheap relative to a full build, and they produce the number that turns a vague “we have data” into a defensible go/no-go input. This is the discipline behind an A3 GenAI Feasibility Assessment: pay for the measurement that de-risks the decision before you pay for the decision. In-house annotators, a labeling vendor, or model-assisted labeling? The sourcing choice is a trade-off between domain expertise, cost, throughput, and control — and it usually is not all-or-nothing. Approach Best when Watch out for In-house / SME annotators Task needs deep domain expertise (clinical, legal, safety) Scarce expert time; slow throughput; annotation is not their day job External labeling vendor High volume, well-specified task, tolerant of a managed quality process Guideline drift, needs a gold-standard audit set, offshore domain gaps Model-assisted (pre-label + human review) Large volume, a model already gets most items right Automation bias — reviewers rubber-stamp wrong pre-labels; needs blind spot-checks Model-assisted labeling is the most misunderstood of the three. Using a capable model to pre-label and having humans correct it can cut cost dramatically, but only if the review step is designed to catch the model’s systematic errors rather than inherit them. When the pre-labeling model and the target model share a failure mode, human reviewers under time pressure tend to accept the confident-looking wrong answer — a pattern worth explicit spot-check design. Self-supervised pre-training changes the arithmetic further by reducing how much labeled data you need at all; our self-supervised learning example covers when learning from unlabeled data is the better lever than buying more labels. In practice we often recommend a hybrid: SMEs write and calibrate the guideline and hold a gold set, a vendor or model-assisted pipeline handles volume, and a continuous audit against the gold set keeps agreement from drifting. The data-centric practices we apply to generative AI treat that audit loop as part of the pipeline, not a one-time setup. FAQ What matters most about data labeling and annotation services in practice? Data labeling turns raw data into supervised signal — a tag on a ticket, a box on an image, a ranked pair of outputs — produced to a defined quality bar by in-house annotators, a vendor, or a model-assisted pipeline. In practice the mechanical marking is the easy part; the hard part is writing a guideline precise enough that independent annotators produce the same label, because that consistency is what makes the data usable. What types of labeling and annotation exist, and which does a given GenAI use case need? The common shapes are classification/tagging, span or token labeling, bounding boxes and segmentation, ranking or pairwise preference data, and free-form reference answers. Cost and difficulty rise from simple classification up to judgment-heavy preference and reference tasks. The use case dictates the shape: intent routing needs classification, RLHF needs preference data, and misjudging which one you need is a common source of budget overrun. How does labeling quality — inter-annotator agreement and consistency — affect whether a use case is feasible? A model cannot learn a distinction that human labelers cannot agree on, so inter-annotator agreement sets a ceiling on achievable accuracy. High chance-corrected agreement (around 95% on a few thousand items) signals a well-defined, automatable task; agreement in the 60–70% range signals a task that belongs in the speculative or research bucket until the guideline is refined or the task redefined. How much labeled data is enough before committing to a build, and how do I estimate labeling cost per item? Reframe the question as the smallest labeled set that lets you measure whether the model clears your accuracy bar — often a few hundred items per class for evaluation, plus a small double-labeled agreement pilot. Estimate cost per item from annotator minutes times loaded cost plus a review pass; a 30-second classification and a 10-minute expert annotation differ by more than an order of magnitude, so the same label count can hide very different budgets. When should we use in-house annotators, a labeling vendor, or model-assisted labeling — and what are the trade-offs? Use in-house or SME annotators when the task needs deep domain expertise, a vendor for high-volume well-specified work, and model-assisted pre-labeling when a model already gets most items right and you can design a review step that catches its systematic errors. A common hybrid has SMEs write and calibrate the guideline and hold a gold set while a vendor or model-assisted pipeline handles volume under continuous audit. How does data labeling readiness feed into the automatable / speculative / research classification in a GenAI feasibility assessment? Labeling readiness is one of the concrete data-readiness signals a feasibility assessment evaluates. High agreement on an adequate labeled set at a known cost per item supports an automatable verdict; unknown cost, bespoke expert annotation, or low agreement pushes the use case toward speculative or research until those unknowns are resolved. Where this leaves the go/no-go call The reason annotation belongs in the feasibility conversation rather than the delivery plan is simple: it changes the classification of the use case, not just its schedule. A task with well-defined labels and measured agreement is automatable and can be priced. A task whose labels nobody can agree on is not a slower build — it is a research question wearing a product’s clothes. Before committing development budget, name two numbers: the labeling cost per item and the target agreement threshold. If you cannot produce either, the honest classification is speculative, and the cheapest next step is the agreement pilot — the same discipline that anchors an A3 GenAI Feasibility Assessment rather than a hopeful build.
