Ask most teams what data annotation is and you will hear some version of the same answer: it is the labelling step you outsource before training, a fixed cost you pay once and forget. That framing is where projects quietly go wrong. Annotation is not a task that ends; it is a pipeline whose quality, consistency, and feedback loops set a hard ceiling on what any downstream model — generative or agentic — can achieve. A model cannot learn a distinction its labels never encoded, and it cannot be corrected against failures nobody labelled. The gap between those two views does not hurt on day one. It surfaces late — usually in production, when a mislabelled edge case drives a silent failure and the schema turns out to have been undefined from the start. This article walks through how annotation actually works as a system, the quality mechanics that keep it from drifting, and — the part most teams miss — how the requirements change the moment a use case stops being a single generative call and becomes an orchestrated agent. What’s worth understanding about data annotation first? At its simplest, annotation attaches structured meaning to raw data: a bounding box around a tumour on a CT slice, a sentiment label on a review, a transcript aligned to an audio clip, a “correct tool call” flag on an agent’s execution trace. The label is the target the model is measured against. That much is uncontroversial. What the one-time framing misses is that the schema — the definition of what each label means and how ambiguous cases are resolved — is the real deliverable, not the labels themselves. Two annotators handed the same image and no schema will disagree on borderline cases, and that disagreement propagates into the model as noise. In our experience, the projects that stall are almost never short of labelling capacity; they are short of an agreed definition of what a correct label looks like. So annotation in practice is three things running together: a schema that defines the target, a workforce (human or model-assisted) that applies it, and a review loop that measures whether the output is consistent. Treating it as only the middle step — the cheap, outsourceable one — is the error the naive view encodes. This is the same principle behind a data-centric approach to AI feasibility: the data definition, not the model architecture, is usually the binding constraint. Annotation solutions sit inside our broader generative AI work precisely because so many feasibility questions resolve to “what would you have to label, and can you label it consistently?” What types of data annotation exist, and which does your use case need? The type of annotation you need is determined by the data modality and the job the model does — not by what a vendor happens to offer as a package. The distinction matters because pricing, tooling, and quality mechanics differ sharply across them. Annotation type Typical output Where it fits Main quality risk Image / video Bounding boxes, segmentation masks, keypoints Detection, segmentation, CV pipelines Boundary ambiguity, occlusion handling Text Spans, entities, sentiment, intent labels Classification, NLP, RAG grounding Undefined class boundaries Audio Transcripts, speaker turns, event tags Speech recognition, diarization Timing alignment, accent coverage Tool-call traces Correct/incorrect action, expected argument Agent evaluation, orchestration debug No canonical “right answer” schema The first three are familiar. The fourth — annotating an agent’s tool-call traces — is the one most teams have never scoped, and it is exactly the one that governs whether an agentic system can be debugged at all. When an agent uses a model as a tool inside an orchestration loop, the interesting failures are not “the model generated bad text.” They are “the agent called the wrong tool,” “it passed the wrong argument,” or “it looped without terminating.” None of those are visible unless someone labelled the trace. How does annotation for training a generative model differ from annotation for evaluating an agentic system? This is the divergence point, and it is worth stating plainly: annotation for training a generator and annotation for evaluating an orchestration loop are different jobs with different economics. Training annotation produces the target distribution the model learns to reproduce — captions, instruction-response pairs, labelled examples. It is largely front-loaded: you label a corpus, you train, and the marginal cost per additional label falls as the dataset grows. Teams scope this reasonably well because it looks like the classic labelling task. Evaluation and tool-call verification annotation is different in kind. It is ongoing by nature, because the failure cases you most need labelled are the ones the system produces after deployment. Every new failure mode an agent exhibits is a new labelling requirement. The question “did the agent take the right action here?” often has no ground truth until a human defines it, and that definition depends on business context the annotator has to be given. Teams that scope annotation as a fixed training cost systematically underestimate this second bucket — the labelling required for agent evaluation, tool-call verification, and failure-case triage. The measurable consequence is concrete. When failure cases are annotated with a consistent schema, a tool-call error becomes traceable — you can point to the exact step and the expected action. In practice, that shortens the debug loop from days to hours (observed pattern across our agentic engagements, not a benchmarked rate), because engineers stop guessing which of a dozen orchestration steps went wrong. Without labelled traces, the same debugging is a reconstruction exercise. If you are scoping this kind of work, our note on what an agentic AI engagement actually delivers covers where evaluation data fits into the delivery plan. What quality measures keep annotation from drifting? Annotation drift is the failure mode that undoes everything downstream. It happens when the meaning of a label changes over time — a new annotator interprets the schema slightly differently, an edge case gets resolved inconsistently, or the data distribution shifts and the schema was never updated. The model then learns a moving target, and its behaviour degrades in ways that are hard to attribute back to the labels. Three mechanisms keep drift bounded, and all three have to be defined before labelling starts, not retrofitted after. Schema definition with resolved edge cases. The schema must specify not just the classes but how borderline and ambiguous cases are handled. An undefined schema is the single most expensive omission — it is what drives the 20–40% re-labelling cost teams incur when they discover, mid-project, that a third of their labels were applied under inconsistent rules (observed range across engagements, not a published benchmark). Inter-annotator agreement thresholds. Measure whether independent annotators produce the same label on the same item. A defined agreement threshold — Cohen’s or Fleiss’ kappa above an agreed floor — turns “the labels feel inconsistent” into a number you can gate on. Below the threshold, the schema is the problem, not the annotators. Review gates. A structured second pass where a fraction of labels is re-checked against the schema, with disagreements fed back into schema clarification. This is the feedback loop that keeps a live pipeline stable as data and annotators change. The ROI here is not abstract. Teams that define agreement thresholds and review gates up front catch labelling drift before it propagates into model behaviour, which is what avoids the re-labelling cost above. Versioning matters too: annotation sets should be tracked with the same discipline as code, which is why data version control tools belong in any serious annotation pipeline. There is a self-supervised angle as well — techniques that learn from unlabelled data can reduce annotation volume, but they do not remove the need for a labelled evaluation set, which is precisely the bucket teams undercount. When should annotation be outsourced versus kept in-house? There is no universal answer, but there is a clear decision rubric. The variable that matters most is not cost per label — it is how much domain context the annotator needs to apply the schema correctly. Factor Lean outsourced Lean in-house Domain expertise required Low (general objects, common language) High (medical, legal, agent business logic) Schema stability Stable, well-defined Evolving with the product Data sensitivity Non-regulated, shareable Regulated or confidential Feedback loop tightness Loose, batch-oriented Tight, needs same-team iteration Volume Large, repetitive Smaller, high-judgement Training-corpus labelling on common modalities often outsources well: the schema is stable, volume is high, and the work is repetitive. Agent evaluation labelling usually does not, because deciding whether an agent took the right action requires understanding the business logic the agent is embedded in — context an external workforce rarely has. Many teams land on a hybrid: outsource the high-volume, stable-schema training data, keep the evaluation and tool-call verification in-house where the judgement lives. The scoping mistake to avoid is assuming the outsourced model that worked for training will extend cleanly to evaluation. It usually will not, and discovering that after signing a fixed-cost contract is how the budget breaks. How do annotation requirements change once a use case becomes an orchestrated agent? This is the question the naive framing never asks, and it is the one that separates a project that ships from one that stalls. A single generative call needs training data and, ideally, an evaluation set. An orchestrated agent — one that uses a model as a tool, chains steps, and acts in the world — needs all of that plus annotated tool-call traces, failure-case labels, and a schema for what “correct behaviour” means at each step of the loop. Whether a use case is a generation problem or an orchestration problem is a feasibility question, not an implementation detail. It determines what must be labelled and how: training data versus evaluation and tool-call verification data. That determination is exactly where annotation-readiness enters our feasibility assessment — because a team that scopes only training annotation for what is actually an orchestration problem has under-budgeted the labelling that governs whether the system can be debugged at all. FAQ How does data annotation solutions work in practice? Annotation attaches structured meaning to raw data — a box on an image, a label on text, a correctness flag on an agent’s tool call — so a model has a target to learn from or be measured against. In practice it is three things running together: a schema defining what each label means, a workforce applying it, and a review loop measuring consistency. The schema, not the labels themselves, is the real deliverable. What types of data annotation exist, and which does my use case need? The main types are image/video (boxes, masks, keypoints), text (spans, entities, sentiment), audio (transcripts, speaker turns), and tool-call traces (correct/incorrect agent actions). Your modality and the model’s job decide which you need. Most teams scope the first three and miss tool-call annotation entirely, which is the one that governs whether an agentic system can be debugged. How does annotation for training a generative model differ from annotation for evaluating an agentic system? Training annotation is front-loaded: you label a corpus once and the marginal cost per label falls as the dataset grows. Evaluation and tool-call annotation is ongoing by nature, because the failures you most need labelled are the ones the deployed system produces. Teams that scope annotation as a fixed training cost systematically undercount this second, continuous bucket. What quality measures keep annotation from drifting? Three mechanisms, all defined before labelling starts: a schema that resolves edge cases, inter-annotator agreement thresholds (e.g. a kappa floor) to catch inconsistency as a number, and review gates that feed disagreements back into schema clarification. Defining these up front is what avoids the 20–40% re-labelling cost that follows an undefined schema. When should annotation be outsourced versus kept in-house? The deciding factor is how much domain context the annotator needs. Stable-schema, high-volume, non-sensitive training data outsources well; evaluation and tool-call verification — which require understanding the agent’s business logic — usually stays in-house. Many teams run a hybrid and go wrong only when they assume the outsourced training model extends cleanly to evaluation. How do annotation requirements change once a use case moves from a single generative call to an orchestrated agent? A single call needs training data and an evaluation set. An orchestrated agent needs all of that plus annotated tool-call traces, failure-case labels, and a schema for correct behaviour at each step of the loop. Deciding whether a use case is a generation or an orchestration problem is a feasibility question that determines the entire annotation scope. Where this leaves the scoping conversation The practical test is simple to state and easy to skip: before you budget annotation, decide whether the use case is a generation problem or an orchestration problem. If a model is called once and its output is the product, front-loaded training annotation with a defined schema and agreement thresholds may be most of what you need. If a model is a tool inside a loop that acts, then the labelling that governs whether you can ever debug it — the tool-call traces, the failure-case triage — is ongoing, and scoping it as a fixed cost is the miss that surfaces in production. The question worth carrying into every feasibility review is not “how much will labelling cost?” but “what would we have to label to know when this system is wrong — and does that ever stop?”
