For data labeling on images, video or 3D point clouds, CVAT is the more natural choice. For text, audio, time series or LLM evaluation, Label Studio fits better, according to a 2026 comparison published on autolabelling.com. The tool still matters less than the workflow you build around it. Northcutt et al. (NeurIPS 2021) estimated an average of at least 3.3% label errors in the test sets of ten widely used computer vision, language and audio datasets. They advise caution about choosing models on test accuracy alone and propose that judging them on correctly labelled test sets may be more useful. So choose the tool by how well it supports ground truth: pre-filling labels, resolving disagreements between annotators, and checking the finished set. Feature lists are a weaker guide.
What should a labelling tool do for a ground-truth dataset?
A labelling tool for ground truth has to produce a small, very accurate answer key. You measure annotators and models against that key, so its errors matter more than errors in bulk training labels. CVAT’s own guide describes a ground-truth dataset as data labelled to the highest achievable accuracy, the reference that model predictions are scored against. The same guide puts typical ground-truth size at 5% to 15% of the total data, labelled by the most experienced annotators, who spend much longer on each frame.
That changes what you need from the software. Drawing speed counts, but four other things count for more:
- Pre-fill: load model predictions or existing metadata so annotators verify and correct instead of starting from a blank image.
- Overlap and adjudication: send the same item to several annotators, measure agreement and settle conflicts.
- Validation: mix in known-good items and score annotators against them.
- Export: hand the result over in a standard export format such as COCO or YOLO.
CVAT vs Label Studio for images, video frames and keypoints
Both tools label images and video. They part ways on data types, on how model assistance works, and above all on which review features are free.
| Need | CVAT | Label Studio |
|---|---|---|
| Data types | Images, video and 3D point clouds | Text, images, audio, video, time series and multimodal data; no point clouds |
| Keypoints and pose | Skeleton templates, set up by hand or loaded from a model; export as CVAT for image or COCO | KeyPointLabels tag, with coordinates stored as a percentage of image size; keypoints must sit under a parent rectangle before export |
| Model pre-fill | Built-in SAM/SAM2, Hugging Face and Roboflow models, and custom models such as YOLO through CVAT AI Agents | Import predictions as JSON, or connect an ML backend for interactive pre-annotations |
| Multi-annotator review | Consensus merging, ground-truth jobs and honeypots; self-hosted CVAT includes the full product for free | Review workflow and ground-truth annotations are Enterprise-only |
| Editions | MIT-licensed Community, hosted Online, and Enterprise | Open-source Community, plus paid Starter Cloud and Enterprise |
| Self-hosting | Docker Compose from the start | Installs with pip, brew or Docker |
For video, CVAT’s track mode interpolates a shape between keyframes. You only correct the frames where the object’s position changes, not every frame. You can also run both tools: the 2026 comparison notes that teams use CVAT for vision datasets and Label Studio for text and evaluation, at the cost of two deployments and two rounds of annotator training.
Which free or open-source data labeling tools are worth considering, and what are their limits?
Both tools are free to self-host, but the free tiers stop in different places. CVAT Community is MIT-licensed. It supports more than 20 import and export formats, including COCO, YOLO and Pascal VOC. The 2026 comparison says self-hosted CVAT gives you the full product, while Label Studio Community holds back role-based access control, SSO, reviewer assignment and QA workflows for Enterprise. In practice, a solo researcher or a small trusted team will not notice the gap. A 20-annotator team with a review stage will run into it quickly.
Label Studio’s documentation confirms the review gap. The annotation review workflow and ground-truth annotations are Enterprise-only. Several people can label the same task in Community, and tasks lock while one person works on them. HumanSignal, the company behind Label Studio, says the open-source edition supports overlapping annotations but gives you no workflow for resolving them, so you export the data and compute agreement yourself. If your ground-truth plan depends on consensus, a small team on Label Studio Community will end up writing that merge-and-adjudicate step as a script.
A worked case: verified text and material tags on supplier photos
Take supplier product photos. Each needs a verified transcription of the printed or embossed text and a packaging-material tag. A reliable ground truth for this has three parts. You pre-fill whatever an existing source can give you, two or more annotators label each item independently, and a written adjudication rule settles disagreements. Illegible text gets its own explicit outcome. Label Studio can combine regions, labels and free-text transcription in one task; its prediction example puts Rectangle, Labels and TextArea results in a single task.
TechnoLynx helps teams in this position set up the CVAT or Label Studio project, the review pipeline, the adjudication rules and the check on existing metadata, as part of its computer vision R&D work. The first step is fixing the label schema: which text gets transcribed, which material classes exist, and what counts as illegible. Before anyone uses the dataset downstream, TechnoLynx validates it for consistency and coverage.
Transcription guidelines need the same care as the tool settings. In a screenshot-transcription study, annotators were told to keep capitalisation and punctuation, to use special marks to separate graphics from text, and to flag any case the instructions did not cover. The same study cross-checked and corrected the transcriptions afterwards. In that study, fixing human error improved accuracy about as much as other technical refinements did.
Can existing metadata or model predictions pre-fill the labels?
Both tools accept pre-filled labels, so annotators verify instead of creating. Label Studio imports predictions in its JSON format under a predictions key. Predictions are read-only, and when the project setting is on, Label Studio copies them into each new annotation for correction. It also has an example for importing OCR pre-annotations from Tesseract output, which suits the text half of this job. For static pre-fill such as supplier metadata, the documentation says running an ML backend may be overkill, and you can import pre-annotated data instead. In CVAT, AI tools auto-label frames with boxes, masks, tags or poses. You first map the model’s labels to the task’s labels, for example “car” to “vehicle”.
Treat supplier metadata as a prior, not as truth. TechnoLynx validates supplier metadata against a manually labelled sample before bootstrapping material tags from it. Map supplier material categories to your own vocabulary before you import. Label Studio makes the comparison easy to trace: when an annotation is created from a prediction, the result IDs stay the same. You can then compare model-generated regions directly with the human-reviewed version. One more risk is annotators anchoring on the pre-filled answer. Your audit sample should therefore include items labelled without pre-fill.
Is majority vote enough for illegible embossed text?
Majority vote is a strong default for embossed text, but not with only two annotators per item or truly unreadable text. A survey of dataset annotation practice (Klie et al., arXiv 2307.08153) found majority voting was the most common adjudication method, at 34% of applicable publications. Ties were broken by extra workers or experts in 8%. The same survey calls majority vote a strong baseline. It recommends expert curation, or majority vote with experts breaking ties, for a high-quality gold standard.
Two annotators cannot outvote each other, so with two, every disagreement goes to an expert. A parent job can merge up to 11 replicas. Consensus merging covers 2D tasks and every annotation type except cuboids. CVAT’s documentation says a 3- or 5-fold consensus on a ground-truth slice of about 3% of the data costs 10-15% of full dataset annotation.
For text nobody can read, give annotators an explicit “illegible” outcome rather than forcing a vote between guesses. The survey also notes that items too ambiguous to agree on can simply be filtered out of the final set. Majority vote is criticised for suppressing minority views on subjective tasks such as hate speech, where a single true answer does not capture the nuances. No tool or process guarantees quality. Annotator training raises agreement, and clear guidelines are essential for consistency.
Keypoint ground truth for outdoor track footage
Say you need a keypoint ground truth on outdoor track footage, as the accuracy baseline for a pose model. The decision comes in three parts: which tool to set up, whether a public dataset can stand in for some of the labelling, and how many frames to label. Each part has its own answer.
Setting up a skeleton template for joint keypoints
CVAT and Label Studio both handle joint keypoints. Pick the one your team already knows. The steps below assume CVAT, with Label Studio equivalents noted:
- Fix the joint schema first. The COCO Keypoints benchmark defines 17 body keypoints, and each prediction carries x, y and a visibility confidence.
- Add visibility rules. Mark each keypoint as visible, occluded or absent, because occlusion is the main difficulty in pose estimation.
- Create the skeleton in CVAT’s configurator, either by hand or “From model” with a human pose estimation model. In Label Studio, use the KeyPointLabels tag.
- Pre-fill points with a detector. In CVAT, go to AI Tools > Detectors and map the model’s points to your skeleton. In Label Studio, the YOLO ML backend fills KeyPointLabels, and each label needs a
model_index. That backend does not train YOLO models, so you retrain outside it. - Export. CVAT exports skeletons as CVAT for image or COCO. In Label Studio, nest each keypoint under its rectangle before export, and remember coordinates are stored as percentages of image size.
Can a public outdoor athletics pose dataset replace labelling from scratch?
Sometimes. Two public datasets cover athletics, and either may replace part of the labelling if its joints and camera views match your footage. AthleticsPose describes itself as the first open 3D human pose dataset of real athletics movements performed outdoors by athletes. AthletePose3D covers 12 sports motions across running, track and field, and figure skating, with about 1.3 million frames and 165,000 postures. It was captured with hardware-synchronised high-speed multi-camera rigs.
Check transfer before you rely on either. If their joints or camera views do not match your footage, they help with pre-training or pre-fill but cannot serve as your accuracy baseline, so label your own footage to cover the gap. Whether you need custom labels at all, or a COCO-pretrained model already covers your classes, is a separate decision to settle first.
How many frames to label for an indoor-outdoor accuracy comparison
No fixed number is defensible. The frame count needed to detect an accuracy drop between indoor and outdoor footage comes from a power calculation. Label and evaluate a pilot set, estimate the size of the drop and its variance, then compute the minimum sample for your chosen confidence and power. Because the pilot estimate is itself uncertain, treat the result as approximate. Datature (2026) suggests 200 to 500 annotated images as a starting point for fine-tuning a COCO-pretrained pose model. That figure is a vendor’s starting point for fine-tuning data.
Correlated frames are the trap. The annotation survey warns that standard sample-size estimates assume independent instances, which often does not hold. Consecutive video frames are about as far from independent as data gets, so sample frames spread across runs and athletes. If a capstone deadline forces a smaller sample than the calculation asks for, report the wider confidence interval rather than hiding it.
Checking the finished ground truth
Datasets considered gold still turn out to contain errors, so estimate the error rate before you publish numbers against it. The annotation survey strongly recommends inspecting a subset of the final dataset and marking each item correct or incorrect to estimate the error rate; it describes sampling that subset at random and recommends reporting a confidence interval. Agreement scores alone are not enough: high agreement shows a reliable process, not correct labels.
CVAT builds this into the job flow. A ground-truth job becomes the reference for every other annotation in the task. Honeypots mix validation frames into regular jobs, so the same frames can check several annotators. The documentation suggests 3 as a starting value for maximum validations per job. Planning this QA across several projects is a larger topic, covered in our piece on ground-truth annotation planning and quality assurance across projects.
TechnoLynx checks the quality and quantity of the data before choosing a model in its computer vision work.
When is a managed labelling platform worth paying for?
A managed platform is worth paying for when keeping a self-hosted tool running costs more engineering time than the subscription. It is also worth it when you need review, permissions or compliance features the free edition lacks. The 2026 comparison says self-hosting trades a vendor invoice for engineering hours. That suits teams with DevOps capacity. Once you add cloud storage, backups, authentication and upgrades, both tools become a standing operations commitment.
The managed options differ. CVAT Online is a hosted edition with Free, Solo and Team tiers. CVAT Enterprise adds SSO/LDAP, audit logs and custom SLAs, on-premises or in a private cloud. HumanSignal says Label Studio Enterprise suits teams with HIPAA or SOC 2 constraints, external annotators, or production-scale volumes. Read that list as a vendor’s sales criteria, not neutral advice. CVAT also sells a fully managed labelling service and says it has more than 300 annotators, which is the vendor’s own claim.
For a Label Studio team, the trigger is the missing review and adjudication workflow. Reviewer assignment and adjudication workflows are Enterprise features, so the choice becomes building them yourself or paying. A CVAT team has no such feature gap, because self-hosted CVAT is the full product. For CVAT teams, the move to a managed tier is mostly about operations, not missing features.
