A team has 500 labelled examples and a rough estimate that a fully-supervised model needs 50,000. Two decisions usually follow, and both skip a question. Either the project is declared infeasible, or an annotation budget is signed off without much scrutiny of what it will buy.
The question that should come first is different: how much signal is already sitting in the unlabelled data? Semi-supervised learning treats a small labelled set as guidance over a much larger unlabelled corpus, rather than as the entire training signal. That reframing is not a trick for squeezing more out of thin data — it changes which use cases are feasible with the data a team actually has, which is why it belongs next to architecture choice rather than after it.
It also has a specific way of going wrong. Push it optimistically and the model’s own confident mistakes become training targets, and accuracy drifts downward while every internal metric looks healthy.
What is semi-supervised learning, and when does it beat full supervision?
Three regimes, cleanly separated:
- Supervised — every training example carries a label. The label set is the signal.
- Unsupervised — no labels. The model learns structure (clusters, densities, embeddings) with no notion of the target you care about.
- Semi-supervised — a small labelled set plus a large unlabelled pool, trained together. The labels tell the model which distinctions matter; the unlabelled data tells it where the data actually lives.
The reason the third regime can win is a structural assumption, not an empirical accident. Semi-supervised methods work when the decision boundary you want falls in a low-density region of the input space — the cluster assumption. If the unlabelled data forms coherent groups that line up with your classes, a handful of labels is enough to name each group, and the unlabelled mass does the work of finding the boundary. When the classes are genuinely clustered and the unlabelled pool is large, semi-supervised training can reach accuracy comparable to a fully-supervised baseline using a fraction of its labelled examples — but only because the unlabelled data is carrying geometric information the labels never had to encode.
Where that assumption fails, it fails hard. If classes overlap in feature space, or the distinction you care about is a fine-grained property rather than a cluster identity (sentiment inside a topic, a defect class inside one product line), the unlabelled data cannot tell you where the boundary sits. It can only tell you where the data is dense — and in that case it will happily place the boundary in the wrong place with high confidence.
What the methods actually do with unlabelled data
“Semi-supervised” covers several mechanisms that behave differently under stress. Three matter in practice.
Self-training / pseudo-labelling. Train on the labelled set, predict on the unlabelled pool, keep the predictions above a confidence threshold as if they were ground truth, retrain. Simple, framework-agnostic, and implementable in a few dozen lines of PyTorch. Also the most fragile: every error the first model made confidently is now a label.
Consistency regularisation. Instead of inventing labels, penalise the model for changing its prediction when the input is perturbed in a way that should not change the answer — augmentation, dropout, adversarial noise. FixMatch and similar methods combine both ideas: pseudo-label a weakly augmented view, then train the strongly augmented view towards that label. The unlabelled data contributes a smoothness constraint rather than a target.
Graph-based label propagation. Build a similarity graph over labelled and unlabelled points and spread label information along edges. Useful when you have a good metric — embeddings from a pretrained encoder, for instance — and a modest dataset that fits in memory.
| Method | What the unlabelled data supplies | Main failure mode | Reasonable first choice when |
|---|---|---|---|
| Pseudo-labelling / self-training | Additional (noisy) training targets | Confirmation bias; errors compound each round | Baseline is already decent; you can hold a clean test set aside |
| Consistency regularisation (e.g. FixMatch) | A smoothness constraint on predictions | Needs meaningful augmentations; weak on tabular data | Images, audio, anything with credible label-preserving augmentation |
| Label propagation | Neighbourhood structure via a similarity graph | Bad distance metric → confidently wrong spread | Strong pretrained embeddings; dataset small enough to graph |
| Fine-tune a pretrained/foundation model | Representations learned elsewhere | Domain gap between pretraining and your data | Your domain resembles public data; labels are the only bottleneck |
The last row is why this topic sits inside a generative-AI discussion rather than beside it. Self-supervised pretraining — masked-token or contrastive objectives run over unlabelled data at scale — is the industrialised version of the same idea, and a pretrained encoder or a general-purpose generative model is often the cheapest semi-supervised system available. In our experience the first thing worth testing is not a clever training loop but a frozen pretrained backbone with a small classifier head on your 500 labels. If that clears the bar, the exotic method was never needed. Choosing between these families is a method decision, and it belongs in the same conversation as the architecture decisions covered in our taxonomy of generative model families and in our generative AI engineering practice.
The preconditions, in the order worth checking
Before spending a sprint on a semi-supervised pipeline, four conditions should hold. They are cheap to check and each one, when violated, predicts a specific failure.
- The unlabelled pool is large relative to the labelled set — a factor of ten or more, otherwise there is little structure to exploit.
- The unlabelled pool comes from the same distribution as deployment data. This is the condition teams most often assume rather than verify. If the labelled set was curated and the unlabelled pool was scraped, they are not the same problem.
- Classes are separable in the representation you are using — check by clustering the embeddings and seeing whether your labelled points land in distinct regions. A ten-minute UMAP or k-means sanity check answers this before any training runs.
- You have, or can afford, a clean fully-labelled held-out test set. Non-negotiable. Without it you cannot detect the failure mode described below.
Condition 4 is where the labelling budget should go first. A common pattern we see is teams spending the whole annotation budget on training data and leaving evaluation to a sample they also trained on — which makes a degrading model indistinguishable from an improving one.
Why pseudo-labelling can quietly make the model worse
The mechanism is unglamorous. The model predicts class A for an ambiguous example with 0.94 confidence. That example enters the next training round as a hard A label. The model becomes more confident about that region, so the next round’s threshold admits more neighbouring ambiguous examples as A. The error does not stay constant; it recruits.
Because the model’s confidence rises throughout, the usual internal signals all point the right way: training loss falls, pseudo-label agreement between rounds rises, the proportion of the unlabelled pool passing the confidence threshold grows. Every dashboard says the run is converging. Accuracy on real ground truth is going the other way.
Practical detection, in rough order of cost:
- Evaluate on the clean held-out set after every self-training round, not once at the end. A non-monotonic curve is the signal.
- Track per-class pseudo-label counts. Runaway growth in one class is confirmation bias becoming visible.
- Compare against the small clean supervised baseline at every round. Semi-supervised training that cannot beat 500 clean labels is not helping, and this comparison is the only honest stopping rule.
- Inspect a random sample of accepted pseudo-labels by hand. Twenty examples is often enough to see the pattern.
Confirmation bias from pseudo-labelling can leave a model less accurate than the smaller, clean supervised baseline it started from, which is why the method is only sound when validated against a held-out, fully-labelled test set. That counter-metric is the whole discipline. Distribution shift in the unlabelled pool behaves the same way — the model learns a boundary that fits the pool rather than the deployment data, and nothing inside the training loop reports it.
Where this lands in a feasibility decision
The useful output of this analysis is not “use semi-supervised learning” or “don’t”. It is a change in what the label estimate means. Instead of asking how many labels a fully-supervised model needs, ask three narrower questions: how many labels do we need for a clean evaluation set, how many for a supervised baseline worth beating, and does the unlabelled pool have exploitable structure. Those three numbers usually add up to far less than the original estimate — and when they don’t, you have learned that early rather than four months into an annotation programme.
The remaining uncertainty is honest: whether the cluster assumption holds in a given feature space is not knowable in advance from domain description alone. It is an empirical question answered by a short experiment, and the experiment is cheaper than the argument about it.
Frequently Asked Questions
What is semi-supervised learning, and when does it beat fully-supervised or fully-unsupervised approaches? It trains on a small labelled set and a large unlabelled pool together, using the labels to say which distinctions matter and the unlabelled data to say where the data lives. It beats full supervision when labels are scarce and expensive but the classes form coherent clusters in feature space. It beats unsupervised learning whenever you care about a specific target rather than generic structure.
How does semi-supervised learning actually use unlabelled data — what do pseudo-labelling, consistency regularisation and self-training do differently? Pseudo-labelling (the core of self-training) converts confident predictions on unlabelled data into training targets, so the unlabelled pool contributes noisy labels. Consistency regularisation never invents a label; it penalises the model for changing its prediction under label-preserving perturbations, so the pool contributes a smoothness constraint. Methods like FixMatch combine the two.
What data conditions have to hold before semi-supervised learning is worth trying, and when do they fail? The unlabelled pool should be roughly ten times the labelled set, drawn from the same distribution as deployment data, and separable in the representation you are using. They fail when the labelled set was curated and the pool was scraped, or when the distinction you care about is a fine-grained property inside a cluster rather than a cluster identity.
How much labelled data is ‘enough’ to start, and how do I decide where the remaining labelling budget goes? Enough means: a clean held-out test set you trust, plus a supervised baseline worth beating. Fund those two first. The remaining budget is best spent on the examples the model is least confident about or that sit closest to the boundary, rather than on more of the easy cases the unlabelled pool already covers.
How does semi-supervised learning relate to self-supervised learning and to pre-trained generative or foundation models? Self-supervised pretraining is the same instinct at industrial scale — learn representations from unlabelled data with a proxy objective, then attach a small supervised head. A frozen pretrained encoder plus a classifier trained on your few hundred labels is usually the cheapest semi-supervised system available, and worth testing before any custom training loop.
What are the main failure modes — confirmation bias, error amplification from bad pseudo-labels, distribution shift in the unlabelled pool — and how do I detect them? All three share a signature: internal metrics improve while true accuracy falls. Detect them by evaluating on a clean labelled test set after every training round, tracking per-class pseudo-label counts for runaway growth, and comparing against the clean supervised baseline continuously rather than once.
How do I evaluate a semi-supervised model honestly when most of my data has no ground truth? You cannot evaluate on pseudo-labels; they encode the model’s own beliefs. Reserve a fully-labelled held-out set that never touches training, size it for the confidence interval you need rather than for convenience, and treat “beats the small clean supervised baseline on that set” as the only acceptance criterion.
When unlabeled data actually improves model accuracy
Consistency regularization and pseudo-labeling deliver measurable gains once your labeled set drops below 10% of available examples. Revisit it when your workload shifts.
