A distilled text-to-speech model ships to two runtimes. On CoreML the mean opinion score holds; on ONNX the same input produces a slightly rougher waveform. The confidence numbers look identical. So which is it — a broken export, or just noisy audio the model was never going to handle well? That question has a precise answer, and getting it wrong is expensive. The naive move is to treat “the model is uncertain” as a single scalar and react the same way every time confidence drops: gather more data, retrain, redistill. Sometimes that works. Often it burns weeks of validation cycles chasing a problem that no amount of training data can fix. The reason is that uncertainty is not one thing. It splits into two kinds — aleatoric and epistemic — and they demand opposite remediation. What is the difference between aleatoric and epistemic uncertainty? Aleatoric uncertainty is irreducible noise in the data itself. A clipped or noisy audio segment feeding an ASR or TTS frontend carries information the model cannot recover, no matter how well trained it is. If two different ground-truth outputs are consistent with the same input, the model is correctly uncertain — and more data will not shrink that uncertainty, because the ambiguity lives in the input, not in the weights. Epistemic uncertainty is the opposite. It reflects gaps in what the model has learned: regions of the input space that were underrepresented in training, or coverage that a distilled or quantised model has genuinely lost. Epistemic uncertainty shrinks as you add data or capacity. When a model is unsure because it has never seen inputs like this one, that is a fixable deficit. The distinction is old — statisticians have separated “risk” from “ignorance” for a century — but it becomes operationally sharp the moment you deploy the same model across runtimes. Here is the claim that matters in production: a low-confidence output tells you nothing actionable until you know which kind of uncertainty produced it. The remediation for one is the wrong move for the other. Why conflating the two wastes retraining cycles Consider the divergence point directly. Suppose your monitoring flags a batch of low-confidence TTS outputs after a distillation pass. Two hypotheses fit the same symptom: Epistemic: the distilled student lost coverage on a phoneme distribution the teacher handled. Fix: more capacity, targeted data, or a less aggressive compression ratio. Aleatoric: the batch happened to contain clipped, low-SNR input audio. Fix: input conditioning, denoising at the boundary, or a fallback path — the model is behaving correctly. If you assume epistemic and retrain, and the real cause was aleatoric, you spend a training cycle and the metric does not move — because you cannot train away noise that lives in the input. If you assume aleatoric and dismiss the drop as “just noise,” and the real cause was epistemic, you ship a model that has quietly lost coverage after compression. Both errors are common. We see teams default to retraining because it feels like doing something, when the diagnostic step upstream would have redirected the effort. This is the same reason a raw confidence score in computer vision needs interpretation before you act on it: the number conflates two mechanisms that call for different responses. How do you tell noisy input from a genuine model gap? You cannot read the answer off a single confidence value. The two uncertainties have to be separated at estimation time, and the estimators differ. Epistemic uncertainty is usually surfaced by disagreement across an ensemble or across sampled sub-models — Monte Carlo dropout, deep ensembles, or a small ensemble of distilled variants. When the members agree, the model is confident about what it knows; when they diverge, that variance is epistemic. Aleatoric uncertainty is instead a property of the input, so it is estimated by having the model predict its own output variance (heteroscedastic regression) or by probing the input’s signal quality directly — SNR, clipping detection, and similar boundary checks on the audio frontend. The practical tell: aleatoric uncertainty is high for a specific input and stays high no matter which model variant you run; epistemic uncertainty varies across model variants for the same input. That asymmetry is the cleanest diagnostic you have without heavy retraining. Run the input through two or three checkpoints or export paths and watch whether the disagreement travels with the input or with the model. A diagnostic rubric for a low-confidence output Observation Likely class First remediation What NOT to do Low confidence, all model variants agree it’s low Aleatoric Input conditioning / denoise / fallback path Retrain or add capacity Low confidence, variants disagree with each other Epistemic Add targeted data or capacity; relax compression Dismiss as “just noise” Confidence gap between CoreML and ONNX, same input Export-path artifact (epistemic-adjacent) Check quantisation, op fidelity, tensor layout Assume the model itself regressed Uniformly low confidence across a whole class of inputs Epistemic (coverage hole) Data audit for that class Patch outputs with heuristics Low confidence correlates with input SNR / clipping Aleatoric Boundary handling at the frontend Chase it in training The rubric is a starting hypothesis, not a verdict — the estimator step above confirms it. But it stops the reflexive retrain. When quality drops after distillation, which uncertainty is it? This is where the split earns its keep. Distillation and quantisation are lossy by design: you trade capacity for size and latency. When a distilled or FP4-quantised model shows a quality regression, the question is whether the loss is a real capacity deficit (epistemic) or whether the harder inputs were always ambiguous (aleatoric) and the smaller model simply surfaces that ambiguity more visibly. Attributing the drop tells you what lever to pull. If epistemic uncertainty rose on the distilled student where the teacher was confident, compression cut into coverage — you need more student capacity, a gentler distillation objective, or targeted data on the affected distribution. This is squarely in the territory of what FP4 and other low-precision formats mean for model quality: precision choices move the epistemic budget, not the aleatoric floor. If instead aleatoric estimates dominate and the epistemic gap between teacher and student is small, the model is fine; redirect the effort to the input boundary. For teams doing this kind of inference optimization work, uncertainty attribution is a supporting diagnostic that keeps the optimisation honest — it tells you whether an observed regression after distillation is a signal to change the model or a signal to leave it alone. We treat it as a standard check before signing off a compression pass in cross-platform inference and porting engagements for media and telecom. Per-platform quality gaps: inherent or an export artifact? Now return to the opening. CoreML holds MOS; ONNX drops. The confidence scores match, so a scalar view says nothing is wrong. Uncertainty attribution says otherwise. If the epistemic estimate is stable across both runtimes and only the aleatoric-linked outputs differ, the runtimes are handling identical model logic and the gap is input-driven — a noise-handling difference at the boundary, not a model deficit. If the epistemic disagreement itself changes between CoreML and ONNX for the same input, the export path altered what the model computes: a quantisation difference, an op that lowered differently, or a tensor layout mismatch. That is not the model regressing — it is the export diverging, and it is diagnosed at the compiler-flag and runtime level, where CoreML and ONNX inference actually differ, not by retraining. This is the operationally relevant framing for multi-runtime deployment: a per-platform quality difference is either inherent (aleatoric, same everywhere) or an artifact of the export path (epistemic divergence introduced by compilation) — and only the second one is a bug you can fix in the pipeline. The measurable payoff is avoided validation cycles and stable per-platform quality bounds, because you stop re-running the whole validation matrix every time one runtime looks slightly worse. Estimating uncertainty without heavy retraining The objection is fair: proper Bayesian uncertainty is expensive, and production teams rarely have the budget to retrain with heteroscedastic heads or maintain large ensembles. The good news is that the diagnostic does not require the gold-standard estimator — it requires enough separation to make a decision. For epistemic signal: run the input through the two or three model checkpoints or export targets you already ship. You are not building an ensemble for accuracy; you are using variant disagreement as a cheap epistemic probe. This is essentially free if you already maintain multiple export paths. For aleatoric signal: measure input quality directly at the boundary — SNR, clipping, silence ratio for audio; blur, exposure, occlusion for vision. These are classical signal checks, not learned components, and they run in microseconds. For confirmation: correlate. If low confidence tracks input-quality metrics and travels with the input across variants, it is aleatoric. If it tracks variant identity, it is epistemic. None of this is a benchmarked accuracy figure — it is a triage procedure we have found reliable across cross-platform inference engagements (observed pattern, not a published measurement). The point is not to quantify uncertainty perfectly. It is to route the remediation correctly, which is a much lower bar and where most of the wasted effort actually hides. FAQ How does aleatoric uncertainty vs epistemic uncertainty actually work? Aleatoric uncertainty is irreducible noise in the input — a clipped or noisy audio segment that no model can fully recover — and adding data will not reduce it. Epistemic uncertainty reflects gaps in what the model has learned and shrinks as you add data or capacity. In practice the distinction decides your remediation: aleatoric means handle the noise at the input or output boundary, epistemic means the model needs more data or capacity. How do I tell whether a low-confidence model output reflects noisy input (aleatoric) or a genuine model gap (epistemic)? Run the same input through two or three model variants or export paths and watch where the disagreement lives. If confidence stays low across all variants and correlates with input-quality metrics like SNR or clipping, it is aleatoric. If the variants disagree with each other on the same input, that variance is epistemic — a model gap. When quality drops after distilling or quantising a model, is that epistemic uncertainty from lost capacity or aleatoric noise in the data? Compare the epistemic estimate between teacher and student. If epistemic uncertainty rose on the distilled student where the teacher was confident, compression cut into coverage — a capacity deficit needing more student capacity or targeted data. If the epistemic gap is small and aleatoric estimates dominate, the harder inputs were always ambiguous and the smaller model merely surfaces that; the fix belongs at the input boundary. How do I estimate each type of uncertainty in a production inference pipeline without heavy retraining? Use variant disagreement across the checkpoints or export targets you already ship as a cheap epistemic probe, and measure input quality directly (SNR, clipping, blur, occlusion) as the aleatoric probe. Then correlate: low confidence that tracks input-quality metrics and travels with the input is aleatoric; low confidence that tracks model identity is epistemic. This is a triage procedure, not a gold-standard Bayesian estimate. Which uncertainty type justifies adding data or model capacity, and which one should redirect effort to input conditioning and fallbacks? Epistemic uncertainty justifies added data or capacity, because it reflects a genuine coverage gap the model can learn to close. Aleatoric uncertainty should redirect effort to input conditioning, denoising, and fallback handling, because the ambiguity lives in the data and cannot be trained away. How does uncertainty attribution help explain per-platform quality differences between CoreML and ONNX runtimes? If the epistemic estimate is stable across both runtimes and only aleatoric-linked outputs differ, the gap is input-driven and the runtimes compute the same logic. If the epistemic disagreement itself changes between CoreML and ONNX for the same input, the export path altered what the model computes — a quantisation, op-lowering, or tensor-layout artifact diagnosed at the compiler level, not by retraining. What are the practical failure modes of treating all model uncertainty as a single confidence score? You react to every low-confidence output the same way, so you retrain to fix input noise that no data can remove, or you dismiss a real capacity gap after compression as “just noise.” Both errors are expensive: the first burns validation cycles with no metric movement, the second ships a model that has quietly lost coverage. A single scalar hides the one thing you need — which mechanism produced the uncertainty. The harder question sitting underneath all of this is where the aleatoric floor actually is for a given input distribution, because that floor is the quality ceiling no amount of capacity will beat. When you next see a per-platform quality gap after a distillation or quantisation pass, resist the reflex to retrain — separate the two uncertainties first, and let the attribution decide whether the fix belongs in the model or at the boundary.
