Introduction

Real-time inference for live-streaming pipelines — whether the workload is text-to-speech for closed captions, computer vision for content moderation, or object detection for overlay graphics — runs on heterogeneous client hardware (iOS, Android, web, set-top, desktop) with a single shared engineering budget: jitter-bounded latency, low dropout rate, and quality measurable end-to-end. The model that holds up in a TensorFlow notebook does not necessarily ship across ONNX Runtime and CoreML at user-acceptable quality; the conversion path silently degrades audio quality, drops framerate under load, and varies per device. See telecommunications for the broader landing this article serves.

The teams that ship real-time cross-platform inference have done the model-compression and per-runtime QA work upfront; the teams that hope to “just convert and deploy” ship pipelines that test in the lab and fail in the field.

What this means in practice

• Cross-platform real-time inference is an engineering discipline, not a one-step model export.
• Latency budgets per platform differ and bound the realistic streaming workload.
• ONNX-to-CoreML conversion has documented quality and performance degradations to manage.
• Multi-runtime QA requires a strategy that does not re-validate each platform from scratch.

How do I deliver real-time TTS inference cross-platform on ONNX and CoreML?

The cross-platform real-time TTS pipeline. Source model: train or fine-tune on the training framework of choice (PyTorch, TensorFlow); the trained model is the source of truth. Export to ONNX: the model is exported to ONNX as the cross-platform intermediate; ONNX Runtime supports Windows, Linux, Android, and web (via wasm). Convert to CoreML for iOS: ONNX is converted to CoreML using coremltools for native iOS/macOS deployment. Per-platform validation: each runtime is tested with the same input set and the outputs compared against the source-model output; deviations beyond tolerance are flagged.

Engineering patterns that work. Trace before training-graph export — quantising and exporting from the same checkpoint avoids subtle differences. Use the supported operator set; ONNX and CoreML each have evolving operator coverage and using cutting-edge ops causes conversion failures or quality drift. Streaming inference uses chunked input and overlap-add for audio; do not export a model that requires a full utterance in one pass if the use case is streaming.

Patterns that fail. Treating model conversion as a one-time export rather than a continuous validation against the source model. Allowing per-platform divergence to accumulate over model updates without re-validation. Choosing model architectures that look great in research but have poor cross-runtime support (custom CUDA kernels, exotic operations). The cross-platform real-time pipeline succeeds when conversion and validation are part of the model release cycle, not afterthoughts.

What latency budget can I realistically hit for streaming TTS on iOS, Android, and Web?

Latency budgets observed in mid-2026 production deployments.

iOS (CoreML on Neural Engine). For a small-to-medium TTS model (say 30-80M parameters), first-frame audio latency from text input to first audio chunk is 50-150ms on recent A-series and M-series hardware. Streaming latency (input chunk to output chunk) is 30-80ms. Older devices add 50-100% latency.

Android (ONNX Runtime, NNAPI or CPU). First-frame latency is 80-200ms on flagship devices with NNAPI; 150-400ms on mid-range with CPU fallback. Streaming latency is similarly 50-150ms flagship, 100-300ms mid-range. The Android fragmentation means worst-case devices serve unacceptable latency for streaming use without aggressive model size reduction.

Web (ONNX Runtime web, WebGPU or wasm). First-frame latency is 200-500ms on modern desktops with WebGPU; 500-1500ms on lower-end devices with wasm fallback. Streaming latency is 100-300ms with WebGPU, 200-500ms with wasm. Web is the most latency-constrained runtime and forces the smallest model.

The implication. The cross-platform model size and quality budget is set by the worst-supported platform, not the best. If the use case requires Web support, the model must fit Web latency, which means smaller than what iOS Neural Engine can run comfortably. This trade-off is the first design decision in a cross-platform real-time pipeline.

Where does ONNX-to-CoreML conversion silently degrade audio quality or performance?

Documented degradation patterns. Quantisation precision: CoreML’s preferred path is INT8 or FP16; FP32 ONNX models convert but lose the performance advantage. INT8 quantisation can introduce audible artefacts in TTS models unless calibration is done carefully with representative audio. Operator coverage: certain ONNX operators have no direct CoreML equivalent; the converter substitutes approximations or splits the operation into multiple steps, increasing latency or changing numerical behaviour. Custom or recently-added operators have the weakest coverage.

Sequence length handling: streaming TTS models often use stateful or recurrent structures; CoreML’s support for these has improved but remains less straightforward than ONNX Runtime’s. Naive conversion of a stateful model can produce non-streaming inference or quality artefacts at chunk boundaries.

Random number generation: TTS models with stochastic components (variational, attention dropout retained at inference for variety) have different RNG behaviour across runtimes; the same seed produces different outputs. Production deployments typically accept this if quality is acceptable rather than try to align RNG across runtimes.

Performance under load: a model that benchmarks well in isolation can degrade when running alongside other ML workloads on the Neural Engine (multi-tenancy contention). The CoreML scheduler does not guarantee real-time priority for one model over another. The mitigation. Validate audio quality with subjective listening tests on each runtime, not just numerical comparison to source. Validate performance on a target device under realistic concurrent workload, not on an idle benchmark device. Build a regression test that catches quality drift between model versions on each runtime.

Which model-compression strategy (distillation vs quantisation) keeps TTS quality acceptable across runtimes?

Distillation. Train a smaller student model to mimic a larger teacher model’s outputs. The student is then exported and runs on the target runtime. Distillation preserves quality better than quantisation when the size reduction is large (4× or more), because the student can develop its own representation rather than being constrained to approximate the teacher in the same parameterisation. The cost is training infrastructure and time; distillation is offline upfront work.

Quantisation. Reduce numerical precision from FP32 to FP16, INT8, or INT4. Post-training quantisation is quick; quantisation-aware training preserves quality better at lower bit widths. Quantisation works well for moderate size reductions (2-4×) and is the standard for shipping a single trained model to multiple precision targets. The cost is per-platform calibration and validation; uncalibrated quantisation can break audio quality.

The practical combination. Most production cross-platform TTS uses both. Distillation produces the cross-platform model size (small enough to fit Web latency). Quantisation produces the per-runtime variant (FP16 for iOS Neural Engine, INT8 for some Android NNAPI paths, FP32 or FP16 for Web wasm/WebGPU). The combination preserves quality better than either alone at the target size.

Quality measurement. MOS (Mean Opinion Score) from listening tests is the gold standard but expensive. PESQ and STOI as objective audio quality metrics correlate with MOS but imperfectly. The production discipline is to maintain a MOS-validated baseline per runtime variant and re-validate at each model update; numerical comparison to source alone is insufficient.

How do I QA a TTS pipeline across multiple runtimes without re-validating per platform from scratch?

The QA strategy. Source-model baseline: produce a reference output set on the source model (PyTorch or TensorFlow); store outputs and ground truth. Per-runtime conversion test: convert the source model to each runtime; produce outputs for the same input set; compare against the source baseline. Cross-runtime consistency test: outputs across runtimes are compared to each other; large divergence flags a conversion or platform-specific issue. Per-device performance test: run on representative devices per platform; measure latency, throughput, dropout under realistic conditions.

The QA scaling pattern. Automate everything: model conversion, output comparison, performance measurement run on every model update through a CI pipeline. Treat each runtime as a continuous integration target, not a manual validation step. Subjective listening tests: human listening for audio quality remains essential but is concentrated on the source-model output and the worst-runtime output; intermediate runtimes are validated through automation against these endpoints.

The pattern that fails. Manual per-runtime validation each release — does not scale, becomes the bottleneck, gets skipped. Validation only against numerical metrics — misses audible quality issues. Validation on benchmark devices only — misses real-world performance under load. Validation only on the model team’s preferred runtime — misses platform-specific degradation. The cross-platform TTS pipeline that ships and scales has CI-integrated multi-runtime QA with periodic subjective validation; the pipeline that ships once and stalls has manual validation that does not survive model updates.

What does “production-ready” mean for cross-platform TTS — measurable in jitter, dropout, and MOS?

Production readiness defined measurably. Jitter: variation in inference latency across requests. For streaming TTS, jitter should be small relative to the audio chunk duration — if chunks are 100ms, jitter beyond 30ms causes audible artefacts or playback dropouts. Production deployments target jitter <20% of chunk duration.

Dropout: failed inferences or missed chunks during streaming. Production targets <0.1% dropout rate for premium-quality streaming; <1% for adequate-quality. Higher dropout rates produce noticeable audio interruptions.

MOS: subjective audio quality on a 1-5 scale. Production-ready TTS typically requires MOS >3.5 across all supported runtimes and devices; flagship platforms typically achieve 4.0+ for the source model and 3.7+ after compression. MOS below 3.5 produces audio that listeners notice as artificial or low quality.

Cross-platform consistency: MOS variation across platforms should be bounded — a model that scores 4.0 on iOS and 3.0 on Web is not a single production model, it is two products. Production targets consistency within 0.5 MOS across runtimes.

Operational readiness. Monitoring: per-runtime jitter, dropout, MOS sampled in production; alerts on degradation. Rollback: ability to revert to previous model version per runtime without affecting others. Versioning: model version explicit in inference outputs for traceability and debugging.

The measurable definition. Production-ready cross-platform TTS has: jitter and dropout within target across all runtimes; MOS within target across all runtimes; monitoring and rollback infrastructure; CI-integrated QA across runtimes; documented model conversion and compression process. Definitions that omit any of these measurements describe a prototype, not a production system.

Limitations that remained

Cross-runtime quality consistency remains an active engineering challenge — perfect cross-runtime equivalence is not achievable in practice, and production deployments accept bounded variation. Web latency budgets remain tight and constrain model size in ways that limit quality for that platform. CoreML’s operator coverage is improving but lags behind PyTorch’s; some model architectures are not yet directly convertible. Android device fragmentation means the worst-case device experience is significantly worse than the median, and production deployments must decide where to draw the device-support line. Monitoring infrastructure for ML inference quality is less mature than for traditional service monitoring; the operational tooling is catching up. These constraints shape what scales and what does not; they do not change the engineering pattern that distinguishes a shipped cross-platform real-time pipeline from a single-platform demo.

How TechnoLynx Can Help

TechnoLynx works on cross-platform real-time ML inference — the model-compression and distillation work, per-runtime conversion and validation, CI-integrated QA, and the production operational infrastructure (monitoring, rollback, versioning) that distinguishes a streaming-grade deployment from a prototype. If your team is shipping real-time AI inference across iOS, Android, and Web and wants the engineering discipline that holds up under load, contact us.

Image credits: Freepik