Modern text to speech is generative AI. Neural systems such as WaveNet, the adversarial EATS model and diffusion-based synthesisers are generative models that produce new waveforms. Older concatenative systems did something else: they cut recorded speech into units and joined the units back together. So is TTS generative AI? Yes for neural synthesis, no for unit selection, which reuses recorded speech.
The label matters because of what comes with it. The first generative voice models were too slow for live use, and once speech has to start inside a conversational turn, latency, GPU serving cost and inference energy decide whether a voice product is viable.
Is text to speech generative AI or a different kind of AI?
Text to speech is a generative task. The model maps a short sequence of characters or phonemes to a long, real-valued audio signal that is not aligned with the input. A survey of audio diffusion models describes speech synthesis as one direction of generative AI, names text to speech and speech enhancement as its two active tasks, and calls diffusion the most popular generative model.
Generative AI covers more than large language models. It includes GANs, diffusion models, autoregressive models, VAEs and normalising flows. WaveNet is autoregressive, WaveGlow is flow-based, EATS is adversarial, and NaturalSpeech 2 is a latent diffusion system. The key feature of generative models here is that the output is sampled, not retrieved. WaveGlow, for example, takes samples from a Gaussian and transforms them into audio conditioned on a mel-spectrogram.
The boundary is blurrier than “neural versus everything else”. The WaveNet paper describes statistical parametric synthesis as using a generative model as well. HMM-based parametric systems generated speech parameters from probability densities and then rebuilt a waveform from those parameters. We read parametric synthesis as a narrower, earlier kind of statistical generation. With neural TTS, deep networks generate the waveform, or a learned latent representation of it, directly. Unit selection, by contrast, builds the utterance from recorded speech instead of a generative model. Its canonical paper, Hunt and Black at ICASSP 1996, describes unit selection in a concatenative system using a large speech database.
How is neural text-to-speech different from older concatenative synthesis?
Speech synthesis went through three broad stages: early formant and concatenative synthesis, then statistical parametric synthesis, then neural networks. The comparison below shows how each one produces speech and how listeners have judged it.
Three generations of speech synthesis compared
| Approach | How the audio is produced | How listeners judged it | Generative? |
|---|---|---|---|
| Concatenative (unit selection) | Recorded utterances are chunked into segments and recombined by unit selection | Prosodic peculiarities and missing coarticulation between words; tends toward hyper-articulated “clear” speech | No: retrieves recorded audio |
| Statistical parametric (HMM, later DNN) | HMMs, and later DNNs, model speech parameters given the text; a waveform is reconstructed from the generated parameters | Rated less natural than WaveNet in English and Mandarin | Partly: a generative model of parameters |
| Neural (acoustic model plus vocoder, or end-to-end) | A network predicts a mel-spectrogram and a neural vocoder synthesises the waveform, or a single model maps text straight to waveform | Rated more human-like, natural, likeable and familiar than concatenative TTS | Yes: samples new waveforms |
In the WaveNet paper’s listening tests, the gap between the best synthetic speech and natural speech shrank from 0.69 to 0.34 MOS points in US English and from 0.42 to 0.13 in Mandarin Chinese. The HiFi-GAN paper notes that most neural speech synthesis models use two stages. Tacotron 2, for example, pairs a sequence-to-sequence spectrogram predictor with a modified WaveNet vocoder. End-to-end models such as EATS drop the intermediate representation and learn the full mapping from text to waveform in one feed-forward network.
There is a catch that naturalness scores hide. A perception study found neural TTS less intelligible than concatenative TTS when listeners heard speech in noise, even though they rated it more natural. The authors point out that the more realistic method produced less intelligible speech in adverse listening conditions. For a voice that will play over a phone line or in a noisy public space, as in AI engineering for media, broadcast and telecom, we would test intelligibility in noise directly and not rely on MOS alone. If you want the plain-language version of what the neural approach brings, see how generative AI benefits text-to-speech.
Are AI voices and AI voice-overs generative AI?
An AI voice or voice-over made by a neural TTS model is generative output, and a sampling-based model like VALL-E produces different audio for the same text. A voice-over recorded by a human is not generative AI. Neither is an older prompt system that plays back recorded segments.
VALL-E treats TTS as conditional language modelling over neural codec tokens and synthesises speech for speakers it has never seen, prompted by an enrolled recording. It was trained on 60,000 hours of English speech from more than 7,000 speakers, which its authors put at hundreds of times more data than earlier TTS systems used. The model carries the emotion of the prompt into its output without emotional fine-tuning, and reproduces the prompt’s reverberation as well. NaturalSpeech 2 reaches zero-shot voices differently: a speech-prompting mechanism conditions a latent diffusion model.
The choice of voice also affects synthesis speed. ElevenLabs’ documentation says its Professional Voice Clones carry extra model complexity and synthesise more slowly than default voices or Instant Voice Clones.
Why the first neural voices were too slow for live use
WaveNet is fully autoregressive: each audio sample is conditioned on every sample before it. The Parallel WaveNet paper says this sample-by-sample generation is poorly suited to massively parallel hardware and hard to deploy in real-time production. The WaveGlow paper puts a number on the gap. The WaveNet implementation it compared against synthesised at 0.11 kHz, while unoptimised WaveGlow reached about 520 kHz on an NVIDIA V100.
Fast vocoders got past the serial bottleneck in different ways:
- Distillation: Parallel WaveNet trains a feed-forward network from a trained WaveNet and runs more than 20 times faster than real time. It was deployed in Google Assistant.
- Flows: WaveGlow avoids autoregression altogether and runs more than 25 times faster than real time on a V100.
- GANs: HiFi-GAN generates 22.05 kHz audio 167.9 times faster than real time on a single V100, and its small-footprint version reaches 13.4 times real time on a CPU. EATS reports 206.8× real time on a V100 and 8.52× on a CPU.
- Fewer diffusion steps: BDDM produces samples in seven steps, 143 times faster than WaveGrad, and DiffGAN-TTS produces high-quality audio in one step. NaturalSpeech 2 settled on 150 Euler solver steps to balance quality against latency.
Autoregression has since come back through codec language models. VALL-E and the “Talker” stage of Qwen3-TTS both generate acoustic tokens one after another, the way an LLM generates text. That brings back a robustness cost: on a set of hard sentences, the NaturalSpeech 2 authors found that autoregressive models such as Tacotron, Transformer TTS and VALL-E had high error rates, while non-autoregressive models had no intelligibility problems.
Read the headline speed-ups with care. The EATS figures come from batches of 30-second utterances, and WaveGlow’s speed per sample falls for shorter utterances because the number of serial steps stays the same. A real-time factor measured on long batched clips tells you little about how fast the first audio arrives for one short sentence.
What changes in latency and GPU serving cost when real-time text to speech is required?
Two numbers become binding. One is time to first audio for each listener. The other is how many concurrent streams a GPU can keep faster than real time, because that number sets the cost per character. In a conversation, speech has to start within roughly 300 ms, a rule of thumb for real-time UX. Our piece on real-time generative AI streaming and latency budgets covers the streaming, cancellation and budget side of that problem.
Real-time factor is quoted in two opposite directions
Real-time factor has two opposite conventions, and results quoted in one are easily misread in the other. In one, lower is faster: an RTF of 1.0 means one second of compute per second of audio. NVIDIA’s Riva documentation uses the inverse, RTFX, defined as audio duration divided by computation time. The two are reciprocals, so Thaki Cloud’s median RTF of 0.100 for VoxCPM2 on a B200 equals 10× in the other convention (1 ÷ 0.100). NVIDIA’s figure of 61.4× on an A100 (7.3 seconds of speech in under 120 ms) is on the higher-is-faster scale.
NVIDIA’s TensorRT work on that pipeline also shows where the time goes. The Tacotron 2 decoder runs about 86 times per second of output audio and is by far the most expensive part of that network. Its many microsecond-scale kernels left the GPU idle and the implementation CPU-bound. Custom TensorRT plugins that fused kernels cut CPU launch time by nearly 10× and GPU time by 5× for decoder iterations at batch size one. For reading vendor latency claims more generally, our guide to reading MLPerf and latency numbers for real-time GenAI goes further.
Where time to first audio goes
ElevenLabs’ latency documentation separates model inference from the other stages of time to first audio:
| Stage | Typical contribution, per ElevenLabs’ documentation |
|---|---|
| Network round trip | 20–200 ms over the public internet, depending on distance |
| Server processing | Single-digit milliseconds for authentication, validation and scheduling |
| Model inference | About 75 ms for Flash models on short inputs; higher for long inputs, under load or with complex voices |
| Audio player buffering | A 500 ms buffer is common |
| Upstream pipeline | In a voice agent, speech recognition and the LLM add their own latency before TTS starts |
On those figures, a default player buffer takes longer than the model itself, and the network can too. A team that spends weeks shaving vocoder milliseconds while leaving a half-second buffer in the client has optimised the wrong stage. Streaming does not cut model latency, but it does cut perceived latency, as the streaming guide linked above explains.
Concurrency sets the cost per million characters
Managed TTS APIs usually charge per character, and Baseten points out that the rate bundles GPU serving cost with the concurrency the provider achieves. On dedicated open-source serving, any increase in concurrency per GPU lowers the price per character directly.
For voice products, TechnoLynx’s Inference Cost-Cut Pack targets inference cost and p95 latency on the workload you actually run.
In Baseten’s Qwen3-TTS measurements, 25 concurrent streams cost $3.04 per million characters with p50 and p90 RTF below the real-time threshold. A single stream ran nearly 6× faster than real time but cost more because the GPU sat underused, and at 32 streams p90 RTF went above real time.
The levers in that work are in the serving stack:
- running the autoregressive token stage and the codec decoder as separate, pipelined engines, so one request’s decode overlaps the next request’s token generation
- capturing CUDA graphs sized from profiled tensor shapes to cut kernel-launch overhead
- a dynamic frame-accumulation count: small for the first chunk to keep time to first audio low, larger afterwards so the decoder can batch
Scheduling policy shapes behaviour under load too. Riva’s documentation says the server does not start inference for a new request until a previous one has finished generating, so requests can time out under high load.
Two company blogs report different results for Qwen3-TTS. Thaki Cloud measured Qwen3-TTS-1.7B on a B200 at a median RTF of 1.196, slower than real time and unable to keep up with even one stream. Baseten reports Qwen3-TTS faster than real time at 25 concurrent streams. Neither result settles the other. We read the gap as evidence that the serving stack decides whether a TTS model is real-time capable more than the model card does.
Is AI text-to-speech bad for the environment, and what drives its energy use?
Multi-purpose generative architectures cost far more energy than task-specific systems in the tasks measured, and serving design shapes a deployment’s inference footprint. Luccioni, Jernite and Strubell found that multi-purpose generative architectures cost orders of magnitude more energy per 1,000 inferences than task-specific systems across the tasks they measured, even with parameter count held constant. A scoping review of generative AI’s carbon footprint adds that inference has become as significant as training, or more, in the overall carbon footprint, and that real-time inference, unlike training, cannot be moved to low-carbon hours or regions.
The drivers that show up in the measurements:
- Generation steps. A study of seven diffusion-based text-to-audio models found energy rising linearly with the number of inference steps. Most Pareto-optimal configurations used 10 to 50 steps, and more steps added energy without improving quality.
- Batch size. In the same study, per-sample energy fell most between batch sizes 1 and 2, and gains largely disappeared beyond 5.
- Idle power. On one B200, Thaki Cloud measured 40.1 J per second of audio for Qwen3-TTS-1.7B, with 86.7% of its power idle, against 25.8 J and 43.4% idle for VoxCPM2. If you rent the whole GPU, idle power is still paid for.
- Measurement boundary. End-to-end measurements can give carbon estimates up to 2.4 times higher than GPU-only ones. Hugging Face’s AI Energy Score counts GPU energy only, and its authors found CPU and RAM use about 30% greater than GPU energy use.
To turn energy into emissions, multiply total energy by data-centre PUE and then by the local grid’s carbon intensity, so the same workload has a different footprint depending on where and when it runs. Treat all of these figures as indicative for speech. The controlled study measures text-to-audio diffusion rather than TTS, and the per-voice numbers come from one vendor’s benchmark on one GPU.
We read cost and energy as pointing to the same fix. A voice endpoint that spends most of its power idle is wasting money and electricity alike, and the cure for both is to raise concurrency until tail RTF approaches the real-time line.
Why the serving stack, not the voice, decides real-time viability
The generative question has a clean answer. The engineering questions do not. One model family can be faster or slower than real time depending on how its stages are pipelined, how its RTF is reported and how full its GPU is kept. A team choosing a voice model is really choosing a serving design, and should measure time to first audio, tail RTF at target concurrency and idle power share on that design before committing.
