Neural text-to-speech systems get listener ratings close to recorded human speech and clearly beat the concatenative and parametric synthesisers that came before them. They can also learn a new voice from under half an hour of data per speaker. Those gains come with costs. Deepgram estimates neural voices cost 4 to 12 times more per character than standard voices, a gap that matters at volume. Autoregressive models infer slowly, and scammers have used convincing cloned voices while four of six tested products checked consent by self-attestation only. A fair verdict on generative AI’s advantages and disadvantages weighs all three: quality, running cost and misuse.
The benefits of generative AI in speech, measured against older synthesis
Text-to-speech went through three broad stages. First came formant and concatenative synthesis, then statistical parametric speech synthesis, and then neural networks. Parametric systems are compact and flexible, but they often sound muffled and full of artefacts.
WaveNet scored above 4.0 for naturalness (mean opinion score, MOS), significantly better than the parametric and concatenative baselines. Compared with the best earlier synthetic system, it cut the gap to natural speech from 0.69 to 0.34 in US English (51%) and from 0.42 to 0.13 in Mandarin (69%). Tacotron 2 reported a MOS of 4.53 against 4.58 for professionally recorded speech, and a Transformer-based model matched Tacotron 2 at 4.39 against a ground truth of 4.44.
MOS is the most common subjective metric for TTS. It reflects human perception well, but running it at scale is expensive. A single MOS figure is evidence about one dataset and one test set, not a ranking that holds everywhere.
Why is generative AI so good at producing natural-sounding speech?
Generative models learn speech from data rather than from hand-built features or rules. WaveNet builds in almost no prior knowledge about audio beyond its receptive field and signal encoding, and it learns a non-linear filter from data that would be very hard to design by hand. Tacotron replaces the hand-engineered linguistic and acoustic features of a traditional pipeline with a single network trained from data alone, and it produces speech that is difficult to distinguish from real human speech.
The sources point to three mechanisms behind the gain in naturalness:
- Prosody and audio are generated together. Microsoft’s documentation for its neural voices says prosody prediction and voice synthesis happen at the same time, which produces more fluid output than traditional synthesis with its trouble over stress and intonation.
- Long-range context. Self-attention links any two frames directly. The Transformer TTS authors say this matters most for prosody in long sentences.
- The vocoder. The vocoder turns acoustic features into a waveform and is vital to how natural the result sounds.
Newer models add control on top. A large language model can pick up emotional intent in the text and pass it to the TTS model, which then adjusts prosody, pitch and energy to match. Users can also describe the voice they want in plain language. Trained on description-based data, such control suits storytelling, audiobooks and personalised voice assistants. For a closer look at whether this counts as “generative” and what changes once voice runs in real time, see our piece on generative AI in text-to-speech and real-time voice.
How little data does a generative voice need?
Generative TTS needs little data per voice once a model shared across speakers has learned most of its parameters. Deep Voice 2 showed that one neural system could learn hundreds of distinct voices from under half an hour of data per speaker while keeping each speaker’s identity almost perfectly. Most parameters are shared across speakers, so each new voice has few parameters of its own to learn.
At the far end, VALL-E produces personalised speech from a 3-second recording of a speaker it has never heard. It was pre-trained on 60,000 hours of English, which its authors describe as hundreds of times more than existing systems. VALL-E’s 3-second prompt thus rests on a very large pre-training set. The sample-efficient diffusion model SESD reached strong results with a far smaller base. Trained on under 1,000 hours, it reached a word error rate of 2.3% against 2.2% for natural human speech, and in speaker-prompted synthesis it beat VALL-E’s 5.9%. It did this with under 2% of VALL-E’s training data.
Microsoft’s custom neural voice needs at least 300 lines (about 30 minutes of speech) and recommends 2,000 lines (two to three hours) for a production voice. Its personal voice feature needs as little as 10 seconds. One constraint is easy to forget: the recording quality of the voice talent sets the upper limit on the quality of the custom voice, and the model copies noise and mispronunciations along with everything else.
What are the main benefits of generative AI for products and services?
Outside speech, one measured benefit is productivity in customer support work. One study followed customer support agents using an AI assistant. The agents resolved 13.8% more issues per hour, and customer satisfaction did not change significantly. The gains were largest for less experienced and lower-skilled agents, at 35%. An agent with two months’ tenure who used the tool performed as well as an unassisted agent with six months’ tenure.
In speech products, the key benefits of generative AI are the ones measured above: near-human voices, new voices from little data, and expressive control. Consumer Reports lists faster audio editing, better film dubbing and automated narration as legitimate uses of voice cloning. These benefits are capabilities, not guaranteed gains. Whether a product gains from it depends on the running costs in the next section.
What drives the inference cost and latency of generative speech?
Inference cost and latency depend on how many sequential steps the model takes, and on the precision and hardware it runs on.
Latency: sequential steps are the main cost
Conversation sets a latency ceiling. A Deepgram production guide, citing Twilio’s latency documentation, says communication breaks down at around 250 to 300 ms, and that time to first byte for conversational agents should ideally sit near 100 ms. Autoregressive models struggle with that ceiling because each frame depends on the frames before it, which forces inference to run in sequence. Some LLM-based controllable TTS models take from several seconds to tens of seconds to produce a 5 to 10 second clip, and they do not support streaming.
Non-autoregressive designs generate frames in parallel. FastSpeech matched autoregressive quality and sped up mel-spectrogram generation by 270x and end-to-end synthesis by 38x compared with autoregressive Transformer TTS. Deepgram puts non-autoregressive MOS at 3.83 to 4.03, against 4.2 to 4.5 for autoregressive systems, and argues the remaining gap hardly matters in conversation. Paired with efficient vocoders, these designs now support conversational latency under 200 ms. Diffusion models face a similar speed trade-off. Their iterative sampling can be too heavy for real time, and the number of refinement steps trades fidelity against speed. Step reduction work such as BDDM (seven steps) and DiffGAN-TTS (one step) targets exactly that cost. When the latency target is tied to streaming output, the design questions overlap with streaming generative AI applications more broadly.
Spend: per-character price, precision and hardware
Deepgram estimates that neural TTS costs 4 to 12 times more per character than standard voices, which works out to between $4,000 and $48,000 per billion characters depending on provider, volume and tier. Once usage grows, inference cost rather than training cost becomes the main economic factor, especially for latency-sensitive and on-prem deployments.
The hardware and the precision you run at change that cost. The authors of Lightning V2 sized hardware to sustain 550 audio-seconds per second. They needed 11 NVIDIA L40S GPUs (about $100,000 in accelerators) or 27 Tenstorrent P100 accelerators (about $27,000), roughly a 3 to 4x difference in upfront accelerator cost for their workload. That came from hardware–software co-design that ran about 95% of operations at reduced fidelity. That result belongs to one model on one pair of platforms. Read it as evidence that cost depends on precision and hardware, not as a verdict on any vendor.
Precision reduction also carries a specific risk for TTS. TTS models are more numerically fragile than LLMs. Small rounding errors build up across diffusion steps and show up as ringing, pitch instability or smearing in the waveform, and tensor-level similarity metrics miss them. The authors conclude that aggressive precision reduction in TTS needs end-to-end validation. In practice, that means listening tests or a perceptual metric on the quantised build before it ships.
Cost and latency drivers at a glance
| Driver | What it changes | Trade-off to expect |
|---|---|---|
| Autoregressive vs non-autoregressive acoustic model | Number of sequential steps, so latency | Autoregressive models put quality ahead of real-time speed; non-autoregressive models are much faster at a lower MOS |
| Diffusion steps | Compute per utterance | More refinement steps improve fidelity and slow synthesis |
| Vocoder type | Waveform generation cost | GAN vocoders run in parallel and in real time at lower cost than autoregressive vocoders |
| Model size (LLM-based TTS) | Memory and compute | Billion-parameter models add naturalness and expressiveness but need significant compute |
| Numerical precision | Compute and accelerator cost | Audible artefacts if not validated end to end |
| Hardware platform | Upfront and running accelerator cost | Results depend on how precision, memory movement and scheduling interact |
Voice cloning without consent and the other ethical concerns of generative AI
A major ethical concern in generative speech is voice cloning without consent. Many cloning products need only a short clip of someone talking to copy their voice. Consumer Reports tested six voice cloning products and cloned a voice from publicly available audio in four of them. Those four (ElevenLabs, Speechify, PlayHT and Lovo) required only a checkbox or similar self-attestation that the user had the right to clone the voice. Descript and Resemble AI had taken steps to make non-consensual cloning harder.
Real-world harm is already documented. In 2024 NHPR reported that state and federal authorities traced robocalls simulating President Biden’s voice, which reached about 20,000 people before the New Hampshire primary, to a Texas company. The FCC has since adopted a declaratory ruling that calls using AI-generated voices are “artificial” under the Telephone Consumer Protection Act. NIST’s Generative AI Profile discusses impersonation fraud and the use of a person’s voice without permission.
Consent is the foundation of lawful voice cloning. One legal overview describes explicit consent as clear, informed, verifiable permission that includes the intended use. It notes that some US state biometric laws require written consent before voice data is collected, and that public audio does not grant permission to clone. Microsoft requires custom neural voice customers to obtain explicit written permission from voice talent, with agreements that cover duration, use and content limits. It also watermarks voices made with its personal voice feature so synthesised speech can be identified, and it tells customers to disclose that a voice is synthetic.
There are quieter disadvantages too. Neural voices still mispronounce names and domain-specific terms, and people who rely heavily on synthesised speech, such as listeners with hearing impairments, face challenges with unclear output. Microsoft also lists financial transactions and personal messaging as high-risk settings to weigh before using high-fidelity synthetic voices.
Generative AI advantages and disadvantages, side by side
| Advantage | Evidence | Matching disadvantage | Evidence |
|---|---|---|---|
| Near-human naturalness | WaveNet cut the gap to natural speech by 51 to 69% | Higher inference cost per character | 4 to 12x standard voices, per Deepgram |
| Less data per voice | Hundreds of voices at under 30 minutes each | Weak consent checks on cloning | Four of six products tested needed only self-attestation |
| Expressive, prompt-driven control | LLM-guided emotion and prosody | Controllable models can be slow | Seconds to tens of seconds per short clip |
| Productivity in customer support | 13.8% more issues resolved per hour | Errors on unusual words and names | Microsoft transparency note |
| Lower accelerator cost through hardware–software co-design | 3 to 4x lower accelerator cost in one study | Numerical fragility | Rounding errors become audible artefacts |
Deciding whether generative speech fits your product
Deepgram’s guidance is to identify your latency ceiling, concurrency needs, regulatory requirements and economic constraints before you compare small quality differences. The consent model belongs on that list from the start. A product that only reads its own copy in a licensed voice faces very different risks from one that lets users upload a voice. Retrofitting consent checks after launch is harder than designing them in.
If your ceiling is conversational (around 100 ms to first audio), the architecture choice is mostly made for you: a non-autoregressive acoustic model with a parallel vocoder, and precision cut only as far as listening tests allow. If the work is offline, such as audiobooks or dubbing, the slower, more expressive models are affordable, and cost per character matters more than latency. For a voice agent that combines both sides, see how conversational AI and generative AI differ before you decide which part of the stack carries the latency budget.
FAQ
Are text-to-speech voices generative AI?
Neural voices built on generative models are. WaveNet, used in some complete TTS systems, is a generative model of time-domain waveforms. Older concatenative and formant engines belong to the earlier, pre-neural stage.
What are some applications of text-to-speech technology?
Legitimate uses include audio editing, film dubbing and automated narration. Expressive, description-driven TTS also suits storytelling, audiobooks and personalised voice assistants.
What does it mean to synthesise speech?
Speech synthesis renders naturally sounding speech signals from text. In effect, a computer imitates the human process of moving the articulators and shaping the airflow from the lungs into sound.
