The GenAI landscape is wider than LLMs

When a procurement team or a product lead says “generative AI,” they almost always mean large language models — GPT-4, Claude, Gemini, Llama. That shortcut is understandable. LLMs are the most visible category of generative model, the most commercially deployed, and the easiest to demo. But the generative AI landscape includes entire families of models that produce images, audio, video, 3D assets, molecular structures, and code — each built on architectures that differ fundamentally from the autoregressive token prediction that defines an LLM.

The reason this matters is not academic. The use case in front of you may be served better, cheaper, and faster by a non-LLM architecture. And the deployment profile — inference cost, latency, fine-tuning budget, output controllability — moves with the architecture, not with the marketing category. Defaulting to an LLM for every generative task is a build decision, not a neutral one.

As reported in Rombach et al. (2022), Stable Diffusion operates in a 64×64 latent space rather than the full 512×512 pixel space, reducing compute requirements by roughly 50× (benchmark, per the published paper). StyleGAN3 (Karras et al., 2021) achieves FID scores below 5 on FFHQ, as reported in the published paper establishing the quality benchmark for unconditional face generation. Neither result is reproducible inside an LLM-shaped architecture, and that gap is the entry point to the wider taxonomy.

Choosing a generative model architecture by deployment constraint

Modality	Primary constraint	Recommended architecture	Key trade-off
Text	Inference cost at scale	LLM (autoregressive transformer)	Quality scales with model size and cost
Images (general)	Generation speed vs quality	Diffusion (Stable Diffusion, DALL·E 3, Imagen)	More denoising steps improve quality but raise latency (~2–10 s per image, observed pattern)
Images (real-time)	Sub-second latency required	GAN (StyleGAN, ESRGAN, pix2pix)	Single-pass generation (milliseconds) but harder to train and less output diversity
Images / molecules (controlled)	Latent-space interpretability	VAE or VAE + diffusion hybrid	Smooth interpolation and control, lower sharpness standalone
Audio / speech	Latency vs naturalness	Neural codec + language model (EnCodec, SoundStream)	Tokenised audio enables LM-style generation; autoregressive decoding adds latency
Video	Compute budget	Temporal diffusion (Sora, Stable Video Diffusion)	~30× the compute cost of single-image generation (observed pattern); quality remains variable
3D assets	Production-readiness	NeRF / score distillation (Point-E, Shap-E, DreamFusion)	Generated assets need significant manual cleanup before production use

The table is the decision tool, not the answer. The rest of the article explains what each row actually means, where it bites, and how we tend to think about it in practice.

How do diffusion models generate images?

Diffusion models generate images by iteratively denoising a random noise sample. The model is trained to reverse a noising process: given a noisy image, predict what the image looked like one step less noisy. Applied repeatedly from pure noise, the procedure produces a clean image drawn from the model’s learned distribution. Stable Diffusion (Stability AI), DALL·E 3 (OpenAI), Imagen (Google), and Midjourney are all diffusion-based.

The training side is straightforward to describe. Gaussian noise is added to images at progressively higher levels, and the model learns to predict and remove that noise at each level. Generation starts from pure noise and applies the denoising prediction repeatedly — typically 20–50 steps — to produce a clean image. Text conditioning, usually through a text encoder like CLIP or T5, turns a prompt into an embedding that steers the denoising trajectory.

What this means in deployment is less commonly explained. Inference is iterative, so each image requires multiple forward passes through the model. In our generative-AI engagements we typically see a 512×512 image at 50 denoising steps land in the 2–10 second range on a consumer GPU, depending on model size and optimisation (observed range across our engagements, not a benchmarked industry rate). Quality scales with compute: more denoising steps generally produce higher-quality images, with diminishing returns past a point. Fine-tuning for a specific style or subject — DreamBooth, LoRA — takes 5–50 reference images of the target and yields a model that reproduces the subject consistently.

The deployment surface is wide: marketing and advertising (product visualisation, campaign imagery), entertainment (concept art, game assets), e-commerce (product photography replacement, virtual try-on), and design (architecture visualisation, interior exploration). We have worked with clients who use diffusion models for retail product visualisation and manufacturing documentation. The architectural fit is real; the cost profile is the part teams underestimate.

GANs: adversarial generation with sharp outputs

Generative Adversarial Networks train two networks simultaneously. A generator produces synthetic images; a discriminator tries to tell them apart from real ones. The adversarial loop pushes both forward — the generator gets more realistic, the discriminator gets more discerning. StyleGAN (NVIDIA), BigGAN, and GigaGAN are the prominent examples.

The structural difference from diffusion is the part that matters operationally. GANs generate in a single forward pass — no iterative denoising — which makes inference fast, in the millisecond range. The cost shows up elsewhere: GANs are harder to train (mode collapse, instability, hyperparameter sensitivity), tend to produce less diverse outputs because the generator can converge to a narrow slice of the data distribution, and are harder to condition on text. Text-to-image control is simply more natural in diffusion architectures.

That cost profile is also why GANs have not gone away. For tasks where single-pass generation speed is the binding constraint, they remain the right answer: real-time image translation (pix2pix, CycleGAN), super-resolution (ESRGAN), face generation and manipulation (StyleGAN), and data augmentation for training other models. The GAN vs diffusion comparison walks through the architectural trade-offs in more detail.

VAEs: structured latent spaces for controlled generation

Variational Autoencoders learn a compressed latent representation of the data and generate new samples by decoding points from that latent space. Unlike GANs, VAEs optimise a well-defined probabilistic objective — the evidence lower bound, or ELBO — which makes training stable and reproducible. The encoder compresses input data into a distribution in latent space; the decoder generates data from points sampled from that distribution. The latent space is continuous and structured, so nearby points produce similar outputs. That property is what enables smooth interpolation between generated samples and controlled manipulation of output attributes.

Standalone, VAE outputs tend to be smoother and less sharp than GAN or diffusion outputs, because the reconstruction term in the objective encourages averaging over possibilities. That makes a pure VAE less suitable for high-fidelity image generation, and well-suited for tasks where the latent structure matters more than pixel sharpness: anomaly detection (outliers have low likelihood in the latent space), data compression, drug discovery (molecular structures generated by sampling the latent space), and representation learning.

The interesting move is what happens when VAEs are combined with other architectures. Stable Diffusion uses a VAE as its image encoder/decoder: images are compressed into a latent space by the VAE encoder, the diffusion process operates inside that much smaller latent space, and the VAE decoder converts the denoised latent back to pixels. The combination is more efficient than operating directly in pixel space, and it is one of the reasons modern image diffusion is tractable at all.

Neural audio and speech models

Generative audio spans text-to-speech, music generation, and sound effect synthesis. The architectures diverge from image generation in instructive ways. Autoregressive models like WaveNet and SoundStorm generate audio sample-by-sample or token-by-token, similar to how an LLM generates text — high quality, but slow inference because of the sequential dependency. Diffusion models adapted for audio (AudioLDM, Stable Audio) apply the denoising framework to spectrograms or latent audio representations, with text-to-audio conditioning following the same pattern as text-to-image.

The most consequential category is neural codec models. EnCodec (Meta) and SoundStream (Google) compress audio into discrete tokens that can then be modelled by autoregressive or masked language models. This is the architecture powering the current generation of voice cloning and music generation systems: audio is tokenised, a language model generates new token sequences, and the codec decoder converts those tokens back to waveforms. It is the cleanest example of how the LLM idea travels — it is the transformer over tokens shape that ports, not the LLM as a product category.

Video generation models

Video generation extends image generation along the temporal axis, and the extra axis brings real problems: temporal consistency (objects must keep their appearance and physics across frames), motion coherence (movement must be physically plausible), and compute cost. As a planning heuristic from our generative-AI engagements, generating one second of video at 30 frames per second requires roughly 30× the computation of a single image (observed pattern, not a benchmarked industry rate, and naive — actual systems share compute across temporally adjacent frames).

The current approaches divide into three families. Diffusion models extended with temporal attention layers (Sora by OpenAI, Runway Gen-2, Stable Video Diffusion) are the dominant pattern. Autoregressive video generation produces frames sequentially, conditioning each on the previous. Frame interpolation approaches generate keyframes and fill the gaps. The technology is advancing rapidly, but it remains compute-heavy and quality-variable. Production-quality video generation at scale is not yet practical for most commercial applications, and a build plan that assumes otherwise will run into both cost and consistency walls.

3D generation models

3D asset generation — meshes, textures, and materials produced from text or image prompts — is the newest frontier and the least mature. Point-E and Shap-E (OpenAI), DreamFusion, and related work use a range of approaches: point cloud generation, neural radiance fields (NeRFs), and score distillation sampling, which optimises a 3D representation to match a diffusion model’s learned distribution across multiple viewpoints.

Practical maturity is the catch. Generated 3D assets typically need significant manual cleanup before they are usable in production pipelines for games, film, or industrial design. The trajectory points toward production-quality output within a couple of years, but that is a market-direction estimate, not an operational benchmark — and not a planning input we would recommend a commercial team rely on today.

Choosing the right generative architecture

The architecture choice falls out of two questions: what is the output modality, and what is the binding deployment constraint?

Output	Architecture	Key trade-off
Text	LLM (autoregressive)	Quality vs inference cost
Images	Diffusion model	Quality vs generation speed
Real-time image transforms	GAN	Speed vs training stability
Structured generation	VAE	Control vs output sharpness
Audio / speech	Neural codec + LM	Quality vs latency
Video	Temporal diffusion	Quality vs compute cost

Defaulting to an LLM for every GenAI use case is a common and expensive mistake. Image, audio, video, and 3D generation typically need a different architecture, and the deployment characteristics — cost, latency, infrastructure — move with the architecture choice. Mismatched model selection is one of the most expensive early decisions in a GenAI project, and we routinely see organisations evaluate fewer than three architecture options before committing (observed pattern across our engagements). A GenAI feasibility assessment is the place to map each use case to the appropriate model architecture before that cost is incurred.

FAQ

What kinds of generative AI models exist beyond LLMs, and when does each architecture make sense?

Beyond LLMs, the main families are diffusion models (image, audio, and increasingly video), GANs (real-time image generation and transforms), VAEs (structured latent spaces for controlled generation, anomaly detection, and molecular design), autoregressive models for audio and code, and 3D-specific approaches like NeRFs and score distillation. The architecture fits the modality and the binding constraint — quality, latency, controllability, or compute budget.

How do GANs, diffusion models, VAEs, and autoregressive models differ in what they generate and what they need to train?

GANs train two networks adversarially and produce sharp outputs in a single forward pass, but are unstable to train and harder to condition on text. Diffusion models train a denoiser and generate iteratively over many steps, trading latency for quality and natural text conditioning. VAEs train an encoder-decoder pair against the ELBO, yielding stable training and a structured latent space at the cost of sharpness. Autoregressive models (LLMs, WaveNet, codec-based audio LMs) generate token-by-token, with quality scaling with model size and inference cost scaling with sequence length.

When is an LLM the wrong default for a generative use case?

An LLM is the wrong default whenever the output is not text. Image, audio, video, and 3D generation each have architectures that fit their modality better — diffusion for images, neural codec models for audio, temporal diffusion for video, NeRF-family approaches for 3D — and the deployment cost and latency profiles differ accordingly. Routing every generative task through an LLM-shaped tool stack is a build decision that locks in the wrong cost curve.

Which generative architecture fits a small-data, high-fidelity image problem?

For small-data, high-fidelity image work, the practical answer is usually a fine-tuned diffusion model with parameter-efficient techniques like LoRA or DreamBooth, which can produce consistent subject- or style-specific outputs from 5–50 reference images. GANs can also fit when single-pass inference latency is the binding constraint, but they need more careful training and are more sensitive to data quality.

How do I match a generative model to a use case before committing to an architecture?

Start from the output modality and the binding deployment constraint — latency, cost per generation, controllability, or training data availability — rather than from a model name. The decision table in this article is the first cut; a feasibility assessment is the second, mapping use case to architecture, training data needs, and inference profile before any build cost is committed.

What are realistic examples of generative AI in production beyond chatbots?

Diffusion models drive product visualisation, marketing imagery, and game and film concept art. GANs power real-time face manipulation, image super-resolution, and data augmentation pipelines. VAEs underpin anomaly detection and molecular design. Neural codec models power voice cloning and music generation systems. Temporal diffusion is moving into short-form video for advertising and prototyping. The chatbot is the most visible deployment of generative AI; it is far from the most economically interesting.