Small language models are language models with far fewer parameters than large language models. Most use the same transformer neural network architecture, sometimes at lower arithmetic precision, and are sized so they can be trained or hosted on consumer hardware such as a personal computer. There is no agreed parameter cut-off. Published definitions put the line anywhere from under one billion parameters to under forty billion. Small models suit narrow domain tasks, tight hardware or budgets, and data that has to stay on the device. A large model still earns its cost on open-domain synthesis, knowledge-heavy questions that retrieval cannot cover, and safety-critical judgement under distribution shift.
How many parameters does a small language model have?
There is no single threshold, because studies define “small” in different contexts.
| Source | Where it draws the line |
|---|---|
| Wikipedia, “Small language model” | Typically fewer than 40 billion parameters |
| Agentic-systems SLM survey (arXiv 2510.03847) | Decoder-only models of roughly 1 to 12 billion, occasionally about 20 billion |
| Comprehensive SLM survey (arXiv 2411.03350) | Reviews models under 7 billion, and notes that other studies use sub-billion or up-to-10-billion boundaries |
| Lu et al., SLM survey and measurements | 100 million to 5 billion, because 7B models were still mostly deployed in the cloud in 2024 |
| Agent-paradigm deployment study (arXiv 2604.19299) | Fewer than 10 billion |
| Role of small models survey (arXiv 2409.06857) | Relative: BERT at 110M is small next to Llama-8B, which is small next to GPT-4 |
Two definitions skip a hard number altogether. The comprehensive SLM survey sets the lower bound at the smallest size where a model shows the abilities a specialised task needs, and the upper bound at the largest size that still fits the available resources. The NVIDIA authors of “Small Language Models are the Future of Agentic AI” (Belcak et al.) argue that a definition tied to parameter count goes stale, and they anchor theirs in what can run on widely available consumer devices with low-latency inference. In our judgement that framing is the most practical: the question is whether a model fits your memory, latency and cost budget on your target hardware, not whether it falls under some number. For where language models sit relative to machine learning and deep learning more broadly, see our comparison of machine learning, deep learning, LLMs and GenAI.
Small language models vs large language models: what the measurements show
The capability gap is real, particularly on knowledge reasoning tasks. The role-of-small-models survey notes that scaling laws favour larger models and that some advanced capabilities only appear beyond certain sizes. The same survey finds small models less robust in deployment: they are more sensitive to distribution shift, perturbations and catastrophic forgetting during continual adaptation. Larger parameter counts are crucial for long-context capability. In Lu et al.’s measurements, Qwen1.5-0.5B and Qwen2-0.5B handled short contexts but lost recognition accuracy as context length grew, while Qwen2.5-3B stayed near-perfect across lengths.
On efficiency per GPU, small models come out well ahead. A task-specific efficiency study of 16 open models from 0.5B to 72B found that 1.5B to 3B models traded 10 to 35% accuracy against 70B+ models on generation tasks for 40 to 90 times the per-GPU throughput (350 to 4,400 tokens per second against 20 to 50). The accuracy gap is task-dependent. On GSM8K maths, accuracy rose from 37.7% at 0.5B to 92.0% for Llama-3.1-70B, yet Qwen2.5-14B reached 80.8% with about a fifth of the parameters. Sentiment classification on IMDB, by contrast, was already at 91.7% with a 0.5B model and barely improved with scale.
Serving cost also favours small models. Belcak et al. state that serving a 7B model is 10 to 30 times cheaper in latency, energy and FLOPs than serving a 70B to 175B model. The agentic-systems survey reports a comparable 10 to 30 times cost reduction for common agent calls. The same NVIDIA paper also records the main counter-argument: a specialised small-model endpoint is harder to keep fully loaded than a shared generalist LLM, so economies of scale can eat into the per-token saving. In our view, that point matters more than the headline multiples for teams with low or uneven traffic, because an idle GPU that serves a cheap model is still an idle GPU.
Where a fine-tuned small model beats a general one
Dedicated small models outperform general LLMs on some domain-specific tasks. The role-of-small-models survey reports that fine-tuning small models on domain-specific data can outperform general LLMs on those tasks. In the comprehensive SLM survey, a fine-tuned model under 1.6 billion parameters scored 75.4% on PubMedQA, against GPT-4’s 74.4% in a few-shot setting. A 220M T5 re-ranker on top of BM25 reached 0.3599 MAP on TREC-DL 2020, against GPT-3’s 0.3163. Results like these are what you should expect where the task is bounded, and they say nothing about general ability.
Token-level and structured-output work is another strong area. The agent-paradigm study of 27 open models under 10B found base small models competitive on classification and token-level tasks such as named entity recognition, sentiment analysis and stock prediction, with base gemma-3-270m outperforming the agent setups by a large margin on most tasks. For tool use, the agentic-systems survey points to the Berkeley Function-Calling Leaderboard: argument correctness and strict schema adherence matter more than parameter count. The same survey says a 3B to 9B model guided by JSON Schema can exceed 99% validity on extraction and templating. That is why small models fit the repetitive, scoped steps inside agent pipelines (see how agentic AI differs from generative AI for what those pipelines are).
Retrieval-augmented generation is effective for small models. The comprehensive SLM survey cites a study in which 7B models with retrieval-augmented generation sometimes matched or beat GPT-3.5 and GPT-4. The design choices behind that kind of system are covered in our retrieval-augmented generation examples and guidance.
Two practical conditions bear on specialisation: data quality and volume. First, data quality: small models benefit more than large ones from clean, filtered data, probably because they have less capacity to absorb noise. Second, volume: Belcak et al. give 10,000 to 100,000 examples as a rule of thumb for fine-tuning a small model, with LoRA or QLoRA to keep the cost down. Privacy is another reason developers favour small models. Running locally keeps messages and calendar data off the cloud, and it lets the application control which queries stay on the device.
Are small language models good enough for coding assistants?
For fast unit tests and localised code generation, yes. A 2025 empirical study of 20 open models from 0.4B to 10B parameters on HumanEval, MBPP, Mercury, HumanEvalPack and CodeXGLUE found that the 3B-to-10B group scored highest, but several models at 3B and below were competitive. Qwen2.5-Coder 3B reached a mean pass@1 of 0.59, among the top five, and Qwen2.5-Coder 1.5B reached 0.54, ahead of CodeLlama 7B (0.37) and StarCoder2 7B (0.47). Both Qwen2.5-Coder models used under 10 GB of VRAM with inference under 0.75 seconds.
Higher accuracy costs memory. Qwen2.5-Coder 7B improved pass@1 over the 1.5B model by 20.37% (65% against 54%), and needed 261% more VRAM to do it. In the study, a 10% performance gain could take nearly four times the VRAM, while inference times stayed statistically similar. Keep the test conditions in mind when reading these figures: each model ran on a single GPU in PyTorch at BF16, with no quantisation, vLLM, TensorRT or FlashAttention, and only open decoder-only models released between 2022 and 2024 were included.
Two of the sources draw the line by task, not by benchmark score. The agentic-systems survey points to models of 3.8B and 7B for unit tests and localised generation, with larger models kept for cross-file refactoring or unfamiliar frameworks. Belcak et al. reach a similar split: small models handle boilerplate and template-based output, while architectural reasoning, adaptive planning and debugging benefit from an LLM.
When should a small model hand off to a large one?
Deployed systems such as Apple Intelligence combine the two. The role-of-small-models survey argues the best setup is hybrid, with small models in specialised or low-cost roles and large models supporting them. Which pattern fits depends on what you are optimising.
| Pattern | Fits when | How it works |
|---|---|---|
| Cascading | Query difficulty varies widely and cost is the main goal | The small model answers. If its confidence falls below a threshold, the query is forwarded to a larger model |
| Front-door routing | Requests fall into known capability classes (extraction, tool use, coding) | A router picks the cheapest competent model and escalates to an LLM when outputs fail validation or confidence is low |
| Speculative decoding | A large model’s output quality is required | The small model drafts candidate tokens and the large model verifies them in parallel, cutting sequential decoding steps and latency |
| Edge-cloud split | Data is sensitive | The small model runs on the device and complex requests go to a cloud LLM, as with Apple Intelligence’s roughly 3B on-device model (2024) |
The agentic-systems survey makes escalation a deliberate decision, triggered by specific conditions: open-domain synthesis with long-range dependencies, knowledge-heavy QA that retrieval cannot answer, safety-critical judgement under distribution shift, dense search over many candidate solutions (such as code repair across large repositories), or a compliance requirement for frontier-grade guardrails. Lu et al. name the weak spot of this on-device filter approach: it is limited both by the small model’s ability and by the module that decides which tasks are easy enough. Before trusting the cost saving, we would measure how often the router keeps a hard query on the small model, because those misroutes are where a cascade quietly loses accuracy.
Can small language models run on a laptop, phone or edge device?
Yes, within memory and runtime limits. Lu et al. measured footprints from 275 MB to 2,456 MB for models of 0.1B to 3B parameters at a 2,048-token context, with memory growing linearly with model size. For comparison, Hugging Face notes that an iPhone 15 has 6 GB of DRAM and an iPhone 15 Pro has 8 GB, and positions its SmolLM models for devices from smartphones to laptops. Context length also drives memory use. At 32,000 tokens, the KV cache took over 80% of memory in Lu et al.’s tests. Meta limited its quantised Llama 3.2 models to short-context applications of up to 8K for this reason.
The comprehensive SLM survey reports that Llama 2 7B needed about 84 seconds to process 100 tokens on a phone with a Snapdragon 685.
Quantisation is a common method for deploying small models on devices. Meta reports that its quantised Llama 3.2 1B and 3B models were 56% smaller on average, used 41% less memory and ran 2 to 4 times faster than BF16, in testing on a OnePlus 12. These are Meta’s own figures. In Lu et al.’s benchmarks, Q4_K_M cut latency by an average of 50%, and 4-bit beat 3-bit in both prefill and decode, because 3-bit’s irregular bit width lacks hardware support. The agentic-systems survey suggests INT4 or INT8 weights with an FP16 KV cache to maximise throughput without hurting structured output.
One open tool for this is llama.cpp. It supports integer quantisation from 1.5-bit to 8-bit, CPU backends for Apple silicon, x86 and RISC-V, CUDA, HIP, Metal, Vulkan and SYCL GPU backends, and CPU+GPU hybrid inference for models larger than available VRAM. It can download and run a GGUF model directly from Hugging Face.
Edge backends are compared on energy efficiency as well as speed. A 2025 comparison of CPU, GPU and NPU backends for small models found specialised backends beat general-purpose CPUs, with an NPU fastest by a wide margin. Low-power ARM CPUs were competitive on energy efficiency, but on energy-delay product the NPU improved on an NVIDIA Jetson Orin Nano Super GPU by 84% for F16 models and 140% for Q4 models. One caveat: the NPU in that study was a RaiderChip design implemented on an FPGA, whose high power draw reduced its advantage. Prefill is where GPUs gain most, because prompt tokens are processed in parallel while decoding produces one token at a time. Decode on small devices is usually limited by memory bandwidth, which is why quantised weights help it so consistently.
Open small language models to look at on Hugging Face
Small models remain highly popular in Hugging Face downloads, according to the role-of-small-models survey. Families named in the sources include:
- Qwen (Alibaba Cloud): sizes from 0.5B upward, including Qwen2.5-3B and the Qwen2.5-Coder line.
- Gemma (Google DeepMind): Gemma-2-2B among others, and gemma-3-270m at the very small end.
- Llama 3.2 (Meta): text-only 1B and 3B models with 128K context, built with pruning and distillation for on-device summarisation, instruction following and rewriting.
- Phi (Microsoft): Phi-3-mini (3.8B), evaluated for edge settings, and Phi-3.5-mini, evaluated on coding.
- SmolLM (Hugging Face): SmolLM2 at 135M, 360M and 1.7B, and SmolLM3, a fully open 3B model trained on 11.2T tokens with up to 128K context and six natively supported languages.
- Granite (IBM), Nemotron (NVIDIA), Mistral Small (Mistral AI): other families that include models under ten billion parameters.
Parameter count does not always predict which of them performs better. Lu et al. found Qwen2 at 1.5B outperforming many 3B models. The coding study likewise concluded that architecture and training quality also play significant roles.
Picking the task first, then the size
Several of these sources conclude that model size alone does not predict outcomes. In the agent-paradigm study, 8B to 10B models combined high energy use with lower effectiveness on financial tasks, and the authors concluded that design should fit the specific task rather than assume bigger is better. That study covered 20 financial datasets, so treat it as evidence for the principle, not a ranking for your domain. The task-specific efficiency study points the same way: 0.5B to 3B models suit constrained deployments best, with diminishing returns beyond 14B.
In practice, the question to settle is not “small or large” but which of your requests a small model handles correctly, at what cost and on which hardware, and which ones it must pass on. Public benchmarks under-represent on-device tasks such as auto-reply or GUI understanding.
FAQ
What is the smallest language model?
The floor depends on the task. Models trained on the TinyStories dataset with as few as 28M parameters can write coherent children’s stories. Among general-purpose open models, SmolLM comes as small as 135M parameters.
What are the best tiny language models?
Hugging Face reports that SmolLM-135M, 360M and 1.7B each beat other models under 200M, 500M and 2B parameters respectively on its common-sense and world-knowledge benchmarks. That is the vendor’s own evaluation, so check it on your workload.
