Download the weights, run the model, cancel the API bill. That is the story most teams tell themselves when they first look at Vicuna, and it is the reason the deployment usually costs more than the API it was meant to replace. An open model is not a hosted API with the price crossed out. It is a set of weights you now have to serve, size, and keep fed on hardware you are paying for whether it is busy or idle. Vicuna is worth understanding precisely because it made this confusion visible at scale. It is one of the models that convinced a lot of teams that self-hosting was suddenly trivial. The model part became easy. The deployment part — the part that actually decides your cost per token — did not. What is Vicuna and how does it differ from the base LLaMA model? Vicuna is an instruction-tuned open large language model built by fine-tuning Meta’s LLaMA weights on conversations. The base LLaMA model is a pretrained transformer: it predicts the next token well, but it has not been shaped to follow instructions or hold a helpful dialogue. Vicuna takes those base weights and fine-tunes them on a large set of user-shared ChatGPT-style conversations, which is what turns a raw language model into something that answers questions in a chat register. That distinction matters more than it looks. When people say “Vicuna AI,” they usually mean the behavior — the chatty, instruction-following assistant. But the thing you deploy is the weights, and the weights inherit LLaMA’s architecture, parameter counts, and — critically — LLaMA’s licensing constraints. Vicuna is released as a set of weight deltas or merged checkpoints precisely because the base weights carry their own terms. You do not get to ignore where the model came from just because the fine-tune has a friendlier name. The practical version of the difference: LLaMA gives you the capability, Vicuna gives you the interaction style, and neither gives you a serving strategy. The interaction style is nearly free to obtain. The serving strategy is the engineering, and it is where the money is. Comparable open instruction-tuned models — Mistral’s instruct checkpoints, the Llama-2 and Llama-3 chat variants, Qwen — occupy the same conceptual slot. The reasoning in this article applies to any of them. Vicuna is the entry point because it is the query people type, not because it is the only model that behaves this way. What GPU and VRAM footprint does Vicuna need? The single number that decides whether a model fits on a given GPU is VRAM, and the dominant term in VRAM is the model weights themselves. A rough planning rule: at 16-bit precision, a model needs about 2 GB of VRAM per billion parameters just to hold the weights, before you account for the KV cache, activations, and framework overhead. This is an architectural approximation, not a benchmark — it follows directly from the byte width of the parameters. Vicuna is distributed at the LLaMA parameter sizes it was fine-tuned from. Using the 2-GB-per-billion rule as the anchor: VRAM planning table for a Vicuna-class model Parameter size Weights at FP16 (~2 GB/B) Weights at INT4 (~0.5 GB/B) Practical single-GPU fit 7B ~14 GB ~4 GB INT4 fits a 8–12 GB card; FP16 needs 16 GB+ 13B ~26 GB ~7 GB FP16 needs a 24–32 GB card; INT4 fits a 12 GB card 33B ~66 GB ~17 GB FP16 needs multi-GPU or 80 GB; INT4 fits a 24 GB card (These figures are architectural approximations from parameter byte-width, not measured throughput. Add headroom for the KV cache, which grows with sequence length and batch size — on long contexts it can rival the weights.) The table already tells you the first hard truth: precision choice moves you across GPU tiers. A 13B Vicuna at FP16 wants a 24 GB-class card; the same model quantised to INT4 fits comfortably on a 12 GB card with room for a batch. If you have skipped the sizing step and assumed “it’s a 13B model, any decent GPU will do,” you have skipped the decision that determines your hardware bill. We see this order-of-operations mistake constantly: the GPU gets bought, then the model gets sized, then the two do not fit. The KV cache is the term people forget. It scales with batch size times sequence length, and on long-context serving it can consume as much memory as the weights. A 128GB GPU changes what fits, but capacity is rarely the real bottleneck — bandwidth and the cache growth pattern usually bite first. How do quantisation and batching change throughput and cost per token? This is where the naive single-request setup leaves most of the hardware idle. A GPU serving one request at a time on an autoregressive model is memory-bandwidth-bound: it streams the entire weight matrix through the compute units to produce a single token for a single user, then does it again. The arithmetic units are mostly waiting. Two levers change the economics: Quantisation shrinks the weights from 16 bits to 8, 4, or fewer bits per parameter. That does two things at once. It lets a larger model fit on a smaller GPU, and — because inference is bandwidth-bound — it reduces how many bytes must move per token, which raises tokens per second. The trade-off is accuracy: aggressive quantisation degrades output quality, and the degradation is task-dependent, so it has to be measured on your workload, not assumed. If you want the mechanics of the most aggressive tier, 4-bit floating point cuts inference cost when it fits the accuracy budget. Batching is the bigger lever for throughput. Because a single request underuses the GPU, serving many requests together amortises the weight-streaming cost across all of them. Continuous batching — the scheduling strategy in serving runtimes like vLLM, TensorRT-LLM, and SGLang — packs requests dynamically as they arrive and finish. In configurations we have worked with, moving from naive one-request-at-a-time serving to a properly batched runtime raises tokens-per-second-per-GPU by several-fold. That is an observed pattern across LLM-serving engagements, not a fixed multiplier — the exact gain depends on request length distribution, model size, and how well the runtime keeps the batch full. The combination is what changes your break-even. A right-sized deployment (correct quantisation plus continuous batching) can serve dramatically more tokens per GPU-hour than a single-request setup on the same hardware, which is the difference between self-hosting being cheaper than a hosted API and being far more expensive. If you want a concrete measurement method for these numbers on your own hardware, our walkthrough on measuring local LLM throughput for cost decisions covers the metrics that actually matter. When does self-hosting Vicuna on GPU beat a hosted API? This is a fixed-cost-versus-variable-cost decision, and it has a clean shape. A hosted API charges per token: your cost scales linearly with usage and is zero when you send nothing. Self-hosting converts that into fixed GPU capacity: you pay for the card (or the cloud GPU-hour) whether it is saturated or idle. Self-hosting wins only above a throughput threshold — the point where your fixed capacity, kept busy, serves tokens more cheaply than the per-token rate would have cost. Decision rubric: self-host vs hosted API Signal Points toward self-hosting Points toward hosted API Sustained request volume High, predictable, continuous Spiky, low, or unpredictable GPU utilisation you can achieve High — you can keep batches full Low — traffic can’t saturate a card Data residency / privacy Strict on-prem or regulatory need No constraint Model customisation Fine-tunes you can’t run on an API Base capability is enough Engineering capacity You can own quantisation + serving No team to run the stack Latency-tail requirements You control the batching trade-off Provider SLA is acceptable The trap is the idle GPU. If your traffic cannot keep a card busy, the fixed cost of that card divided across a small number of tokens produces a higher cost per token than the API — sometimes far higher. Self-hosting does not lower cost; it changes the cost structure. It only lowers cost when volume is high enough and utilisation is good enough to exploit the structure. This is the same divergence point that shows up across GPU workloads: raw hardware cost is not the business case, serving throughput and workload fit are. It is also why the same reasoning appears in comparable model deployments — DeepSeek on H100 has the same inference-economics shape, just with different weights and a different card. Before you can even evaluate the rubric, you need to know your token volume. A token calculator estimates inference cost before you deploy, which is the input to both sides of the break-even. What are the realistic quality and licensing limits? Two limits get glossed over in the “just self-host it” pitch. The first is quality. Vicuna and its open siblings are genuinely capable, but they are not automatically drop-in equivalents to the largest hosted frontier models on every task. For summarisation, extraction, classification, and constrained chat, an open model at 13B–33B is frequently good enough — and the quality gap that remains is often narrower than the cost gap. For open-ended reasoning at the frontier, the gap can be real. This has to be evaluated on your actual task, not on a leaderboard headline. The second is licensing. Because Vicuna derives from LLaMA, its usage terms are constrained by the base model’s licence, not by an unconditional open-source grant. Some LLaMA-family licences carry field-of-use restrictions or acceptable-use terms that matter for a commercial production workload. “Open weights” is not the same as “unrestricted commercial use.” Reading the specific licence for the specific checkpoint you deploy is not optional legal paranoia — it is a gate on whether the deployment is viable at all. How do I size a Vicuna deployment before buying hardware? Sizing runs backward from your targets, not forward from a GPU you already like. Start from three numbers: your target latency (time to first token and inter-token latency), your target concurrency (how many simultaneous requests), and your sustained request volume. From concurrency and sequence length you compute the KV-cache memory you need; add it to the weight footprint at your chosen precision to get total VRAM; that determines the GPU tier. From volume and the batched throughput you can achieve on that tier, you compute cost per token and compare it against the hosted-API rate. The generative AI deployment work we lead treats this as one connected calculation, because the numbers are coupled — lowering precision changes both VRAM and throughput, raising concurrency changes both cache pressure and utilisation. Doing it as a sequence of independent guesses is how teams end up with a GPU that fits the model but never pays for itself. FAQ What does working with Vicuna AI involve in practice? Vicuna is a LLaMA-derived transformer that has been instruction-tuned on conversational data so it follows instructions and holds a chat-style dialogue. In practice, “using Vicuna” means obtaining a friendly assistant behavior easily, but deploying it means serving a set of weights on GPU hardware you size and pay for — the behavior is nearly free, the serving strategy is the engineering. What is Vicuna and how does it differ from the base LLaMA model it is fine-tuned from? The base LLaMA model is a pretrained next-token predictor that has not been shaped to follow instructions. Vicuna fine-tunes those weights on ChatGPT-style conversations, which turns raw language modelling into instruction-following chat. LLaMA supplies the capability and the architecture (including parameter counts and licence terms); Vicuna supplies the interaction style layered on top. What GPU and VRAM footprint does Vicuna need to run at its different parameter sizes? As an architectural planning rule, weights need roughly 2 GB of VRAM per billion parameters at FP16 and about 0.5 GB per billion at INT4, plus headroom for the KV cache. A 7B model fits a small card in INT4; a 13B model wants a 24 GB-class card at FP16 but fits a 12 GB card quantised; a 33B model needs multi-GPU or 80 GB at FP16. Precision choice moves you across GPU tiers. When does self-hosting Vicuna on GPU beat paying per-token for a hosted API? Self-hosting converts variable per-token cost into fixed GPU capacity, so it only wins above a throughput threshold where you can keep the card busy. High, sustained, predictable volume with good achievable utilisation favours self-hosting; spiky or low traffic that leaves the GPU idle favours the hosted API. An idle GPU produces a higher cost per token than the API, not a lower one. How do quantisation and batching change Vicuna’s throughput and cost per token on a single GPU? Quantisation shrinks weights so a model fits smaller hardware and — because inference is bandwidth-bound — moves fewer bytes per token, raising throughput at some accuracy cost. Continuous batching amortises the weight-streaming cost across many concurrent requests, which is the larger lever. Together they can raise tokens-per-second-per-GPU by several-fold versus a naive single-request setup (observed across serving engagements, not a fixed multiplier), which shifts the break-even against a hosted API. What are the realistic quality and licensing limits of using Vicuna for a production workload? On summarisation, extraction, classification, and constrained chat, open models at 13B–33B are often good enough, though the gap versus the largest frontier hosted models can be real on open-ended reasoning — it must be measured on your task. On licensing, because Vicuna derives from LLaMA it inherits the base model’s terms, which may carry field-of-use or acceptable-use restrictions; “open weights” is not the same as unrestricted commercial use. How do I size a Vicuna deployment for my latency and concurrency targets before buying hardware? Work backward from three numbers: target latency, target concurrency, and sustained request volume. Concurrency and sequence length give the KV-cache memory; add it to the weight footprint at your chosen precision to get total VRAM and the GPU tier; then combine volume with the batched throughput on that tier to compute cost per token against the API rate. Treat it as one coupled calculation, because precision and concurrency each move two variables at once. The question that actually decides a Vicuna deployment is not “can we run it” — you can, in an afternoon — but “will the card we buy stay busy enough to pay for itself at our concurrency and latency targets.” That is a GPU-audit question before it is a model question: whether the deployment is sized correctly for your VRAM, batching, and throughput targets before you commit the hardware.
