The pitch for “deepseek inference” is seductive: pull the open weights, point a request at them, and watch your per-token bill collapse against a hosted API. That story holds at demo scale and quietly falls apart the first time real volume arrives. The thing you actually adopt when you self-host DeepSeek is not a cheaper model — it is an operational layer you now own: the serving runtime, the GPU memory budget, the batching policy, the quantization trade-offs, and the safety and policy filtering that a hosted API used to hide behind its endpoint.

That distinction is the whole article. Teams that treat inference as infrastructure ship something they can operate and cost-account. Teams that treat it as a config change ship something that either blows the GPU budget silently or fails a policy review after launch. DeepSeek can genuinely undercut a hosted API on per-token cost at sustained volume — but only if utilization, batching, and model selection are managed deliberately, not assumed.

How does DeepSeek inference actually work in practice?

“Inference” here means running a forward pass through the model for every request and streaming out tokens. For a hosted API, all of that happens on someone else’s hardware and you pay per token. When you self-host DeepSeek, you are standing up a serving stack — most commonly vLLM, SGLang, or TensorRT-LLM — that loads the weights onto one or more GPUs, holds a key-value (KV) cache per active sequence, and schedules incoming requests across the available compute.

The mechanical reality is that a single request rarely saturates a modern GPU. Generation is memory-bandwidth bound: each new token requires reading the full set of weights (or the active experts, in DeepSeek’s mixture-of-experts variants) plus the growing KV cache. One request at a time leaves most of the silicon idle. This is why the economics of self-hosted inference are decided by batching, not by the model weights themselves. Continuous batching — where the scheduler packs new requests into in-flight batches as older sequences finish — is what turns an idle GPU into a cost-efficient one. Serving runtimes like vLLM built their reputation on exactly this, and prefix-reuse techniques such as KV-cache and prefix reuse for production LLM serving push utilization further by avoiding recomputation of shared context.

The practical consequence: your cost per thousand requests is a function of how many requests you can keep concurrently in flight without breaching your p95 latency target. That number is not on any spec sheet. You measure it under load, on your traffic shape.

Self-hosting DeepSeek versus a hosted generation API

The real decision is not “DeepSeek or GPT” as models — it is “own the serving stack or rent it.” Each side wins under different conditions, and the crossover is a function of volume, latency tolerance, and how much operational surface your team can carry.

Dimension	Self-hosted DeepSeek	Hosted generation API
Per-request marginal cost at high volume	Low if GPU utilization is high; approaches zero as batching improves	Fixed per-token; predictable but does not decay with volume
Cost at low / bursty volume	Poor — you pay for idle GPUs	Excellent — you pay only for what you use
Latency control	Full control over batching, quantization, hardware placement	Bounded by the provider’s queueing and rate limits
Data residency & governance	You own the boundary; nothing leaves your infrastructure	Depends on provider terms and region
Safety / policy filtering	You must build it — no filter ships in the weights	Provider-side filters included (but opaque)
Operational burden	High: patching, scaling, on-call, model updates	Low: an HTTP endpoint
Time to first working demo	Days to weeks	Hours

The row that surprises teams is the safety one. A hosted API quietly runs its own content filters on both prompt and completion. Open weights ship with none of that. If your workload is customer-facing, you inherit responsibility for a filtering layer — the kind of toxicity and policy screening covered in benchmarking toxicity in customer-facing GenAI — and that layer is not optional in most enterprise reviews. Treating it as a follow-up task is one of the most common reasons a working DeepSeek pilot gets rolled back after the first budget or safety surprise.

For the broader trade-off between adopting a raw model versus a managed platform, choosing between an agent platform and a model for production walks the same fork from the platform angle.

How do you control DeepSeek inference cost as volume grows?

The naive cost model divides the hosted API’s per-token price by some optimism factor and calls it savings. The honest model starts from GPU-hours and works toward cost-per-thousand-requests.

Here is a worked example with explicit assumptions — illustrative, not a benchmark of any specific deployment:

• Assume one GPU node rented at roughly $2/hour (cloud on-demand pricing varies widely; treat this as a placeholder, not a quote).
• Assume, under continuous batching, the node sustains on the order of 40 concurrent requests at your p95 latency ceiling, each producing ~500 output tokens in ~5 seconds.
• That is roughly 40 requests / 5 s = 8 requests per second, or ~28,800 requests per hour, for ~$2.
• Cost per thousand requests ≈ $2 / 28.8 ≈ $0.07 — if the node stays saturated.

Now change one assumption. If real traffic only fills the node to 10% average utilization because it arrives in daytime bursts, your effective cost per thousand requests rises roughly tenfold, and the hosted API — which charges nothing when idle — becomes cheaper. This is the crossover that teams miss: self-hosting is a bet on sustained utilization (this is an observed-pattern from feasibility work; the exact numbers are workload-specific, not a published rate). A token size calculator for latency and cost budgets helps pin the token-count side of that math before you commit hardware.

The lever that most directly moves this number, aside from batching, is model selection. You do not need to serve the largest DeepSeek variant for every request. Routing cheap requests to a smaller model and reserving the large one for genuinely hard queries — the pattern described in how LLM routing cuts cost without losing reliability — often does more for the bill than any kernel-level optimization.

What do quantization, batching, and GPU memory budget actually buy you?

These three knobs are where “deepseek inference” stops being a model choice and becomes an engineering problem. Each buys something real and costs something real.

Quantization shrinks the weights from 16-bit down to 8-bit or 4-bit representations. The immediate win is memory: a model that needed multiple GPUs in FP16 might fit on one in INT4, which changes your cost structure entirely because GPU count is the dominant line item. The immediate risk is quality drift — quantization can degrade instruction-following and reasoning in ways that do not show up on generic benchmarks but do show up on your task. The trade-off between numerical precision and output fidelity is a first-class decision, not a free optimization, and it interacts with the runtime: model optimization for edge inference — distillation, quantisation, and runtime fit covers where that line usually sits.

Batching buys throughput at the cost of tail latency. Larger batches keep the GPU busier and lower cost per request, but a request that lands at the back of a full batch waits longer. Continuous batching mitigates this, but there is no free lunch: your p95 latency target caps how aggressively you can batch. This is the knob you tune against a latency SLO, not against a cost target directly.

GPU memory budget is the constraint that ties the other two together. The KV cache grows with both batch size and context length, and it competes with the weights for the same HBM. Long-context requests can eat memory that would otherwise hold more concurrent sequences, which is why context-heavy workloads sometimes cost more per token than short-turn chat despite using the same model. Prefix-reuse and tiered KV strategies — see hierarchical caching for low-latency LLM inference — exist precisely to reclaim that memory.

A quick diagnostic: is your DeepSeek serving setup actually cheaper?

Run through this before claiming a saving over a hosted API:

• Do you know your sustained GPU utilization under real traffic, not peak?
• Is your cost expressed as cost-per-thousand-requests, not per-token in isolation?
• Have you measured p95 latency under your target batch size, not median?
• Have you validated output quality after quantization, on your task, not a public benchmark?
• Is there a safety/policy filter in the path that a hosted API would have provided?
• Is there a human review path for high-risk outputs?

If more than one of these is a “we’ll get to it,” you have a demo, not a production inference path.

Where does DeepSeek inference fit in a full generation stack?

Inference is one layer among several, and the mistake is treating it as the whole system. A production generation stack already has model selection (which model handles which request), safety filtering (before and after generation), observability (so you can see when quality or latency drifts), and a human review path for the outputs that matter. DeepSeek inference slots underneath all of that as the execution engine — it does not replace any of it.

This mirrors a point we make about consumer-grade demos generally: the impressive part you saw in the demo sits on top of a production stack that the demo hid. When that stack is missing, the model looks like the product. In practice, keeping generated output usable past the first month — rather than watching it get rolled back after the first budget or safety surprise — depends more on the surrounding layers than on the raw model. Monitoring in particular is not optional; how to monitor ML models in production covers what drift detection looks like for a generation workload.

The realistic expectation for latency and controllability: self-hosting gives you more control than a hosted API, not less — you decide the batching, the quantization, the hardware — but you only realize that control if you have the observability to see what your choices are doing. Control you cannot measure is not control.

FAQ

What should you know about deepseek inference in practice?

DeepSeek inference means running the model’s forward pass on hardware you control, using a serving runtime such as vLLM, SGLang, or TensorRT-LLM that loads the weights, holds a KV cache per request, and schedules requests across GPUs. In practice it means you own an operational layer — batching, memory budget, and scheduling — not just a model file. Generation is memory-bandwidth bound, so a single request rarely saturates a GPU; batching is what makes it efficient.

What are the trade-offs between self-hosting DeepSeek inference and using a hosted generation API?

Self-hosting can drive per-request cost toward zero at high, sustained utilization and gives you full control over latency, data residency, and safety filtering — but you carry the operational burden and pay for idle GPUs at low volume. A hosted API charges predictably per token, includes provider-side safety filters, and is far cheaper for bursty or low-volume traffic. The crossover depends on your sustained utilization, not on the model.

How do you account for and control DeepSeek inference cost as request volume grows past demo scale?

Model cost as cost-per-thousand-requests derived from GPU-hours and sustained utilization, not as a per-token discount against the API. The dominant levers are continuous batching (to keep GPUs saturated) and model selection (routing cheap requests to smaller models). Self-hosting is a bet on high sustained utilization; if traffic is bursty and utilization stays low, a hosted API is usually cheaper.

What do quantization, batching, and GPU memory budget buy you, and where do they hurt quality or latency?

Quantization shrinks weights to fit fewer GPUs but can degrade instruction-following and reasoning on your specific task. Batching raises throughput and lowers cost but increases tail latency, so it must be tuned against a p95 SLO. GPU memory budget ties them together — the KV cache competes with weights for HBM, and long-context requests eat the memory that would otherwise hold more concurrent sequences.

Where does DeepSeek inference fit into a production generation stack with model selection, safety filters, and human review?

It sits underneath them as the execution engine, not as a replacement for them. A production stack still needs model selection, pre- and post-generation safety filtering, observability for drift, and a human review path for high-risk outputs. Open weights ship none of these, so treating inference as the whole system is how a working pilot fails a later budget or safety review.

What latency and controllability expectations are realistic for DeepSeek inference in an enterprise workflow?

Self-hosting gives you more control than a hosted API — you set batching, quantization, and hardware placement — but only if you have observability to see what those choices do to latency and quality. Realistic p95 latency depends on your batch size and context length, measured under real traffic rather than assumed. Control you cannot measure is not control.

Before committing to a self-hosted path, the honest test is whether the inference layer survives contact with cost accounting, latency SLOs, and a safety review at once — the failure class where a one-line API swap hides the production stack underneath. That is precisely what a [generative AI feasibility assessment](generative AI) is built to surface, and it is the same discipline we bring to the wider generative AI practice.