You spun up DeepSeek-V3 on the SGLang runtime, sent a handful of prompts, saw sensible answers, and told the channel the endpoint is up. That is a smoke test, not a validated deployment — and the gap between those two claims is where production incidents live. The search that brings most teams here — sglang deepseek-v3 — usually means “get the model serving quickly and confirm it works”. The trouble is that “it works” collapses two separate questions into one, and the collapse is exactly what fails under load. One question is whether the served system matches the serving configuration you agreed to run: the quantisation format, the context length, the batching policy, the decode parameters, the radix-cache behaviour. The other is whether the deployed endpoint actually meets the workload it was commissioned for — the concurrency, the latency budget, the request-shape distribution it will see in production. The first is verification. The second is validation. A demo answers neither. How does sglang deepseek-v3 work, and why does the smoke test mislead? SGLang is an inference-serving runtime built around RadixAttention, a KV-cache reuse scheme that shares prefix computation across requests via a radix tree. DeepSeek-V3 is a large mixture-of-experts model. Put them together and you get a high-throughput endpoint whose behaviour depends heavily on how it is configured — which experts are routed, what precision the weights are served at, how the cache is sized and evicted, how requests are batched. If you want the mechanics of the cache layer, we cover it in how prefix reuse cuts LLM inference cost and latency and in KV-cache reuse for production LLM serving. Here is why the smoke test misleads. When you send three prompts to a freshly launched server, you exercise the model at concurrency one, on short contexts, with a cold cache, at whatever precision the launch flags happened to set. Production runs that endpoint at concurrency fifty, with long shared prefixes, a warm and occasionally evicting cache, under a p99 latency budget. Almost none of the conditions that determine whether the endpoint is fit for purpose are present in the smoke test. Plausible output at concurrency one tells you the graph loads and decodes. It tells you nothing about whether the served model matches your reference weights, or whether it holds latency under real traffic. The reframe worth signing against: “the DeepSeek-V3 endpoint is up” is a status; “the DeepSeek-V3 endpoint meets its verified serving spec and its validated workload” is a claim. Only the second is reviewable. Verification: does the served system match the agreed serving specification? Verification is the internal-consistency question. You commissioned the endpoint to run a specific configuration; verification checks that what is actually serving matches that specification, independent of whether the specification was the right one. Concretely, for an SGLang/DeepSeek-V3 deployment, verification checks: Quantisation format. If the spec says FP8 weights with a named calibration, verify the loaded checkpoint is that format — not a silently-substituted BF16 or a differently-calibrated FP8. Context length. The served maximum context matches the spec, and requests near that limit behave as specified rather than truncating or erroring differently. Batching and scheduling policy. Continuous batching, chunked prefill, and the scheduler settings are the ones agreed, because they change latency distribution and, at the margins, output. Decode parameters. Temperature, top-p, top-k, and repetition penalties applied at the server default match the spec — a mismatch here quietly changes every response. Radix-cache behaviour. Cache sizing and eviction match the spec, since prefix-reuse settings interact with correctness expectations under certain sampling regimes. Verification produces a config-conformance record: for each spec item, the observed served value and a pass/fail. This is cheap to automate and it is the first thing a reviewer should be able to read. In our experience across reliability engagements, most “the model got worse after the update” incidents are verification failures — a launch flag drifted, a checkpoint was swapped, a decode default changed — not model failures at all (observed pattern; not a benchmarked rate). Validation: does the endpoint meet the real workload? Validation is the external-fitness question. Given a verified configuration, does the endpoint meet the workload it was commissioned for? This is where you leave the demo entirely and drive the endpoint with representative traffic. Validation of an SGLang/DeepSeek-V3 endpoint has three axes worth separating: Throughput and latency under concurrency. Drive the server at the concurrency levels production will see, with request shapes (prompt length, output length, prefix-sharing rate) drawn from the real distribution — not uniform toy prompts. Measure sustained throughput and the p95/p99 latency, not the mean. A server that holds p50 while p99 collapses under a burst has failed validation even though the average looks fine. Behaviour under cache pressure. Because RadixAttention’s benefit depends on prefix reuse, validation must include the traffic mix where the cache fills and evicts. An endpoint validated only against high-reuse traffic can degrade badly when the request mix shifts. Output fitness for the task. Whatever the endpoint is commissioned to do — extraction, classification, generation graded by a rubric — validation scores that task on a held-out set, at the served configuration, not on the reference weights in a notebook. The claim-class discipline matters here. Throughput and latency numbers from a load run are benchmark-class evidence only if the harness, traffic model, and hardware are named and the run is reproducible; a one-off wrk invocation on a shared box is not. We treat the load/regression suite as the thing that turns those numbers into evidence, the same way the end-to-end ML pipeline places reliability gates at each stage so that each claim traces to a gate rather than a vibe. How do you prove output equivalence to the reference DeepSeek-V3 weights? This is the question that quantisation and cache optimisation make sharp, and it is the one most deployments skip. When you serve DeepSeek-V3 at FP8 instead of the reference precision, or when radix-cache reuse changes numerics at the margins, the served outputs can drift from the reference weights. Sometimes the drift is negligible; sometimes it moves a classification boundary or degrades a reasoning chain. You cannot know which without measuring. Output equivalence is not bit-exactness — quantised inference is not expected to be bit-identical. It is a bounded-divergence claim against a defined tolerance. A workable approach: Assemble a fixed evaluation set that covers the task’s decision-relevant cases, including the hard and boundary ones. Generate reference outputs from the reference-precision weights with sampling fixed (greedy or a fixed seed) so the comparison is deterministic. Generate served outputs from the SGLang endpoint at the production configuration with the same sampling settings. Score divergence with a metric appropriate to the task — exact match and token-level agreement for structured output, a rubric or judge model for open generation, distributional checks for classification logits. Declare a tolerance before running, and treat a breach as a fail, not a discussion. The output-equivalence check is what lets you say “the FP8-served endpoint matches the reference within tolerance X on evaluation set Y” — a signable statement. Without it, “we quantised it and it seemed fine” is the whole of your evidence. What evidence should the serving eval harness produce? The verification and validation work only pays off if it leaves reproducible artifacts. A serving eval harness and load/regression suite for an SGLang/DeepSeek-V3 endpoint should produce, at minimum, the evidence in the table below. This is exactly what populates the release-readiness section of a validation pack; caching this evidence and re-running it cheaply is part of why hierarchical caching speeds regression suites. Release-readiness evidence checklist for an SGLang/DeepSeek-V3 endpoint Evidence artifact What it proves Question it answers Class Config-conformance record Served config == serving spec Verification benchmark (config diff) Load-test report (named harness, traffic model, hardware) Sustained throughput, p95/p99 under real concurrency Validation benchmark Cache-pressure profile Latency holds when the radix cache evicts Validation benchmark Output-equivalence report Served outputs within tolerance of reference weights Verification of numerics + validation of fitness benchmark Task eval score on held-out set Endpoint meets the commissioned task quality Validation benchmark Reproduction manifest (image, flags, seeds, dataset hash) Anyone can re-run and get the same numbers Both — The reproduction manifest is the one teams forget and the one reviewers care about most. A number without a way to regenerate it is a screenshot, not evidence. When the runtime is upgraded or the model is re-quantised, what re-runs? V&V is not a one-time event, and the trigger conditions are specific. An SGLang version bump can change scheduling, kernel selection, or cache semantics; re-quantisation changes the served numerics; a config change to batching or decode defaults changes latency and output. Each of these invalidates part of the evidence. The cheap discipline is a change-triggered matrix: a runtime upgrade re-runs the full load and cache-pressure suite and the output-equivalence check; a re-quantisation re-runs output equivalence and task eval; a decode-default change re-runs output equivalence. The recurring cost of skipping this is the one the ROI case turns on — you end up re-litigating what “the endpoint works” meant after every update, in production, under incident pressure. A reliability practice worth its name makes the re-run automatic and the evidence the gate. Who owns verification versus validation? Ownership blurs in exactly the way that lets things fall through. A practical split we see hold up: the platform team owns verification — they run the config-conformance and reproduction manifest because they control the runtime and the flags. ML engineering owns output equivalence and task eval, because they own the reference and the tolerance. QA or a release function owns the go/no-go read of the assembled pack. What matters less is the exact boundary and more that each artifact in the table above has one named owner, so that “someone checked latency” is never the answer to “who signs this?” You can see the whole of this discipline as one question about your production AI reliability practice: does standing up DeepSeek-V3 on SGLang leave you with signable serving evidence, or with a passing smoke test and a hope? A reliability audit of the serving stack asks precisely that. FAQ What does working with sglang deepseek-v3 involve in practice? SGLang is an inference-serving runtime built around RadixAttention, a KV-cache prefix-reuse scheme, and DeepSeek-V3 is a large mixture-of-experts model served on it. In practice the endpoint’s behaviour depends on its configuration — quantisation, context length, batching, decode parameters, and cache settings — so “it works” only means something once you separate whether it matches its serving spec from whether it meets its real workload. What does verification of an SGLang DeepSeek-V3 deployment check? Verification checks that the served system matches the agreed serving specification, independent of whether that spec is fit for purpose. It confirms the quantisation format, served context length, batching and scheduling policy, decode parameters, and radix-cache behaviour all match what was commissioned, and produces a config-conformance record with a pass/fail per spec item. What does validation look like for this endpoint under real concurrency? Validation drives the endpoint with representative traffic at production concurrency levels, using request shapes from the real distribution rather than toy prompts. It measures sustained throughput and p95/p99 latency (not the mean), checks behaviour when the radix cache fills and evicts, and scores output fitness on a held-out task set at the served configuration. How do you prove output equivalence to the reference DeepSeek-V3 weights after quantisation or radix-cache optimisation? You declare a tolerance in advance, generate reference outputs from the reference-precision weights with sampling fixed, generate served outputs at the production configuration with identical sampling, and score divergence with a task-appropriate metric. Equivalence is a bounded-divergence claim within that tolerance, not bit-exactness, and a breach is treated as a fail. What reproducible evidence should the serving eval harness and load/regression suite produce? At minimum: a config-conformance record, a load-test report naming its harness and traffic model and hardware, a cache-pressure profile, an output-equivalence report, a task eval score on a held-out set, and a reproduction manifest capturing image, flags, seeds, and dataset hash. The manifest is what turns any number into re-runnable evidence rather than a screenshot. How does V&V change when the runtime is upgraded or the model is re-quantised? Each change invalidates part of the evidence, so a change-triggered matrix decides what re-runs: a runtime upgrade re-runs the full load, cache-pressure, and output-equivalence suites; a re-quantisation re-runs output equivalence and task eval; a decode-default change re-runs output equivalence. Automating this avoids re-litigating what “works” means in production after every update. Who owns verification versus validation of the served endpoint? A workable split gives the platform team verification (config conformance and reproduction manifest, since they own the runtime and flags), ML engineering output equivalence and task eval (since they own the reference and tolerance), and QA or a release function the go/no-go read of the assembled pack. The essential point is that every evidence artifact has one named owner.
