A classifier trained in PyTorch outputs a logit. The serving path in C++ turns that logit into a probability with sigmoid. On paper the transform is one line — 1 / (1 + exp(-x)) — and most teams porting an inference path treat it as such: copy the formula into the target runtime, assume the outputs match, move on. That assumption is where a port silently changes what the model decides. The problem is not that the math is hard. It is that sigmoid(x) = 1/(1+exp(-x)) is numerically fragile in exactly the region where classifiers live, and each target runtime — a CPU vector unit, a CUDA kernel, a WebAssembly build, a WebGL fragment shader — handles that fragility differently. A large negative logit that overflows exp() on one backend but saturates cleanly on another does not just change a probability by a rounding digit. It can flip a classification. And because the two runtimes agree on almost every other input, the divergence hides until a validation set — or worse, production traffic — lands on the boundary. How should you think about logit sigmoid in practice? A logit is a raw, unbounded pre-activation score. The final linear layer of a binary classifier emits it: any real number, positive or negative, with no probabilistic interpretation on its own. The sigmoid function maps that unbounded score into the open interval (0, 1) so it reads as a probability. Large positive logits push toward 1, large negative logits toward 0, and the logit 0 maps exactly to 0.5 — which is, for most binary classifiers, the decision threshold. That last point is what makes the transform load-bearing rather than cosmetic. The classifier’s decision is sigmoid(logit) > threshold, and because sigmoid is monotonic, that is equivalent to logit > logit_threshold. So a port that reproduces logits perfectly but computes sigmoid slightly differently still produces the same class labels — right up until floating-point behaviour near the extremes breaks monotonicity or saturates inconsistently. The interesting failures do not live at the boundary the model cares about; they live at the numeric edges the formula was never stable across. If you want to see where this transform sits relative to everything else that has to survive a port, our walkthrough on mapping the serving path with a machine learning architecture diagram treats the activation step as one node among many that need equivalence checks, not a footnote. Why does the naive sigmoid formula overflow, and how is it stabilised? Write out 1 / (1 + exp(-x)) and feed it a large negative logit — say x = -800. The term exp(-x) becomes exp(800), which overflows the representable range of an IEEE-754 float long before that. In float32, exp() overflows to infinity above roughly x ≈ 88; in float64, above roughly x ≈ 709, per the IEEE-754 double-precision exponent range. Once you have infinity in the denominator, the result is 0 — which happens to be the correct limit here, so a large negative logit often survives by luck. The mirror case is the killer: the naive formula does not overflow for large positive logits, but a symmetric hand-port that rearranges to exp(x)/(1+exp(x)) does, and now a confident positive prediction returns NaN. The standard fix is a branch on the sign of the input: sigmoid(x) = 1 / (1 + exp(-x)) for x >= 0 sigmoid(x) = exp(x) / (1 + exp(x)) for x < 0 Both branches keep the exponent argument negative, so exp() stays in [0, 1] and never overflows. This is the formulation PyTorch, TensorFlow, and most mature numerics libraries use internally — torch.sigmoid does not run the textbook one-liner. The trap in porting is that when you reimplement the activation by hand in C++ or a shader, you tend to copy the textbook formula, not the branch-stabilised one the reference runtime actually executed. The reference path was never numerically naive; your port often is. How can logit-to-sigmoid numerics differ across runtimes? Even with a stabilised formula, exp() itself is not a single fixed function across runtimes. It is an approximation, and each backend chooses a different one. CPU vector units — SIMD paths (SSE, AVX, AVX-512) often route exp() through a vectorised polynomial approximation with different accuracy than the scalar libm version. On the same CPU, the vectorised and scalar results can disagree in the last few bits. Compiler flags matter here: -ffast-math relaxes IEEE compliance and can reorder or approximate more aggressively, which is one reason GCC optimization flags like -O2, -O3, and -march change a port’s behaviour rather than only its speed. CUDA kernels — the device expf() intrinsic and the --use_fast_math variant trade accuracy for throughput. A kernel compiled with fast-math can be several ULP off the CPU reference, and fused multiply-add reassociation changes intermediate rounding. WebAssembly — WASM specifies IEEE-754 arithmetic, but transcendental functions like exp() are library calls, so results depend on the emscripten/libm build compiled in, not the WASM spec itself. WebGL — the harshest case. Fragment shaders may run in mediump (roughly 16-bit) precision on mobile GPUs unless you force highp, and even highp is only required to be float32-ish. A logit computed correctly can be crushed to a coarse grid before sigmoid ever sees it. None of these are bugs. They are legitimate accuracy-versus-speed choices baked into each runtime. The failure is assuming the choices agree. What precision differences shift a classification decision boundary? Here is the mechanism, made concrete. Suppose a logit sits at -0.0001 — essentially on the boundary, mapping to a probability just under 0.5. The reference float64 path in Python computes sigmoid and returns 0.49997, classified as the negative class. A WebGL mediump port quantises the logit to the nearest representable value, which rounds to +0.0002, and now sigmoid returns 0.50005 — the positive class. Same input, same weights, opposite decision. (Illustrative: the exact rounding depends on the shader precision the device grants.) For most inputs this never happens, because most logits are far from zero and the sign is unambiguous regardless of precision. The population that flips is the set of near-boundary examples, and its size depends on how your logit distribution stacks up against the threshold. A well-separated classifier has almost no boundary-adjacent mass and ports cleanly; a classifier operating in a hard domain with many marginal cases has a fat boundary region and is fragile to any precision downgrade. This is why “it passed on my test images” is not evidence of a safe port — it is evidence that your test set happened to miss the boundary. How does a porting assessment verify output-equivalence? You do not eyeball this. You measure it, against the profiled Python baseline, on inputs chosen to stress the boundary rather than the easy interior. The check has three layers. Sigmoid-parity check across candidate runtimes Check What it measures Pass criterion (illustrative) Evidence class Sweep-domain parity sigmoid(x) over a logit sweep incl. ±extremes max abs error ≤ tolerance vs float64 reference benchmark (named test set) Overflow / NaN guard Response to logits at ±800, ±88 no NaN/Inf on any backend benchmark Boundary-mass parity Class labels on the validation set zero decision-boundary flips benchmark Precision-mode audit Shader/kernel precision actually granted highp / float32 confirmed, not assumed observed-pattern The tolerance in row one is not a universal number — it is derived from how close your logit distribution runs to the threshold. A classifier with a wide margin can tolerate a looser sigmoid tolerance than one whose logits crowd zero. Row three is the one teams skip and regret: label parity on the validation set is the outcome that matters, because it collapses all the upstream numerical differences into the single question of whether the port changes what the model decides. In the porting assessments we run, the boundary-mass check is where silent divergences surface that the sweep-domain error alone would have passed. When is a divergence large enough to block a port? Not every mismatch is a blocker. The decision rule we apply: if the sweep-domain error is non-zero but no validation-set label flips and the flip-free margin holds across the boundary region, the divergence is tolerable — you document the tolerance and ship. If any label flips, or if the precision-mode audit reveals the runtime is silently running below float32, the port is blocked until the activation path is corrected (force highp, disable fast-math for the activation kernel, or promote the logit to float32 before the transform). The line is not “the numbers differ” — numbers always differ across runtimes. The line is “the decision differs.” FAQ How does logit sigmoid actually work? A logit is a raw, unbounded pre-activation score from the classifier’s final linear layer; sigmoid maps it into (0, 1) so it reads as a probability, with logit 0 mapping to 0.5 — usually the decision threshold. Because sigmoid is monotonic, the decision sigmoid(logit) > threshold is equivalent to a logit threshold. In practice this means the transform is load-bearing: a port that reproduces logits but computes sigmoid inconsistently near the numeric extremes can still change the class label. Why does the naive sigmoid formula overflow, and how is it stabilised? 1/(1+exp(-x)) sends exp() toward infinity for large negative logits (float32 overflows around x ≈ 88, float64 around x ≈ 709 per IEEE-754), and the mirror rearrangement exp(x)/(1+exp(x)) overflows for large positives. The fix is a sign branch that keeps the exponent argument negative in both cases, which is what PyTorch and TensorFlow use internally. Ports fail when they copy the textbook one-liner instead of the branch-stabilised form the reference runtime actually ran. How can logit-to-sigmoid numerics differ across CPU-vector, GPU, WASM, and WebGL runtimes? exp() is an approximation, and each backend picks a different one: CPU SIMD paths use vectorised polynomials that disagree with scalar libm in the last bits, CUDA offers fast-math intrinsics several ULP off the reference, WASM depends on the libm build compiled in, and WebGL may silently run mediump (~16-bit) unless highp is forced. These are legitimate accuracy-versus-speed choices, not bugs — the failure is assuming they agree. What float-precision differences between runtimes can shift a classification decision boundary? Only near-boundary logits flip. A logit at -0.0001 mapping to 0.49997 in float64 can round to +0.0002 in WebGL mediump and cross to the positive class — same input, opposite decision. Most logits are far from the threshold and are sign-stable regardless of precision, so the flip population equals the boundary-adjacent mass of your logit distribution; well-separated classifiers port cleanly, marginal ones are fragile. How does a porting assessment verify output-equivalence of the sigmoid path against the Python baseline? It measures sigmoid parity against the float64 Python reference across a logit sweep that includes the extremes, guards against NaN/Inf, audits the precision actually granted by the runtime, and — the decisive layer — checks class-label parity on the validation set. Label parity collapses all upstream numerical differences into the one question that matters: does the port change what the model decides? When is a logit-sigmoid divergence large enough to block a port versus tolerate it? If the sweep-domain error is non-zero but no validation-set labels flip and the flip-free margin holds across the boundary, document the tolerance and ship. If any label flips, or the precision audit shows the runtime running below float32, block the port until the activation path is corrected. The line is not that the numbers differ — they always do across runtimes — it is whether the decision differs. Where this leaves a port The logit-sigmoid step is a small function that carries a large risk, because it is the last node before the model’s answer and the one most tempting to hand-copy. Getting it right on a GPU performance and porting assessment means treating output-equivalence as something you verify against the profiled baseline, not something you assume from a shared formula. The same discipline extends past the activation: once you accept that numerics do not travel for free, the CUDA-versus-OpenCL target-runtime decision becomes a parity question as much as a performance one. The concrete failure class to name in the assessment: a serving-path port that changes the decision boundary without changing the code that looks like it matters. The numerical-parity check on the logit-sigmoid path exists to catch exactly that before the port ships.
