Logit vs Sigmoid: How the Sigmoid Turns Logits into Probabilities

A model’s final linear layer emits a logit. Somewhere downstream, someone reads that number as a probability, applies a sigmoid a second time, or pairs a manual sigmoid with a loss that already expects raw logits. The training run either fails to converge or quietly produces NaN a few hundred steps in — and the bug is nowhere near where the symptom shows up.

This is one of the smallest details in a neural network and one of the most common sources of silent training failure. The confusion is understandable: a logit and a probability are two numbers describing the same thing, and the sigmoid is the one-line function that converts one into the other. But treating them as interchangeable is where things break. Getting the logit/sigmoid boundary right is a low-level prerequisite for anything from a binary classifier to a GAN discriminator, and it is worth being precise about.

What is a logit, and what does the sigmoid actually do?

A logit is the raw, unbounded score a model produces before any probability transformation. It lives on the whole real line: it can be −7.3, 0, or +12.1. Larger means “more confident this is the positive class,” but the number itself is not a probability. It is a pre-activation value.

The sigmoid is the function that maps that unbounded score into the open interval (0, 1):

sigmoid(z) = 1 / (1 + exp(-z))

Feed it a logit of 0 and you get 0.5. Feed it +5 and you get roughly 0.993. Feed it −5 and you get roughly 0.007. The sigmoid is monotonic, so ordering is preserved — a higher logit always yields a higher probability — but the mapping is nonlinear and saturates hard at both ends. That saturation is the source of most of the trouble later.

The inverse matters too. The logit is the inverse of the sigmoid: given a probability p, the logit is log(p / (1 - p)) — the log-odds. This is why the word “logit” is used for the pre-sigmoid score in the first place. It already sits in log-odds space. When people say a network “outputs logits,” they mean it outputs values that, once passed through a sigmoid, become calibrated log-odds probabilities. The network is not producing a raw guess and hoping; it is producing the log-odds directly.

Why is a logit not a probability?

The practical distinction is about where each number is safe to use.

A logit is what you want for loss computation and gradient flow. It is unbounded, so it does not saturate, and the gradients through it stay well-behaved across the full range of model confidence. A probability is what you want for interpretation, thresholding, and reporting — the number you show a downstream system that needs to decide whether 0.87 clears the bar.

The mistake is using the probability where the logit belongs. Once you have squashed a logit through a sigmoid, you have thrown away numerical headroom. A logit of +40 and a logit of +80 both map to a probability indistinguishable from 1.0 in floating point. If your loss then takes the log of that probability — as cross-entropy does — you are taking log(1.0), which is 0, and the gradient vanishes. The model has plenty of room to be more or less confident in logit space, but you have collapsed that information before the loss ever saw it.

This is the same class of numerical-conditioning concern that governs why weight initialization controls training stability: the arithmetic in the forward and backward pass has to stay inside a range where gradients carry usable signal. The sigmoid boundary is one specific place where that range is easy to leave by accident.

Why the “with-logits” loss is preferred over manual sigmoid plus cross-entropy

Here is the single most important operational point in this article. When you need a binary cross-entropy loss, do not apply a sigmoid yourself and then feed the probability into a cross-entropy function. Feed the raw logits into a fused “with-logits” loss instead.

In PyTorch that means nn.BCEWithLogitsLoss rather than nn.Sigmoid followed by nn.BCELoss. In TensorFlow it is tf.nn.sigmoid_cross_entropy_with_logits. The difference is not stylistic. The fused version computes the sigmoid and the log together using the log-sum-exp trick, which rearranges the arithmetic to avoid ever materializing a saturated probability that then gets logged.

Consider the mechanism. The naive path computes p = sigmoid(z), then loss = -[y·log(p) + (1-y)·log(1-p)]. When z is large and positive, p rounds to exactly 1.0 in float32, so log(1-p) becomes log(0) = -inf, and the gradient is NaN. When z is large and negative, log(p) blows up the same way. The fused formulation never computes p explicitly; it uses the identity that lets it express the loss directly in terms of z as max(z, 0) - z·y + log(1 + exp(-|z|)), which stays finite for every input. This is a benchmark-class property in the sense that it is a reproducible numerical fact about the two code paths, verifiable by running both on the same inputs — the naive path produces NaN where the fused path returns a finite loss.

The payoff is concrete: correct logit/sigmoid handling avoids the manual-sigmoid-plus-BCE bug that produces vanishing or NaN gradients, which is a debugging cost you pay in wasted training runs that never converge rather than in any visible error message. In our experience reviewing training code, this is one of the first things worth checking when a binary or multi-label model trains for a while and then silently stalls or diverges.

Quick reference: which loss for which output

Model output	Loss to use	Do NOT
Raw logit, binary/multi-label	`BCEWithLogitsLoss` / `sigmoid_cross_entropy_with_logits`	apply sigmoid first
Raw logits, multi-class (mutually exclusive)	`CrossEntropyLoss` (expects logits, applies log-softmax internally)	apply softmax first
Already a probability from an external source	`BCELoss` (needs `sigmoid` applied)	pass to a with-logits loss
Logit, inference-time reporting	apply `sigmoid` after the loss step, for display only	feed the sigmoid output back into a loss

The pattern across every row: the loss function owns the activation. If the loss name contains “with-logits” or is CrossEntropyLoss, it applies the sigmoid or softmax internally, and your job is to hand it the raw scores.

How does the sigmoid on a logit work inside a GAN discriminator?

In a generative adversarial network, this boundary is not a corner case — it is the entire adversarial signal. A discriminator’s job is to output one number per sample: how real does this look? That number is a logit. The sigmoid of it is the discriminator’s probability that the input is real rather than generated.

Both the discriminator and the generator are trained on losses derived from that logit. The standard formulation uses BCEWithLogitsLoss against real-labels for genuine samples and fake-labels for generated ones. If you instead apply a sigmoid manually inside the discriminator’s forward pass and then use a plain BCE loss, you reintroduce exactly the saturation problem — and in a GAN it is worse, because a well-trained discriminator is supposed to become very confident, driving logits to large magnitudes. That is the regime where the naive path saturates and the gradient to the generator vanishes. The generator stops receiving a usable learning signal precisely when the discriminator is doing its job well.

This is why logit-based loss formulations improve numerical stability in GAN training: they keep the adversarial gradient usable across the full range of discriminator confidence, rather than collapsing to zero once the discriminator is sure. Getting this one boundary right is often the difference between a discriminator that trains an adversarial pair to convergence and one that stalls after the first phase. If you are architecting a training stack for generative models, the surrounding concerns — checkpointing, serving, and monitoring GANs and diffusion models — are covered in our write-up on MLOps system design for serving GANs and diffusion in production, and the broader feasibility questions live on our generative AI practice page.

When should you use sigmoid versus softmax?

These get conflated because both turn logits into probabilities, but they answer different questions.

Use the sigmoid when each output is an independent yes/no. Binary classification (one output) or multi-label classification (several outputs, any number can be true at once — an image can be both “outdoor” and “contains a dog”). Each logit is squashed independently, and the resulting probabilities do not sum to 1 across outputs.

Use the softmax when the outputs are mutually exclusive classes and exactly one is correct. It normalizes a vector of logits so they sum to 1, coupling them together — raising one class’s probability necessarily lowers the others. This is the right choice for single-label multi-class classification.

The failure here is applying softmax to a multi-label problem, which forces the classes to compete when they should not, or applying sigmoid where you need a proper distribution over exclusive classes. The choice of optimizer interacts with this too — the difference between SGD and Adam and how to choose matters more once the output activation and loss are correctly matched, because only then does the gradient you are optimizing mean what you think it means.

Common bugs at the logit/sigmoid boundary

A short diagnostic list — these are the patterns worth scanning for when a model’s training behaves strangely:

Double sigmoid. A sigmoid inside the model’s final layer and a with-logits loss that applies its own sigmoid. The loss sees probabilities-of-probabilities, gradients are tiny, and the model learns almost nothing. Symptom: loss barely moves, accuracy stuck near the prior.
Manual sigmoid plus BCE (or plus cross-entropy). The classic NaN generator. Works for small logits, then produces inf/NaN once confidence grows and a saturated probability gets logged.
Reading a logit as a probability. Thresholding on the raw logit at 0.5 instead of thresholding the probability at 0.5 (which corresponds to a logit threshold of 0). Off-by-a-whole-transformation errors in decision logic.
Feeding inference-time sigmoid output back into a loss during fine-tuning or distillation, when the loss expected logits.
Softmax where sigmoid belongs on a multi-label head, quietly capping the model’s ability to predict multiple positives.

The unifying diagnosis is always the same question: at this exact line of code, is the tensor a logit or a probability, and does the next operation expect the one it is receiving?

FAQ

What does working with logit sigmoid involve in practice?

A logit is an unbounded pre-activation score on the whole real line; the sigmoid maps it into (0, 1) via 1 / (1 + exp(-z)). In practice, the network outputs logits and the sigmoid converts them into calibrated probabilities — but only at the right point in the pipeline. Logits are used for loss and gradient computation; sigmoid outputs are used for interpretation and thresholding.

What is the difference between a logit and a probability, and why does the sigmoid connect them?

A logit is a raw log-odds score that can be any real number; a probability is bounded to (0, 1). The sigmoid connects them because the logit is defined as log(p / (1-p)) — the log-odds — so the sigmoid is its inverse. The distinction matters numerically: logits keep gradient headroom, while probabilities saturate and lose information at the extremes.

Why is a numerically stable ‘sigmoid-with-logits’ loss preferred over applying sigmoid manually before a cross-entropy loss?

The fused with-logits loss (BCEWithLogitsLoss, sigmoid_cross_entropy_with_logits) computes the sigmoid and the log together using the log-sum-exp trick, so it never materializes a saturated probability that gets logged into log(0) = -inf. The manual path produces NaN once logits grow large. This is a reproducible numerical fact, not a preference — the same inputs yield a finite loss on the fused path and NaN on the naive one.

How does the sigmoid on a logit function inside a GAN discriminator’s real/fake decision?

The discriminator outputs one logit per sample; its sigmoid is the probability the input is real. Both discriminator and generator losses derive from that logit, so using a logit-based loss keeps the adversarial gradient usable even when the discriminator becomes very confident. Applying a manual sigmoid there reintroduces saturation exactly when the discriminator is working well, starving the generator of signal.

When should I use sigmoid versus softmax on model outputs?

Use sigmoid for independent yes/no outputs — binary or multi-label problems where several classes can be true simultaneously and probabilities need not sum to 1. Use softmax for mutually exclusive classes where exactly one is correct and the probabilities must sum to 1. Applying softmax to a multi-label head wrongly forces classes to compete.

What are the common bugs — double sigmoid, vanishing gradients — that come from mishandling logits and sigmoids?

The frequent ones are the double sigmoid (a sigmoid in the model plus a with-logits loss, giving tiny gradients and stalled learning), the manual-sigmoid-plus-BCE path (NaN once logits grow), thresholding on a raw logit as if it were a probability, and using softmax where sigmoid belongs on a multi-label head. The unifying check is asking, at each line, whether the tensor is a logit or a probability and whether the next operation expects that form.

When a training run stalls or goes NaN with no obvious cause, the logit/sigmoid boundary is worth ruling out first — it is a single-line fix in most codebases, and it sits directly underneath the feasibility question of whether a GAN-based approach will train stably at all.