Pick the wrong weight initializer and your first training run can diverge before it ever gets interesting. Not because the architecture is wrong, and not because the learning rate is off — but because the variance of activations drifts layer to layer until gradients either explode or vanish. LeCun initialization is one of the three variance-scaling schemes that exist to stop exactly this, and it does so by scaling weight variance by 1/fan_in. The catch most teams miss: it is not interchangeable with Xavier/Glorot or He. Each scheme is tuned to a different activation function, and using the wrong one is a silent way to make training slower and less stable than it needs to be. That is the whole point of this article. Initialization looks like a knob you can leave at the framework default, and most of the time you can — because the default usually matches the activation you are using. When you reach for a self-normalizing network with SELU, or a linear/tanh-like activation, the default may not be LeCun, and the mismatch shows up as sluggish convergence or step-zero instability. Getting the variance scaling right before the first full run is cheaper than debugging a divergent run afterwards. What does LeCun initialization actually do? Initialization schemes all try to solve one problem: keep the variance of the signal roughly constant as it flows through the layers of a network. If each layer amplifies the variance of its inputs, activations blow up by the output layer. If each layer shrinks it, activations collapse toward zero and gradients vanish. Either way, training stalls. LeCun initialization draws each weight from a distribution with variance 1/fan_in, where fan_in is the number of input connections into the neuron. In the Gaussian form, weights are sampled from a normal distribution with mean zero and standard deviation sqrt(1/fan_in). There is a uniform variant that produces the same variance from a bounded range. The reasoning is straightforward once you write out the variance of a weighted sum: if you sum fan_in independent terms, each contributing weight variance times input variance, the total variance is preserved only when the per-weight variance is 1/fan_in. That is the entire derivation, and it is why the scheme is sometimes just called “LeCun normal” or “LeCun uniform” in framework code — for example, tf.keras.initializers.LecunNormal in TensorFlow, or torch.nn.init combined with a fan_in mode in PyTorch. The assumption baked into that derivation is that the activation function does not itself change the variance much. That assumption holds for linear activations and, closely enough, for the self-normalizing regime that SELU was designed to produce. It does not hold for ReLU, which zeroes out roughly half its inputs and therefore halves the variance — which is precisely why He initialization exists. LeCun vs Xavier/Glorot vs He: when should you use each? The three schemes differ only in how they compute the target variance, and that difference maps directly onto which activation function you are using. This is the decision most teams get wrong by defaulting instead of matching. Initializer Variance scale Designed for Typical framework default context LeCun 1/fan_in SELU, self-normalizing nets; linear/tanh-like activations SELU layers (must set explicitly in most frameworks) Xavier / Glorot 2/(fan_in + fan_out) tanh, sigmoid, symmetric saturating activations Dense layers with tanh/sigmoid; often the framework default He / Kaiming 2/fan_in ReLU and ReLU variants (LeakyReLU, GELU-ish) ReLU-family conv and dense layers The pattern is worth stating as a rule of thumb: match the initializer’s variance assumption to what the activation does to variance. ReLU throws away half the signal, so He compensates with a factor of two. Xavier balances forward and backward variance across both fan-in and fan-out because tanh and sigmoid are symmetric and near-linear around zero. LeCun assumes the activation preserves variance, which is exactly what SELU is constructed to do — its scale and alpha constants are chosen so that, given LeCun-initialized weights, activations converge toward zero mean and unit variance. Pair SELU with He and you break that self-normalizing property; the network no longer sits at the fixed point SELU was designed around. If you take one thing from the comparison, it is that these are not ranked from worst to best. There is no “best initializer.” There is only the one whose variance assumption matches your activation. Getting that match right is a step-zero decision, not a hyperparameter you tune later. Why is LeCun the right match for SELU and self-normalizing networks? SELU — the Scaled Exponential Linear Unit — is unusual because its two constants are not free parameters. They are derived so the activation has a stable fixed point at zero mean and unit variance, but only under a specific initialization: zero-mean weights with variance 1/fan_in. That is LeCun initialization. The self-normalizing behavior that makes SELU attractive — activations that keep their statistics without batch normalization — is a mathematical consequence of the SELU constants and the LeCun variance scaling working together. Swap in a different initializer and you leave the region where the fixed-point analysis holds. The network may still train, but you have thrown away the reason to use SELU in the first place. This is why the original self-normalizing-network work specifies LeCun-normal initialization as a requirement, not a suggestion. In practice, when we see a self-normalizing network underperform, a mismatched initializer is one of the first things worth checking — it is a cheap check and a common cause. The same logic extends to networks built around linear or tanh-like activations where variance is approximately preserved. LeCun is a reasonable default there too, though Xavier is often used interchangeably for tanh because the difference between 1/fan_in and 2/(fan_in + fan_out) is small when fan-in and fan-out are similar. How does fan_in scaling keep activation variance stable across layers? Think of a single forward pass through a layer as a weighted sum of fan_in inputs. If the inputs each have variance v and the weights each have variance w, and both are independent and zero-mean, the output of the linear part has variance fan_in × w × v. For that output to also have variance v — so nothing amplifies or shrinks as you stack layers — you need fan_in × w = 1, which means w = 1/fan_in. That is LeCun. Extend that across, say, twenty layers. If each layer multiplies variance by a factor slightly above one, twenty layers compound it into a large number and activations saturate. Slightly below one, and they compound toward zero. The 1/fan_in scaling sets the per-layer factor to approximately one so the compounding cancels out. This is the same compounding logic behind He (which corrects for ReLU’s variance halving) and Xavier (which balances the forward and backward passes) — the schemes differ only in what correction factor they apply on top of the basic fan-in count. The dependency to keep in mind: this analysis assumes the activation function’s effect on variance is what the scheme expects. Change the activation and you change the required scaling. Change the numerics underneath the activation — and you can quietly change the effective variance the scheme was tuned to preserve. That is the part porting teams skip. Can porting to a new target invalidate your initializer’s assumptions? Yes, and this is the failure mode most teams do not anticipate because it does not show up on the platform where the model was developed. A run that converged cleanly on the origin platform can regress on the target for reasons that have nothing to do with the code and everything to do with the numerics underneath it. Variance-scaling initialization is a statistical argument that assumes the arithmetic behaves like real-number arithmetic. Move to reduced precision — FP16, BF16, or lower — and the effective variance of the initialized weights and of the activations they produce can shift from what the scheme intended. Rounding at low precision does not just add noise; it can bias variance, especially near the small-magnitude values that sqrt(1/fan_in) produces for large layers. If you are moving a workload to a target where precision changes, the numerical-behavior implications are worth understanding before you assume convergence carries over — the trade-offs are covered in our explainer on how 4-bit floating point works and when to use it. Precision is not the only lever. A different math library or BLAS backend on the new target can change accumulation order and rounding behavior, and aggressive compiler flags — fast-math in particular — can reorder or relax floating-point operations in ways that perturb the variance a scheme was tuned to preserve. We walk through which flags actually change numerical results in what -O3, -march, and fast-math actually change. The broader discipline of re-validating numerical behavior when a workload moves targets is the subject of our overview of what it means to move a workload to new hardware. Initialization assumptions belong on that re-validation checklist alongside the more obvious operator and kernel checks. If you build accelerated training and inference systems that have to survive a move to new GPU hardware, treat the initializer’s variance assumption as one of the things that can silently break in the port — not as a settled decision from the origin platform. How do you verify at step zero that initialization is behaving as intended? The cheapest verification happens before you commit to a full training run. You do not need to train to know whether variance scaling is working — you can measure it at initialization and after a single forward and backward pass. Step-zero initialization check: Measure per-layer activation variance after one forward pass on a representative batch. With correct initialization, variance should stay in the same order of magnitude across layers, not trend toward zero or blow up by the last layer. Measure per-layer gradient norms after one backward pass. Gradient norms that shrink geometrically from output to input (vanishing) or grow geometrically (exploding) signal a variance mismatch at initialization, not a learning-rate problem. Confirm the initializer matches the activation. SELU → LeCun; ReLU-family → He; tanh/sigmoid → Xavier. A mismatch here is the first thing to correct before touching anything else. On the target platform, repeat steps 1–2 after porting. If activation variance or gradient norms drift relative to the origin platform, suspect precision or math-library differences before suspecting the model. Log these metrics per run. Step-zero gradient-norm stability and epochs-to-convergence are the measurable signals that tell you whether the initialization decision paid off. Tracking them is part of the same discipline covered in our guide to what to track for GPU inference and training workloads. This check takes minutes and catches a class of problems that otherwise cost you a divergent run and hours of debugging. The measurable outcomes are concrete: epochs-to-convergence, gradient-norm stability across layers at step zero, and the number of training-run restarts you avoid by getting variance scaling right before the first full run — these are observed engineering signals from real training loops, not published benchmarks. FAQ What does working with LeCun initialization involve in practice? LeCun initialization draws each weight from a distribution with variance 1/fan_in — a normal distribution with standard deviation sqrt(1/fan_in), or an equivalent bounded uniform range. In practice it keeps the variance of activations roughly constant as signal flows through the layers, so activations neither blow up nor collapse toward zero across a deep stack. It is called LecunNormal / LecunUniform in framework code and is the initializer SELU networks are designed around. What is the difference between LeCun, Xavier/Glorot, and He initialization, and when should each be used? They differ only in the variance scale they target: LeCun uses 1/fan_in, Xavier/Glorot uses 2/(fan_in + fan_out), and He uses 2/fan_in. Use LeCun with SELU and self-normalizing or linear/tanh-like activations, Xavier with tanh and sigmoid, and He with ReLU-family activations. The rule is to match the initializer’s variance assumption to what the activation does to variance — there is no single best choice, only the correct match. Why is LeCun initialization the right match for SELU and self-normalizing networks? SELU’s scale and alpha constants are derived so activations converge to zero mean and unit variance, but only when weights are initialized with LeCun’s 1/fan_in variance. The self-normalizing property is a mathematical consequence of the SELU constants and LeCun scaling working together. Pair SELU with a different initializer and you leave the region where that fixed-point analysis holds, discarding the reason to use SELU in the first place. How does the fan_in variance scaling in LeCun initialization keep activation variance stable across layers? A layer’s linear output has variance equal to fan_in × weight_variance × input_variance. Setting weight variance to 1/fan_in makes the per-layer factor approximately one, so variance is preserved rather than compounding upward or downward across stacked layers. Without that scaling, small per-layer amplification or shrinkage compounds geometrically over many layers into saturated or vanishing activations. Can porting to a new target — different precision or math library — invalidate the assumptions a chosen initializer relied on? Yes. Variance-scaling initialization assumes arithmetic behaves like real numbers; reduced precision can bias the effective variance of small-magnitude initialized weights, and a different math library or aggressive fast-math flags can change accumulation order and rounding. A run that converged on the origin platform can regress on the target for these numerical reasons alone. Initialization assumptions belong on the numerical-behavior re-validation checklist when a workload moves hardware. How do you verify at step zero that your initialization is behaving as intended before committing to a full training run? Measure per-layer activation variance after one forward pass and per-layer gradient norms after one backward pass on a representative batch. Correct initialization keeps activation variance in the same order of magnitude across layers and avoids geometrically vanishing or exploding gradient norms. Confirm the initializer matches the activation, repeat the check on the target platform after porting, and log the metrics per run — the whole check takes minutes and catches problems that would otherwise cost a divergent run. Initialization is one of the quiet decisions that never announces itself as the cause when a run goes wrong. The variance argument that makes it work is simple, but it rests on assumptions about the activation function and the numerics underneath — and those assumptions are exactly what a port to new hardware can break. The failure class here is a numerical-behavior regression that survives every functional test and only shows up as a training run that will not converge. Re-validating the initializer’s variance assumption at step zero, on the target, is the check that keeps that failure from reaching a full run.
