LeCun Initialization Explained: Why Weight Init Governs Training Stability

LeCun initialization sets weight variance to 1/fan_in so signals neither vanish nor explode.

LeCun Initialization Explained: Why Weight Init Governs Training Stability
Written by TechnoLynx Published on 11 Jul 2026

LeCun initialization draws each weight from a zero-mean distribution with variance 1/fan_in, where fan_in is the number of input units in the weight tensor, and sets the biases to zero. In the normal form that means a standard deviation of sqrt(1/fan_in). In the uniform form it means a range of [-sqrt(3/fan_in), sqrt(3/fan_in)]. The point of the rule is to keep the variance of each layer’s output equal to the variance of its input, so that a deep stack neither shrinks nor magnifies the signal exponentially as it passes through.

Why initial weight variance decides whether a deep network trains

A forward pass multiplies the signal by a fresh weight matrix at every layer, so any per-layer gain or loss compounds with depth. For zero-mean, independent weights and inputs, the variance of a unit’s pre-activation is roughly fan_in × Var(w) × Var(x). To keep the forward signal from vanishing or exploding you therefore need fan_in × Var(w) = 1, which is exactly Var(w) = 1/fan_in. The same derivation run on the backpropagated gradients gives the mirror condition, fan_out × Var(w) = 1.

The practical consequence is blunt. Weights drawn too small lead to slow learning, and weights drawn too large lead to divergence, even though both choices break the symmetry that all-zero weights would leave in place. The deeplearning.ai notes reduce the goal to two rules of thumb: activations should have zero mean, and their variance should stay the same across every layer, so that the backpropagated gradient is never multiplied by values too small or too large on its way to the input.

Sun’s survey of optimisation for deep learning names gradient explosion and vanishing as a major challenge, with careful initialisation and normalisation as the main solutions. It adds that proper initialisation is very important for the algorithm to start training.

Worked example: what a gain of 2 or 0.5 per layer does

Take a purely linear network of 20 layers, all the same width, and ignore activations so that only the weight scale matters (an assumption made to isolate the mechanism, not a model of a real network). Each layer multiplies the signal variance by fan_in × Var(w).

Per-layer gain Variance after 20 layers What happens
2 2^20 ≈ 10^6 Signal explodes
1 (LeCun) 1^20 = 1 Signal arrives at the same scale it started
0.5 0.5^20 ≈ 10^-6 Signal vanishes

A factor that looks harmless at one layer becomes a six-orders-of-magnitude error at twenty.

LeCun normal initialization: where the 1/fan_in rule comes from

The rule traces back to Efficient BackProp by LeCun, Bottou, Orr and Müller, published in 1998 in Neural Networks: Tricks of the Trade. That paper recommends drawing weights from a zero-mean distribution with standard deviation m^(-1/2), where m is the fan-in of the unit. The stated aim, for the sigmoid the paper used, was to keep each unit’s output at roughly 1. Şimşekli and colleagues (2021) trace the variance-1/d scheme to tanh activations (Kalman and Kwasny, 1992) and note that the literature now calls it “LeCun initialization”.

The reason a tanh-style unit tolerates a linear analysis is its slope of about one near zero: with small pre-activations, tanh(z) ≈ z. Sun’s survey describes both LeCun and Xavier initialisation as designed for sigmoid-type activations that have slope 1 in this linear regime.

The normal and uniform variants carry the same variance. A uniform distribution on [-a, a] has variance a²/3, so setting a²/3 = 1/fan_in gives a = sqrt(3/fan_in), the limit Keras uses for LecunUniform. Keras’s LecunNormal draws from a truncated normal centred on zero with stddev sqrt(1/fan_in), and JAX’s lecun_normal is likewise defined as variance scaling with scale 1.0, mode “fan_in” and a truncated normal.

For a dense layer, fan_in is the input dimension; for a convolution, it is input channels times kernel height times kernel width.

How does LeCun initialization differ from Xavier and He?

All three schemes share one goal, preserving signal magnitude across layers, and differ in which fan they scale by and which activation they target. LeCun preserves the forward signal for linear and tanh-like units. He (Kaiming) initialisation doubles LeCun’s variance to compensate for ReLU.

Scheme Weight variance Normal stddev (Keras) Designed for Balances
LeCun 1/fan_in sqrt(1/fan_in) Linear, tanh-type Forward signal
Xavier / Glorot 2/(fan_in + fan_out) sqrt(2/(fan_in + fan_out)) tanh, sigmoid Forward and backward together
He / Kaiming 2/fan_in sqrt(2/fan_in) ReLU family One direction only

The factor of 2 in He initialisation has a simple cause. For a zero-mean symmetric input, ReLU outputs zero with probability 1/2, so it cuts half the signal on average, and doubling the weight variance puts it back. Hanin and Rolnick (2018) report that, for ReLU networks, drawing weights from a symmetric distribution with variance 2/fan-in avoids the exploding or vanishing mean activation length failure mode.

Xavier’s motivation was different. Glorot and Bengio studied why gradient descent from random initialisation did so poorly in deep networks and found that the logistic sigmoid, because of its non-zero mean, can drive the top hidden layer into saturation. They reasoned that training gets harder when the singular values of each layer’s Jacobian are far from 1, and proposed their scheme to keep them closer. He initialisation, by contrast, does not try to balance forward and backward propagation the way Xavier does; it balances just one.

Which direction should you care about? For rectifier networks, the 2025 arXiv study Weight Initialization and Variance Dynamics in Deep Neural Networks and Large Language Models argues it is usually more important to keep the forward signal stable, and reports that fan-in He initialisation converged faster and more stably than Xavier in its experiments. That is one author’s controlled experiments, not a universal ranking, but it agrees with the variance argument.

One naming trap is worth knowing. The deeplearning.ai initialisation notes label N(0, 1/n^[l-1]), which is LeCun’s 1/fan_in variance, as “Xavier initialization”, while Keras reserves the Glorot name for the 2/(fan_in + fan_out) form. When a paper or a colleague says “Xavier”, check which variance they actually mean before comparing results.

Why LeCun normal is the initializer SELU expects

LeCun normal is the initialiser to be used with SELU, the scaled exponential linear unit of self-normalising neural networks (SNNs). Klambauer and colleagues (2017) designed SELU so that activations automatically converge towards zero mean and unit variance, giving the effect of batch normalisation without explicit normalisation layers. Their weight initialisation sets zero mean and τ = 1 per unit, and the PyTorch route gives each weight variance 1/N, which corresponds to LeCun normal.

The pairing matters because the self-normalising property rests on a stable fixed point. SELU’s constants are chosen so that, given zero-mean, unit-variance inputs and weights at this scale, the output is again zero mean and unit variance. The paper proves that this fixed point is stable and attracting, so activations pulled away from it drift back as they pass through layers. SELU damps variance for negative net inputs and increases it for positive ones, so a layer with too much variance passes on less and a layer with too little passes on more. From that, the authors show the activation variance is bounded above, ruling out exploding gradients, and bounded below, ruling out vanishing ones.

Start the weights at a different scale and the network begins away from the point the proofs are built around. The Keras SELU documentation says this directly: the constants preserve mean and variance between layers only when the weights are initialised correctly with LecunNormal and the number of input units is large enough, and SELU is to be used together with LecunNormal and with AlphaDropout rather than regular dropout. The SELU paper also notes that SNNs cannot be derived with ReLU, sigmoid, tanh or leaky ReLU units, because the activation needs negative and positive values, saturation regions and a slope greater than one.

Setting LeCun initialization in Keras, PyTorch and JAX

Keras exposes LeCun initialisation by name, passed through the layer’s kernel_initializer argument:

import keras

layer = keras.layers.Dense(
    256,
    activation="selu",
    kernel_initializer=keras.initializers.LecunNormal(),
    bias_initializer="zeros",
)

JAX provides jax.nn.initializers.lecun_normal(), a specialisation of variance_scaling with scale 1.0, fan-in mode and a truncated normal. Keras’s general VarianceScaling initialiser with scale=1.0 and mode="fan_in" gives the same stddev, sqrt(1/n).

The route the SNN authors give is kaiming_normal_ with mode="fan_in" and nonlinearity="linear", which corresponds to LeCun normal because fan-in is used with a gain of 1. The torch.nn.init documentation warns against the obvious-looking nonlinearity="selu": for self-normalising networks you should pass "linear" instead, which gives the initial weights the 1/N variance needed for a stable fixed point in the forward pass.

The SNN authors recommend initialising the linear and convolutional layers of SELU networks this way:

import torch.nn as nn

def lecun_normal_init(m):
    if isinstance(m, (nn.Linear, nn.Conv1d, nn.Conv2d, nn.Conv3d)):
        nn.init.kaiming_normal_(m.weight, mode="fan_in", nonlinearity="linear")
        if m.bias is not None:
            nn.init.zeros_(m.bias)

model.apply(lecun_normal_init)

When the architecture depends on it, as SELU networks do, set the initialiser explicitly like this. One difference remains between frameworks: Keras and JAX draw from a truncated normal, while PyTorch documents kaiming_normal_ simply as a Kaiming normal distribution.

Initialisation is necessary for stable training, not sufficient

LeCun initialization fixes the starting point, and training then moves the weights away from its assumptions. Sun’s survey notes that networks deeper than about ten layers usually also need normalisation layers and skip connections. It also shows that, even with He initialisation, the expected empirical variance of the signal grows on the order of exp(L/d) for depth L and width d, so for a fixed width, adding depth can make signal propagation unstable. For activations other than ReLU, the survey points out, the right variance has to be worked out case by case, for example from the fixed points of a mean-field variance recursion.

In transformers, the same 2025 study describes LayerNorm limiting forward variance drift and AdamW’s preconditioning and weight decay damping gradient variance during backpropagation. Its practical advice is to initialise transformer projections with a small standard deviation, monitor per-layer standard deviations and gradient norms, and adjust warm-up, residual scaling or weight decay if layers fail to expand or stagnate. Once you are there, the deep learning optimizer and the normalisation scheme carry much of the stability that initialisation alone carried in a plain feed-forward stack. The wider set of habits that keep a run healthy (data checks, learning-rate schedules, monitoring) is covered in our best practices for training deep learning models.

Variance preservation also rests on an assumption that is being questioned: Şimşekli and colleagues (2021) point out that SGD can produce heavy-tailed weight distributions with potentially infinite variance, and propose schemes that preserve lower moments, recovering LeCun and Kaiming initialisation as the limit. For most practitioners that is a research direction rather than a reason to change defaults. The question that matters day to day is narrower: does your initialiser’s variance assumption match your activation?

Back See Blogs
arrow icon