Reach for the L2 penalty because the tutorial used it, and you may be quietly discarding the one behaviour that would have solved the problem in front of you. L1 and L2 look like the same knob — add a penalty term, turn down overfitting — but they do structurally different things. L1 (Lasso) drives coefficients to exactly zero and performs feature selection. L2 (Ridge) shrinks weights smoothly toward zero without ever eliminating them. That difference is not cosmetic, and picking the wrong one wastes model capacity on sparse-signal problems or throws away useful correlated features on dense ones.

The confusion is understandable. Both penalties get bolted onto the same loss function, both are controlled by a strength parameter usually called lambda or alpha, and both reduce the gap between training error and validation error. If your mental model stops at “regularization fights overfitting,” the two look substitutable. They are not, and the reason lives in the geometry of the constraint they impose.

What does regularization actually do to a model?

Start with the failure it is meant to prevent. A model with more parameters than the signal warrants will fit noise in the training data — it memorizes idiosyncrasies that do not generalize. You see this as a widening spread between training error and held-out validation error: the model looks excellent on data it has seen and mediocre on data it has not. That spread is the generalization gap, and it is the quantity regularization is designed to shrink.

Regularization does this by adding a penalty on the size of the model’s weights to the loss the optimizer minimizes. Instead of minimizing prediction error alone, the model minimizes prediction error plus some cost for having large coefficients. The optimizer is now negotiating: fit the data well, but do not pay too much for it in weight magnitude. This is the same bias-variance tension every practitioner meets. A little penalty adds bias but cuts variance, and if the variance reduction outweighs the added bias, held-out performance improves.

The part most explanations skip is that how you measure “size of the weights” changes everything. L1 measures it as the sum of absolute values. L2 measures it as the sum of squares. Those two ways of counting weight magnitude produce solutions with completely different structure — and that is the whole story.

What is the mathematical difference between the L1 and L2 penalties?

The penalties differ only in a norm, but the consequences are large. For a weight vector w, the L1 penalty adds lambda * sum(|w_i|) to the loss; the L2 penalty adds lambda * sum(w_i^2). In scikit-learn, Lasso implements the first and Ridge implements the second; in PyTorch you get L2 nearly for free through the weight_decay argument on optimizers like SGD and Adam.

The behavioural difference comes from the gradient of each penalty. The L2 penalty’s gradient is proportional to the weight itself, so a large weight gets pushed down hard and a small weight barely gets nudged. As a weight approaches zero, the force pushing it further shrinks proportionally — so L2 asymptotes toward zero but essentially never arrives. The L1 penalty’s gradient is a constant magnitude regardless of how small the weight already is. That constant push does not relent near zero, so it can drive a coefficient the rest of the way to exactly zero and hold it there.

Geometrically, this is the diamond-versus-circle picture. If you draw the region where the penalty stays under a fixed budget, L1 traces a diamond (with sharp corners on the axes) and L2 traces a circle. The optimal solution lands where the loss contours first touch that region. A diamond’s corners sit on the axes, so solutions are pulled onto axes — meaning some coefficients become exactly zero. A circle has no corners, so the solution touches it at a generic point where every coefficient is small but nonzero. The corners are the entire reason L1 produces sparsity and L2 does not.

Why does L1 produce sparse models while L2 does not?

Sparsity is not a side effect you tune for — it is a direct consequence of that constant gradient and those diamond corners. When L1 zeroes a coefficient, the corresponding feature is dropped from the model entirely. This is feature selection performed as part of training, not as a separate preprocessing step. A model that starts with 500 candidate features may end up using 150, with the other 350 coefficients set to exactly zero.

That behaviour is worth real money in production. Correct regularization choice measurably reduces the generalization gap, and in the L1 case it cuts the number of features the model has to compute at inference — often shrinking model footprint by roughly 30–60% with negligible accuracy loss (an observed pattern across sparse-signal tabular work, not a single benchmarked figure). On sparse-signal problems, where most candidate features carry no real information, L1 removes the irrelevant ones that would otherwise degrade held-out AUC by several points. The reduced feature count also means fewer inputs to collect, store, and validate — a maintenance win that outlasts the model itself.

L2 keeps every feature. It shrinks the coefficients of unimportant features toward zero but never sets them to zero, so the model stays dense. On problems where the signal is genuinely spread across many correlated inputs — where most features contribute a little — that density is a feature, not a bug. L2 does not force you to pick a single winner among correlated predictors.

When should you choose L1, L2, or Elastic Net?

The choice is driven by what you believe about your feature space, not by which one a framework defaults to. Use the following as a first-pass rubric, then confirm empirically.

Situation	Prefer	Why
Many candidate features, few expected to matter (sparse signal)	L1 (Lasso)	Zeroes irrelevant coefficients; performs feature selection during training
Signal spread across many features, all plausibly relevant (dense signal)	L2 (Ridge)	Shrinks smoothly without discarding useful features
Groups of highly correlated features you want to keep together	L2 or Elastic Net	L1 alone tends to pick one from a correlated group arbitrarily
Sparse signal and correlated features	Elastic Net	Combines L1 selection with L2’s stable handling of correlation
You want interpretability from a small feature set	L1 (Lasso)	Fewer nonzero coefficients are easier to explain and audit
You mainly want a smaller generalization gap, feature count irrelevant	L2 (Ridge)	Reliable variance reduction without structural side effects

Elastic Net is the deliberate compromise: it applies both penalties, controlled by a mixing ratio. It matters most when your data is both sparse and correlated — a common combination in real tabular problems. Pure L1 on correlated features has a known weakness: faced with a group of highly correlated predictors, it tends to keep one and zero the rest, and which one it keeps can be unstable across resamples of the data. Elastic Net’s L2 component stabilizes that behaviour, keeping correlated groups intact while still driving genuinely irrelevant features to zero. If you have ever seen a Lasso model’s selected feature set flip between two training runs on nearly identical data, correlated features and pure L1 are usually the culprit — that is the case where reaching for Elastic Net pays off.

How do you tune the regularization strength in practice?

Neither penalty helps if lambda is wrong. Too small and the penalty does nothing — you keep the overfitting you were trying to remove. Too large and the model underfits, biased so heavily toward small weights that it cannot represent the real signal. The right value is found empirically, and the honest answer is that it depends on your data.

The standard method is cross-validation over a grid of lambda values, typically spaced logarithmically because the useful range spans orders of magnitude. Scikit-learn ships LassoCV, RidgeCV, and ElasticNetCV precisely because sweeping this parameter is routine work. Watch the training and validation error curves together as lambda changes: the goal is the point where validation error is lowest, which is usually where the two curves are close but validation error has not yet started climbing from underfitting. A useful diagnostic is to plot the number of nonzero coefficients against lambda for L1 — you can watch features drop out and choose a point that trades feature count against validation error explicitly. This is the same discipline we bring to any hyperparameter that governs model behaviour; it sits close to the reasoning in deep learning optimizers and how to choose one, where the “right” setting is likewise a property of the data, not a default.

One practical caution: regularization penalizes weight magnitude, so features on wildly different scales get penalized unfairly. A feature measured in the thousands and one measured in fractions will not be treated comparably unless you standardize inputs first. Skipping standardization is one of the most common reasons a correctly chosen penalty produces disappointing results.

FAQ

How does L1 and L2 regularization work in practice?

Both add a penalty on weight magnitude to the loss the optimizer minimizes, so the model trades a little prediction accuracy on training data for smaller weights and better generalization. In practice this narrows the gap between training and validation error. The difference is that L1 can eliminate features entirely while L2 only shrinks them, which changes what the trained model looks like and what it costs at inference.

What is the mathematical difference between the L1 (Lasso) and L2 (Ridge) penalties?

L1 adds the sum of absolute weight values (lambda * sum(|w_i|)) to the loss; L2 adds the sum of squared weights (lambda * sum(w_i^2)). The L2 gradient is proportional to each weight, so it shrinks smoothly and asymptotes toward zero, while the L1 gradient is a constant magnitude that can push a coefficient all the way to exactly zero. Geometrically, L1’s constraint region is a diamond with corners on the axes and L2’s is a circle without corners.

Why does L1 produce sparse models with weights driven to exactly zero while L2 does not?

L1’s constant-magnitude gradient does not relent as a weight approaches zero, so it drives coefficients to exactly zero and holds them there; its diamond-shaped constraint region has corners on the axes where solutions land. L2’s gradient shrinks proportionally near zero and its circular constraint region has no corners, so weights become small but never exactly zero. That is why L1 performs feature selection and L2 keeps a dense model.

When should you choose L1 over L2, and when is Elastic Net the right compromise?

Choose L1 when you have many candidate features but expect only a few to matter, or when you want a small, interpretable feature set. Choose L2 when the signal is spread across many plausibly relevant features and you mainly want to reduce the generalization gap. Elastic Net is the compromise when your data is both sparse and correlated — L1 handles the selection while L2 stabilizes how correlated feature groups are treated.

How do L1 and L2 relate to overfitting and the bias-variance tradeoff?

Both penalties fight overfitting by adding bias (pulling weights toward zero) in exchange for lower variance, which shrinks the spread between training and validation error. If the variance reduction outweighs the added bias, held-out performance improves. Too little regularization leaves the overfitting in place; too much biases the model so hard it underfits.

How do you set and tune the regularization strength (lambda / alpha) in practice?

Sweep lambda over a logarithmically spaced grid using cross-validation — scikit-learn provides LassoCV, RidgeCV, and ElasticNetCV for exactly this. Watch training and validation error together and pick the value where validation error is lowest before it climbs from underfitting; for L1 you can also plot nonzero-coefficient count against lambda. Standardize features first, since penalizing raw magnitude treats differently scaled features unfairly.

How does regularization interact with correlated features, and which penalty handles them better?

Pure L1 tends to keep one feature from a correlated group and zero the rest, and which one it keeps can be unstable across resamples. L2 handles correlated features more gracefully, shrinking them together rather than picking arbitrarily. Elastic Net gives you the best of both: L1’s selection plus L2’s stable treatment of correlated groups.

Regularization is one of the earliest places the broader question shows up: what can this model actually be asked to learn, and how do you constrain it so the answer is reliable rather than lucky? Choosing L1 versus L2 is a small decision with a clear structural logic, but it belongs to the same discipline that governs feasibility across a whole generative AI build — knowing what a model can and should do, and scoping it deliberately. For a compact, side-by-side treatment of the two penalties without the tuning detail, see our companion piece on how L1 vs L2 regularization work and what they mean in practice. The failure mode to watch for is not choosing wrong once — it is never revisiting the choice when the feature space changes underneath the model.