A training run stalls. Loss plateaus three epochs in and refuses to move. The obvious next step is to open the dashboard, stare at the loss curve, and start another hyperparameter sweep. That instinct is exactly where teams lose days. wandb.watch() is the Weights & Biases hook that logs a model’s gradients and parameters over the course of training. Most people reach for it to “see the training curves” — but its real value is diagnostic. Read correctly, the gradient and parameter histograms it produces answer a much sharper question than “is training going well?”: is this model failing because the engineering is wrong, or because the problem itself is unresolved research? Those two failure modes look identical on a loss plot. They look completely different in a gradient histogram. What should you know about wandb.watch() in practice? At its core, wandb.watch(model) registers hooks on a PyTorch (or Keras) model. Under the hood, it attaches backward hooks to the parameters and, optionally, forward hooks to the modules. On each logging step it captures the current values of the tensors it is tracking, bins them into histograms, and ships those to the W&B backend where they render as distributions over training time. The signature is deliberately small. You pass the model, choose what to log (log="gradients", log="parameters", or log="all"), and set how often (log_freq, expressed in training steps). A typical call looks like this: import wandb wandb.init(project="training-diagnosis") model = MyModel() wandb.watch(model, log="all", log_freq=100) That single line is where the misunderstanding starts. People treat it as a switch — flip it on, get pretty charts, hope instability resolves itself. What you have actually done is instrument the two quantities that tell you why a network is or is not learning. The switch metaphor is the problem, because it stops you from asking the histograms a question. We see this regularly in engagements where a team has been sweeping learning rates for a week. The instrumentation was on the whole time. Nobody was reading it. What does wandb.watch() actually log — gradients, parameters, or both? It logs whatever you tell it to, at whatever cadence you set, and the defaults matter more than most people notice. log="gradients" (the default) captures the gradient of each tracked parameter tensor after the backward pass. These are the histograms that reveal exploding or vanishing gradients. log="parameters" captures the weight values themselves. Watching how the parameter distribution shifts — or fails to shift — over epochs tells you whether a layer is actually learning. log="all" logs both. For diagnosis, this is usually what you want, because the two signals are complementary: gradients tell you what the optimizer is trying to do, parameters tell you what actually changed. log_freq controls how often histograms are captured, counted in training steps rather than epochs. A common default is every 100 or 1000 steps. This is the single biggest overhead lever: capturing and binning tensors on every step of a large transformer adds measurable wall-clock cost to training. In configurations we have profiled, dropping log_freq from every step to every few hundred steps removes almost all of the instrumentation overhead while preserving enough temporal resolution to spot instability (an observed pattern across debugging engagements, not a published benchmark). The histograms are a diagnostic sample, not a continuous trace — treat them accordingly. How do I read gradient histograms to tell a bug from a research problem? This is the whole point, and it is where naive and expert use diverge. The naive reading looks at a gradient histogram and asks “does this look healthy?” The expert reading asks “which of the two failure classes am I looking at?” There are exactly two, and they demand opposite responses: The engineering is wrong. Gradients are exploding, vanishing, or a layer’s gradient distribution is pinned at zero. The data pipeline is feeding garbage. An activation is saturated. These have known, bounded fixes — clipping, normalization, initialization, a corrected loader. The problem is unresolved research. Instrumentation is clean. Gradients flow, parameters move, nothing is pathological — and the model still cannot learn a reliable baseline. No amount of hyperparameter tuning will close that gap, because the gap is not in the implementation. The histograms are what let you tell these apart before you spend another week. A stalled loss curve is consistent with both. A gradient histogram is not. A worked diagnostic pass Assume a run where validation loss plateaus early. Here is the bounded inspection we would run against the wandb.watch(log="all") output, in order: Signal in the histogram What it indicates Class First response Gradient magnitudes growing across steps, tails widening toward large values Exploding gradients Engineering Gradient clipping; check learning rate; inspect for a bad batch Gradient histogram collapsing toward zero in early layers Vanishing gradients Engineering Residual connections, normalization, revisit initialization/activation One module’s gradient distribution flat at zero throughout Dead layer / disconnected module Engineering Confirm the module is in the graph; check requires_grad and the forward path Parameter histograms static across epochs while gradients are non-zero Optimizer not applying updates (LR, param groups, frozen weights) Engineering Verify optimizer sees the params; check LR schedule and freezing logic Activations saturated (bimodal, pinned at bounds) Saturated non-linearity starving gradients Engineering Rescale inputs, revisit activation choice, add normalization Gradients flow, parameters move, distributions look textbook — loss still won’t drop below a useless baseline Clean instrumentation, no learnable signal Research Stop tuning. Revisit problem framing, labels, or feasibility The value of that table is that every engineering row has a fix you can attempt today. The final row does not — and recognizing it early is what saves the budget. If you have exhausted the engineering rows and still land on the last one, that is not a failure of the instrumentation. It is the instrumentation doing its job. Which signals mean a fixable engineering defect? Four patterns cover most of what turns up. Exploding gradients show histograms whose spread widens and shifts toward large magnitudes step over step; the fix is clipping and a look at the learning rate. Vanishing gradients show early-layer gradient distributions collapsing toward zero while later layers still move — a classic depth/initialization problem. Saturated activations show forward-pass distributions pinned at their bounds, which starves the gradients feeding back through them. Static parameter norms — weights that do not move even though gradients are non-zero — almost always point at the optimizer: a parameter group that never got registered, an accidental requires_grad=False, or a schedule that zeroed the learning rate. None of these require research. Each is a defect with a known class of fix. The reason logging them matters is speed of attribution: a gradient histogram tells you which of the four you have in one glance, whereas a loss curve tells you only that something is wrong. That is the difference between a bounded inspection and an open-ended sweep, and it is the same discipline we apply when scoping a hyperparameter sweep in an AI proof of concept — instrument first so the sweep answers a question rather than replacing one. Why is a clean-but-stuck run evidence of a research question? Here is the reframe that changes how a project is run. When gradients flow cleanly, parameters visibly update, activations are well-behaved, and the model still cannot reach a reliable baseline, you have ruled out the engineering. What remains is the task itself: the signal may not be learnable from the data you have, the labels may be inconsistent, or the formulation may be wrong. That distinction has direct commercial consequences. A fixable defect is production work with a known end. An unresolved research question is bounded R&D with an uncertain end — and continuing to sweep hyperparameters against it is spend with no expected return. Clean instrumentation is the evidence that lets you make that call deliberately instead of by exhaustion. In our R&D consulting engagements, this is precisely the boundary an early risk assessment is built to surface: training-time gradient and parameter evidence is what separates “we can fix this” from “this needs research we should scope before we commit.” That is also why we treat W&B not as a dashboard but as part of the evidence trail. Alongside gradient logging, structured W&B Tables for logging experiment data in a first MLOps deployment and W&B Sweeps for hyperparameter search close the loop — but the sweep only makes sense after the gradient histograms have confirmed you are on the fixable side of the boundary. FAQ What does working with wandb watch involve in practice? wandb.watch(model) registers backward hooks (and optional forward hooks) on a PyTorch or Keras model, samples the tracked tensors at a set step cadence, bins them into histograms, and logs them to the W&B backend. In practice it instruments the two quantities — gradients and parameters — that reveal why a network is or is not learning, rather than just whether the loss is moving. What does wandb.watch() actually log — gradients, parameters, or both — and how often? It logs what you specify via log=: "gradients" (the default), "parameters", or "all" for both. Cadence is set by log_freq, counted in training steps, and defaults to a value like every 100–1000 steps. For diagnosis, log="all" is usually best because gradients show what the optimizer is trying to do and parameters show what actually changed. How do I read gradient histograms to tell whether a training failure is an engineering bug or a research problem? Ask which of two classes you are in, not whether the plot “looks healthy.” Exploding, vanishing, saturated, or zero-pinned distributions are engineering defects with known fixes. When gradients flow, parameters move, and distributions look textbook but the model still won’t reach a reliable baseline, the failure is not in the implementation. Which parameter and gradient signals indicate a fixable engineering defect? Exploding gradients (widening magnitudes), vanishing gradients (early-layer collapse to zero), saturated activations (distributions pinned at bounds), and static parameter norms (weights not moving despite non-zero gradients). Each maps to a known fix — clipping, normalization/initialization, activation or input rescaling, and checking the optimizer’s parameter groups and freezing logic respectively. When clean instrumentation shows a model still won’t learn a reliable baseline, why is that evidence of a research question rather than a bug? Because clean gradient and parameter histograms rule out the engineering failure modes: the optimizer is applying updates and nothing is pathological. What remains is the task — learnability of the signal, label quality, or problem formulation — none of which hyperparameter tuning can fix. That makes it bounded R&D to scope, not production debugging. What are the common pitfalls of wandb.watch(), and how do I avoid them? The three recurring ones are logging overhead (mitigate by raising log_freq so you sample every few hundred steps rather than every step), missing modules (confirm the layers you care about are actually in the graph and tracked), and misread histograms (read them as a two-class diagnosis, not a vague health check). Treat the histograms as a diagnostic sample, not a continuous trace. The question worth asking before the next sweep The most expensive mistake in a stalled training run is not a bad hyperparameter — it is spending a week tuning a problem no tuning can close. Before you launch another sweep, look at the gradient and parameter histograms and answer one thing: have I confirmed I am on the fixable side of the engineering-versus-research boundary? If the instrumentation is clean and the model still won’t learn, that is not a reason to tune harder. It is the evidence that the problem needs research scoped, not more production spend — the same signal an early risk assessment is designed to catch before the budget goes with it.
