A training run finishes, the loss curve looks clean, and a wandb sweep shows one config edging out the rest. Someone screenshots the chart, drops it in Slack, and the model ships. Six weeks later a reviewer asks why that model — which dataset version, which eval split, which threshold — and nobody can reconstruct the answer. The charts are still there. The evidence is not. That gap is the whole point of this article. Weights & Biases (wandb) is an excellent experiment tracker, but a monitoring harness needs something a tracker does not give you by default: durable, versioned artefacts that map onto specific harness sections a reviewer can sign against. A dashboard shows you what happened during training. A harness has to prove why this model was fit to ship — and keep proving it after the model is retrained. Those are different jobs, and the difference is where most teams quietly lose their audit trail. What does “harness-grade evidence” actually mean here? Start with the word most people skip past. A production AI monitoring harness is not a collection of dashboards; it is a structured record an engineering reviewer signs against before release and re-checks against drift over time. It has named sections — an eval-harness section, a drift-telemetry section, a release-readiness decision — and each section demands evidence that is reproducible and attributable to a specific model version. wandb can feed every one of those sections, but only if you instrument it deliberately. Left to defaults, wandb records the training story beautifully and the reliability story barely at all. The fix is not more logging. It is logging the right things as versioned objects instead of ephemeral scalars. The mental shift is this: treat wandb runs, model artefacts, and evaluation tables as the reproducible substrate behind the harness’s eval-harness and drift-telemetry sections, rather than as a parallel, unaccountable record that lives next to the harness and contradicts it at handoff. This is the same principle we develop in how an experiment tracker feeds a production AI monitoring harness — wandb is one concrete instance of that pattern, and the discipline generalizes to any tracker. The four things a wandb run must carry to be signable A run that only carries loss and accuracy is a training artefact. A run that a reviewer can sign against carries four kinds of object, and each one maps to a harness section. wandb object What to log Harness section it feeds Why it must be versioned Dataset artefact Train/val/test splits as a wandb.Artifact with a content hash Eval-harness (provenance) So “which data” is answerable in minutes, not archaeology Model artefact Checkpoint + config + framework version, tagged with the producing run Release-readiness So the shipped weights trace to exactly one run Evaluation table Per-class / per-slice metrics via wandb.Table, not just a scalar Eval-harness (coverage) So a reviewer sees where the model is weak, not just the headline number Baseline metric set The metrics tagged as the signed reference point at release Drift-telemetry So future drift is measured against a fixed, agreed anchor The distinction between the third and fourth rows is the one teams get wrong most often. Logging eval/accuracy = 0.94 gives you a chart. Logging a wandb.Table of per-slice results and then tagging one run’s metric set as the release baseline gives you something a drift monitor can compare against later. In our experience across reliability engagements, the single highest-leverage change is promoting the release-time metric set to a named, immutable baseline — this is an observed pattern, not a benchmarked figure, but it is remarkably consistent. How do you log experiments and sweeps so they survive to handoff? Reproducibility at handoff is mostly about breaking the chain of implicit context. A run reproduces when someone who was not in the room can rebuild the “why” from the record alone. Three habits carry most of that weight. First, log the dataset as an artefact and consume it by reference (run.use_artifact(...)), so the run’s lineage graph names its exact inputs rather than a mutable file path. Second, capture the environment — framework versions, CUDA build, git SHA — into the run config, because a metric with no environment is a metric you cannot defend when it moves. Third, for sweeps, log the search space and the objective into the sweep config itself, so the winning configuration is explained by a recorded space rather than by tribal memory of “we tried a few things.” For teams already on PyTorch Lightning, most of this comes almost free through the logger integration — we walk through the specifics in capturing training telemetry with PyTorch Lightning and W&B that feeds drift monitoring. The framework matters less than the discipline: a scalar logged and forgotten is training noise; the same scalar logged against a versioned artefact and a signed baseline is evidence. Release-readiness: what to log versus what is just training noise Not everything worth watching during training belongs in a release-readiness review. Reviewers drown in gradient-norm plots and learning-rate schedules that have no bearing on whether the model is fit to ship. The filter is simple to state and hard to apply: log for the reviewer’s decision, not for the training loop’s diagnostics. Use this as a triage rubric before a release review: Ship-decision evidence (keep, promote to the harness): per-slice eval tables, calibration on the held-out set, the exact dataset artefact version, the model artefact hash, and the metric set you are proposing as the signed baseline. Diagnostic-only (keep in wandb, do not put in front of the reviewer): loss curves, gradient norms, LR schedules, throughput. Useful for debugging a regression; irrelevant to whether this checkpoint is signable. Noise (do not log as evidence at all): transient per-step scalars with no held-out meaning, and any metric computed on data the model also trained on. The ROI here is concrete. When eval and regression results are versioned and reproducible, the time to answer “why did we ship this model” drops from days of reconstruction to minutes of lookup, and the audit-evidence pack becomes a byproduct of training rather than a separate write-up effort. That release-readiness decision does not live only in the harness — it also feeds the AI-infrastructure vertical lens, which is why wandb-logged eval and drift metrics should map cleanly onto the readiness gate that infrastructure teams own. The gate-placement question across the full lifecycle is covered in where reliability gates belong at each stage of an ML pipeline. Versioning runs and baselines so drift has a fixed reference Drift is only measurable against something. If your baseline is “the metrics from whichever run we shipped, probably,” you do not have a drift monitor — you have a moving target. The wandb primitive that fixes this is the artefact alias. When a model passes release-readiness, promote its model artefact and its evaluation table to a stable alias — production or baseline-v3 — rather than leaving them pinned to a run ID nobody will remember. The alias is the signed reference point. Every subsequent retrain logs its own run, produces its own eval table, and is compared against the aliased baseline, so the drift-telemetry section always has a named anchor. The mechanics of treating W&B artefacts as versioned, sign-off-grade evidence are worked through in detail in versioning condition-monitoring evidence with W&B artefacts, and the same versioning shape applies to any monitored model. This is also what keeps the record honest after a retrain. Point the new run at the same dataset-artefact lineage where the splits are unchanged, re-run the eval table, and either the numbers hold against the baseline or the delta is itself the evidence a reviewer needs. Retraining does not invalidate the harness; it appends a comparable entry to it — provided you kept the baseline fixed and the eval reproducible. Where wandb stops being harness-grade evidence A dashboard is not a sign-off. This is the boundary that matters most, and it is worth stating plainly rather than discovering it during an audit. wandb records what you told it to record. It does not, by itself, enforce that the eval ran on held-out data, that the baseline was agreed rather than assumed, or that the reviewer who “signed” actually reviewed the right table. Those are process guarantees the harness has to carry — the tracker supplies the evidence, the harness supplies the accountability. A pretty sweep chart with no held-out eval table and no fixed baseline is exactly the naive artefact this whole article is trying to replace. For teams building this out as a formal deliverable, the reproducible-evidence pattern connects directly to our work on production AI reliability, where wandb outputs become the substrate behind the harness’s eval and drift sections rather than a parallel record. Where wandb also stops short: it is a tracker, not a policy engine. It will happily version a baseline that was chosen for the wrong reasons. The judgement about what to sign against — which slices matter, which drift threshold triggers re-review — is engineering work that no logging tool performs for you. FAQ How do you use wandb in a production AI reliability workflow? Treat wandb runs, artefacts, and eval tables as the reproducible evidence behind a monitoring harness, not as a standalone training dashboard. Log datasets and models as versioned artefacts, capture per-slice evaluation as tables, and tag a release-time metric set as the signed baseline. The outputs then feed the harness’s eval-harness and drift-telemetry sections directly. How do you log experiments, sweeps, and evaluation runs in wandb so they are reproducible at handoff? Log the dataset as an artefact and consume it by reference so the run’s lineage names its exact inputs, capture the environment (framework versions, git SHA) in the run config, and record the search space and objective in the sweep config. A run reproduces when someone who was not present can rebuild the “why” from the record alone, which means breaking every chain of implicit context. How do wandb model and dataset artefacts map onto the eval-harness and drift-telemetry sections of a monitoring harness? Dataset artefacts and per-slice evaluation tables feed the eval-harness section by supplying provenance and coverage. Model artefacts feed the release-readiness decision by tracing shipped weights to exactly one run. The metric set tagged as the release baseline feeds the drift-telemetry section as the fixed reference every later retrain is compared against. What should you log in wandb to support a release-readiness review versus what is just training noise? Log the ship-decision evidence — per-slice eval tables, held-out calibration, dataset artefact version, model hash, and the proposed signed baseline. Keep loss curves, gradient norms, and throughput as diagnostics but out of the reviewer’s view. Do not log transient per-step scalars or any metric computed on training data as evidence. How do you version wandb runs and baselines so drift can be measured against a signed reference point? When a model passes release-readiness, promote its model artefact and eval table to a stable alias such as production or baseline-v3 rather than leaving them pinned to a forgettable run ID. That alias becomes the signed reference point, and every subsequent retrain is compared against it, so the drift-telemetry section always has a named anchor. What are the limits of wandb — where does a dashboard stop being harness-grade evidence a reviewer can sign against? wandb records only what you tell it to and does not enforce that eval ran on held-out data, that the baseline was agreed, or that a reviewer checked the right table. Those are process guarantees the harness carries, not the tracker. wandb is also not a policy engine — the judgement about which slices and drift thresholds matter is engineering work no logging tool performs. How do you keep wandb records updated when the model is retrained or updated? Point each retrain’s run at the same dataset-artefact lineage where the splits are unchanged, re-run the eval table, and compare against the fixed aliased baseline. Retraining appends a comparable entry to the harness rather than invalidating it, as long as the baseline stays fixed and the evaluation stays reproducible — the delta against baseline is itself the evidence a reviewer needs. The question worth carrying out of this is not “how do I log more in wandb,” but “what would I need to hand a reviewer, six months from now, for a model I have already forgotten the details of.” Answer that, instrument backward from it, and the audit-evidence pack stops being a separate project — it becomes the residue of doing the training properly the first time.
