An experiment tracker is not a place data scientists log loss curves and forget once a model ships. It is the lineage substrate the monitoring harness reads from — the record of which data, code, config, and metrics produced the model now in production, and how the next candidate stacks up against it. The moment a tracker’s value is decided is not during exploration; it is at the handoff from training to release. That is where a well-wired tracker turns into the signed, reproducible baseline every deployment decision leans on, and where a neglected one turns into archaeology. Teams that treat the tracker as a training-time toy hit the same wall every release. A shipped model misbehaves, someone asks “which run produced this?”, and the answer takes days to reconstruct because the run that mattered was one of four hundred, its config lived in a notebook cell that has since been edited, and the eval numbers were pasted into a Slack thread. A team that instead wires the tracker into its eval harness and release-readiness review can answer the same question with a single diff against a known baseline. This is the divergence that matters, and it is the subject of this article. What an experiment tracker actually records The common shorthand — “it logs metrics” — is true but misleading, because it makes the tracker sound interchangeable with a metrics dashboard. What a tracker records is a run: a bundle that ties four things together so they can never drift apart. Data — a reference or hash of the exact dataset split used, not just “the training set”. Which version, which filtering, which label revision. Code — the commit SHA of the training code, ideally with a dirty-working-tree flag so an uncommitted change cannot masquerade as a clean run. Config — every hyperparameter, seed, and environment detail that shaped the result: learning rate, augmentation policy, precision mode, the CUDA and framework versions in the container. Metrics and artefacts — the scalar time series (loss, mAP, calibration error) and the produced weights, checkpoints, and evaluation reports as versioned artefacts. The point of bundling these is reproducibility of causation, not just of numbers. A dashboard can tell you a model scored 0.91 mAP. Only a tracker can tell you that build produced 0.91 mAP from this data, this commit, and this config — which is the difference between a chart and evidence. When a run in a tool like Weights & Biases or MLflow is logged with disciplined lineage, the whole bundle is addressable by a single run ID, and that ID becomes the thing a downstream reviewer cites. Our own practice around Weights & Biases as a feeder into a monitoring harness leans hard on this: the run ID is the primary key of release evidence. How is experiment tracking different from generic logging or a metrics dashboard? Generic logging is append-only text keyed by time. A metrics dashboard is a visualization layer keyed by a metric name. Both answer what happened. An experiment tracker is keyed by the run and answers what produced this, which is a fundamentally different query. You cannot ask a Grafana panel “reconstruct the exact conditions of the model currently in production” — it was never designed to hold code and data provenance alongside the numbers. You can ask a tracker that, because provenance is its native record. This distinction is why we treat the tracker as infrastructure rather than convenience tooling. A dashboard is something you look at. A tracker is something the release process reads from programmatically, and that reading is where it earns its keep. How the tracker’s run lineage feeds the harness The monitoring harness — the collection of eval suites, regression gates, and drift monitors that decides whether a build is fit to ship and stays fit — does not itself store the training history. It references it. The tracker is the feeder; the harness is the consumer. Concretely, two sections of the harness read from the tracker’s lineage. The eval-harness section needs a frozen metric set and a reference run to evaluate against. When a new candidate is scored, the harness pulls the shipped model’s run — its exact eval config and its recorded numbers — from the tracker and runs the candidate through the same evaluation. Because the eval config came from lineage rather than being re-specified by hand, the comparison is apples-to-apples by construction. This is the same discipline that governs where reliability gates belong at each stage of an ML pipeline: the gate is only meaningful if the thing it compares against is pinned. The regression-suite section needs a baseline to diff against. The tracker supplies build N-1 as a signed reference. The regression suite asserts that build N is no worse than build N-1 on the frozen metric set — and if it regresses, the diff points at exactly which metric moved and, via lineage, what changed in data or config to move it. Training telemetry captured through integrations like PyTorch Lightning with W&B flows straight into this diff, which is why the capture discipline at training time determines the quality of the release-time comparison. None of this requires the harness to contain the tracker. It requires the tracker to expose stable, addressable lineage that the harness can cite as evidence. That separation of concerns — tracker as system of record, harness as system of judgment — is what makes both auditable. What a reproducible baseline looks like A reproducible baseline is not a number. It is a run ID that resolves to the full bundle: the data reference, the code commit, the config, the frozen eval metric set, and the artefact hash of the weights. When a reviewer signs a release-readiness decision, they are not signing “0.91 mAP looks good.” They are signing “build N was evaluated against baseline run abc123 on the frozen metric set and is no worse; here is the diff.” That signature is defensible precisely because anyone can re-resolve abc123 and re-run the comparison. The table below contrasts what a reviewer can and cannot do depending on where the tracker sits in the workflow. Capability at release time Tracker used only for exploration Tracker wired into the harness Identify which run produced the shipped model Days of archaeology across notebooks and threads Single run-ID lookup Prove build N cleared the same bar as N-1 Re-litigated per release, often by hand Automated diff against pinned baseline Reconstruct why a shipped model behaves as it does Frequently impossible after code drift Resolve run → data, code, config recovered Attribute a regression to a specific change Guesswork Diff isolates the moved metric + lineage delta Sign a release-readiness decision defensibly Signature rests on pasted numbers Signature rests on a re-runnable reference (Evidence class: observed pattern across TechnoLynx MLOps-hardening engagements; not a benchmarked rate.) The recurring shape is that teams without wired-in lineage pay the reconstruction cost on every release, while teams with it pay the setup cost once. That is where the return on discipline shows up — not in training speed but in triage speed, which in our experience is where the days actually go. What breaks at release time when tracking is missing The failure is rarely visible during training. It surfaces at the handoff. A model that scored well in exploration ships, then behaves oddly in production — a calibration drift, a class the model quietly stopped detecting, a latency regression from a precision change nobody logged. The reviewer asks the two questions the harness is supposed to answer: which run is this, and did it clear the bar? Without lineage, neither has an answer. The concrete breakages we see repeatedly: No reference run. The shipped model’s config was never pinned, so there is nothing for the eval harness to compare a candidate against. Each release invents its own baseline, which means the regression suite is comparing against a moving target. Config drift went unrecorded. A precision mode changed from FP16 to FP8, or an augmentation policy was tweaked, and because the tracker only held loss curves, the change is invisible until it manifests as a production regression. Capturing precision-mode changes matters enough that we treat it as first-class lineage in work like TensorBoard logging for anomaly models. Artefacts drifted from metrics. The weights that shipped are not provably the weights that produced the logged eval numbers, because the checkpoint was overwritten. Versioning artefacts as immutable references — the pattern behind W&B Artifacts for condition monitoring — is what closes this gap. Each of these is a lineage hole, and each turns a routine regression diff into an investigation. The cost is not hypothetical: it is the difference between signing a release in an afternoon and blocking one for a week. Does experiment tracking change across CV, LLM, and perception workloads? The mechanism is the same — bundle data, code, config, metrics — but what has to be pinned shifts with the workload, and getting this wrong is a common source of false confidence. For computer-vision models on an inspection line, the dataset reference must include label revisions and the exact augmentation policy, because a relabelling pass can move mAP more than an architecture change. For LLM workloads, the config that matters expands to include the prompt template, the decoding parameters, and the served runtime (a model behaves differently under vLLM than under a raw transformers loop), so the tracker’s “config” has to reach into serving, not just training. For perception stacks in automotive, the frozen metric set is often a scenario suite rather than a single scalar, and lineage has to tie a run to the scenario coverage it was evaluated against, which is why perception validation leans on tracking as the substrate for release sign-off. The unifying rule is that lineage must capture whatever variable actually moves the metric for that workload. A tracker configured for CV loss curves will silently miss the decoding parameter that shifted an LLM’s behaviour. The discipline transfers; the specific fields do not. FAQ What does working with an experiment tracker involve in practice? A tracker records each training or evaluation run as a bundle that ties together the data reference, the code commit, the full config, the metric time series, and the produced artefacts, all addressable by a single run ID. In practice it means every model has a re-resolvable provenance record, so you can reconstruct exactly what produced it rather than reasoning from pasted numbers. What does an experiment tracker actually record — data, code, config, metrics, artefacts? All of them, and the value is in bundling them so they cannot drift apart: a dataset reference (with version and label revision), the training code commit SHA, every hyperparameter and environment detail, the scalar metrics, and the weights and reports as versioned artefacts. The purpose is reproducibility of causation — proving this data, this code, this config produced these numbers — not just reproducibility of the numbers. How does experiment tracking differ from generic logging or a metrics dashboard? Generic logging is keyed by time and a dashboard is keyed by a metric name; both answer what happened. A tracker is keyed by the run and answers what produced this, holding code and data provenance alongside the numbers. You cannot ask a dashboard to reconstruct the exact conditions of the shipped model, because it was never designed to store that. How does the tracker’s run lineage feed the eval and regression sections of a monitoring harness? The harness references lineage rather than storing it. The eval section pulls the shipped model’s run and its frozen eval config so a candidate is scored apples-to-apples; the regression section pulls build N-1 as a signed baseline and asserts build N is no worse on the frozen metric set. The tracker is the feeder; the harness is the consumer that cites its lineage as evidence. What does a reproducible baseline look like, and how is it used to compare a new model build against the shipped one? It is a run ID that resolves to the full bundle — data, code, config, frozen metric set, and artefact hash. A reviewer compares a new build by re-running it through the pinned baseline’s evaluation and signing that the result is no worse, backed by a diff anyone can re-resolve and re-run rather than a number pasted into a thread. What breaks at release time when experiment tracking is missing or incomplete? The failure surfaces at the training-to-release handoff: there is no reference run to compare against, unrecorded config drift (like a precision-mode change) becomes invisible until it manifests as a production regression, and shipped artefacts cannot be proven to match the logged metrics. Each hole turns a routine regression diff into a multi-day investigation. How does experiment tracking differ across CV, LLM, and perception workloads in practice? The mechanism is identical but the fields that must be pinned differ: CV needs label revisions and augmentation policy, LLMs need prompt template, decoding parameters, and served runtime, and perception needs scenario-coverage lineage. The rule is to capture whatever variable actually moves the metric for that workload — a tracker tuned for CV loss curves will silently miss an LLM decoding parameter. The tracker earns its place the moment someone has to sign a release. If a new build cannot be diffed against a pinned, re-resolvable baseline, the reviewer is signing a number, not a decision — and that gap is exactly where a release-readiness review leans on baseline lineage as its evidence. If your release process still answers “which run was that?” with archaeology, the failure class is a lineage hole, and the fix is upstream in how runs are recorded, not downstream in the harness. The reliability practice behind this — the eval and regression sections that cite tracker lineage as evidence — is detailed on our [production AI reliability](production AI reliability) work, which the home page points to as the through-line across these engagements.
