MLOps Tracking skill for Claude Code

Tracking is working when anyone can take the model serving production, find the exact run that produced it, see the code commit, data version, parameters, and evaluation results, and reproduce it. A dashboard full of unnamed runs with loss curves is logging, not tracking.

Ground rules

• Every run records: git commit (and dirty flag), data version or hash, full config, environment (package versions, hardware), seed, metrics, and the produced artifacts.
• One experiment per question or project, consistent run names, and tags for filtering.
• Models move to production through the registry with an alias or stage change that a person or a CI gate approved, never by copying a file from someone’s run.
• Do not log secrets, raw personal data, or full datasets as run parameters or artifacts unless the store is approved for that data.
• Pick one tool per team. Running MLflow and W&B side by side doubles the work and splits the truth.

Choose the tool

Situation	Choice
Open source, self-hosted, models must stay in your infra	MLflow tracking server with a database backend and object storage for artifacts
On Databricks	Managed MLflow (registry under Unity Catalog)
Rich interactive dashboards, sweeps, team collaboration, SaaS acceptable	Weights & Biases
Very small project, one person	MLflow with a local mlruns or sqlite store is enough to start

Workflow

- [ ] 1. Define naming, tags, and what every run must log
- [ ] 2. Stand up the tracking backend (server, DB, artifact store, auth)
- [ ] 3. Instrument training: config, metrics, artifacts, lineage
- [ ] 4. Version data and link it to runs
- [ ] 5. Log models with signatures and register candidates
- [ ] 6. Define promotion: evaluation gate, aliases, approvals
- [ ] 7. Make serving load by alias; record deployments
- [ ] 8. Audit: can you trace production back to data and code?

1. Conventions

experiment: <team>/<project>/<task>            e.g. search/ranking/ltr
run name:   <model>-<key change>-<date>         e.g. lgbm-add-ctr-feats-2026-10-12
tags:       owner, purpose (baseline|ablation|sweep|candidate), dataset_version, git_commit

Required metrics are named the same across runs (val/auc, test/recall@10), with the split in the name so nobody compares validation to test.

2. Backend (MLflow)

mlflow server \
  --backend-store-uri postgresql://mlflow:***@db:5432/mlflow \
  --artifacts-destination s3://ml-artifacts/mlflow \
  --host 0.0.0.0 --port 5000

• Put it behind authentication (MLflow’s basic auth app, an auth proxy, or SSO); an open tracking server leaks models and data paths.
• Back up the database; artifacts live in object storage with versioning or lifecycle rules.
• Clients set MLFLOW_TRACKING_URI; never hardcode it.

3. Instrument training (MLflow)

import mlflow, subprocess

mlflow.set_experiment("search/ranking/ltr")
commit = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
dirty = bool(subprocess.check_output(["git", "status", "--porcelain"], text=True).strip())

with mlflow.start_run(run_name="lgbm-add-ctr-feats-2026-10-12") as run:
    mlflow.set_tags({"git_commit": commit, "git_dirty": dirty,
                     "dataset_version": DATA_VERSION, "purpose": "candidate"})
    mlflow.log_params(flatten(config))                 # full config, flattened keys
    for epoch, m in train(config):
        mlflow.log_metrics({"train/loss": m.loss, "val/auc": m.auc}, step=epoch)
    mlflow.log_metrics({"test/auc": test_auc, "test/recall_at_10": r10})
    mlflow.log_artifact("reports/eval.html")
    model_info = mlflow.lightgbm.log_model(
        model, name="model",                            # MLflow 3: `name` replaces artifact_path
        input_example=X_val[:5],                        # infers the signature
        registered_model_name="search-ranker",
    )

• Autologging (mlflow.autolog()) is a fine start, but still add tags, data version, and final test metrics yourself.
• Log evaluation reports, confusion matrices, and feature importances as artifacts; they are what reviewers read.
• MLflow 3 treats logged models as first-class objects with their own IDs; check your version’s docs for the model URI forms it returns.

3b. Instrument training (W&B)

import wandb

run = wandb.init(project="search-ranking", name="lgbm-add-ctr-feats",
                 config=config, tags=["candidate"], job_type="train")
data = run.use_artifact("train-data:v12")             # lineage: this run consumed v12
for epoch, m in train(config):
    run.log({"train/loss": m.loss, "val/auc": m.auc}, step=epoch)
run.summary["test/auc"] = test_auc

art = wandb.Artifact("search-ranker", type="model", metadata={"test_auc": test_auc})
art.add_file("model.txt")
logged = run.log_artifact(art)
run.link_artifact(logged, target_path="wandb-registry-model/search-ranker")
run.finish()

Registry target paths follow wandb-registry-<registry>/<collection>; confirm the form your W&B version expects. W&B records git commit and environment automatically when run inside a repo; verify it appears in the run overview.

4. Data versioning

Approach	When
Immutable, dated snapshot paths in object storage plus a content hash tag	Simple and effective for most teams
Tracking-tool artifacts for datasets (W&B artifacts, MLflow datasets)	Small to medium datasets; gives lineage graphs
DVC or lakeFS	Large datasets with branching and Git-style workflows
Table versions (Delta, Iceberg time travel)	Data lives in a lakehouse; log the table version number

Whatever you choose, the run must record an identifier that resolves to exactly the data used. “s3://bucket/latest/” is not a version.

5-6. Registry and promotion

Use aliases (MLflow) or registry tags/aliases (W&B) rather than legacy stages:

from mlflow import MlflowClient
client = MlflowClient()
client.set_registered_model_alias("search-ranker", "challenger", version=17)

# after the evaluation gate passes and an approver signs off
client.set_registered_model_alias("search-ranker", "champion", version=17)

Promotion gate (in CI or a scheduled job):

1. Load models:/search-ranker@champion and @challenger.
2. Evaluate both on the same frozen test set and recent production slice.
3. Require the challenger to beat champion on the primary metric by a set margin without regressing guardrails (latency, fairness slices, calibration).
4. Record the comparison as an artifact and tag the version with the result.
5. Move the champion alias only after required approval; keep the previous version number for rollback.

For deciding whether an online A/B result is significant, use experiment-analysis.

7. Serving and deployment records

• Serving loads by alias (models:/search-ranker@champion) or a pinned version, never by run path. Pinned versions are safer for strict reproducibility; aliases are simpler for promotion. Choose one and document it.
• Log each deployment (version, time, environment, who) as a tag on the model version or in the deployment system, so “what was live on Oct 3?” has an answer.
• Rollback = move the alias back to the previous version and redeploy.

8. Audit

Pick the production model and walk backward: registry version -> run -> commit, config, data version -> evaluation artifacts. Any missing link is a fix item. Run this check quarterly or before audits.

Gotchas

• Logging metrics with inconsistent names or steps makes runs incomparable; fix names in a shared helper.
• Logging every batch from a large job can overwhelm the tracking server; log every N steps.
• Parameter values have length limits in MLflow; log large configs as an artifact file plus flattened key parameters.
• Artifacts uploaded from training machines need write credentials; scope them to the artifact bucket only.
• Deleting runs in the UI does not always delete artifacts in object storage; set lifecycle rules and a garbage-collection job.
• Notebook runs without a commit are not reproducible; require scripts or log the notebook file as an artifact.

Output contract

**Tool:** <MLflow server URL / W&B entity/project>, auth <method>
**Conventions:** experiments <pattern>, run names <pattern>, required tags <list>
**Lineage:** data versioned via <method>; runs record commit + data version <yes>
**Registry:** model <name>, aliases champion/challenger, promotion gate <script/CI job>
**Serving:** loads <alias or pinned version>; rollback <procedure>
**Audit result:** production model <version> -> run <id> -> commit <sha> -> data <version>