Before touching a training pipeline, we want five questions answered — and not by reading the model code first. They take an afternoon, and the answers say more about how long the work will take than the code does.
1. Can you retrain last month’s model today?
Not “roughly”. Exactly — same data, same code, same hyperparameters, same weights within tolerance. If the answer is no, everything downstream is guesswork: you cannot attribute a regression, you cannot roll back with confidence, and you cannot debug a production incident by reproducing it.
The usual culprits are a mutable data source, an unpinned dependency, and a preprocessing step that lives in someone’s notebook. All three are cheap to fix, and none of them get fixed until somebody asks the question out loud.
2. Where does the training data come from, and who can change it?
Follow one feature back to its origin. Count the systems it passes through and note who has write access to each. The answer is often uncomfortable: a table that feeds a production model turns out to be maintained by hand in a spreadsheet, or a “cleaned” dataset turns out to have been cleaned once, years ago, by someone who has since left.
If a human can silently change training data, the model has a dependency nobody is monitoring.
3. What happens when the pipeline fails at 3am?
Three sub-questions: does anyone find out, is there a runbook, and can the job be safely re-run? A pipeline that fails loudly and can be replayed is a minor irritation. One that fails silently and cannot be replayed is a data-loss incident on a delay.
4. How does a model get from a training run to serving traffic?
Ask someone to walk you through it while you write it down. If the description contains the phrase “and then I”, you have found the bottleneck and also the key-person risk. The goal is not full automation on day one; it is that the path is written down and repeatable by a second person.
5. What would tell you the model got worse?
Most teams monitor infrastructure — latency, error rate, GPU utilisation — and nothing about the thing the model is for. Prediction distribution, input drift, and whatever downstream business metric the model exists to move all need to be on a chart somewhere, with a threshold and an owner.
The specific failure this catches is quiet degradation: the model does not break, it slowly stops being right, and nobody notices for a quarter because every dashboard is green.
Why these five
Because the answers are almost never about machine learning. They are about reproducibility, ownership, and observability — ordinary software engineering discipline applied to a system that happens to contain a model. A team that can answer all five has a delivery problem you can attack directly. A team that cannot has usually just found what the delivery problem was.
Either way, it is a better first week than reading model code.