The third model is where it breaks
The first model in production is usually fine. The people who built it know its data, its quirks and its users. It runs on a scheduled job someone set up, and when it misbehaves, the person who wrote it notices.
The trouble starts with the third, fourth and fifth. Different people built them, in different ways, on different schedules. Nobody keeps a list. A change upstream breaks two of them and nobody finds out for weeks. Someone from risk asks which models touch customer decisions, and assembling the answer takes days.
Our position: MLOps is an operating routine for a portfolio of models. Tools help, and some are very good. But buying a platform does not create the routine, and a team that has the routine can run it on modest tooling.
Why the model is the small part
The best-known argument for this comes from a group of Google engineers. Their paper, Hidden Technical Debt in Machine Learning Systems, showed that only a small fraction of a real-world machine learning system is the model code. The rest is the infrastructure around it: collecting and checking data, computing features, serving predictions, monitoring, configuration. That surrounding work is where most of the cost of ownership sits, and where most failures begin.
It is also why MLOps cannot be bolted on at the end. The decisions that make a model operable are made during the build: how features are computed for live scoring, how training data is versioned, how a prediction reaches the person or system that acts on it. Get those wrong and no platform will rescue the model afterwards.
What goes wrong
- Tool first. A platform is bought before anyone has agreed what it should do. It becomes one more place where models are scattered.
- Notebook to production. Training code that ran once on a laptop is copied into a scheduled job. Nobody can reproduce the model that is actually running.
- Training and serving disagree. Features are computed one way for training and another way for live scoring. The model is correct; the numbers it receives are not.
- No inventory, no owner. Nobody can say how many models are running, what each one decides, or who answers for it.
- No retirement. Models are added and never removed. Each one still has to be monitored, secured and explained.
The six practices that matter
Strip MLOps down to what keeps models reliable and you are left with six practices. None of them requires a particular vendor.
- An inventory with owners. Every model in production, the decision it serves, the data it uses, how much that decision matters, and a named owner. This is the first thing to build and the cheapest.
- Reproducible training. Code, data version, parameters and environment recorded, so you can rebuild the model that is running and explain how it was made.
- One way to deploy. A shared release path with tests, approval where the decision warrants it, and a rollback.
- Shared monitoring. Data quality, drift, model performance and the business measure, watched the same way across models. We cover what to watch in Model drift: what to monitor and when to retrain.
- Retraining and retirement rules. Written down per model: what triggers a retrain, who approves the new version, and when a model is switched off.
- A review cadence that holds. A regular, short review of the portfolio: what changed, what alerted, what should be retired. It only works if it survives a busy quarter.
Start with the inventory. If you do one thing this month, list every model in production with its decision, its data and its owner. The gaps in that list are your MLOps roadmap.
How much automation you need
Google's architecture guide to MLOps and continuous delivery describes three levels. At level 0, every step is manual. At level 1, the training pipeline is automated, so models can be retrained continuously. At level 2, the pipelines themselves are built, tested and deployed automatically through CI/CD.
It is tempting to treat level 2 as the goal for everything. It is not. A model retrained twice a year to support a quarterly planning decision can live well on a manual process, provided that process is documented and reproducible. A fraud model scoring live transactions on fast-moving data probably needs automated training and release. Match the automation to how often the model has to change and how many models share the machinery.
| Situation | Reasonable level | What must be true regardless |
|---|---|---|
| A few models, slow-changing data, periodic decisions | Manual, but documented and reproducible | Inventory entry, named owner, monitoring, a written retraining rule |
| Models retrained regularly as new data arrives | Automated training pipeline | Validation against the current model before release, rollback, data-quality checks |
| Many models, frequent code changes, several teams | Automated build, test and release of pipelines | Shared standards, a review cadence, retirement rules |
Controls that scale with the decision
Not every model deserves the same paperwork. A model that sets credit limits or prioritizes patients needs explanations, review, an audit trail and a clear human override. A model that orders an internal ticket queue needs much less. Scale documentation and review to how much the decision matters, and record that judgment in the inventory so it is visible and can be challenged.
Where a recognized framework helps, use it. We follow the NIST AI Risk Management Framework where it applies; our article on the NIST AI RMF in practice shows what that looks like day to day. A framework organizes the work. It does not replace owners who read the alerts.
A checklist for your model portfolio
- Can you list every model in production, with its decision, its data and its owner?
- Could you rebuild the model that is running today from what you have recorded?
- Is there one release path, with tests and a rollback?
- Are features computed the same way for training and for live scoring?
- Is every model monitored for data quality, drift, performance and its business measure?
- Are retraining and retirement rules written down for each model?
- Does a portfolio review happen on a fixed cadence, and did the last one actually take place?
- Is the level of documentation and review matched to how much each decision matters?
If the answers are mostly yes, your tooling choices are a detail. If they are mostly no, a new platform will not fix it.
