Model drift: what to monitor and when to retrain.

Models decay quietly

A model is trained on the past and scored on the present. When the present stops looking like the past, the model keeps producing numbers with exactly the same confidence. Nothing crashes. The value drains away, and often nobody notices until a business owner asks why the results have slipped.

That is drift, and it is normal. Customers change, products change, a competitor moves its prices, an upstream system renames a field. The question is not whether your models will drift. It is whether you will know when it matters, and whether anyone has agreed in advance what happens next.

Our position: monitor the decision the model serves, not only the model, and retrain on a trigger that was agreed when the decision was framed, not on a calendar.

Two kinds of drift, and one impostor

Data drift is a change in the inputs: the mix of customers, products or transactions the model sees is no longer the mix it was trained on. Concept drift is a change in the relationship between the inputs and the outcome. The same customer profile that used to signal loyalty now signals churn. The research literature has studied the second problem for years; the survey by Gama and colleagues in ACM Computing Surveys describes it as change over time in the relationship between input data and the target variable.

The difference matters because of timing. Data drift you can see straight away, since you have the inputs the moment you score them. Concept drift you usually see late, because you need the true outcome, and outcomes arrive after a delay: a churn label after the renewal date, a fraud label after the chargeback. A model can look stable on every input chart while its predictions are quietly going wrong.

Then there is the impostor. A broken join, a change of units, a field that started arriving empty: these look like drift on a chart, but they are data-quality failures. Retraining on them teaches the model the bug.

What goes wrong

• Nobody owns the alerts. Monitoring exists and alerts fire, into a shared channel that nobody is accountable for reading.
• Everything alerts. Every column gets a drift test with a default threshold. Within weeks the team has learned to ignore them all, including the one that mattered.
• Retraining on a calendar. A monthly retrain feels safe. It retrains whether or not anything changed, can absorb a data bug into the model, and can still be too slow when something changes in a week.
• The metric holds and the value falls. The model's accuracy looks fine while the business measure drops, because the threshold, the offer or the workflow around the model changed.

What to monitor, layer by layer

Think of monitoring as five layers, from the plumbing up to the decision. Each layer catches something the others miss, and each has an obvious owner.

Two practical rules. Limit input checks to the features that drive the model, because monitoring every column produces noise, not safety. And track performance by segment, because an average can hold steady while one region or product line falls apart.

When to retrain, and when not to

Google's architecture guide to MLOps and continuous training lists several ways to trigger a training pipeline: on demand, on a schedule, when new training data becomes available, when model performance degrades, and when data distributions change significantly. All of them are legitimate. The point is to choose deliberately for each model and to write the choice down.

In our own builds, the trigger is set before any engineering starts. The Data IQ™ Decision Brief records when a model is reopened, for example when lift falls below a floor agreed with the business. That turns retraining from a habit into a decision with an owner.

Retrain when

• Performance on fresh outcomes falls below the agreed floor.
• Input or prediction drift is sustained, and has been confirmed not to be a data fault.
• New labelled data covers situations the model has never seen.

Do not retrain when

• The pipeline is broken. Fix it first.
• The business changed the threshold, the offer or the workflow. Re-validate the decision instead.
• The shift is a seasonal pattern the model already handles.

Reopen the decision, not only the model, when

• The decision itself has changed, or the people acting on it have.
• The agreed measure no longer reflects the value the business wants.

Before a retrained model goes live, compare it with the current one on the same recent data, check it segment by segment, and keep the previous version ready to roll back.

Owners and runbooks

Monitoring without an owner is decoration. The paper on hidden technical debt in machine learning systems, written by engineers at Google, argued that comprehensive live monitoring of system behavior, combined with automated response, is critical. Automated response helps. A named person who reads the alert and knows what to do is essential.

Every production model needs three things: a named owner for its alerts, a runbook that says what each alert means and what to do about it, and a regular review of whether the model still earns its place. When we hand a model over, those go with it. Once you run more than a handful of models, this becomes a portfolio routine, which we cover in MLOps in practice.

A checklist for each production model

• Is there a named owner for this model's alerts?
• Are data-quality checks separate from drift checks, so a pipeline fault is not mistaken for drift?
• Are input checks limited to the features that drive the model?
• Is performance tracked by segment once outcomes arrive, and is the delay before outcomes arrive known?
• Is the business measure the model serves monitored too?
• Is the retraining trigger written down and agreed with the business owner?
• Is a retrained model compared with the current one before release, with rollback ready?
• Is there a rule for retiring a model that no longer earns its place?

If you cannot answer the first question, start there. Everything else depends on someone being responsible for the answer.