10 min read
Six months after deployment, the retention team starts to notice that the model "doesn't work like it used to". The weekly interventions keep running, customers flagged as high risk keep appearing in the contact queue, but the recovery rate has dropped compared to the first quarter. Nobody can point to a moment when something broke. The model keeps returning probabilities, the process keeps executing, the dashboards remain open. The feeling is of gradual erosion.
That erosion is the natural state of any production model. The assumptions it was trained on are a snapshot of the business at the moment of training, and the business never stops moving. Customers change their behaviour, new products appear, the acquisition channel brings a different kind of audience, the macroeconomic cycle shifts the propensity to buy. The model doesn't notice. It keeps applying the rules it learned, and the signal it was producing at the start becomes progressively noisier. A monitoring protocol agreed before deployment is what turns that erosion into an actionable signal, rather than a vague complaint that arrives when the damage is already done.
Degradation is gradual, not a collapse
There are two ways a model can drift away from the reality it should predict. The first is data drift, which occurs when the distribution of the variables the model uses as input changes. If the propensity-to-repurchase model was trained on a base where 60% of customers were over 35, and six months later that proportion has dropped to 40%, the input the model sees in production is different from what it saw in training, even if the target variable still behaves similarly. The model doesn't detect this difference on its own; it simply applies the same transformations to data that no longer resembles the originals.
The second is concept drift, which occurs when the relationship between inputs and the output the model tries to predict changes. The input distribution may still look the same, but the underlying phenomenon has moved. A 40-year-old customer with three recent purchases was still a good repurchase candidate at the time of training; after a catalogue change, that same profile no longer behaves that way. The inputs look the same, the model processes them the same way, and predictions start ordering cases incorrectly.
Both forms of drift coexist and reinforce each other. A model rarely degrades from a single clean cause. What usually happens is that the customer base evolves (data drift), the marketing team changes the campaign mix (concept drift induced by process change), and both effects accumulate without anyone linking them to the performance drop. Degradation is gradual because the two drifts advance in parallel at different speeds, and the model's signal becomes noisy before it becomes directly wrong.
How to detect it: the metrics that matter
Detecting drift requires watching two things in parallel. The first is the distribution of input variables between two time windows — a reference window (the training window or a recent stable window) and the current one. The most widely used metric for this is the PSI, Population Stability Index, which measures how much a variable's distribution shifted between two periods. PSI is calculated by partitioning each variable into bins and comparing the proportion of observations falling in each bin across the two windows. The reference thresholds are well established: below 0.1 the variable is considered stable, between 0.1 and 0.25 it falls in the watch zone, above 0.25 there is significant drift that warrants review. PSI is calculated variable by variable and also on the model's aggregated predictions, which gives two different views of the same question.
A PSI value above the threshold is not, by itself, an indication that the model has lost predictive capacity. It means the input moved. If the variable that shifted is less important to the model, the prediction may still be useful. If the variable is one of the heaviest contributors to the model's output, the drift signal is a priority. That's why PSI is read alongside each variable's importance, not in the abstract.
The second thing to watch is the model's actual performance on recent data, in a rolling window. This includes the metrics the model was evaluated against originally (AUC, precision in the top decile, lift over baseline, whatever corresponds to the problem type) calculated on new cases as real feedback arrives. The usual trap is that this feedback is slow to arrive. A 90-day repurchase propensity model doesn't know until 90 days later whether it was right. During that blind window, PSI and other distribution metrics are the only available indicators. The real performance arrives later, confirms or refutes, and allows recalibration.
What to investigate before deciding to retrain
The metric says there's drift. The next question is not whether to retrain, but what is causing that drift. A PSI of 0.3 on a critical variable can have three very different origins, and each calls for a different action.
The first path is to review the data pipeline. If a source feeding the model changed its schema, if a field is now returning null where it previously had a value, if an integration started dropping records, the drift the PSI sees doesn't reflect any real change in the business — it reflects a bug. Before considering retraining, rule out that the input is broken. It's the cheapest check and the one that closes or opens fastest.
The second path is to interpret the drift as a business signal. If the age distribution of customers shifted because the acquisition channel changed, if average propensity dropped because a competitor launched an aggressive promotion, if the average ticket moved because a new product segment came in, the drift is detecting a real phenomenon that matters independently of the model. That information has value on its own: a well-instrumented model's monitoring system is also an early detection mechanism for business pattern changes that would otherwise be hard to see until they show up in aggregate numbers months later.
The third path is to retrain. It's justified when pipelines are healthy and the drift represents a real change in the phenomenon the model needs to keep predicting. But retraining is the most expensive of the three actions and is not always the right one. Drift coming from a transient operational change (a two-week marketing campaign) may resolve itself by waiting. Drift coming from a structural change (the customer base shifted permanently) does warrant rebuilding the model.
When to trigger retraining
Assuming the previous investigation ruled out bugs and confirmed structural change, the decision of when to retrain rests on two dimensions.
The first is technical. How fast is the predictive signal moving? Is the performance drop concentrated in a specific customer segment or is it generalised? Are the variables that drifted the ones the model weighted most heavily, or are they secondary? These questions are answered by the technical team looking at the numbers and understanding which part of the model became most exposed. The technical criterion defines when degradation exceeds the threshold that justifies rebuilding the model.
The second is operational. Retraining a model isn't free, even when the pipeline is automated. There's team time to validate the new model, an internal approval process, connected systems that consume the prediction and need to be tested against the new output, and a shadow mode phase: the new model runs in parallel to the one in production without touching decisions, so the team can compare both models' predictions against the real feedback arriving. Only when the new one demonstrates better performance in that comparison does it get promoted to production and the previous one retired. That process has a natural cadence driven by the team's own rhythm. A retention team that operates with a monthly model review can retrain without friction four times a year; a team running weekly campaigns with a strong operational dependency on the prediction has a narrower window.
Retraining is triggered when both conditions align. The technical criterion creates the need (there is enough structural drift to justify the effort), and operational reality defines when it's viable to move (there is team capacity and it fits the model's use cycle). A useful monitoring protocol declares both thresholds before deployment, so the response is ready when drift appears and doesn't have to be improvised during the first crisis.
The protocol is agreed before deployment
The conditions that determine whether a model reaches production are developed in detail in ML projects that reach production. Monitoring is one of those conditions. A model that enters production without an agreed monitoring protocol enters with a debt that gets called in later — when the business team senses something dropped, the technical team starts reviewing data to understand why, and the conversation about what to do happens under pressure with no prior reference point.
The protocol that works declares four things before day one: which metrics will be monitored, at what frequency, and over which rolling window; which thresholds trigger watch and which trigger action; who looks at the metrics and on what joint business-technical review cadence; what specific action follows each exceeded threshold, from "nothing for now" to "retrain next cycle" via "review the specific variable that drifted". None of this can be improvised six months later.
There is a less obvious consequence of thinking about monitoring before deployment. The model that is trained when you already know how it will be watched is, again, a different model. If the agreed cadence is reviewing performance every six weeks with high drift tolerance, the model can be more complex and use more variables. If the cadence is monthly with low tolerance and the constraint is that the model must be stable between retrainings, the design shifts toward more stable variables and an architecture less sensitive to input distribution changes. Surveillance shapes how the model is built from the start.
Proactivity doesn't end at deployment
A production model with a monitoring protocol designed before deployment is a model with guardrails. Degradation remains inevitable: the world changes, data moves, assumptions age. What changes is that it's no longer a surprise that arrives when the damage is done. The same proactive logic that characterises building a model — testing hypotheses, validating before deploying, closing scope before starting — is the one that applies after deployment. Anticipating erosion rather than reacting to it is what distinguishes a Data Science project that lives a year in production from one that becomes operational noise in the first semester.