7 min read
When a company wants to predict which customers are going to cancel or stop buying, it finds that most of those in the base won't do so. That asymmetry is the norm, not the exception: in any classification problem where the class of interest is the minority (2%, 5%, 10% of the total), the model can learn to produce performance metrics that look good in the report while missing almost every case the business wanted to catch. The result is a technically correct and operationally inapplicable model that reaches production answering with precision the wrong question.
The conversation that separates one from the other happens before training, and it's a discussion about what error the business can afford and what error it cannot. That agreement determines the metric, the model's calibration, and the intervention strategy that follows. Without it, the pipeline reaches production with the data scientist's criterion (maximise aggregate performance) when the business's criterion was something else.
When the default metric hides the problem
Suppose a churn model trained on a base where 2% of customers actually leave in the next quarter. A trivial model that predicts "won't churn" for every single customer is right 98% of the time. Accuracy 0.98 on the test set, and the metric looks excellent. It is also the operational definition of a useless model, because it detected none of the real churns — which is the only thing it was built for.
AUC (the area under the ROC curve, a measure of how well the model orders cases by probability of being positive) holds up a little better against this deception because it doesn't depend on the threshold. But it has its own blindness in extreme imbalance. An AUC of 0.87 can coexist with a recall of 0.18 on the minority class. The model orders reasonably well between negatives and positives, but when it comes time to classify, it keeps sending most of the real positives to the negative side. Both numbers are real and compatible. The first is the one that appears in the technical report; the second is the one that determines whether the solution has value.
The metric that reveals this problem is specific to the type of error the business cannot afford. Recall, when the cost of missing a positive case is high. Precision, when the cost of intervening on a negative is high. The data scientist can facilitate the choice, but the input that determines it belongs to the team that bears the consequences. Business variables like the expected cost of losing a customer and the cost of the campaign that would be activated to retain them enter this decision before any algorithmic consideration.
Two problems with the same technique and opposite decisions
In a repurchase propensity problem with an email campaign, the false positive costs little. An email goes to someone who probably wasn't going to repurchase; the cost is the send and some inbox noise. The false negative has an opportunity cost. It's not contacting someone who would have repurchased if they had received the message. In that case recall matters more. The model is calibrated to capture most potential buyers, even if it means contacting some who weren't.
In fraud detection the analysis is the opposite. The false positive has a real cost. The legitimate customer whose transaction was blocked calls in frustration or, worse, switches provider. The false negative also costs, but there are margins to tolerate it depending on the volume and average fraud amount. Precision matters more. The model is calibrated so that when it says "fraud", it is almost always right, even if that means missing some real cases.
Both problems use the same type of model. What changes between them is the relative cost of the two types of error, and therefore the metric governing training and validation. If the average cost of an undetected fraud is €400 and the cost of blocking a legitimate transaction is €20 in customer service, the loss function assigns twenty times more weight to the false negative than to the false positive. That ratio is defined by whoever knows the business. Without that ratio, the model optimises with equal implicit costs, which never match the real costs.
Segmenting the ranking opens up different strategies
Choosing the metric is one part of the work. The other is understanding that the model's output is a ranking, not a label. Each customer receives an estimated probability, and that distribution of probabilities can be split into segments that admit different operational strategies.
A churn model might identify that customers with probability above 80% (the top decile of the distribution) account for 40% of the real churns that will occur in the quarter. That small, concentrated segment might justify an expensive intervention: a call from an account manager, a renewal offer with a discount, a retention meeting. Customers with probability between 60% and 80% (the second decile) are a larger group with a lower concentration of real churns; there, a cheaper intervention, like an automated personalised communication, may have positive ROI. Customers with probability below 60% probably don't justify any specific retention intervention.
Without this exercise, the default practice is to apply a single classification threshold, typically 0.5 — a value with no business justification — and execute the same action for all positives. That uniformity wastes budget on the positives with lower probability and underinvests in those with higher probability.
The threshold is not a technical parameter
The model does not produce binary predictions directly. It produces a ranking with probabilities. The classification threshold is the cut that converts those probabilities into a decision. Above the threshold the system activates the action; below it, nothing. That cut is defined by whoever knows the consequences, not by the data scientist.
By default, most libraries use 0.5. That value has no business justification. It is the point where the model estimates the two outcomes are equally probable, and it almost never coincides with the point where the costs of the two errors balance. In repurchase propensity with a low contact cost, lowering the threshold to 0.25 can significantly increase recall with an acceptable drop in precision. In fraud with a high blocking cost, raising the threshold to 0.7 can improve precision at the price of missing some real cases.
The threshold is calibrated on the precision-recall curve on validation data, with the business's real costs as reference. That cut is defined by the team that knows the consequences, not the data scientist. The decile segmentation seen earlier is an operational extension of the same principle: there is no single optimal threshold, there are optimal thresholds by segment according to the relative cost of the intervention associated with each one.
Defining the metric is defining the expected outcome
The conversation about what error the business can afford is the highest-return project that can be done without data yet — not an optional preliminary step before the project. It determines the metric, the model's calibration, the intervention strategy by segment, and the criterion against which the outcome will be evaluated in production.
Without that conversation, the metric is chosen by default and the model is trained against a criterion nobody agreed to. When it reaches production and someone asks "does it work?", if nobody agreed in advance how to answer that question, the answer will be ambiguous. The discussion that didn't happen before training happens after, when any change to the model or the threshold carries a re-validation and re-deployment cost that is no longer marginal.
A model reaches production answering a question with technical precision — the question the metric optimised. If that question doesn't match the one the business needed to answer, the operations team finds itself with a system that "works" according to the data scientist and isn't useful according to them. Closing that gap is not algorithmic sophistication. It is agreeing before training what the model needs to optimise and why. Any project that skips that conversation pays for it later, and pays dearly.