10 min read
A B2B SaaS company is losing accounts slowly. The customer stops using the product for weeks, reduces login frequency, opens a ticket that never gets resolved, and cancels at the third month of the contract. The retention team wants to get ahead of it. They ask for a model that will flag, a quarter in advance, which accounts are going to cancel so they can intervene before renewal. That request has a technical name — churn prediction, also called customer attrition modelling — and a set of operational variants that condition everything that follows.
The churn model, in its standard form, is a binary classifier. It assigns each customer an estimated probability of churning during a defined horizon (the next 30, 60 or 90 days, depending on the business cycle). That probability is converted into a decision (contact / do not contact) by applying a cut-off threshold — one that is often set without a business justification. There are variants: a propensity ranking that orders the entire customer base without a binary cut-off, or segmented models that train separate classifiers for audiences with different dynamics. All three share the same underlying problem. The model produces a probability. Someone has to decide what to do with it, and that decision depends on how much each type of error costs.
Two errors in churn detection
In any classification there are two ways to be wrong. The false negative is the customer the model flagged as safe who ended up cancelling. The false positive is the customer the model flagged as at-risk who had no intention of leaving. Each has its own cost, and that cost is rarely symmetric.
The cost of a false negative depends on the lost customer's value. In segments with high LTV variance, that value is estimated with a pLTV (predicted lifetime value) model, which warrants its own article. In a SaaS with annual contracts at €5,000 per account, missing a cancellation costs five thousand euros of LTV plus the cost of acquiring a replacement (which in B2B is usually several multiples of the initial CAC). The cost of a false positive depends on the intervention that triggers when the model says "at risk". If the intervention is an automated email, the marginal cost is near zero. If the intervention is a thirty-minute call from a senior account manager, the cost is those thirty minutes multiplied by the fully loaded salary of the manager, plus the opportunity cost of not attending to accounts that actually needed attention.
The ratio between those two costs — how many false positives are accepted for each false negative avoided — is the input that determines how the model is trained, calibrated and evaluated. Without that ratio made explicit, the pipeline uses implicit equal values (a false positive weighs the same as a false negative), which almost never matches the business reality.
How the FN/FP ratio defines the threshold and the metric
The model delivers probabilities, not labels. For the output to be actionable, each probability must be converted into a binary decision, and that is done by applying a cut-off threshold that is often set without a business justification.
The optimal threshold is derived from the cost matrix. If a false negative costs twenty times more than a false positive, the threshold drops (the model flags more customers as at-risk, accepting more false positives in exchange for capturing more of those who would actually cancel). If a false positive costs three times more than a false negative, the threshold rises (the model becomes more conservative, only flags when the estimated probability is high, accepts losing some real cancellations in exchange for not spending interventions on false alarms). The business cost ratio is what moves the threshold.
The metric to optimise during training follows the same logic. Recall, precision, F-beta and expected value are four technical formulations of different points on that trade-off, and each encodes a different assumption about the FN/FP ratio. Choosing the metric without having discussed the ratio means choosing the default assumption — that the two errors weigh equally. The discussion on misleading accuracy in imbalanced problems and the operational segmentation of the ranking into deciles is developed in detail in Model performance vs. a working solution. What follows here is the specific derivation of each metric from the ratio.
Three scenarios, three metrics
The three scenarios that follow carry equal weight in frequency. None is the general case from which the others are exceptions. They are three different cost configurations, and each one imposes a different metric.
Recall-dominant case: the cost of losing a customer is much greater than the cost of contact
The high-LTV B2B SaaS described at the outset falls into this category. A cancelled account is worth five thousand euros per year; a retention call takes half an hour of a manager's time. The FN/FP ratio is around three hundred, possibly more if LTV is properly accounted for rather than just the current contract.
With that ratio, recall is the governing metric. Recall measures, of the customers who actually cancelled, what fraction the model detected. A recall of 0.85 means the model captured 85% of real cancellations; it missed 15%. That 15% is lost revenue. The model's precision (of those it flagged as at-risk, how many were really going to cancel) matters less, because false positives each cost thirty minutes. Many false positives are accepted in exchange for capturing nearly all the true positives.
The threshold is lowered aggressively, sometimes down to 0.15 or 0.2. The model flags a wide volume of accounts, the manager works through the list, and the total cost is the sum of the interventions spent on false alarms minus the value rescued from accounts that would actually have left. In this regime the rescued value dominates, and any model that does not prioritise recall leaves money on the table.
Precision-dominant case: high cost of intervention or low-value churn
A B2C ecommerce with low margin per customer (say the average LTV is €180) and a retention strategy based on 20% discounts sent to those flagged as at-risk. The false negative costs €180 per lost customer (less the margin already extracted during the active cycle). The false positive costs the discount spent on someone who was not going to leave, plus the erosion of the segment's overall margin if the practice becomes expected. The FN/FP ratio here can be close to 1 or even inverted, depending on how much the discount is spent on accounts that already had a repeat purchase scheduled.
In that regime, precision is the governing metric. Precision measures, of the customers the model flagged as at-risk, what fraction actually were. A precision of 0.6 means that of every ten discounts sent, six went to accounts that were genuinely going to cancel; four were gifts to customers who would have stayed anyway. If the product margin is low, four out of ten discounts gifted to those who did not need them destroy the campaign's economics.
The threshold is raised. The model only flags when the estimated probability is high (0.7, 0.8), and the volume of flagged accounts drops. The marketing team receives a short, well-qualified list; some real positives that fell below the threshold are missed, but the total campaign cost is lower than the value rescued from the true positives captured.
Neutral or generalist case: balanced costs
A mid-price monthly subscription service (say €25 per month) with low-cost automated interventions (a personalised email sequence generated by a system that is already built). The value of a lost customer is moderate, the cost of the intervention is nearly zero, but the volume of interventions creates noise in the base (channel fatigue, a drop in future open rates) and there is a deferred cost that is difficult to quantify.
In this regime no metric clearly dominates, and it is worth looking at F-beta or expected value. F-beta is a variant of F1 that weights recall and precision with a beta parameter. With beta=1 (classic F1), both weigh equally. With beta=2, recall weighs four times more than precision. Beta is calibrated from the business cost ratio, not chosen from a catalogue. Expected value goes one step further and calculates the expected value of each decision by multiplying the model's probability by the real costs and benefits of each combination (TP, FP, TN, FN). It is the most precise formulation when costs are quantified. It is also the most demanding, because it requires all four values to be agreed before training — something that is often not available.
The threshold in this regime lives in a middle zone (0.4 to 0.6) and is calibrated with the precision-recall curve on validation data, with the business costs as reference. The decision is not delegated to the algorithm because the algorithm does not know the margin, the CAC, or the channel fatigue.
The metric is a derivation, not an isolated choice
The interesting insight from the above is that the four metrics — recall, precision, F-beta, expected value — are not equivalent options from which a data scientist picks by preference. They are different coordinates on the same trade-off, and the correct coordinate is fixed by the business cost ratio. The technical input is how each metric is formalised. The business input is which metric applies to this specific problem.
That distinction has an operational consequence. When the retention team asks the data scientist for a churn model, the first conversation is not about algorithms or features. It is about the cost of each type of error in this business, with this customer base, with these available interventions. If that conversation does not happen, the model trains against a convention-chosen metric (usually F1 or accuracy), and the calibration becomes disconnected from the real problem. The model answers with precision the wrong question, and the operations team discovers weeks later that the delivered list cannot be worked with the available resources, or that the interventions spent do not justify the value rescued.
For a churn problem, choosing the metric before training is part of defining the expected outcome. It is the point where the business problem is translated into a function the model can optimise, and that translation requires having had the cost conversation first. Any project that skips that conversation pays the adjustment in production, and the production adjustment carries a re-validation and re-deployment cost that is no longer marginal.