7 min read
An ecommerce customer base. Two clearly distinct profiles: those who buy once a quarter, are subscribed to the newsletter and respond to reactivation campaigns; and those who arrive from a price comparison site, buy at the moment they find the best price, and do not return regardless of what they are offered. Both groups are in the same table. The repurchase propensity model trains on both without distinction.
The result is predictable. The model learns to rank the first group well, which has stable behavioural signals and majority representation in the dataset. The second group receives predictions that systematically overestimate its repurchase probability. Aggregate performance looks reasonable. Nobody notices until the marketing team reviews conversions by segment and discovers that half of last month's contacts were customers who were never going to come back. The root of the error lies in the problem definition.
Two populations, two dynamics, one single output variable
The confusion starts before opening the notebook. In both groups the target variable is repurchase in the next 90 days. The logic seems correct because the question is the same for both. The error is in assuming that the same question has the same causal structure in different populations.
For a recurring customer, the probability of repurchase depends on behavioural variables: time since last purchase, historical frequency, response to previous communications, categories explored. There is clear signal. The model can generalise that signal because the patterns are consistent within the group.
For a one-shot customer, those same variables are noisy or directly irrelevant. Historical frequency is low by definition. Response to communications may be high if they arrived via an active comparison channel, but that does not predict repurchase. The category they bought may be the only one they ever looked for at that price. The variables that explain the recurring customer's behaviour have no predictive power over the one-shot customer, and vice versa.
When a model trains on both, the correlations of the majority group dominate. The training gradient adjusts weights to reduce error on the whole set, and that set has more recurring customers than one-shots. The result is a model that works as a recurring-customer classifier applied mechanically to everyone.
What are the signs that two populations are mixed?
There are detectable signals before training the first model:
- Appearance of a bimodal distribution in the purchase frequency variable. This is the most direct. If the histogram of purchases per customer has two separate peaks, there are two populations. One with a mean of 1–1.5 purchases per period, another with a mean of 4–6. A single model will try to learn the dynamics of both with the same function.
- Correlations that reverse sign between segments. If the dataset is split between customers with a single purchase and customers with two or more, and the correlation between "time since last purchase" and "probability of repurchase" is calculated in each subgroup, that correlation can have the opposite sign. In recurring customers, more time without buying may predict an imminent repurchase (seasonal cycle, a drop in engagement that precedes an effective campaign). In one-shots, more time simply confirms they are not coming back.
- If the business team describes in the initial meeting that there are customers who "always come back" and customers who "bought on price", that is a hypothesis about two populations with different dynamics. This is an operational signal — and the most important one — and the one most often lost because it does not appear in the data until someone puts it on the table.
Three alternatives and a criterion for choosing
Once the two populations are identified, there are three ways to handle them.
The first is to build two separate models, one per segment. Each model learns the correlations of its population without interference. It is the option with the highest predictive power when the segments are large and have clearly different dynamics. The cost is operational: two training pipelines, two sets of relevant features, two monitoring processes in production. It is worth it when the minority segment has enough representation to train a reliable model and when error in that segment has significant business consequences.
The second is to include the segment as a categorical feature. The model learns that segment membership modifies the predictions. It is simpler than two separate models and works when the dynamics are different but not completely independent. The risk is that the model may ignore the feature if the effect is not strong enough or if there is correlation with other variables that already partially capture it.
The third is to explicitly model the interaction between the segment feature and the most important variables. Instead of letting the model discover the interactions, they are encoded — for example, "time since last purchase × recurring segment". This gives the model the information that the same variable has different effects in each population. It is more interpretable than two separate models and more expressive than a categorical indicator without interactions.
The criterion for choosing comes from the business, not the model. How large is the minority segment? How much does an error in that segment cost? How different is the observable dynamic in the data? If the segment is small, the dynamic is different and the cost of error is low, the categorical feature is sufficient. If the segment has scale, the dynamic is clearly different and the cost of error is high, two separate models.
The decision that is not made in the data lake
The critical moment of this analysis is the first conversation with the business team, before any data exploration.
"Do all the customers in the base buy for the same reasons?" is a business question. The answer is usually known by someone who works every day with the acquisition channel or with the CRM team. If the answer is "there are customers who come for price and don't come back, and customers who have a buying habit", that is the two-population hypothesis, stated in the language of business before looking at any data.
Data exploration confirms or refutes that hypothesis. Without the question, the exploration doesn't find the signal either, because nobody knows what to look for.
Aligning the problem definition with the business carries the same technical weight as training the model, and in propensity or behaviour prediction projects it is where the most important decisions are made. The data team and the business team must reach the same reading of the populations before writing the first script, so that the predictions do not end up pointing at the wrong population.
What to watch for when the model is already in production
If the prior analysis was not done and the model is already running, there are degradation signals that point specifically to this problem.
The first signal is conversion rates consistently lower in a subset of customers with predictions similar to the rest. If customers contacted with probability between 0.6 and 0.7 convert at 8% in one group and at 1% in another, the model has different calibration in the two groups. The predictions are not comparable across segments.
The second is that the poorly performing subset has a characteristic in common that the model does not have as a relevant feature. If that characteristic is "first purchase only", the hypothesis is clear.
The correction in production follows the same logic as the initial design: segment, analyse the dynamics separately, decide whether two separate models or an explicit interaction is the right answer. The cost of correcting late is higher than defining correctly from the start, but the analysis is the same.
Separating the populations before modelling is recognising that there are two different problems that look the same from the outside — more than a technical sophistication of the pipeline.