7 min read
The model was good. The validation metrics were solid, the backtesting closed, the technical team had reason to be satisfied. Three months after going into production, nobody was consulting it before making decisions. The business process that was supposed to consume its output kept operating exactly as before.
The requirements that lead a Data Science project to be used by the business are different from the criteria that make a model pass technical validation. Most projects don't reach production, and of those that do, a significant fraction ends up without real use — the algorithm is rarely the cause. Those that do work share three conditions, and all three are defined before the first model is trained. Without any one of them, the most likely result is a technically correct deliverable that nobody consumes.
A sponsor with authority over the process
The first condition is a distinction that tends to be overlooked. Someone who wants the model is not the same as someone with authority to change the process that will consume it. The first may be the marketing director who has spent months asking for something with machine learning, or the analytics lead convinced that data will transform operations. Their support is necessary, but not sufficient. For the project to end up being used, someone else needs to be in the room from the start.
The sponsor is not asked to understand how the model works. The question that needs a clear answer is: who has the authority to redesign the decision process around the model's output? If the repurchase propensity model produces a customer prioritisation every week, someone has to have the authority to make the contact team work with that prioritisation instead of the previous criterion. If that person is not involved from the start, the model reaches production and finds that the decision process did not expect it.
The criterion for knowing whether the right sponsor is present is concrete. If the answer involves convincing someone else who is not in the conversation, the project has a problem before it starts. Projects that end up being used almost always have that person in the scope definition, with visibility into what process will change when the model is available and with authority to make that change happen.
An integration designed from the start
The model reaches production and the question arises of how the business team will access the output. When the integration was not thought through from the start, the technical team has to build it after the model is already trained, with constraints that were unknown when the solution was designed. The output may not have the format the destination system expects. The process may need data the model does not produce. The update frequency may not match the frequency at which the team needs the information.
These problems are solvable, but each has a cost. The cost accumulates. At some point in the late integration process, the project loses the support it had when it started, because the time elapsed has already exceeded the expectation of when something useful was going to be available. Projects that work design the integration alongside the model. From week one, it is known which system houses the decision process, what format that system expects, what the team must do differently when the model emits its output.
There is a less obvious consequence of designing the integration from the start. The model that is trained when it is already known how it will be consumed is a different model from one trained without that information. The variable that predicts best may not be the variable available in production. The decision threshold may change if the business process has capacity constraints. The integration is an input to the model design — more than a final technical step.
A cadence that matches the process
Data Science models have an update cadence. Business processes have a decision cadence. When the two do not match, the model loses value even if the prediction is correct. A churn probability model that updates monthly for a retention team that contacts customers every week has a problem. The team needs to know which customers are at risk this week, and the model tells them which customers were at risk at the start of the month. By the fourth week, part of that information no longer reflects current reality.
The inverse case also occurs. A model that generates daily predictions for a monthly decision process creates unnecessary work. The team receives updates they cannot consume at that frequency. Cadence is a question about when the business team makes the decision the model informs — beyond the computational cost of updating the model. If the supply chain team decides on replenishment on Monday mornings, the demand forecasting model needs its output available before that meeting, not the previous Thursday and not in real time.
This alignment requires a conversation between whoever builds the model and whoever will use it, and it must occur before the model is trained, because the update cadence affects how much historical data is used to train it and what type of validation makes sense. Teams that define it from the start end up with a model the business knows how to consume in the rhythm of the real process.
What changes when the three conditions are met
The three conditions cannot be resolved retroactively. What was not agreed before training the first model appears as debt once the process is already underway, and that debt is paid with integration adjustments, model re-validation and erosion of the receiving team's trust.
The first months of production show the difference. The sponsor with authority can redesign the process when the model recommends it. The integration designed from the start does not have to adapt to constraints nobody knew about. The aligned cadence makes the output arrive when the business team needs it. What projects that reach production have in common is not the technology stack, the type of model, or the size of the technical team. It is that someone, before the first model was trained, answered the three questions that define those conditions: who has authority over the process, how will the output reach the place of decision, and when does it need to be available.
These questions come before any technical choice. Answering them well allows for choosing the simplest algorithm that solves the problem and reaching production with that. The distinction between archived projects and projects in use is rarely made by the data or the models. It is made by the conversations that happened (or did not happen) before the project started.