6 min read
A team automates the extraction of data from invoices. They write rules: find the invoice number where it usually appears, extract the amount following the known format, flag as "review" any case that does not fit. The system handles 8 out of every 10 invoices without human intervention. The remaining 2 are reviewed by a person.
Months later the main supplier changes. Formats multiply. The rules start failing more often. An AI-based extractor is brought in, trained on thousands of scanned invoices. The system now handles 9.5 out of every 10. The person who used to review 2 cases a day now reviews 10.
Nothing failed technically. The AI system makes fewer errors and processes more volume than the previous rules. The arithmetic breaks somewhere else. The type of review the system demands changed, and nobody designed the human flow that corresponds to that new demand. From this point, the question is where human work ends up living when automation shifts from rules to models, and how to design that place before the system goes into production.
Where the human lives in a rules-based process
In a rules-based automated process, human work has a predictable location: the exceptions. What the rule could not classify falls into a review queue. The person resolves the case, and when the process is well designed, that resolution feeds back into the rule so that next time that variant is covered automatically.
The human acts only on what the system explicitly flagged as uncertain. They know that what arrives in their queue is unusual and worth looking at carefully. They also know how the system will react, because the rule is deterministic. Given the same input it produces the same output, always. Over time, if the process is well designed, the volume of exceptions tends to fall. Each reviewed case is an opportunity to refine the rule.
Where the human lives in an AI process
With AI, the geography of human work changes. The system returns an answer for almost all cases, including those that would previously have fallen into the review queue. Each answer carries an implicit uncertainty that rules did not have.
The human no longer intervenes only when the system flagged a case as uncertain. They also intervene when the system has responded, to verify whether it responded correctly. And that verification work cannot be done by negative sampling, because AI system errors do not always announce themselves. An extractor that infers the amount on an invoice may confuse the subtotal with the total. The invoice goes out as processed, nothing fails technically, and the error appears weeks later when the payment system generates a discrepancy with the supplier. For that error not to reach production, you have to validate beforehand — and that validation has a cost in people's time.
The human ends up working more, on cases the system considers resolved, using judgement that was previously reserved only for exceptions. The type of review AI demands is different from the one rules demand, and if the process was designed for the second, the first will overwhelm it.
What happens when validation is not designed
The most expensive error in an AI automation project is usually the absence of a well-defined human place beside it. When validation is thought about after the system is implemented, it appears in one of two forms.
In the first, someone decides to review everything. The person reviews every system output, one by one. The manual process had a volume; the new one has that same volume plus the time to run the system and wait for its result. Automation did not reduce the work — it reorganised it with an additional step.
In the second, someone decides to review nothing. The system's errors enter the data flow undetected. Weeks later some report does not add up, some customer complains, some payment is made at the wrong amount. The organisation loses trust in the system, and the process reverts to manual with the additional conviction that "AI doesn't work".
Both conclusions are preventable. They are avoided by designing from the start where the human will be and with what criterion they will intervene.
The threshold as a business decision
An AI pipeline that works in production has explicit pieces from the start: the model that returns the answer, a confidence measure associated with each answer, and a threshold that separates what the system resolves on its own from what goes to human validation.
That threshold is a business decision that determines how much human work is accepted and what error level is tolerable when validation does not occur. You can start conservatively, with a high threshold that sends many cases to review, and adjust it as the system proves reliable on certain types of case. Or you can start with a lower threshold, accepting a known percentage of errors in exchange for processing speed.
What does not work is not taking that decision. A system in production without an explicit threshold and without a defined human flow discovers its own operation on real operations, and that exploration is paid in hours of people resolving consequences that the system design generated. The cost appears four weeks after the system went live.
The mixed pipeline as the default architecture
Most processes worth automating are neither purely rules-based nor purely AI. They are mixed: a deterministic part that rules handle well, and a part with variability that requires a model. The design consists of identifying where one ends and the other begins.
Rules cover the predictable: known formats, structure validations, numerical thresholds. AI covers the variable: formats the rules never saw, content ambiguity, cases that are recognisable but cannot be enumerated in advance. The human is placed at the point of highest uncertainty, where the system has the least confidence in its own output. There they calibrate the system and validate the boundary between what the model resolves with confidence and what needs review. It is work of lower volume than the old exceptions, but of greater responsibility per case.
The question that structures this design is where the human will be, and what they will see when they intervene. Answering that question before building the system determines whether the result is a more efficient process or a more complicated one with the same operating cost as the manual process it replaced.