The story is usually told like this: at the Dutch Tax Administration, an algorithm went wrong. A risk-classification model that wrongly marked thousands of families as fraudsters. Blame the technology, hold a few accountability meetings about model choices and data quality, and move on with the agenda.
That telling is attractive because it's comfortable. If the technology failed, then the technology was at fault — and the organization gets to cast itself as the victim of a difficult discipline rather than the party at the controls. But a model doesn't decide that a citizen is suspicious. An organization decides that. The parliamentary inquiry into the childcare benefits scandal didn't speak of a technical incident. It spoke of unprecedented injustice — "ongekend onrecht". Those are not words about software. Those are words about governance.
Anyone who reads the scandal as a technology story draws the wrong lesson for their own organization. Especially now, as companies of one hundred to five hundred people are pulling AI into their core processes, it's worth looking at what actually broke.
What actually broke
Not the model. The decision architecture around it.
The Tax Administration's risk-classification system did exactly what it was built to do: it flagged files with an elevated risk profile. What was missing wasn't inside the system but around it. Nowhere was it recorded who was allowed to decide that a flag from the system could lead to consequences for a family. Nowhere did it say who had the right — no, the duty — to stop the system when its outputs no longer matched reality. And for the citizen who ended up inside it, there was no workable path of appeal in practice. Once labeled a fraudster, you had to prove you weren't. Suspicion had become the institutional default, and no one had ever taken that default as an explicit decision.
The system did exactly what it was allowed to do. Nobody had written down what it was not allowed to do.
That is the core. There was no malicious algorithm and there was no malicious civil servant. There was an organization that had handed decisions to a system without first recording who was allowed to make which decision, who could intervene, and how the person being decided about could defend themselves. The technology filled a vacuum the organization had allowed to form. Vacuums are dangerous because nobody feels responsible for them — and that's exactly why they grow.
Robodebt: the same shape, a different country
This is not a Dutch anomaly. Australia's Robodebt was the same failure in the same shape: a system that automatically raised debt notices against welfare recipients, based on an income average that structurally misread the reality of irregular earnings. Hundreds of thousands of citizens received debt letters that didn't hold up; the scheme was later ruled unlawful. There too, the temptation was to frame it as an IT failure. There too, the actual chain was the same: a political-administrative appetite for strictness, a system executing that appetite at scale, and nobody who had recorded where the execution should stop. Automated decision-making without recorded decision rights doesn't fail occasionally. It is the default failure mode.
What this means for your organization
You are not the Tax Administration. That's exactly your advantage. In an organization of one hundred to five hundred people, feedback loops are short: you see sooner when an automated decision comes out wrong, and you can intervene sooner. But that advantage evaporates the moment you delegate decisions to systems without writing down the boundary conditions. Being small doesn't protect you from the same pattern — it only gives you the chance to choose, right now, before it does.
The choice is concrete, and it doesn't have to wait for an AI strategy. Two instruments are enough to start:
- The Decision Rights Matrix (Beslisrechtenmatrix). For every decision you (partly) hand to a system: who decides, who advises, who gets informed, and on what timeline. Including the explicit stop right: who may pause the automation, and on what grounds. No matrix, no delegation. Decision-making that nobody has recorded is decision-making nobody can stop.
- The Non-Automation List (Niet-automatiseringslijst). A named list of decisions that stay human, with the reason attached. Dismissal, credit refusal, price differentiation between customers — whatever they are in your context. The strongest governance signal an organization can send is not a forty-page model policy. It's a short list of names and reasons stating what will never be automated.
Order matters: make decision rights explicit first, then delegate. Anyone who delegates first and builds the governance afterwards is constructing the appeals process when the first victims already exist. That was the order of the benefits scandal. It's also the order you see in almost every AI adoption that moves too fast: the tooling is in place, the ownership question was never answered.
The question isn't whether, but where not
AI is an accelerant. That is its power and its risk in a single word: the AI Accelerator amplifies what is already there — including the decisions nobody owns. An organization with clear decision rights gets faster, better decisions. An organization with unowned decisions gets faster injustice, at greater scale, with an audit trail nobody can explain. What we call AI noise — the noise that appears when tooling moves faster than the operating model — was carried to its ultimate consequence in the benefits scandal.
So the question is not whether your organization will use AI in decisions. It will. The question is whether you have written down where it may not. Anyone who can't answer that today doesn't have their decision architecture — and will discover the answer at the moment it hurts.
The benefits scandal wasn't a technical accident. It was an organization that had forgotten who was deciding. That mistake is free to prevent — but impossible to repair after the fact.