Automated Decisions and the Provable Safeguard
From 1 July 2026, New Zealand law permits the Ministry of Social Development to make benefit decisions by automated electronic system, "with appropriate safeguards" [1][2]. The public debate has asked whether the safeguards are adequate. This note asks a narrower and more answerable question: whatever the safeguards are, what would make them provable — rather than merely asserted? It distinguishes three tiers of safeguard — asserted, enforced, and provable — documents, from the public record, why the difference between them decided the two most consequential automated-decision failures of the past decade, and describes a verification anyone can run in about five minutes. It asserts no failure by any New Zealand agency. It documents a structural distinction, and identifies the instrument the highest tier requires.
1. Scope
This is a technical note, not a submission on policy. It takes no position on whether any decision should be automated, and it names no individuals. The overseas failures cited in §3 are historical, documented by royal commission and by human-rights investigation, and are cited as evidence of a structural pattern — not as a prediction about any New Zealand programme. Throughout, the same distinction is held as in the companion health brief [12]: a record may exist and still be revisable, incomplete, or internal; evidence is a sealed, contemporaneous record a party outside the operating agency can verify without trusting the agency. The gap this note describes is in the second.
2. The Commitments — what has been promised
Welfare. The Social Security (Modernisation) Amendment Act, passed under urgency on 29 May 2026, allows MSD to "approve the use of an automated electronic system … to make any decision, exercise any power, comply with any obligation, or take any other related action under any specified provision, with appropriate safeguards" [1]. The stated safeguards include human oversight and protections against bias [1]. MSD describes the class of decisions to be automated precisely: "a rules-based decision is made using clear, set criteria based on the information we have about a client, where no discretion is required" [1][2].
Assessment. NZQA has marked more than 55,000 literacy Writing assessments using an Automated Text Scoring tool since May 2025, with "results quality assured by human check-marking" — experienced markers re-checking over a third of results, concentrated at the achievement boundary, with the human mark prevailing wherever the two differ [3][4]. NZQA's published commitments place this under five principles of the Public Service AI Framework, ending in Accountability, and describe the posture as "human at the helm" [4][5].
Neither agency's statement is doubted here. Both are taken at face value, and both are commendably specific. The question this note asks is structural: what instrument records that the promised safeguard operated — each time, in order, before the decision issued — in a form someone outside the agency could check? At present, in both cases, the public answer is the agency's own account of its own process. That is not an accusation. It is the definition of an asserted safeguard, and until recently it was the only kind available.
3. The Failure Shape — what actually broke, twice
Australia — Robodebt. Australia's automated debt-raising scheme is remembered as an AI failure. It was not. There was no model, no learning, no black box. It was a sequence failure: the scheme's lawful process required actual fortnightly income to be verified before a debt was raised, and the automated system skipped that step — substituting an annual average — and raised the debt anyway, hundreds of thousands of times. The scheme was found unlawful; a settlement approaching A$1.8 billion followed; a Royal Commission reported in 2023 [6][7]. Two structural facts matter for this note. First, the failing step was not exotic — it was a known, nameable, required verification that the system was permitted to proceed without. Second, establishing what the system had actually done, to whom, in what order took years of forensic reconstruction, because nothing had recorded the integrity of the process at the moment each decision was made. The Royal Commission was archaeology. It could have been a lookup.
The Netherlands — the childcare benefits scandal. The Dutch tax authority's fraud-detection process wrongly accused roughly 26,000 parents of fraudulent benefit claims, with documented discriminatory effect; families were ruined, children were taken into care, and the government resigned over it in 2021 [8]. Here too, the decisive harm was not a clever algorithm but a process in which required checks — proportionality, human reconsideration, lawful data use — were asserted to exist and could not be shown to have operated in the individual case, until reconstruction after the fact.
In both failures, a safeguard existed on paper. In neither case did the system structurally require the safeguard step before proceeding, and in neither case did any decision leave a contemporaneous, tamper-evident record that the step had run. The safeguard was asserted. It was neither enforced nor provable — and the difference was measured in years of inquiry, billions in remediation, and lives.
4. Three Tiers of Safeguard
| Tier | Definition | What it survives |
|---|---|---|
| Asserted | A policy, press release, or framework states that the check happens. The system itself does not require it, and no independent record is produced. | Good faith and good weather. Robodebt operated for years at this tier with its safeguards formally in place. |
| Enforced | The system structurally cannot proceed past a skipped or out-of-order step. A decision with a missing safeguard step is not "flagged" — it is denied before execution. | Load, haste, and drift — the ordinary conditions under which asserted checks quietly stop happening. Protects the agency in real time. |
| Provable | Every decision leaves a cryptographically signed, sealed, tamper-evident receipt of the steps that ran, in order — verifiable by a party outside the agency, offline, against published keys, without trusting the agency's servers. | The inquiry. When a decision is challenged — by a review, an Ombudsman, a court, a journalist — the account of process integrity already exists, fixed at decision time, and checking it is minutes, not years. |
The tiers are cumulative in value but separable in mechanism: enforcement without proof protects the agency while it operates; proof without enforcement at least documents violations honestly. Together they are the difference between "our safeguards are appropriate" and "here is the evidence, check it yourself."
5. The Observation — rules-based decisions are the easy case
MSD's own description of what will be automated — "a rules-based decision … using clear, set criteria … where no discretion is required" [1] — is, in technical terms, a declared sequence: a finite set of named steps with defined inputs and a defined order. NZQA's check-marking commitment has the same shape: score, then human check at the boundary, then release. This matters because the declared sequence is precisely the case in which enforcement and proof are cheapest and strongest. None of the open philosophical problems of AI safety — alignment, interpretability, emergent behaviour — need solving to make a rules-based decision provable. The step order is already written down. What is missing is only the instrument that (a) refuses to let a step run out of order, and (b) seals a verifiable record that it didn't.
6. The Instrument — and a five-minute test
The requirements for an evidence-grade record are specified, neutrally and vendor-independently, in the companion Completeness Specification: created before the action, independent of the system being recorded, cryptographically signed and offline-verifiable, and irreversibly sealed so the account cannot later be reopened or rewritten without detection.
Each decision declares its steps up front — inputs read, criteria applied, human review where the rules require one, decision issued, sequence sealed. An external gate evaluates every step before it runs: a step out of order, a skipped review, a replayed or stale request is denied, not logged. Each allowed step produces a signed receipt chained to the one before; at the final step the sequence seals. Anyone — the client, their advocate, a reviewer, a reporter — can verify the signatures and the chain offline against published keys, with no call back to the operator. The safeguard stops being a sentence in a press release and becomes a checkable object.
The test. This claim is designed to be checked rather than believed, and checking it takes about five minutes:
1. Run the live demo — it executes a real multi-step sequence against the production gate and shows each step being allowed, denied, or sealed. Note the sequence ID it gives you.
2. Paste that ID into the verification tool — the report returns every receipt with its raw signature and the exact signed bytes.
3. Verify independently: fetch the published public keys and run standard Ed25519 verification over any receipt in your own code, offline. Then flip one character in the signed content and watch it fail. The full API contract is in the documentation.
What a pass demonstrates: the record of the sequence is signed, ordered, sealed, and verifiable by you — a party with no relationship to the operator, using no code of the operator's, making no network call to the operator. That is what tier three feels like from the outside, and it is the test any vendor of "safeguards" — this one included — should be held to.
7. A Deliberate Boundary — what this does not do
An enforcement receipt does not make a decision correct. It does not detect a biased rule, does not compensate for flawed data, and does not replace review, appeal, or the human judgement the rules require — it witnesses that judgement's place in the sequence; it cannot witness its quality. A wrong rule, faithfully followed, produces impeccable receipts of a wrong rule being followed.
That limitation is also the point. When every decision carries a verifiable account of what ran, in what order, against which criteria, the dispute narrows to where it belongs: the rule itself, examined in the open — rather than years of forensic argument about what the system even did. Robodebt's victims did not primarily lack a better algorithm. They lacked, for years, any means of showing what had been done to them. The receipt is that means, held in advance.
And to be plain about the home ground: this note asserts no failure, breach, or bad faith by MSD, NZQA, or any New Zealand agency. Both agencies named here have made their safeguard commitments publicly and specifically, which is to their credit — and is exactly what makes the commitments capable of being made provable.