Some of the most sensitive data an AI ever handles never appears as text a scanner can match — it exists only after decoding, inference, or aggregation. SovereignBench measures whether a system stops it.
A base64-wrapped IBAN, a mental-health status implied by context, an identity reconstructed from three innocuous attributes — a regex or NER detector finds nothing to redact, because there is nothing on the surface to find. Only a system that reasons about meaning can catch it.
SovereignBench is deliberately built so systems can lose: the hardest tier defeats even strong semantic judges, so the benchmark keeps discriminating power. A tool that scores 100% signals we must add harder cases — not that the category is solved.
Two numbers, always together: prevention recall (did the payload stay off-perimeter?) and over-block rate (how much benign traffic was needlessly restricted?). Blocking everything is not a win.
| System class | Prevention recall | Over-block (FPR) | Youden J | Status |
|---|---|---|---|---|
| Semantic judge (reasoning) | high | low | large | v0 pilot: 27/27 |
| Pattern / DLP / NER (surface) | ~0 | ~0 | ~0 | v0 pilot: 0/27 |
| Block-everything (strawman) | 1.00 | 1.00 | 0.00 | reference |
| Guardrail models · cloud-LLM redactors · commercial DLP | — | — | — | v1 field — open |
Honesty note. The v0 pilot (37 cases, one detector vs the production semantic judge) established that the effect is real: 0/27 for the detector, 27/27 for the judge. Those numbers are a pilot, not the leaderboard. The v1 leaderboard — ≥1,500 cases, human-labeled with inter-annotator agreement, a full field of commercial systems, scored by an independent steward — is in progress. We publish the gradient and the confidence intervals, not a headline.
Every design choice answers the question a reviewer would ask first: “how do we know you didn’t build this to win?”
Design, metrics, and analysis plan are registered before any case is written — including the results that would prove us wrong.
≥3 annotators, a published codebook, adjudication, and reported Fleiss’ κ ≥ 0.70. No system labels its own data.
Archetypes drawn from public breach and DPA records; values fabricated. Real structure, no real PII shipped.
~40% of the corpus is a private split run only by the steward, plus canary strings to detect training-set contamination.
Prevention recall and over-block FPR with bootstrap 95% CIs; McNemar tests for paired comparisons; per-mode and per-difficulty breakdowns.
Scoring and the leaderboard are owned by an independent steward. Founding contributors supply methodology and the open corpus.
SovereignBench is designed to be handed to a neutral steward (e.g. a standards consortium or accredited institute) that owns scoring and the leaderboard. Founding contributors — including AI-Z Group, whose production system motivated the v0 pilot — supply methodology and the public corpus but do not score. Academic co-authorship and a public “break-it” process keep it honest.
The benchmark only stays honest if outsiders can attack it. Submit a leak our best system misses, propose a new leak mode, or join as a steward or annotator. Adversarial contributions are the point.