The leak no detector can see.
Some of the most sensitive data an AI ever handles never appears as text a scanner can match. It exists only after decoding, inference, or aggregation. SovereignBench measures whether a system stops it, and fixes how any leak rate must be stated so that numbers can finally be compared.
What exists, what does not.
A benchmark that overstates its own maturity is marketing. This is the state of each part, as it is today.
Reporting standard
Version 0.1: ten rules for stating a leak rate, a calculator and a report checker that run in your browser.
Benchmark v1
Specification, codebook, datasheet, generator and scorer exist as drafts. 123 public-split cases are drafted; the target is at least 1,000.
Independent scoring
No steward is appointed and no system has been scored by a third party. No results are published.
Pre-registration
The analysis plan is written (2026-07-06) but not yet deposited with a timestamping registry.
Pattern and DLP tools operate on surface forms. These leaks have none.
A base64-wrapped IBAN, a mental-health status implied by context, an identity reconstructed from three innocuous attributes: a regex or NER detector finds nothing to redact, because there is nothing on the surface to find. Only a system that reasons about meaning can catch it.
SovereignBench is deliberately built so systems can lose. The hardest tier is designed to defeat even strong semantic judges, so the benchmark keeps discriminating power. A tool that scores 100 % would signal that harder cases must be added, not that the category is solved.
A leak rate is only a number until it says what it counts.
The same system, on the same traffic, honestly yields several different "leak rates" depending on the unit, what is excluded and how strictly partial protection is judged. The standard fixes those choices in advance, so two reports can be set side by side.
Illustrative arithmetic, not a measurement of any system. Scale 0 to 10 %. Which of these a buyer should care about depends on the question, which is why the standard requires the unit, the counts and the fallback share to be stated, and a strict headline.
Say the unit and show the counts
Requests, not values, as the headline. Always k of n in plain numbers.
Fix the class set and score strictly
Name the protected classes up front. A partly masked value counts as leaked.
Give the interval and the fallback share
95 % Wilson interval; with zero leaks, the upper bound, never "0 %". Report separately what was protected only because a component failed safe.
Over-block, identity, frozen set, repeats
Leakage never without over-blocking. Name the exact build. Freeze and hash the test set before measuring. Repeat runs and keep the raw records.
Who stops the leak, and at what utility cost?
Two numbers, always together: prevention recall (did the payload stay off-perimeter?) and over-block rate (how much benign traffic was needlessly restricted?). Blocking everything is not a win.
| System class | Prevention recall | Over-block (FPR) | Youden J | Status |
|---|---|---|---|---|
| Pattern / DLP / NER (surface-form) | — | — | — | awaiting independent scoring |
| Guardrail models | — | — | — | awaiting independent scoring |
| Cloud-LLM redactors | — | — | — | awaiting independent scoring |
| Reasoning-based / semantic systems | — | — | — | awaiting independent scoring |
| Block-everything (reference) | 1.00 | 1.00 | 0.00 | J = 0 by definition |
No self-reported scores, from anyone, including the operator of this site. A result enters the leaderboard only when an independent party has run it on the full corpus under the pre-registered protocol. We will report the full gradient and confidence intervals, never a vendor headline. Until then the board stays empty on purpose: a benchmark whose author fills in the winner is marketing, not measurement.
Designed to survive a hostile review.
Every design choice answers the question a reviewer would ask first: "how do we know you didn't build this to win?" Each pillar below states whether it is already in place.
Hypotheses locked first
Design, metrics and analysis plan are written down before any system is run, including the results that would prove us wrong.
Written · deposit pendingLabeled by people, not our model
At least three annotators, a published codebook, adjudication, and a reported Fleiss' κ with a target of 0.70. No system labels its own data.
Codebook drafted · κ not yet measuredScenarios from real incident types
Archetypes drawn from public breach and data-protection-authority records; values are fabricated. Real structure, no real personal data.
Partly authored · 20 of 123 drafted cases incident-derivedYou can't tune to what you can't see
A private split, run only by the steward, plus canary strings to detect training-set contamination.
Designed · steward neededFrontier, not a headline
Prevention recall and over-block rate with bootstrap 95 % intervals, McNemar tests for paired comparisons, breakdowns by mode and difficulty.
Implemented in the scorerWe don't grade our own homework
Scoring and the leaderboard belong to an independent steward. Founding contributors supply methodology and the open corpus.
Steward soughtAn open benchmark, headed for independent ownership.
SovereignBench is designed to be handed to a neutral steward (a standards consortium or an accredited institute) that owns scoring and the leaderboard. Until one is appointed, AI-Z GmbH operates this site as founding contributor. AI-Z develops a product of the kind this benchmark measures. That is a conflict of interest, which is why the operator accepts no self-reported scores, publishes the standard before any result, and will not score systems.
Drafts, not a release. The files above are working drafts and are publicly readable as such. The public-split corpus and the scorer are served from /benchmark/ in the same state. The formal v1 release follows the registry deposit and the appointment of a steward. The held-out private split is never published. Corpus licence CC BY-SA 4.0, code Apache-2.0. No real personal data is distributed.
Break it.
The benchmark only stays honest if outsiders can attack it. Submit a leak our best system misses, propose a new leak mode, comment on the reporting standard, or join as a steward or annotator. Adversarial contributions are the point.
Messages go to the operator's mailbox (info@ai-z-group.com) until a dedicated, steward-run address exists.