How to state a leak rate.
When a vendor says "99 % accurate", nobody can tell what was counted. This standard fixes ten choices so that two leak rates can be set side by side. It sets no pass mark. It only decides how a result must be stated.
Why a reporting standard
A leak rate depends on choices that are rarely written down: whether a request or a single value is the unit, which data classes count, whether a half-masked value is "protected", how many requests were never inspected because a component failed over to a safe default, which software build was tested, and whether the test set was used to tune the system. Change any of them and the same system produces a different number. A claim that does not say which choices were made cannot be compared with anything.
The standard applies to any system that decides whether a text may leave a perimeter, for literal personal data as well as for the semantic leaks SovereignBench targets. It prescribes how a result is stated, not how a system is built.
The ten rules
The unit is the request
The headline counts requests (one task sent to the system), not values. A value-level rate is diluted by unproblematic values and may be added as a second figure, labelled as such.
Show k of n
State the number of requests that contained a protected value (n) and the number of those that leaked (k) as plain integers. A percentage alone is not a report.
Name the protected classes first
List the data classes that count (for example direct identifiers and the special categories of GDPR Article 9) before measuring. The headline covers requests containing at least one such value.
Partial protection is a leak
A value that is only partly masked counts as leaked. A lenient figure may be reported next to the strict one, clearly labelled lenient. The headline is the strict one.
95 % interval; bound, not zero
Give a 95 % Wilson interval. If no request leaked, report the upper bound of the interval. "0 % leakage" without a sample size is not a result.
Separate recognition from fail-safe
Report how many requests were protected because the system recognised the data, and how many only because a component was unavailable and the system failed over to a safe default. The second group says nothing about detection quality.
Leakage never without over-blocking
Report in the same place how many benign requests were needlessly restricted or degraded (k of n, with an interval). A system that blocks everything leaks nothing.
Name exactly what was tested
Product version, build or commit identifier, the configuration that affects routing or filtering, and the measurement date. A development build is labelled as one.
Freeze and hash the test set
Publish the hash of the test set, its languages and whether it is synthetic or real. State that it was fixed before the measurement and not used for tuning.
Repeat, and keep the raw records
Run the measurement at least three times and report the spread. Keep the per-request records and publish their hash so a third party can recompute every figure.
Beyond the ten rules: do not call a test result "anonymous" or a protection "guaranteed". Say what was tested and what the result was. Where a regulator or auditor applies a legal test, it applies it to the documented, repeatable evidence this standard asks for, not to a single percentage.
Claims that do not meet the standard
| Claim | What is missing |
|---|---|
| "99.5 % accuracy" | Unit, counts, class set, strictness, test set. Token-level accuracy is dominated by tokens that are not personal data at all. |
| "2× higher precision than X" | A ratio without an absolute value, a test set or a build. Precision says nothing about what leaked (that is recall). |
| "No sensitive data ever leaves" | An absolute statement that no finite test can establish. The most a test supports is an upper bound at a stated sample size (R5). |
| "0 % leakage in our tests" | Sample size and upper bound. 20 clean requests are compatible with a leak rate of up to 16 %. |
| "F1 0.93 on a public dataset" | Often measured on the distribution the model was trained on, at token level, with lenient overlap. It says little about whole requests (R1, R4, R9). |
| "Independently validated" | Scope, date, build and a link to the report. Independence needs a named party and a published method. |
A statement that meets the standard
Fill the fields; every placeholder corresponds to a rule.
Sample-size calculator
How many requests make a result mean something? Both tools use the Wilson score interval at 95 %.
Interval for a result
Requests needed to show a bound with zero leaks
| Upper bound with 0 leaks | Requests needed |
|---|
One leak in 100 requests does not show a rate below 1 %: the interval for 1 of 100 reaches about 5.4 %. If the true rate is clearly above the target, more requests do not rescue the claim; they only narrow the interval around the real number.
Report checker
Paste a leak report as JSON (schema: leak-report.schema.json). The page checks it against R1 to R10 and recomputes the interval. Nothing is uploaded: the check runs in your browser. A pass means the report is stated correctly under this draft standard. It says nothing about whether the numbers are true or the result is good.
Check a report
Open questions in v0.1
- Does "failed over to a safe default" count as protected? v0.1 counts it as protected for the leak rate but demands it be reported separately (R6). Some buyers will want to treat it as a failure of the control. Both views stay computable from the same report.
- Which classes are critical? The standard requires a named list (R3) and does not supply one. A shared minimal list is a candidate for v0.2.
- Strict versus lenient. v0.1 makes strict the headline (R4). Whether to define degrees of partial masking is open.
- Measurement point. A system's own account of what it sent is weaker than a capture of what left the perimeter. v0.1 asks for the build and configuration (R8) but does not yet require a network capture.
- Multi-turn conversations and attachments are not covered.
Use and versioning
Licence of this text: CC BY 4.0 (proposed). Versions are dated and kept; a report states the version of the standard it follows (schemaVersion). Send comments to info@ai-z-group.com.