SovereignBench: Measuring Semantic Data Leakage in AI Systems

0Status of this draft

1Introduction

The dominant control for “shadow AI” and cross-border data flow is redaction at the perimeter. It works well for literal personal data — an email address, an IBAN, a phone number — where a pattern exists to match. But a competent reader (or model) routinely recovers sensitive information from text that contains no matchable token: a base64 blob, a diagnosis implied by a treatment schedule, an identity reconstructed from three individually-innocuous attributes. For these, a surface-form redactor has, by construction, nothing to redact. If the perimeter relies on such tools, the data leaves.

This gap is under-measured. Literal-PII benchmarks assume a surface form; AI-safety suites target harmful generation, not confidential egress; “data leakage” in the LLM literature usually denotes training-set contamination, an unrelated phenomenon. SovereignBench measures the missing quantity directly, and — critically — is engineered to withstand the obvious criticism of a benchmark whose result favours a contributor’s product.

2Related work and scope

SovereignBench is deliberately narrow and complementary. It does not re-measure literal PII (covered by redaction benchmarks such as REDACT [7]), training-data contamination (the other “leakage”; [3]), harmful-content safety (MLCommons AILuminate [1]), or multi-agent tool leakage (AgentLeak [2]). It operationalises the outbound-governance case behind OWASP LLM02 “Sensitive Information Disclosure” [4], now the second-ranked LLM risk, and GDPR Art. 9/10 special categories. Explicit scoping is itself a validity measure: we do not relabel an existing benchmark.

3Construct and threat model

A semantic leak is sensitive information a competent reader can recover from a text but which no surface-form detector can locate, because the sensitive token does not appear literally — it exists only after decoding, inference, or aggregation. We measure the perimeter question: if this message is forwarded to an external model, does confidential content leave — regardless of whether it is spelled out?

4Taxonomy

Nine leak modes, each anchored to an external reference so the taxonomy is not ours alone: (1) encoding/transformation smuggling; (2) mosaic/aggregation (k-anonymity break, cf. statistical disclosure control); (3) special-category context without keywords (GDPR Art. 9/10); (4) business-confidential inference; (5) fragmented/cross-message identifiers (re-identification, cf. Sweeney [5]); (6) cross-lingual transliteration; (7) cloaking/fiction framing; (8) indirect reference/definite description; (9) format/unit obfuscation. Each case is tagged mode × language × domain × difficulty tier T1–T4.

5Corpus and provenance

Target ≥1,000 public + ≥500 held-out private cases, ≥40% benign controls. No single provenance is trusted; five streams, each tagged per case: incident-derived (archetype from public breach/DPA records — US HHS Breach Portal, UK ICO register, EDPB decisions, VERIS/VCDB — with fabricated values, so no real PII is shipped); real document templates (Arztbrief, bank statement, HR record) with fabricated values; expert red-team under written protocol; crowd-sourced adversarial submissions; and, in v2, naturalistic consented enterprise logs. A validator enforces that no protect-case payload appears as a surface form. Canary strings seed a subset to later detect training-set contamination.

6Ground truth

The v1 design specifies that every case is labeled by ≥3 trained annotators against a published codebook (leak? / mode / severity / recoverability), with disagreements adjudicated. We will report Fleiss’ κ; cases below κ = 0.70 will be revised or dropped. As of this draft no agreement statistic has been produced. Annotation is blind to system output, and no system under test labels its own data — this eliminates the circularity of an LLM grading cases its own family produced.

7Systems and metrics

Every system implements one adapter — text → restrict | allow — and is scored blind. The baseline field spans pattern/DLP (Presidio, MS Purview, Google Cloud DLP, AWS Comprehend, and commercial DLP), guardrail models (Llama Guard, NeMo Guardrails), cloud-LLM-as-redactor, and semantic judges.

We report two numbers, always together: Prevention Recall = P(restrict | protect) and Over-Block FPR = P(restrict | control), summarised by Youden’s J at the operating point and plotted as a prevention-vs-utility frontier. A system that restricts everything scores J = 0. How any such rate must be stated (unit, counts, strictness, interval, fail-safe share, build identity, frozen test set) is specified separately in the Leak Rate Reporting Standard. All headline numbers carry bootstrap 95% confidence intervals; paired system comparisons use McNemar’s exact test.

8Results

8.1 v0 pilot (37 cases)

On the 27 protect cases of the pilot that have no surface form, a high-recall PII detector recovers 0/27 (0%), zero in every mode, as the corpus construction predicts. Results of semantic systems are deliberately not reported here: the pilot was run by a contributor, not by an independent party, and no such figure is a leaderboard entry. They will be reported by the steward on the v1 corpus.

8.2 v1 pipeline validation

The v1 scorer was exercised on the draft public split with three stand-ins, not products: a regex-style surface detector, a block-everything strawman and a keyword heuristic standing in for a semantic judge (deliberately blind to the hardest tier T4). This validates the pipeline and the metrics only. The table is not a result about any real system.

We report the gradient, not a headline. If any system saturates the benchmark, we add harder T4 cases rather than celebrate — a benchmark with no headroom has no future signal.

9Threats to validity

Construct: “leak” is judgment-laden → codebook + κ. Internal (gaming): authored-to-win → held-out private split, external co-authors, pre-registration, blind scoring. External: synthetic values ≠ real distribution → real-incident structure, real document templates, naturalistic stretch goal. Contamination: canaries + private split + versioned re-release. Grader bias: human gold; any LLM assistance is a third-party model, never a system under test, validated against human labels.

10Governance and reproducibility

Scoring and the leaderboard are owned by a neutral steward; founding contributors supply methodology and the public corpus but do not grade. The dataset is to ship with a Datasheet [6], is versioned, and is licensed CC BY-SA 4.0 (code Apache-2.0). Drafts of the pre-registration, codebook, datasheet, generator and scoring harness are publicly readable (see the home page); the formal release follows the registry deposit and the appointment of a steward.

11Conclusion

Perimeter DLP creates a false sense of safety against a class of leaks it cannot see. SovereignBench makes that gap measurable, and does so under a design a competitor could accept as fair. The intended outcome is not that any one system wins, but that the field can finally distinguish surface redaction from genuine data governance.

References

1. MLCommons. AILuminate: v1.0 of the AI Risk and Reliability Benchmark. arXiv:2503.05731, 2025.
2. El Yagoubi, Badu-Marfo, Al Mallah. AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems. arXiv:2602.11510, 2026.
3. On benchmark data contamination in LLMs (“training on the benchmark”). arXiv:2409.01790, 2024.
4. OWASP. Top 10 for LLM Applications (2025): LLM02 Sensitive Information Disclosure. OWASP GenAI Security Project.
5. L. Sweeney. k-anonymity: a model for protecting privacy. IJUFKS, 2002.
6. T. Gebru et al. Datasheets for Datasets. CACM, 2021.
7. G. Vats et al. REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection. arXiv:2606.19881, 2026.