# SovereignBench v1 — Pre-Registration

*Pre-registration is written and time-stamped **before** cases are authored or any
system is run, then deposited (OSF). It locks hypotheses, metrics, and the analysis
plan so results can't be reverse-fit. This is the single strongest defence against
"you designed it to win."*

**Registered:** 2026-07-06 (draft; deposit to OSF before authoring the full corpus)
**Authors:** AI-Z Group (founding contributor) + external academic co-author (TBD:
HPI / KIT / Fraunhofer AISEC) + neutral steward (TBD: MLCommons / OWASP GenAI)

## 1. Research question
Do systems that reason about *meaning* (semantic redactors / LLM judges) prevent
sensitive-data egress that **surface-form** detectors (regex / NER / DLP) provably
cannot — and at what over-blocking cost?

## 2. Hypotheses (directional, pre-specified)
- **H1.** Surface-form/DLP systems achieve Prevention Recall ≈ 0 on the protect set
  (95% CI upper bound < 0.10). *Rationale: the corpus is validated so no payload
  appears as a surface form.*
- **H2.** At least one semantic system achieves Recall ≥ 0.70 with Over-Block FPR
  ≤ 0.15 (Youden's J ≥ 0.55), i.e. it opens a gap DLP cannot.
- **H3.** No system reaches Recall = 1.0 on Tier T4 (headroom exists; the benchmark
  retains discriminating power). *If violated, T4 is hardened before release.*
- **H4.** The semantic-vs-DLP recall difference is significant (McNemar, α = 0.05,
  Holm-corrected across systems).

## 3. Design
- **Units:** individual prompts (cases). **Between-systems**, within-case (paired).
- **Corpus:** ≥ 1,000 public + ≥ 500 held-out private; 9 leak modes × 6 domains ×
  4 difficulty tiers × {de, en}; ≥ 40% benign controls. Private split run only by
  the steward (anti-gaming). Generator + validator: `gen_corpus_v1.py`.
- **Ground truth:** ≥ 3 human annotators, codebook `CODEBOOK.md`, adjudication;
  inclusion requires Fleiss' κ ≥ 0.70 on the labeled set. **No system-under-test
  labels its own data.**
- **Systems:** ≥ 8 across DLP, guardrail models, cloud-LLM-redactors, semantic
  judges; each via the same adapter (`score_v1.py`), scored blind.

## 4. Primary & secondary outcomes (locked)
- **Primary:** Prevention Recall and Over-Block FPR, summarised by Youden's J at the
  system's default operating point; reported as the Recall–(1−FPR) frontier.
- **Secondary:** per-mode recall, per-difficulty recall, severity-weighted recall.
- All headline numbers: bootstrap 95% CIs (2,000 resamples, case-level).

## 5. Analysis plan (locked)
- H1/H2: point estimate + bootstrap CI vs the pre-set thresholds.
- H3: T4 recall < 1.0 with CI upper bound < 1.0.
- H4: McNemar exact test, paired protect cases, Holm correction.
- **Exclusions (pre-specified):** cases with κ < 0.70 → dropped before scoring;
  cases where the payload leaks as a surface form (validator fail) → fixed or
  dropped; duplicate near-identical cases collapsed. Exclusions reported with counts.

## 6. What would falsify our thesis (stated up front)
If a surface-form/DLP system reaches Recall ≥ 0.30, H1 fails and the "detectors are
blind to semantic leaks" claim is **wrong** — we report it. If no semantic system
beats J = 0.55, the category has no measurable advantage — we report that too.

## 7. Deviations
Any change after registration is logged in a dated `DEVIATIONS.md` with rationale.