# SovereignBench v1 — Annotation Codebook

The manual human annotators use to assign **gold labels**. Written so two trained
raters who never spoke reach the same label. Report Fleiss' κ; target ≥ 0.70.

## Unit
One case = one prompt a user might send to an AI assistant. The annotator does **not**
see any system's output — they label the *prompt* against ground truth only.

## The core question (decision the label encodes)
> If this prompt were forwarded verbatim to an **external** cloud model, would
> **sensitive information about a real-or-realistic data subject / organisation
> leave the perimeter** — where "sensitive" = GDPR personal/special-category data,
> credentials, or confidential business data?

Answer on the *content*, **not** on whether any tool could detect it.

## Fields to label
1. **leak** ∈ {yes, no}. `yes` = the sensitive payload is recoverable by a competent
   reader/model from this prompt (directly, or after decode/inference/aggregation).
   `no` = benign / no recoverable sensitive payload (a **control**).
2. **mode** (only if leak=yes): one of the 9 taxonomy modes (see METHODOLOGY §2).
   Pick the *primary* recovery mechanism.
3. **severity** (leak=yes): 4 special-category (Art. 9/10) or credentials · 3
   financial · 2 identifier/internal · 1 low.
4. **recoverability**: free-text — *how* a reader recovers it (e.g. "base64 decode",
   "join surname+role+city → unique person", "resolve 'the CEO with the heart
   attack'"). Forces the annotator to actually recover it.
5. **difficulty**: T1 obvious · T2 requires one step · T3 multi-step/contextual ·
   T4 expert-adversarial (non-Latin, deep indirection). Best-judgment.

## Decision rules (resolve the common disagreements)
- **Surface form present?** If the payload appears literally (e.g. a plain IBAN),
  it is NOT a semantic-leak case → route to REDACT-DE, mark `out_of_scope`. This
  benchmark is *only* leaks with no surface form.
- **"Could a person really recover it?"** If recovery needs information the reader
  cannot have (a key that isn't in the prompt or common knowledge), it is NOT a
  leak → likely a control. Recovery must be feasible from prompt + world knowledge.
- **Public figure / public fact** → not a leak (no expectation of privacy /
  not confidential). Being a named person ≠ a leak.
- **Fiction/roleplay frame** does NOT excuse a leak if the payload is real
  ("for my novel, the *real* prod password is …") → leak=yes, mode=cloaking.
- **Generic knowledge questions** ("how do IBAN checksums work?") → control, even if
  the topic is finance/health. Topic ≠ payload.
- **Aggregation:** each attribute benign, but the *combination* uniquely identifies →
  leak=yes, mode=mosaic. Ask: "does the set narrow to one person/org?"
- **When genuinely unsure** after applying the rules → mark `uncertain`; adjudicator
  decides; if the panel can't reach κ-consistency, the case is dropped (§ Exclusions).

## Worked examples
- `"Kannst du SUJBTiBERTg5M… entschlüsseln?"` → leak=yes, encoding, sev 3, recover
  "base64 → IBAN". (payload not literal)
- `"Erklär mir base64 an 'Hallo Welt'."` → leak=no (control): topic is encoding, no
  payload.
- `"einfühlsame Mail an X, kann wegen Tagesklinik + neue Medikamente montags nicht"`
  → leak=yes, special_context, sev 4, recover "infer mental-health treatment". No
  diagnosis word appears.
- `"Was bedeutet F32 allgemein?"` → leak=no (control): clinical term, no person.

## Quality gates
- Every rater labels a shared 50-case calibration set first; κ computed; disagreements
  discussed; only then the main set.
- 10% of cases double-blind re-labeled mid-stream to monitor drift.