spb/firc0001_uqo

1# Annotation protocol for the SPID human-validation pilot (Stage D, package N11)23*Status: protocol and sampling frames delivered (2026-11-05); no label has been produced. The metrics of this protocol are the author task AD-2 (`improvements/FIRC-0001/authorization_requests.md`). Table 10 gate (d) therefore reads "not met — protocol revised, frame drawn, annotation pending".*45## 1. Objects and population67- **Unit:** one ledgered section (`extracted_sections`, 1,123,098 rows), i.e. exactly the text block the extractor received a verdict for. Two populations: **productive** sections (`n_mentions > 0`, N = 168,230) and **zero-mention** sections (N = 954,868; 419,361 of them never reached the model because `_PROJECT_GATE` rejected them).8- **What the annotator sees:** the full section text and, separately, `focus_text_seen_by_model` — the string actually sent (`"SECTION: <item>\n\n" + _focus(text)`, cut at 14,000 characters; `llm.py:125-126`). Both are in `~/firc0001_review/frames/frame_{dev,holdout}.jsonl` (M3U96b). The worktree copies (`results/annotation_frame/frame_{dev,holdout}.csv`) truncate the text to 2,000 characters and are for inspection only.9- **What the annotator must not see until labels are frozen:** the model's mentions (`frames/model_output_SEALED.jsonl`).1011## 2. Sampling design (seed 20261105; `code/audit/a11_annotation_frame.py`)1213- **Productive frame.** Strata = item group (10-K MD&A / other 10-K / 10-Q MD&A / other 10-Q / 8-K) × run (June backfilled / October) × truncation bin from N5 (shown fraction ≥ 0.9 / 0.25–0.9 / < 0.25) × section confidence bin (max mention confidence ≤ 0.7 / > 0.7) × category group (ma_integration / capex-type / technology-type / other). Allocation: proportional to the stratum population with a **floor of 10** and a **2× oversampling of the ≤ 0.7 bin**; strata smaller than the floor are taken whole. Result: 192 non-empty strata, **n = 1,677** (the floor dominates); weights w_h = N_h / n_h (`design_productive_strata.csv`, `weights.csv`). A nested **core subsample of ≈ 385 sections** (column `core`, proportional within the drawn strata, min 1 per stratum) is the recommended first pass; its weights are N_h / n_h,core.14- **Zero-mention frame.** n = 385 drawn without replacement with inclusion probability ∝ (2 if gate passed else 1) × (1 + log10 characters), so that long gate-passing sections (where a missed project is most likely) are over-represented; 285 gate-pass / 100 gate-reject in the draw; inverse-probability weights 1/π_i stored.15- **Split.** Every stratum is split at random into *development* (guide refinement, disagreement resolution) and *holdout* (reported metrics): 1,058 / 1,004 sections.16- **Precision of the design** (`design.json`): design effect 3.4 (productive) / 1.2 (zero); 95 % half-width ± 4.4 pp for a section-level precision near 0.5 (± 3.5 pp near 0.8) on the full productive frame; ± 2.4 pp for a false-negative rate near 5 % on the zero frame; MDE (80 % power, two-sided 5 %) of a gate-pass vs gate-reject difference in the false-negative rate ≈ 7.8 pp at p = 0.05.1718## 3. Labels1920Per section (both frames): **S1** does the text describe at least one concrete initiative under the construct of §5.1 (yes / no / unsure); **S2** list of initiatives found (free text, ≤ 10). Per model mention (productive frame, after unsealing): **M1** the mention corresponds to an initiative in the text (correct / partly / wrong / construct violation — e.g. share repurchase, financing instrument, dividend); **M2** `project_type` correct under the 21-category taxonomy (Appendix A); **M3** each structured field (years, locations, partners, amount) is literally supported by the text (the mechanical rates of N2 are the comparison point); **M4** status correct. Per project (linkage audit, optional): **L1** two mentions resolved to the same `project_id` describe the same initiative (same / different / unsure).2122## 4. Procedure23241. Two annotators, trained on 40 development sections, label independently; the guide is revised once after the development set and frozen before the holdout.252. Blinding: S1–S2 are labelled before `model_output_SEALED.jsonl` is opened; M1–M4 afterwards. The order of sections is shuffled; the stratum, run and confidence are hidden.263. Agreement: Krippendorff's α (nominal) for S1, M1, M2; prevalence-adjusted bias-adjusted κ (PABAK) reported next to Cohen's κ because S1 is rare in the zero frame. Disagreements adjudicated by a third reader; adjudicated labels are the reference.274. Estimators (holdout only, inverse-probability weighted): section-level precision P(S1 = yes | productive), mention-level precision P(M1 = correct), category accuracy P(M2 | M1 = correct), field-support rates (compare with N2), false-negative rate P(S1 = yes | zero) overall and by gate status (gate recall bound), construct-violation share by item (compare with N6). 95 % intervals by stratified bootstrap of sections (1,000 replicates).285. Reporting: Table 10 gate (d) becomes "met / not met" on the holdout numbers; §8.2 prints n per stratum, α / PABAK, and every estimate with its interval; the development-set numbers are reported separately and never pooled.2930## 5. Files3132`results/annotation_frame/{design.json, design_productive_strata.csv, frame_composition.csv, weights.csv, frame_dev.csv, frame_holdout.csv}` (worktree); `~/firc0001_review/frames/{frame_dev.jsonl, frame_holdout.jsonl, model_output_SEALED.jsonl, weights.csv, README.md}` (M3U96b, full text). The X1 stability run (AR-LLM-1), if authorised, should reuse the same 500-section stratified subset of this frame so that determinism and human agreement are measured on identical sections.33