# Annotation protocol for the SPID human-validation pilot (Stage D, package N11)

Status: protocol and sampling frames delivered (2026-11-05); no label has been produced. The metrics of this protocol are the author task AD-2 (improvements/FIRC-0001/authorization_requests.md). Table 10 gate (d) therefore reads "not met — protocol revised, frame drawn, annotation pending".

# 1. Objects and population

• Unit: one ledgered section (extracted_sections, 1,123,098 rows), i.e. exactly the text block the extractor received a verdict for. Two populations: productive sections (n_mentions > 0, N = 168,230) and zero-mention sections (N = 954,868; 419,361 of them never reached the model because _PROJECT_GATE rejected them).
• What the annotator sees: the full section text and, separately, focus_text_seen_by_model — the string actually sent ("SECTION: <item>\n\n" + _focus(text), cut at 14,000 characters; llm.py:125-126). Both are in ~/firc0001_review/frames/frame_{dev,holdout}.jsonl (M3U96b). The worktree copies (results/annotation_frame/frame_{dev,holdout}.csv) truncate the text to 2,000 characters and are for inspection only.
• What the annotator must not see until labels are frozen: the model's mentions (frames/model_output_SEALED.jsonl).

# 2. Sampling design (seed 20261105; code/audit/a11_annotation_frame.py)

• Productive frame. Strata = item group (10-K MD&A / other 10-K / 10-Q MD&A / other 10-Q / 8-K) × run (June backfilled / October) × truncation bin from N5 (shown fraction ≥ 0.9 / 0.25–0.9 / < 0.25) × section confidence bin (max mention confidence ≤ 0.7 / > 0.7) × category group (ma_integration / capex-type / technology-type / other). Allocation: proportional to the stratum population with a floor of 10 and a 2× oversampling of the ≤ 0.7 bin; strata smaller than the floor are taken whole. Result: 192 non-empty strata, n = 1,677 (the floor dominates); weights w_h = N_h / n_h (design_productive_strata.csv, weights.csv). A nested core subsample of ≈ 385 sections (column core, proportional within the drawn strata, min 1 per stratum) is the recommended first pass; its weights are N_h / n_h,core.
• Zero-mention frame. n = 385 drawn without replacement with inclusion probability ∝ (2 if gate passed else 1) × (1 + log10 characters), so that long gate-passing sections (where a missed project is most likely) are over-represented; 285 gate-pass / 100 gate-reject in the draw; inverse-probability weights 1/π_i stored.
• Split. Every stratum is split at random into development (guide refinement, disagreement resolution) and holdout (reported metrics): 1,058 / 1,004 sections.
• Precision of the design (design.json): design effect 3.4 (productive) / 1.2 (zero); 95 % half-width ± 4.4 pp for a section-level precision near 0.5 (± 3.5 pp near 0.8) on the full productive frame; ± 2.4 pp for a false-negative rate near 5 % on the zero frame; MDE (80 % power, two-sided 5 %) of a gate-pass vs gate-reject difference in the false-negative rate ≈ 7.8 pp at p = 0.05.

# 3. Labels

Per section (both frames): S1 does the text describe at least one concrete initiative under the construct of §5.1 (yes / no / unsure); S2 list of initiatives found (free text, ≤ 10). Per model mention (productive frame, after unsealing): M1 the mention corresponds to an initiative in the text (correct / partly / wrong / construct violation — e.g. share repurchase, financing instrument, dividend); M2 project_type correct under the 21-category taxonomy (Appendix A); M3 each structured field (years, locations, partners, amount) is literally supported by the text (the mechanical rates of N2 are the comparison point); M4 status correct. Per project (linkage audit, optional): L1 two mentions resolved to the same project_id describe the same initiative (same / different / unsure).

# 4. Procedure

1. Two annotators, trained on 40 development sections, label independently; the guide is revised once after the development set and frozen before the holdout.
2. Blinding: S1–S2 are labelled before model_output_SEALED.jsonl is opened; M1–M4 afterwards. The order of sections is shuffled; the stratum, run and confidence are hidden.
3. Agreement: Krippendorff's α (nominal) for S1, M1, M2; prevalence-adjusted bias-adjusted κ (PABAK) reported next to Cohen's κ because S1 is rare in the zero frame. Disagreements adjudicated by a third reader; adjudicated labels are the reference.
4. Estimators (holdout only, inverse-probability weighted): section-level precision P(S1 = yes | productive), mention-level precision P(M1 = correct), category accuracy P(M2 | M1 = correct), field-support rates (compare with N2), false-negative rate P(S1 = yes | zero) overall and by gate status (gate recall bound), construct-violation share by item (compare with N6). 95 % intervals by stratified bootstrap of sections (1,000 replicates).
5. Reporting: Table 10 gate (d) becomes "met / not met" on the holdout numbers; §8.2 prints n per stratum, α / PABAK, and every estimate with its interval; the development-set numbers are reported separately and never pooled.

# 5. Files

results/annotation_frame/{design.json, design_productive_strata.csv, frame_composition.csv, weights.csv, frame_dev.csv, frame_holdout.csv} (worktree); ~/firc0001_review/frames/{frame_dev.jsonl, frame_holdout.jsonl, model_output_SEALED.jsonl, weights.csv, README.md} (M3U96b, full text). The X1 stability run (AR-LLM-1), if authorised, should reuse the same 500-section stratified subset of this frame so that determinism and human agreement are measured on identical sections.