SPB Git
spboucher.ai

spb/wp29_uqo

Public
13.5 KB

# WP29 — Content-coding protocol for the mandate corpus (codebook)

Version 1.0 — 2026-10-29. Applies to research/corpus/*.md (13 files, hashes in research/corpus_manifest.csv).

# 1. Scope, provenance and limits of the coding

  • Corpus. Thirteen files retrieved on 2026-10-29 (UTC 2026-10-30T03:01). They contain eleven distinct CLAUDE.md mandates (WP15, WP18, WP19, WP20, WP21, WP22, WP23, WP24, WP25, WP26, WP28), one supplementary design brief (WP27 DESIGN-BRIEF.md, which is not a CLAUDE.md) and one byte-identical duplicate (WP15_PROMPT_PROMPT_INITIAL_CLAUDE.md, SHA-256 2dd19c59…1b77, identical to WP15_CLAUDE.md). The duplicate is recorded but coded once. WP27's own CLAUDE.md returned HTTP 404 and is therefore absent; nothing is inferred about it.
  • Who coded. A single automated + LLM-analyst pass (Claude Code subagent, model Claude Fable 5.1, 2026-10-29). The automated variables were computed by research/code_corpus.py; the analyst variables were produced by the same subagent after reading each document in full. There was no human second coder, so no intercoder reliability (kappa, agreement rate) is reported and none should be inferred. The analyst columns are the subagent's reading of the text and may contain judgement errors; every yes/partial/no judgement is accompanied by a verbatim quote so that it can be re-checked against the source.
  • What the coding can and cannot support. The corpus is small (11 mandates) and researcher-selected (the author's own instruction files for earlier UQO working papers). It is used to motivate design patterns for WP29's protocol, not to estimate the prevalence of practices across autonomous-research mandates in general. Denominators are reported as "x of 11 mandates" and no inferential statistics are computed.
  • What is coded. Only what the instruction documents say. The coding does not assess whether the earlier working papers were good, whether the instructions were followed, or whether the papers were produced without human assistance. A mandate that requires a provenance manifest is coded as requiring one; whether such a manifest exists is outside this coding.
  • Verbatim vs interpretation. In corpus_coding.csv, verbatim material is always enclosed in double quotes "…"; everything outside the quotes is analyst interpretation. Quotes are copied exactly (including capitalisation, spelling and Markdown markup) and truncated with … only when the sentence exceeds roughly 40 words; nothing is paraphrased inside quotation marks.

# 2. Files produced

File Produced by Content
research/code_corpus.py script (stdlib only) computes the automated variables
research/corpus_automated.csv script 13 rows (one per file, duplicate flagged)
research/corpus_coding.csv analyst (written with csv module, RFC-4180) 13 rows: 11 mandates + 1 design brief + 1 duplicate-note row
research/corpus_summary.md analyst counts, patterns, excerpts

# 3. Variable types

  • (A) automated — computed by code_corpus.py from the file bytes; fully reproducible by re-running the script.
  • (I) analyst-interpreted — a categorical judgement made after reading the whole document; always accompanied by a supporting verbatim quote or by none.
  • (V) verbatim extraction — text copied from the document, or metadata copied from corpus_manifest.csv.

# 4. Automated variables (corpus_automated.csv) — type (A)

Variable Definition
file file name in research/corpus/
wp leading WP\d+ token of the file name
document_type claude_md; design_brief if the name contains BRIEF; duplicate_claude_md if the SHA-256 equals that of an earlier file in sorted order
duplicate_of file name of the first file with the same SHA-256, else empty
sha256, sha256_matches_manifest SHA-256 of the bytes; yes if equal to the manifest value
manifest_url, manifest_retrieved_utc copied from corpus_manifest.csv
bytes, lines, words byte length; newline count (+1 if no trailing newline); len(text.split())
n_headings lines matching ^#{1,6}\s+\S (ATX Markdown headings)
n_phase_labels_distinct number of distinct N in matches of \bphase\s*(\d+) (case-insensitive)
n_stage_labels_distinct distinct capital letters X in (?i:\bstage)\s+([A-Z])\b
n_gate_labels_distinct distinct N in \bgate\s*(\d+) (case-insensitive)
n_numbered_items_in_phase_sections numbered list items (^\s*\d+\.\s) located inside a section whose heading contains one of phase(s), stage(s), sequence, execution order, work sequence, execution protocol, execution phases, milestones; a section extends to the next heading of equal or higher level
n_numbered_items_total all numbered list items in the file
n_numbered_phases mechanical phase count. Rule, in order: distinct Phase N labels if any; else distinct Stage X labels; else distinct Gate N labels; else numbered items inside phase/sequence sections. n_numbered_phases_rule records which branch fired. Known limits: WP20 states a 6-step "Mandatory sequence: (1) … (6)" inline with parentheses (not matched) and 7 numbered gates (matched → 7); WP21 lists steps "A … G" inline (→ 0); WP28's "Stage A–G" are message-protocol steps, not phases, but its 13 Phase N labels take precedence. The analyst variable n_phases_declared is authoritative.
n_tables Markdown table separator rows (`
n_code_blocks pairs of ``` fences
language_of_document, fr_token_share heuristic: share of French function-word tokens (le, la, les, des, est, une, ce, cette, dans, pour, sur, avec, pas, aux, qui, que, ci-dessous, dossier, appelle, correspond, auteur) among French + English function-word tokens; en < 0.015 ≤ mixed < 0.50 ≤ fr. Tokens shared with English in this corpus (conventions, journal, du, et) are excluded. Result: all files en except WP18 and WP20 (mixed: a French prefatory paragraph on an English body).
mentions_<family> / mentions_<family>_n boolean / count of case-insensitive regex matches. Families: preregist pre-?regist\w*; estimand \bestimands?\b; hypothes \bhypothes\w*; falsif \bfalsif\w*; baseline \bbaselines?\b; leakage `leakage

# 5. Analyst variables (corpus_coding.csv)

Allowed values for every yes/no-type variable are yes / partial / no (plus n/a for the duplicate row). partial means the mandate addresses the feature but only indirectly, conditionally, or without an operational mechanism (e.g. it requires that assumptions be stated but does not state them). Each value is followed, in the same cell, by a supporting verbatim quote "…" or by an analyst remark outside quotes.

Variable Type Definition and allowed values
wp V WP15 … WP28; WP27_BRIEF for the design brief; WP15_PROMPT for the duplicate row
file V corpus file name
document_type I claude_md (an instruction file addressed to Claude Code); design_brief (WP27, an internal description of the simulator, not a mandate); duplicate
source_url, retrieved_utc, sha256 V copied from corpus_manifest.csv
research_domain I free label: real estate economics; finance / market microstructure; macro-finance theory; econometric theory / statistical computing; applied econometrics (meta-analysis); etc.
research_type I empirical, theoretical, simulation, design-science, or mixed (…) with the components in parentheses
explicit_question V the research question verbatim (≤ 40 words; the first RQ when several are listed), or none when the document states objectives but no question
explicit_estimand_or_hypotheses I + V yes = named estimand(s) / hypotheses / theorems to prove, or a pre-specified primary outcome; partial = research questions or candidate contributions treated "as hypotheses" without formal statements; no
stated_assumptions I + V yes = the mandate itself states substantive model/identification assumptions or requires a numbered assumptions register; partial = requires that assumptions be stated with each result but does not state them; no
falsification_or_null_result_conditions I + V yes = explicit instruction to attempt disproof, required counterexamples/placebos, or a declared acceptable null outcome; partial = failures/negative results must be reported but no disproof condition is defined; no
literature_or_novelty_requirement I + V yes = verified literature review and/or novelty gate required before claiming a contribution; no
data_access_controls I + V yes = explicit rules on credentials, secrets, scraping, licensing, privacy or permission; partial = only generic "do not pretend access / log unavailable files"; no
temporal_information_controls I + V yes = look-ahead / as-of-date / point-in-time / vintage / chronological-split rules; partial = vintage awareness without a leakage rule; no (including not applicable)
baseline_requirement I + V yes = named comparison baselines/benchmark methods; partial = a "baseline scenario" or benchmark model, not comparison methods; no
n_phases_declared I integer count of execution phases/stages the document declares, or none; the labelling scheme is noted in parentheses
dependency_or_gate_structure I + V explicit = named gates or "do not start X until Y" conditions; implicit = ordering only; none
stopping_rules I + V yes = conditions under which work must stop / a verdict be issued; partial = escalation or fallback rules only ("if blocked, document and deliver partial result"); no
recovery_or_resume_instructions I + V yes = resumable/idempotent/checkpointed execution required; partial = phase logs, caching or retry limits that enable resumption only indirectly; no
artifact_requirements V (list) semicolon-separated list of required deliverables (paper, tables, figures, code, data, README, logs, audits …)
provenance_requirements I + V yes = hashes/manifests/pinned revisions/traceable number-to-file mapping/versioned outputs; partial = "every number must map to a result file" or logs without hashes/manifests; no
review_or_self_audit_stage I + V yes = a dedicated audit/referee/validation stage; partial = audit items inside a final checklist only; no
disclosure_or_AI_use_statement I + V yes = an AI-use / authorship-disclosure instruction; no (statements about prose style, e.g. "not AI-generated prose", are not disclosure)
budget_or_resource_limits I + V yes = an explicit budget (API cost tiers, token budgets) or numeric resource cap; partial = resource estimation/profiling or "within available resources" without a cap; no
integrity_prohibitions I + V yes = explicit "never invent / fabricate …" prohibitions; no
contradictory_or_non_operational_requirements I free text listing instructions that cannot be verified from the document's own criteria, or that conflict with another instruction; none found otherwise
statements_added_during_implementation V passages that the document itself marks as added later or reconstructed (headers such as "added 2026-10-27", bracketed reconstruction notes, prefatory mapping notes dated after the programme); none identified otherwise
analyst_notes I anything else relevant

# 6. Procedure

  1. Compute hashes and automated variables (python3 -I research/code_corpus.py); confirm every hash equals the manifest (sha256_matches_manifest = yes for all 13 files) and identify duplicates.
  2. Read each distinct document in full (the two long documents, WP15 at 2 165 lines and WP28 at 1 055 lines, were read in consecutive line ranges).
  3. For each analyst variable, locate the supporting passage, copy it verbatim, then assign the value. Where no passage exists, assign no/none without quote.
  4. Write corpus_coding.csv with Python's csv module (QUOTE_ALL), re-read it with the same module and check the row count (13) and column count.
  5. Tabulate counts over the 11 mandates only (brief and duplicate excluded) for corpus_summary.md.

# 7. Known weaknesses of this coding

  • Single coder; no reliability statistics (see §1).
  • Keyword counts are crude (see the budget/gate caveats in §4) and are reported only as descriptive context.
  • The partial category absorbs heterogeneous cases; the quotes allow a reader to re-cut the categories.
  • The design brief (WP27) is a different genre; its row is kept separate and excluded from all "of 11" counts.
Files · 452964ee0fbbaa40fbbed2c87a54ca798ffd6a85
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere