…
N1 on M4M64a from a read-only rsync of the M3U96b parquet corpus (33.5 GB):
- corpus counts reproduced exactly (10,720,890 chunks / 1,342,283 sections / 527,548 filings) + macros
- tokenizer audit: max_seq_length 256, 94.8 % of chunks truncated, 62 % of token mass dropped, 51 % of keyword chunks have all AI terms beyond the window (scored at half)
- regex AI-lexicon baseline over all sections and chunks: firm-year Spearman 0.24 with AI Adopt, 4.8 % of the semantic top decile carries a CORE AI term, 84 % zero firm-years
- human-validation frames (seed 20261105): 400 stratified + 600 top-firm + 600 median-firm + 100 top-by-year chunks, blind coding sheet + codebook
- X2 (AR-FIRC0002-2): stripped / narrow anchors embedded locally, scored against the stored chunk embeddings; pipeline reproduced to 1e-8; firm-year rho 0.976 (0.935 narrow); FM slope unchanged, EW sort t 1.92
- X3: negation probe (negated sentences still above the 99.8th percentile); laptop T5/T7/T9 rows for stripped anchors and keyword rates
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>