SPB Git

Repository Public Research preview

validation_codebook.md 1.9 KB

# Codebook — FIRC-0002 N1 human-validation frames (seed 20261105)

Files: validation_coding_sheet_blind.csv (A stratified 400 + D top-by-year 100, shuffled, no scores shown — code this one), validation_stratified_key.csv (same A rows with decile, scores and keyword counts — for the analyst, not the coders), validation_top_firms.csv (B: 20 top chunks x 30 top firms of Table 4, IDA / HE flagged in special), validation_median_firms.csv (C: 20 top chunks x 30 median firms), validation_top_by_year.csv (D with scores).

Columns to fill (two readers, independently; kappa computed on label_ai_substantive):

  • label_ai_substantive: 1 if the passage describes the FIRM'S OWN adoption, deployment, investment, product or infrastructure in artificial intelligence / machine learning (including generative AI); 0 otherwise. Generic risk-factor boilerplate, third-party / customer AI, pure automation without AI, cybersecurity without AI, generic data centres or cloud capacity without AI purpose are 0.
  • label_ai_mention: 1 if AI / machine learning is mentioned at all (any register), 0 otherwise.
  • label_boilerplate: 1 if the passage is standardized language repeated across filings (risk factors, safe-harbor statements, exhibit lists, accounting policies), 0 if firm-specific narrative.
  • label_topic: one of AI | automation | cybersecurity | data_center_or_cloud | IT_generic | analytics_or_data | utility_load_or_energy | financial_boilerplate | other.
  • reader: initials. notes: free text.

Precision by decile = mean of label_ai_substantive within each 2012 / 2024 decile (join on sample_id to the key file); precision by sector from the sector column; the B/C frames give the top-firm precision (IDA / HE in particular). Text is clipped at 2,000 characters (text_truncated_2000 in the key files); the full chunk is in filing_chunks by chunk_id.

Files main
Menu

Theme