FIRC0002-DATA-10 AI_C-06 AI_C-07 EDIT-05 DATA-11 THEO-14 ADVE-03 AI_C-17 (N1, X2, X3): corpus audits on M4M64a, tokenizer truncation, keyword baseline, validation frames, alternative anchors
N1 on M4M64a from a read-only rsync of the M3U96b parquet corpus (33.5 GB): - corpus counts reproduced exactly (10,720,890 chunks / 1,342,283 sections / 527,548 filings) + macros - tokenizer audit: max_seq_length 256, 94.8 % of chunks truncated, 62 % of token mass dropped, 51 % of keyword chunks have all AI terms beyond the window (scored at half) - regex AI-lexicon baseline over all sections and chunks: firm-year Spearman 0.24 with AI Adopt, 4.8 % of the semantic top decile carries a CORE AI term, 84 % zero firm-years - human-validation frames (seed 20261105): 400 stratified + 600 top-firm + 600 median-firm + 100 top-by-year chunks, blind coding sheet + codebook - X2 (AR-FIRC0002-2): stripped / narrow anchors embedded locally, scored against the stored chunk embeddings; pipeline reproduced to 1e-8; firm-year rho 0.976 (0.935 narrow); FM slope unchanged, EW sort t 1.92 - X3: negation probe (negated sentences still above the 99.8th percentile); laptop T5/T7/T9 rows for stripped anchors and keyword rates Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
15 changed files +20,001 −0
added
code/n1/firm_lists_t4.json
+492 −0
@@ -0,0 +1,492 @@ | ||
| 1 | +{ | |
| 2 | + "source": "author's data/processed_r3k/analysis_panel.parquet (read-only), same rule as r3k_07 tab_topfirms: mean z_AI_COMPOS over firm-years with >=5 years", | |
| 3 | + "n_firms_ranked": 2067, | |
| 4 | + "top30": [ | |
| 5 | + { | |
| 6 | + "ticker": "NET", | |
| 7 | + "cik": "0001477333", | |
| 8 | + "company_name": "CLOUDFLARE CLASS A", | |
| 9 | + "sector": "Information Technology", | |
| 10 | + "z": 3.4503346414142033, | |
| 11 | + "yrs": 6 | |
| 12 | + }, | |
| 13 | + { | |
| 14 | + "ticker": "HPE", | |
| 15 | + "cik": "0001645590", | |
| 16 | + "company_name": "Hewlett Packard Enterprise", | |
| 17 | + "sector": "Information Technology", | |
| 18 | + "z": 3.3420230352865614, | |
| 19 | + "yrs": 10 | |
| 20 | + }, | |
| 21 | + { | |
| 22 | + "ticker": "IBM", | |
| 23 | + "cik": "0000051143", | |
| 24 | + "company_name": "IBM", | |
| 25 | + "sector": "Information Technology", | |
| 26 | + "z": 3.2989450574249752, | |
| 27 | + "yrs": 15 | |
| 28 | + }, | |
| 29 | + { | |
| 30 | + "ticker": "IDA", | |
| 31 | + "cik": "0001057877", | |
| 32 | + "company_name": "IDACORP", | |
| 33 | + "sector": "Utilities", | |
| 34 | + "z": 3.263430359939158, | |
| 35 | + "yrs": 15 | |
| 36 | + }, | |
| 37 | + { | |
| 38 | + "ticker": "INTC", | |
| 39 | + "cik": "0000050863", | |
| 40 | + "company_name": "Intel", | |
| 41 | + "sector": "Information Technology", | |
| 42 | + "z": 3.2392914326890816, | |
| 43 | + "yrs": 15 | |
| 44 | + }, | |
| 45 | + { | |
| 46 | + "ticker": "AI", | |
| 47 | + "cik": "0001577526", | |
| 48 | + "company_name": "C3 AI CLASS A", | |
| 49 | + "sector": "Information Technology", | |
| 50 | + "z": 3.2290574514020682, | |
| 51 | + "yrs": 5 | |
| 52 | + }, | |
| 53 | + { | |
| 54 | + "ticker": "TYL", | |
| 55 | + "cik": "0000860731", | |
| 56 | + "company_name": "Tyler Technologies", | |
| 57 | + "sector": "Information Technology", | |
| 58 | + "z": 2.7884845514026493, | |
| 59 | + "yrs": 15 | |
| 60 | + }, | |
| 61 | + { | |
| 62 | + "ticker": "ADSK", | |
| 63 | + "cik": "0000769397", | |
| 64 | + "company_name": "Autodesk", | |
| 65 | + "sector": "Information Technology", | |
| 66 | + "z": 2.7161008136817286, | |
| 67 | + "yrs": 15 | |
| 68 | + }, | |
| 69 | + { | |
| 70 | + "ticker": "NSIT", | |
| 71 | + "cik": "0000932696", | |
| 72 | + "company_name": "INSIGHT ENTERPRISES INC", | |
| 73 | + "sector": "Information Technology", | |
| 74 | + "z": 2.659393754232204, | |
| 75 | + "yrs": 15 | |
| 76 | + }, | |
| 77 | + { | |
| 78 | + "ticker": "UBER", | |
| 79 | + "cik": "0001543151", | |
| 80 | + "company_name": "Uber", | |
| 81 | + "sector": "Industrials", | |
| 82 | + "z": 2.6338311711173734, | |
| 83 | + "yrs": 6 | |
| 84 | + }, | |
| 85 | + { | |
| 86 | + "ticker": "HE", | |
| 87 | + "cik": "0000354707", | |
| 88 | + "company_name": "HAWAIIAN ELECTRIC INDUSTRIES", | |
| 89 | + "sector": "Utilities", | |
| 90 | + "z": 2.605238281784658, | |
| 91 | + "yrs": 15 | |
| 92 | + }, | |
| 93 | + { | |
| 94 | + "ticker": "CMI", | |
| 95 | + "cik": "0000026172", | |
| 96 | + "company_name": "Cummins", | |
| 97 | + "sector": "Industrials", | |
| 98 | + "z": 2.5274964304398404, | |
| 99 | + "yrs": 15 | |
| 100 | + }, | |
| 101 | + { | |
| 102 | + "ticker": "ROK", | |
| 103 | + "cik": "0001024478", | |
| 104 | + "company_name": "Rockwell Automation", | |
| 105 | + "sector": "Industrials", | |
| 106 | + "z": 2.5139409733125087, | |
| 107 | + "yrs": 15 | |
| 108 | + }, | |
| 109 | + { | |
| 110 | + "ticker": "U", | |
| 111 | + "cik": "0001810806", | |
| 112 | + "company_name": "UNITY SOFTWARE", | |
| 113 | + "sector": "Information Technology", | |
| 114 | + "z": 2.382778471578561, | |
| 115 | + "yrs": 5 | |
| 116 | + }, | |
| 117 | + { | |
| 118 | + "ticker": "APTV", | |
| 119 | + "cik": "0001521332", | |
| 120 | + "company_name": "Aptiv", | |
| 121 | + "sector": "Consumer Discretionary", | |
| 122 | + "z": 2.3770917443570845, | |
| 123 | + "yrs": 13 | |
| 124 | + }, | |
| 125 | + { | |
| 126 | + "ticker": "AMKR", | |
| 127 | + "cik": "0001047127", | |
| 128 | + "company_name": "AMKOR TECHNOLOGY", | |
| 129 | + "sector": "Information Technology", | |
| 130 | + "z": 2.340889154397485, | |
| 131 | + "yrs": 15 | |
| 132 | + }, | |
| 133 | + { | |
| 134 | + "ticker": "ADBE", | |
| 135 | + "cik": "0000796343", | |
| 136 | + "company_name": "Adobe Inc.", | |
| 137 | + "sector": "Information Technology", | |
| 138 | + "z": 2.3375034278556113, | |
| 139 | + "yrs": 15 | |
| 140 | + }, | |
| 141 | + { | |
| 142 | + "ticker": "ALOY", | |
| 143 | + "cik": "0001567900", | |
| 144 | + "company_name": "REALLOYS INC", | |
| 145 | + "sector": "Materials", | |
| 146 | + "z": 2.272966673991874, | |
| 147 | + "yrs": 10 | |
| 148 | + }, | |
| 149 | + { | |
| 150 | + "ticker": "PRGS", | |
| 151 | + "cik": "0000876167", | |
| 152 | + "company_name": "PROGRESS SOFTWARE CORP", | |
| 153 | + "sector": "Information Technology", | |
| 154 | + "z": 2.23440769804261, | |
| 155 | + "yrs": 15 | |
| 156 | + }, | |
| 157 | + { | |
| 158 | + "ticker": "SEIC", | |
| 159 | + "cik": "0000350894", | |
| 160 | + "company_name": "SEI INVESTMENTS", | |
| 161 | + "sector": "Financials", | |
| 162 | + "z": 2.177808536691798, | |
| 163 | + "yrs": 15 | |
| 164 | + }, | |
| 165 | + { | |
| 166 | + "ticker": "PAR", | |
| 167 | + "cik": "0000708821", | |
| 168 | + "company_name": "PAR TECHNOLOGY", | |
| 169 | + "sector": "Information Technology", | |
| 170 | + "z": 2.1484856859284407, | |
| 171 | + "yrs": 15 | |
| 172 | + }, | |
| 173 | + { | |
| 174 | + "ticker": "MCHP", | |
| 175 | + "cik": "0000827054", | |
| 176 | + "company_name": "Microchip Technology", | |
| 177 | + "sector": "Information Technology", | |
| 178 | + "z": 2.1183842550114766, | |
| 179 | + "yrs": 15 | |
| 180 | + }, | |
| 181 | + { | |
| 182 | + "ticker": "AKAM", | |
| 183 | + "cik": "0001086222", | |
| 184 | + "company_name": "Akamai Technologies", | |
| 185 | + "sector": "Information Technology", | |
| 186 | + "z": 2.1174466561603773, | |
| 187 | + "yrs": 15 | |
| 188 | + }, | |
| 189 | + { | |
| 190 | + "ticker": "QUBT", | |
| 191 | + "cik": "0001758009", | |
| 192 | + "company_name": "QUANTUM COMPUTING", | |
| 193 | + "sector": "Information Technology", | |
| 194 | + "z": 2.067038614432889, | |
| 195 | + "yrs": 6 | |
| 196 | + }, | |
| 197 | + { | |
| 198 | + "ticker": "MANH", | |
| 199 | + "cik": "0001056696", | |
| 200 | + "company_name": "MANHATTAN ASSOCIATES", | |
| 201 | + "sector": "Information Technology", | |
| 202 | + "z": 2.0582787700732594, | |
| 203 | + "yrs": 15 | |
| 204 | + }, | |
| 205 | + { | |
| 206 | + "ticker": "ANET", | |
| 207 | + "cik": "0001596532", | |
| 208 | + "company_name": "Arista Networks", | |
| 209 | + "sector": "Information Technology", | |
| 210 | + "z": 2.0155927306014525, | |
| 211 | + "yrs": 11 | |
| 212 | + }, | |
| 213 | + { | |
| 214 | + "ticker": "NEO", | |
| 215 | + "cik": "0001077183", | |
| 216 | + "company_name": "NEOGENOMICS", | |
| 217 | + "sector": "Health Care", | |
| 218 | + "z": 2.000620541542724, | |
| 219 | + "yrs": 15 | |
| 220 | + }, | |
| 221 | + { | |
| 222 | + "ticker": "INTU", | |
| 223 | + "cik": "0000896878", | |
| 224 | + "company_name": "Intuit", | |
| 225 | + "sector": "Information Technology", | |
| 226 | + "z": 1.9921753925837509, | |
| 227 | + "yrs": 15 | |
| 228 | + }, | |
| 229 | + { | |
| 230 | + "ticker": "AWR", | |
| 231 | + "cik": "0001056903", | |
| 232 | + "company_name": "AMERICAN STATES WATER", | |
| 233 | + "sector": "Utilities", | |
| 234 | + "z": 1.9840208245179025, | |
| 235 | + "yrs": 15 | |
| 236 | + }, | |
| 237 | + { | |
| 238 | + "ticker": "NTLA", | |
| 239 | + "cik": "0001652130", | |
| 240 | + "company_name": "INTELLIA THERAPEUTICS", | |
| 241 | + "sector": "Health Care", | |
| 242 | + "z": 1.9615978690589329, | |
| 243 | + "yrs": 9 | |
| 244 | + } | |
| 245 | + ], | |
| 246 | + "median30": [ | |
| 247 | + { | |
| 248 | + "ticker": "TFX", | |
| 249 | + "cik": "0000096943", | |
| 250 | + "company_name": "TELEFLEX", | |
| 251 | + "sector": "Health Care", | |
| 252 | + "z": -0.04799515072905726, | |
| 253 | + "yrs": 15 | |
| 254 | + }, | |
| 255 | + { | |
| 256 | + "ticker": "ENVA", | |
| 257 | + "cik": "0001529864", | |
| 258 | + "company_name": "ENOVA INTERNATIONAL", | |
| 259 | + "sector": "Financials", | |
| 260 | + "z": -0.049405447759540054, | |
| 261 | + "yrs": 11 | |
| 262 | + }, | |
| 263 | + { | |
| 264 | + "ticker": "PVLA", | |
| 265 | + "cik": "0001583648", | |
| 266 | + "company_name": "PALVELLA THERAPEUTICS INC", | |
| 267 | + "sector": "Health Care", | |
| 268 | + "z": -0.0519447341795096, | |
| 269 | + "yrs": 11 | |
| 270 | + }, | |
| 271 | + { | |
| 272 | + "ticker": "FORM", | |
| 273 | + "cik": "0001039399", | |
| 274 | + "company_name": "FORMFACTOR INC", | |
| 275 | + "sector": "Information Technology", | |
| 276 | + "z": -0.053527226711414494, | |
| 277 | + "yrs": 15 | |
| 278 | + }, | |
| 279 | + { | |
| 280 | + "ticker": "WGS", | |
| 281 | + "cik": "0001818331", | |
| 282 | + "company_name": "GENEDX HOLDINGS CLASS A", | |
| 283 | + "sector": "Health Care", | |
| 284 | + "z": -0.05778369343064989, | |
| 285 | + "yrs": 5 | |
| 286 | + }, | |
| 287 | + { | |
| 288 | + "ticker": "SFM", | |
| 289 | + "cik": "0001575515", | |
| 290 | + "company_name": "SPROUTS FARMERS MARKET INC", | |
| 291 | + "sector": "Consumer Staples", | |
| 292 | + "z": -0.0578582464199853, | |
| 293 | + "yrs": 12 | |
| 294 | + }, | |
| 295 | + { | |
| 296 | + "ticker": "AYI", | |
| 297 | + "cik": "0001144215", | |
| 298 | + "company_name": "ACUITY", | |
| 299 | + "sector": "Industrials", | |
| 300 | + "z": -0.05819267548244656, | |
| 301 | + "yrs": 15 | |
| 302 | + }, | |
| 303 | + { | |
| 304 | + "ticker": "PBI", | |
| 305 | + "cik": "0000078814", | |
| 306 | + "company_name": "PITNEY BOWES", | |
| 307 | + "sector": "Industrials", | |
| 308 | + "z": -0.05828594542188567, | |
| 309 | + "yrs": 15 | |
| 310 | + }, | |
| 311 | + { | |
| 312 | + "ticker": "PMT", | |
| 313 | + "cik": "0001464423", | |
| 314 | + "company_name": "PENNYMAC MORTGAGE INVESTMENT TRUST", | |
| 315 | + "sector": "Financials", | |
| 316 | + "z": -0.062372824296726005, | |
| 317 | + "yrs": 15 | |
| 318 | + }, | |
| 319 | + { | |
| 320 | + "ticker": "MO", | |
| 321 | + "cik": "0000764180", | |
| 322 | + "company_name": "Altria", | |
| 323 | + "sector": "Consumer Staples", | |
| 324 | + "z": -0.0623764397427811, | |
| 325 | + "yrs": 15 | |
| 326 | + }, | |
| 327 | + { | |
| 328 | + "ticker": "ASH", | |
| 329 | + "cik": "0001674862", | |
| 330 | + "company_name": "ASHLAND", | |
| 331 | + "sector": "Materials", | |
| 332 | + "z": -0.06523471554258894, | |
| 333 | + "yrs": 9 | |
| 334 | + }, | |
| 335 | + { | |
| 336 | + "ticker": "VCEL", | |
| 337 | + "cik": "0000887359", | |
| 338 | + "company_name": "VERICEL", | |
| 339 | + "sector": "Health Care", | |
| 340 | + "z": -0.06578478422959973, | |
| 341 | + "yrs": 15 | |
| 342 | + }, | |
| 343 | + { | |
| 344 | + "ticker": "WMK", | |
| 345 | + "cik": "0000105418", | |
| 346 | + "company_name": "WEIS MARKETS", | |
| 347 | + "sector": "Consumer Staples", | |
| 348 | + "z": -0.06606932787899573, | |
| 349 | + "yrs": 15 | |
| 350 | + }, | |
| 351 | + { | |
| 352 | + "ticker": "GO", | |
| 353 | + "cik": "0001771515", | |
| 354 | + "company_name": "GROCERY OUTLET HOLDING", | |
| 355 | + "sector": "Consumer Staples", | |
| 356 | + "z": -0.06607835943388912, | |
| 357 | + "yrs": 6 | |
| 358 | + }, | |
| 359 | + { | |
| 360 | + "ticker": "CHCT", | |
| 361 | + "cik": "0001631569", | |
| 362 | + "company_name": "COMMUNITY HEALTHCARE TRUST INC", | |
| 363 | + "sector": "Real Estate", | |
| 364 | + "z": -0.06704766756086793, | |
| 365 | + "yrs": 10 | |
| 366 | + }, | |
| 367 | + { | |
| 368 | + "ticker": "LAMR", | |
| 369 | + "cik": "0001090425", | |
| 370 | + "company_name": "LAMAR ADVERTISING COMPANY CLAS", | |
| 371 | + "sector": "Real Estate", | |
| 372 | + "z": -0.06863556937877867, | |
| 373 | + "yrs": 15 | |
| 374 | + }, | |
| 375 | + { | |
| 376 | + "ticker": "USFD", | |
| 377 | + "cik": "0001665918", | |
| 378 | + "company_name": "US FOODS HOLDING", | |
| 379 | + "sector": "Consumer Staples", | |
| 380 | + "z": -0.06908411252334369, | |
| 381 | + "yrs": 9 | |
| 382 | + }, | |
| 383 | + { | |
| 384 | + "ticker": "LECO", | |
| 385 | + "cik": "0000059527", | |
| 386 | + "company_name": "LINCOLN ELECTRIC HOLDINGS", | |
| 387 | + "sector": "Industrials", | |
| 388 | + "z": -0.07443288296568197, | |
| 389 | + "yrs": 15 | |
| 390 | + }, | |
| 391 | + { | |
| 392 | + "ticker": "NNN", | |
| 393 | + "cik": "0000751364", | |
| 394 | + "company_name": "NNN REIT", | |
| 395 | + "sector": "Real Estate", | |
| 396 | + "z": -0.0762083428574501, | |
| 397 | + "yrs": 15 | |
| 398 | + }, | |
| 399 | + { | |
| 400 | + "ticker": "NVR", | |
| 401 | + "cik": "0000906163", | |
| 402 | + "company_name": "NVR, Inc.", | |
| 403 | + "sector": "Consumer Discretionary", | |
| 404 | + "z": -0.07730804483118364, | |
| 405 | + "yrs": 15 | |
| 406 | + }, | |
| 407 | + { | |
| 408 | + "ticker": "TRGP", | |
| 409 | + "cik": "0001389170", | |
| 410 | + "company_name": "Targa Resources", | |
| 411 | + "sector": "Energy", | |
| 412 | + "z": -0.07811492684546463, | |
| 413 | + "yrs": 15 | |
| 414 | + }, | |
| 415 | + { | |
| 416 | + "ticker": "ABG", | |
| 417 | + "cik": "0001144980", | |
| 418 | + "company_name": "ASBURY AUTOMOTIVE GROUP INC", | |
| 419 | + "sector": "Consumer Discretionary", | |
| 420 | + "z": -0.07892092416888237, | |
| 421 | + "yrs": 15 | |
| 422 | + }, | |
| 423 | + { | |
| 424 | + "ticker": "PRCH", | |
| 425 | + "cik": "0001784535", | |
| 426 | + "company_name": "PORCH GROUP", | |
| 427 | + "sector": "Financials", | |
| 428 | + "z": -0.07933894431531485, | |
| 429 | + "yrs": 6 | |
| 430 | + }, | |
| 431 | + { | |
| 432 | + "ticker": "TCBI", | |
| 433 | + "cik": "0001077428", | |
| 434 | + "company_name": "TEXAS CAPITAL BANCSHARES", | |
| 435 | + "sector": "Financials", | |
| 436 | + "z": -0.07994522483620843, | |
| 437 | + "yrs": 15 | |
| 438 | + }, | |
| 439 | + { | |
| 440 | + "ticker": "FIX", | |
| 441 | + "cik": "0001035983", | |
| 442 | + "company_name": "Comfort Systems USA", | |
| 443 | + "sector": "Industrials", | |
| 444 | + "z": -0.08079212860633334, | |
| 445 | + "yrs": 15 | |
| 446 | + }, | |
| 447 | + { | |
| 448 | + "ticker": "FULC", | |
| 449 | + "cik": "0001680581", | |
| 450 | + "company_name": "FULCRUM THERAPEUTICS", | |
| 451 | + "sector": "Health Care", | |
| 452 | + "z": -0.08101700621605758, | |
| 453 | + "yrs": 6 | |
| 454 | + }, | |
| 455 | + { | |
| 456 | + "ticker": "PRKS", | |
| 457 | + "cik": "0001564902", | |
| 458 | + "company_name": "UNITED PARKS AND RESORTS", | |
| 459 | + "sector": "Consumer Discretionary", | |
| 460 | + "z": -0.08181751534620181, | |
| 461 | + "yrs": 12 | |
| 462 | + }, | |
| 463 | + { | |
| 464 | + "ticker": "CMPR", | |
| 465 | + "cik": "0001262976", | |
| 466 | + "company_name": "CIMPRESS PLC", | |
| 467 | + "sector": "Industrials", | |
| 468 | + "z": -0.08313034804525632, | |
| 469 | + "yrs": 15 | |
| 470 | + }, | |
| 471 | + { | |
| 472 | + "ticker": "NOVT", | |
| 473 | + "cik": "0001076930", | |
| 474 | + "company_name": "NOVANTA", | |
| 475 | + "sector": "Information Technology", | |
| 476 | + "z": -0.0837210271419093, | |
| 477 | + "yrs": 15 | |
| 478 | + }, | |
| 479 | + { | |
| 480 | + "ticker": "IMXI", | |
| 481 | + "cik": "0001683695", | |
| 482 | + "company_name": "INTERNATIONAL MONEY EXPRESS INC", | |
| 483 | + "sector": "Financials", | |
| 484 | + "z": -0.08374579297115683, | |
| 485 | + "yrs": 8 | |
| 486 | + } | |
| 487 | + ], | |
| 488 | + "special": [ | |
| 489 | + "IDA", | |
| 490 | + "HE" | |
| 491 | + ] | |
| 492 | +} | |
| \ No newline at end of file | ||
added
code/n1/n1_01_corpus_counts.py
+110 −0
@@ -0,0 +1,110 @@ | ||
| 1 | +"""n1_01_corpus_counts.py — corpus counts by form / year (DATA-10, T1 macros). | |
| 2 | + | |
| 3 | +Reproduces (or documents the discrepancy with) the manuscript's 10,720,890 chunks / | |
| 4 | +1,342,283 sections / 527,548 filings from the parquet copy, and emits LaTeX macros. | |
| 5 | +Outputs: results/corpus_counts.json, corpus_counts_by_form_year.csv, corpus_macros.tex | |
| 6 | +""" | |
| 7 | +import os, sys | |
| 8 | +import pandas as pd | |
| 9 | +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | |
| 10 | +from n1_common import * # noqa | |
| 11 | + | |
| 12 | +c = connect(); views(c) | |
| 13 | +out = {"targets": {"chunks": TARGET_CHUNKS, "sections": TARGET_SECTIONS, "filings": TARGET_FILINGS}} | |
| 14 | + | |
| 15 | +with Timer("filings"): | |
| 16 | + out["filings_total"] = c.execute("SELECT count(*) FROM filings").fetchone()[0] | |
| 17 | + out["filings_distinct_accession"] = c.execute("SELECT count(DISTINCT accession_number) FROM filings").fetchone()[0] | |
| 18 | + out["issuers_cik"] = c.execute("SELECT count(DISTINCT cik) FROM filings").fetchone()[0] | |
| 19 | + out["issuers_ticker"] = c.execute("SELECT count(DISTINCT ticker) FROM filings").fetchone()[0] | |
| 20 | + out["companies_rows"] = c.execute("SELECT count(*) FROM companies").fetchone()[0] | |
| 21 | + out["filings_by_form"] = c.execute("""SELECT form_type, count(*) n, count(DISTINCT cik) issuers, | |
| 22 | + sum(CASE WHEN parse_status='complete' THEN 1 ELSE 0 END) parsed, | |
| 23 | + min(filing_date)::VARCHAR dmin, max(filing_date)::VARCHAR dmax FROM filings GROUP BY 1 ORDER BY 1""").df().to_dict("records") | |
| 24 | + out["parse_status"] = c.execute("SELECT form_type, parse_status, count(*) n FROM filings GROUP BY 1,2 ORDER BY 1,2").df().to_dict("records") | |
| 25 | + out["download_status"] = c.execute("SELECT download_status, count(*) n FROM filings GROUP BY 1 ORDER BY 1").df().to_dict("records") | |
| 26 | + | |
| 27 | +with Timer("sections"): | |
| 28 | + out["sections_total"] = c.execute("SELECT count(*) FROM sections").fetchone()[0] | |
| 29 | + out["sections_distinct_id"] = c.execute("SELECT count(DISTINCT section_id) FROM sections").fetchone()[0] | |
| 30 | + out["sections_filings_with_section"] = c.execute("SELECT count(DISTINCT accession_number) FROM sections").fetchone()[0] | |
| 31 | + out["sections_by_form_name"] = c.execute("""SELECT f.form_type, s.section_name, count(*) n, | |
| 32 | + median(s.word_count) med_words, avg(s.word_count) mean_words, sum(s.word_count) words | |
| 33 | + FROM sections s JOIN filings f USING (accession_number) GROUP BY 1,2 ORDER BY 1,2""").df().to_dict("records") | |
| 34 | + out["sections_words_total"] = c.execute("SELECT sum(word_count) FROM sections").fetchone()[0] | |
| 35 | + out["sections_orphan_no_filing"] = c.execute("SELECT count(*) FROM sections s LEFT JOIN filings f USING (accession_number) WHERE f.accession_number IS NULL").fetchone()[0] | |
| 36 | + | |
| 37 | +with Timer("chunks"): | |
| 38 | + out["chunks_total"] = c.execute("SELECT count(*) FROM chunks").fetchone()[0] | |
| 39 | + out["chunks_distinct_id"] = c.execute("SELECT count(DISTINCT chunk_id) FROM chunks").fetchone()[0] | |
| 40 | + out["chunks_filings"] = c.execute("SELECT count(DISTINCT accession_number) FROM chunks").fetchone()[0] | |
| 41 | + out["chunks_null_section"] = c.execute("SELECT count(*) FROM chunks WHERE section_id IS NULL").fetchone()[0] | |
| 42 | + out["chunks_orphan_no_section"] = c.execute("SELECT count(*) FROM chunks ch LEFT JOIN sections s USING (section_id) WHERE ch.section_id IS NOT NULL AND s.section_id IS NULL").fetchone()[0] | |
| 43 | + wc = c.execute("""SELECT count(*) n, min(word_count) mn, quantile_cont(word_count, 0.05) p05, quantile_cont(word_count, 0.25) p25, | |
| 44 | + median(word_count) med, avg(word_count) mean, quantile_cont(word_count, 0.75) p75, quantile_cont(word_count, 0.95) p95, max(word_count) mx, | |
| 45 | + avg(CASE WHEN word_count = 512 THEN 1.0 ELSE 0 END) share_eq512, avg(CASE WHEN word_count > 512 THEN 1.0 ELSE 0 END) share_gt512, | |
| 46 | + avg(CASE WHEN word_count < 512 THEN 1.0 ELSE 0 END) share_lt512, avg(char_count) mean_chars, median(char_count) med_chars, sum(word_count) words | |
| 47 | + FROM chunks""").df().iloc[0].to_dict() | |
| 48 | + out["chunk_word_count"] = wc | |
| 49 | + out["chunks_per_filing"] = c.execute("""WITH t AS (SELECT accession_number, count(*) n FROM chunks GROUP BY 1) | |
| 50 | + SELECT f.form_type, count(*) filings, avg(n) mean_chunks, median(n) med_chunks, min(n) mn, max(n) mx, sum(n) chunks | |
| 51 | + FROM t JOIN filings f USING (accession_number) GROUP BY 1 ORDER BY 1""").df().to_dict("records") | |
| 52 | + byfy = c.execute("""SELECT f.form_type, EXTRACT(YEAR FROM f.filing_date)::INT AS year, count(*) chunks, | |
| 53 | + count(DISTINCT ch.accession_number) filings, count(DISTINCT f.cik) issuers, sum(ch.word_count) words | |
| 54 | + FROM chunks ch JOIN filings f USING (accession_number) GROUP BY 1,2 ORDER BY 1,2""").df() | |
| 55 | + byfy.to_csv(f"{RES}/corpus_counts_by_form_year.csv", index=False) | |
| 56 | + out["chunks_by_form"] = byfy.groupby("form_type")[["chunks", "filings", "words"]].sum().reset_index().to_dict("records") | |
| 57 | + out["chunks_by_year"] = byfy.groupby("year")[["chunks", "filings"]].sum().reset_index().to_dict("records") | |
| 58 | + | |
| 59 | +with Timer("chunk_scores / embeddings"): | |
| 60 | + out["chunk_scores_total"] = c.execute("SELECT count(*) FROM chunk_scores").fetchone()[0] | |
| 61 | + out["chunk_scores_by_form"] = c.execute("SELECT form_type, count(*) n, count(DISTINCT accession_number) filings, count(DISTINCT cik) issuers FROM chunk_scores GROUP BY 1 ORDER BY 1").df().to_dict("records") | |
| 62 | + out["chunk_scores_by_section"] = c.execute("SELECT form_type, section_name, count(*) n, avg(score_ai_adoption) mean_adopt, avg(score_ai_infrastructure) mean_infra FROM chunk_scores GROUP BY 1,2 ORDER BY 1,2").df().to_dict("records") | |
| 63 | + out["chunks_without_score"] = c.execute("SELECT count(*) FROM chunks ch LEFT JOIN chunk_scores cs USING (chunk_id) WHERE cs.chunk_id IS NULL").fetchone()[0] | |
| 64 | + out["scores_without_chunk"] = c.execute("SELECT count(*) FROM chunk_scores cs LEFT JOIN chunks ch USING (chunk_id) WHERE ch.chunk_id IS NULL").fetchone()[0] | |
| 65 | + import pyarrow.parquet as pq, glob as _glob | |
| 66 | + _files = sorted(_glob.glob(f"{DATA}/embeddings/*/*.parquet")) | |
| 67 | + _rows = [pq.ParquetFile(f).metadata.num_rows for f in _files] # metadata only, one file at a time | |
| 68 | + out["embeddings_total"] = int(sum(_rows)); out["embeddings_files"] = len(_files) | |
| 69 | + out["embeddings_rows_per_file"] = {"min": int(min(_rows)), "max": int(max(_rows)), "mean": float(sum(_rows) / len(_rows))} | |
| 70 | + out["embeddings_by_model"] = [{"model_name": os.path.basename(os.path.dirname(_files[0])).split("=")[-1], "n": int(sum(_rows))}] | |
| 71 | + out["filing_scores_total"] = c.execute("SELECT count(*) FROM filing_scores").fetchone()[0] | |
| 72 | + out["filing_scores_by_form"] = c.execute("SELECT form_type, count(*) n, sum(chunk_count) chunks FROM filing_scores GROUP BY 1 ORDER BY 1").df().to_dict("records") | |
| 73 | + out["annual_final_rows"] = c.execute("SELECT count(*) FROM annual_final").fetchone()[0] | |
| 74 | + out["annual_final_years"] = c.execute("SELECT fiscal_year, count(*) n, count(DISTINCT cik) ciks FROM annual_final GROUP BY 1 ORDER BY 1").df().to_dict("records") | |
| 75 | + | |
| 76 | +out["acceptance"] = { | |
| 77 | + "chunks_match": out["chunks_total"] == TARGET_CHUNKS, "chunks_diff": out["chunks_total"] - TARGET_CHUNKS, | |
| 78 | + "sections_match": out["sections_total"] == TARGET_SECTIONS, "sections_diff": out["sections_total"] - TARGET_SECTIONS, | |
| 79 | + "filings_match": out["filings_total"] == TARGET_FILINGS, "filings_diff": out["filings_total"] - TARGET_FILINGS, | |
| 80 | + "embeddings_vs_chunks_diff": out["embeddings_total"] - out["chunks_total"], | |
| 81 | + "chunk_scores_vs_chunks_diff": out["chunk_scores_total"] - out["chunks_total"], | |
| 82 | +} | |
| 83 | +dump_json(out, f"{RES}/corpus_counts.json") | |
| 84 | + | |
| 85 | + | |
| 86 | +def fmt(n): return f"{int(n):,}".replace(",", "{,}") | |
| 87 | +forms = {r["form_type"]: r for r in out["filings_by_form"]} | |
| 88 | +cf = {r["form_type"]: r for r in out["chunks_by_form"]} | |
| 89 | +macros = [ | |
| 90 | + "% corpus macros generated by code/n1/n1_01_corpus_counts.py (FIRC-0002 Stage D, N1) from the parquet copy of M3U96b", | |
| 91 | + f"\\newcommand{{\\mnChunksCorpus}}{{{fmt(out['chunks_total'])}}}", | |
| 92 | + f"\\newcommand{{\\mnSectionsCorpus}}{{{fmt(out['sections_total'])}}}", | |
| 93 | + f"\\newcommand{{\\mnFilingsCorpusN}}{{{fmt(out['filings_total'])}}}", | |
| 94 | + f"\\newcommand{{\\mnIssuersCorpusN}}{{{fmt(out['issuers_cik'])}}}", | |
| 95 | + f"\\newcommand{{\\mnEmbeddingsCorpus}}{{{fmt(out['embeddings_total'])}}}", | |
| 96 | + f"\\newcommand{{\\mnChunkScoresCorpus}}{{{fmt(out['chunk_scores_total'])}}}", | |
| 97 | + f"\\newcommand{{\\mChunkWordsMedian}}{{{out['chunk_word_count']['med']:.0f}}}", | |
| 98 | + f"\\newcommand{{\\mChunkWordsMean}}{{{out['chunk_word_count']['mean']:.0f}}}", | |
| 99 | + f"\\newcommand{{\\mChunkShareFull}}{{{100*out['chunk_word_count']['share_eq512']:.1f}}}", | |
| 100 | +] | |
| 101 | +for ft in ["10-K", "10-Q", "8-K"]: | |
| 102 | + key = ft.replace("-", "") | |
| 103 | + if ft in forms: | |
| 104 | + macros.append(f"\\newcommand{{\\mnFilings{key}}}{{{fmt(forms[ft]['n'])}}}") | |
| 105 | + macros.append(f"\\newcommand{{\\mnParsed{key}}}{{{fmt(forms[ft]['parsed'])}}}") | |
| 106 | + if ft in cf: | |
| 107 | + macros.append(f"\\newcommand{{\\mnChunks{key}}}{{{fmt(cf[ft]['chunks'])}}}") | |
| 108 | +open(f"{RES}/corpus_macros.tex", "w").write("\n".join(macros) + "\n") | |
| 109 | +print(json.dumps(out["acceptance"], indent=1)) | |
| 110 | +print("chunks", out["chunks_total"], "sections", out["sections_total"], "filings", out["filings_total"], "embeddings", out["embeddings_total"], "scores", out["chunk_scores_total"]) | |
added
code/n1/n1_02_tokenizer_audit.py
+139 −0
@@ -0,0 +1,139 @@ | ||
| 1 | +"""n1_02_tokenizer_audit.py — WordPiece truncation audit of the 512-word chunks under all-MiniLM-L6-v2 | |
| 2 | +(AI_C-06 (a), EDIT-05). Tokenizer only, no inference. | |
| 3 | + | |
| 4 | +Samples (deterministic, hash-based, seed 20261105): | |
| 5 | + R : 10,000 random chunks; | |
| 6 | + K : 10,000 random chunks containing >= 1 CORE AI keyword (regex, lower-cased); | |
| 7 | + T : the 5,000 top-scoring chunks on score_ai_adoption and 5,000 on score_ai_infrastructure. | |
| 8 | +For every sampled chunk: WordPiece token count (with [CLS]/[SEP]), truncation flag at the configured | |
| 9 | +max_seq_length (read from the cached model's sentence_bert_config.json), share of tokens dropped, | |
| 10 | +and the token position of every AI keyword hit (fast-tokenizer offsets) -> hits inside / beyond the window. | |
| 11 | +Outputs: results/tokenizer_audit.json, tokenizer_sample.csv (one row per sampled chunk). | |
| 12 | +""" | |
| 13 | +import os, sys, re, json, glob | |
| 14 | +import numpy as np, pandas as pd | |
| 15 | +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | |
| 16 | +from n1_common import * # noqa | |
| 17 | +os.environ.setdefault("HF_HUB_OFFLINE", "1"); os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") | |
| 18 | +from transformers import AutoTokenizer | |
| 19 | + | |
| 20 | +# --- configured max_seq_length: cached model config + pipeline code (no override in src/embeddings/models.py) | |
| 21 | +snap = glob.glob(os.path.expanduser("~/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/*/"))[0] | |
| 22 | +sbert_cfg = json.load(open(snap + "sentence_bert_config.json")) | |
| 23 | +tok_cfg = json.load(open(snap + "tokenizer_config.json")) | |
| 24 | +MAXLEN = int(sbert_cfg["max_seq_length"]) # 256 (includes [CLS] and [SEP]) | |
| 25 | +CONTENT = MAXLEN - 2 | |
| 26 | +tok = AutoTokenizer.from_pretrained(snap) | |
| 27 | +print("max_seq_length (sentence_bert_config.json) =", MAXLEN, "| tokenizer model_max_length =", tok_cfg.get("model_max_length"), "| fast =", tok.is_fast) | |
| 28 | + | |
| 29 | +c = connect(); views(c) | |
| 30 | +SCORE_COLS = "score_ai_adoption, score_ai_infrastructure, score_ai_disruption_risk, score_automation_labor, score_cybersecurity" | |
| 31 | + | |
| 32 | +def pull(sql, frame): | |
| 33 | + df = c.execute(sql).df(); df["frame"] = frame; return df | |
| 34 | + | |
| 35 | +with Timer("samples"): | |
| 36 | + base = f"""SELECT ch.chunk_id, ch.accession_number, cs.cik, cs.ticker, cs.form_type, cs.fiscal_year, cs.section_name, | |
| 37 | + ch.chunk_order, ch.word_count, ch.char_count, ch.chunk_text, {SCORE_COLS} | |
| 38 | + FROM chunks ch JOIN chunk_scores cs USING (chunk_id)""" | |
| 39 | + R = pull(base + f" WHERE hash(ch.chunk_id || '{SEED}') % 1072 = 0 LIMIT 10000", "R") | |
| 40 | + K = pull(base + f" WHERE hash(ch.chunk_id || '{SEED}') % 4 = 0 AND regexp_matches(lower(ch.chunk_text), '{CORE_RE}') LIMIT 10000", "K") | |
| 41 | + T1 = pull(base + " ORDER BY cs.score_ai_adoption DESC LIMIT 5000", "T_adopt") | |
| 42 | + T2 = pull(base + " ORDER BY cs.score_ai_infrastructure DESC LIMIT 5000", "T_infra") | |
| 43 | + df = pd.concat([R, K, T1, T2], ignore_index=True) | |
| 44 | + print({k: len(v) for k, v in [("R", R), ("K", K), ("T_adopt", T1), ("T_infra", T2)]}) | |
| 45 | + | |
| 46 | +core_re = re.compile(CORE_RE, re.I); acr_re = re.compile(ACRONYM_RE) | |
| 47 | + | |
| 48 | +def analyse(text): | |
| 49 | + enc = tok(text, add_special_tokens=True, truncation=False, return_offsets_mapping=True) | |
| 50 | + ids = enc["input_ids"]; offs = enc["offset_mapping"] | |
| 51 | + n = len(ids) | |
| 52 | + # char position of the last token kept (content tokens 1..CONTENT) | |
| 53 | + if n > MAXLEN: | |
| 54 | + cut_char = offs[CONTENT][0] # start of the first dropped content token | |
| 55 | + else: | |
| 56 | + cut_char = len(text) | |
| 57 | + starts = np.array([o[0] for o in offs[1:-1]], dtype=np.int64) # content tokens | |
| 58 | + hits = [m.start() for m in core_re.finditer(text)] + [m.start() for m in acr_re.finditer(text)] | |
| 59 | + hits.sort() | |
| 60 | + tok_pos = np.searchsorted(starts, np.array(hits, dtype=np.int64), side="right") if hits else np.array([], dtype=int) | |
| 61 | + inside = int((tok_pos <= CONTENT).sum()); outside = int((tok_pos > CONTENT).sum()) | |
| 62 | + first_hit_tok = int(tok_pos[0]) if len(tok_pos) else -1 | |
| 63 | + words_before_cut = len(text[:cut_char].split()) | |
| 64 | + return n, cut_char, inside, outside, first_hit_tok, words_before_cut | |
| 65 | + | |
| 66 | +with Timer(f"tokenizing {len(df):,} chunks"): | |
| 67 | + res = [analyse(t) for t in df["chunk_text"].tolist()] | |
| 68 | +df[["n_tokens", "cut_char", "hits_inside", "hits_beyond", "first_hit_token", "words_in_window"]] = pd.DataFrame(res, index=df.index) | |
| 69 | +df["truncated"] = df["n_tokens"] > MAXLEN | |
| 70 | +df["share_tokens_dropped"] = np.where(df["truncated"], (df["n_tokens"] - MAXLEN) / df["n_tokens"], 0.0) | |
| 71 | +df["share_chars_dropped"] = 1 - df["cut_char"] / df["char_count"].clip(lower=1) | |
| 72 | +df["n_hits"] = df["hits_inside"] + df["hits_beyond"] | |
| 73 | +df["hit_class"] = np.select([df.n_hits == 0, (df.hits_inside > 0) & (df.hits_beyond == 0), (df.hits_inside == 0) & (df.hits_beyond > 0)], | |
| 74 | + ["none", "inside_only", "beyond_only"], "both") | |
| 75 | +df["tokens_per_word"] = df["n_tokens"] / df["word_count"].clip(lower=1) | |
| 76 | + | |
| 77 | +def summarise(g): | |
| 78 | + full = g[g.word_count >= 500] | |
| 79 | + return { | |
| 80 | + "n": int(len(g)), "n_full_512w": int(len(full)), | |
| 81 | + "tokens_median": float(g.n_tokens.median()), "tokens_mean": float(g.n_tokens.mean()), | |
| 82 | + "tokens_p05": float(g.n_tokens.quantile(.05)), "tokens_p95": float(g.n_tokens.quantile(.95)), | |
| 83 | + "tokens_per_word_median": float(g.tokens_per_word.median()), | |
| 84 | + "share_truncated": float(g.truncated.mean()), "share_truncated_full_512w": float(full.truncated.mean()) if len(full) else None, | |
| 85 | + "share_of_token_mass_dropped": float((g.n_tokens - g.n_tokens.clip(upper=MAXLEN)).sum() / g.n_tokens.sum()), | |
| 86 | + "mean_share_tokens_dropped_if_truncated": float(g.loc[g.truncated, "share_tokens_dropped"].mean()) if g.truncated.any() else 0.0, | |
| 87 | + "mean_share_chars_dropped": float(g.share_chars_dropped.mean()), | |
| 88 | + "words_in_window_median": float(g.loc[g.truncated, "words_in_window"].median()) if g.truncated.any() else None, | |
| 89 | + "share_with_ai_hit": float((g.n_hits > 0).mean()), | |
| 90 | + "hits_total": int(g.n_hits.sum()), "hits_beyond_window": int(g.hits_beyond.sum()), | |
| 91 | + "share_hits_beyond_window": float(g.hits_beyond.sum() / max(g.n_hits.sum(), 1)), | |
| 92 | + "share_hit_chunks_all_hits_beyond": float((g.hit_class == "beyond_only").sum() / max((g.n_hits > 0).sum(), 1)), | |
| 93 | + "hit_class_counts": g.hit_class.value_counts().to_dict(), | |
| 94 | + "mean_score_adopt_by_hit_class": g.groupby("hit_class").score_ai_adoption.mean().round(4).to_dict(), | |
| 95 | + "mean_score_infra_by_hit_class": g.groupby("hit_class").score_ai_infrastructure.mean().round(4).to_dict(), | |
| 96 | + "median_score_adopt_by_hit_class": g.groupby("hit_class").score_ai_adoption.median().round(4).to_dict(), | |
| 97 | + "p90_score_adopt_by_hit_class": g.groupby("hit_class").score_ai_adoption.quantile(.9).round(4).to_dict(), | |
| 98 | + } | |
| 99 | + | |
| 100 | +out = {"max_seq_length": MAXLEN, "content_tokens": CONTENT, "model_max_length_tokenizer": tok_cfg.get("model_max_length"), | |
| 101 | + "pipeline_override": "none: src/embeddings/models.py calls SentenceTransformer(model_name, device) without setting max_seq_length; config/embeddings.yaml lists max_tokens: 256 for all-MiniLM-L6-v2 (informational only)", | |
| 102 | + "chunker": "512 words, overlap 64, min 50 (src/parsing/chunker.py)", "seed": SEED, | |
| 103 | + "frames": {fr: summarise(g) for fr, g in df.groupby("frame")}} | |
| 104 | +# by form / section for the random frame | |
| 105 | +Rdf = df[df.frame == "R"] | |
| 106 | +out["random_by_form"] = {k: {"n": int(len(g)), "tokens_median": float(g.n_tokens.median()), "share_truncated": float(g.truncated.mean()), | |
| 107 | + "share_token_mass_dropped": float((g.n_tokens - g.n_tokens.clip(upper=MAXLEN)).sum() / g.n_tokens.sum())} | |
| 108 | + for k, g in Rdf.groupby("form_type")} | |
| 109 | +out["random_by_section"] = {k: {"n": int(len(g)), "tokens_median": float(g.n_tokens.median()), "share_truncated": float(g.truncated.mean())} | |
| 110 | + for k, g in Rdf.groupby("section_name")} | |
| 111 | +out["random_tokens_hist"] = np.histogram(Rdf.n_tokens, bins=[0, 64, 128, 192, 256, 320, 384, 448, 512, 640, 768, 1024, 100000])[0].tolist() | |
| 112 | +out["random_tokens_hist_bins"] = [0, 64, 128, 192, 256, 320, 384, 448, 512, 640, 768, 1024, "inf"] | |
| 113 | +# position of the first AI hit (K frame): share of hit chunks whose FIRST hit lies beyond the window | |
| 114 | +Kdf = df[(df.frame == "K") & (df.n_hits > 0)] | |
| 115 | +out["K_first_hit_beyond_window_share"] = float((Kdf.first_hit_token > CONTENT).mean()) | |
| 116 | +out["K_first_hit_token_quantiles"] = Kdf.first_hit_token.quantile([.1, .25, .5, .75, .9]).round(0).to_dict() | |
| 117 | +# top-scoring chunks: how many are truncated, and does the score depend on hits inside the window | |
| 118 | +for fr in ["T_adopt", "T_infra"]: | |
| 119 | + g = df[df.frame == fr] | |
| 120 | + out["frames"][fr]["share_top_chunks_truncated"] = float(g.truncated.mean()) | |
| 121 | + out["frames"][fr]["share_top_chunks_short_lt_256_tokens"] = float((g.n_tokens <= MAXLEN).mean()) | |
| 122 | + out["frames"][fr]["words_median"] = float(g.word_count.median()) | |
| 123 | +# correlation between score and hits inside vs beyond (random+K pooled) | |
| 124 | +P = df[df.frame.isin(["R", "K"])] | |
| 125 | +from scipy.stats import spearmanr | |
| 126 | +out["pooled_RK_spearman_adopt_vs_hits_inside"] = float(spearmanr(P.score_ai_adoption, P.hits_inside).correlation) | |
| 127 | +out["pooled_RK_spearman_adopt_vs_hits_beyond"] = float(spearmanr(P.score_ai_adoption, P.hits_beyond).correlation) | |
| 128 | +# regression-style contrast: among chunks with exactly the same number of hits, inside vs beyond | |
| 129 | +tmp = P[P.n_hits.between(1, 3)] | |
| 130 | +out["pooled_RK_1to3hits_mean_adopt_inside_only"] = float(tmp[tmp.hit_class == "inside_only"].score_ai_adoption.mean()) | |
| 131 | +out["pooled_RK_1to3hits_mean_adopt_beyond_only"] = float(tmp[tmp.hit_class == "beyond_only"].score_ai_adoption.mean()) | |
| 132 | +out["pooled_RK_nohit_mean_adopt"] = float(P[P.hit_class == "none"].score_ai_adoption.mean()) | |
| 133 | +out["pooled_RK_1to3hits_n"] = tmp.hit_class.value_counts().to_dict() | |
| 134 | +dump_json(out, f"{RES}/tokenizer_audit.json") | |
| 135 | +cols = ["frame", "chunk_id", "accession_number", "cik", "ticker", "form_type", "fiscal_year", "section_name", "chunk_order", "word_count", "char_count", | |
| 136 | + "n_tokens", "truncated", "share_tokens_dropped", "share_chars_dropped", "words_in_window", "n_hits", "hits_inside", "hits_beyond", "first_hit_token", "hit_class", | |
| 137 | + "score_ai_adoption", "score_ai_infrastructure", "score_ai_disruption_risk", "score_automation_labor", "score_cybersecurity"] | |
| 138 | +df[cols].to_csv(f"{RES}/tokenizer_sample.csv", index=False) | |
| 139 | +print(json.dumps({k: out["frames"][k] for k in out["frames"]}, indent=1, default=str)[:4000]) | |
added
code/n1/n1_03_keyword_baseline.py
+158 −0
@@ -0,0 +1,158 @@ | ||
| 1 | +"""n1_03_keyword_baseline.py — regex AI-lexicon baseline vs the embedding scores (AI_C-07, EDIT-05, THEO-14). | |
| 2 | + | |
| 3 | +Section level (1,342,283 sections, the pipeline's text unit): CORE and EXT keyword counts (see n1_common), | |
| 4 | +filing level: hits per 1,000 words (plain and section-weighted with the pipeline's operational weights), | |
| 5 | +firm-year level: mean over filings (mirrors eq. annual of the paper) and pooled rate -> keyword_firm_year.csv. | |
| 6 | +Chunk level (10,720,890 chunks): CORE hit counts -> chunk_kw.parquet, then agreement with score_ai_adoption / | |
| 7 | +score_ai_infrastructure: hit share by score decile, AUC, Spearman, mean score by hit bucket. | |
| 8 | +Firm-year agreement with annual_final (ai_adoption_wmean, ai_infrastructure_wmean, composite): Spearman / Pearson | |
| 9 | +overall and by year, quintile transition, top-quintile overlap, share of zero-hit firm-years. | |
| 10 | +Outputs: results/keyword_baseline.json, keyword_firm_year.csv, keyword_chunk_deciles.csv, keyword_corr_by_year.csv, | |
| 11 | + results/sec_kw.parquet, results/chunk_kw.parquet (node only, inputs for n1_04). | |
| 12 | +""" | |
| 13 | +import os, sys, json | |
| 14 | +import numpy as np, pandas as pd | |
| 15 | +from scipy.stats import spearmanr, pearsonr | |
| 16 | +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | |
| 17 | +from n1_common import * # noqa | |
| 18 | + | |
| 19 | +c = connect(); views(c) | |
| 20 | +out = {"lexicon": {"core_terms": CORE_TERMS, "ext_extra_terms": EXT_EXTRA_TERMS, "acronym_case_sensitive": ACRONYM_RE, | |
| 21 | + "note": "reviewer-assembled lexicon; CORE = unambiguous AI terms common to published AI dictionaries (Babina, Fedyk, He & Hodson 2024 JFE; Eisfeldt, Schubert & Zhang 2023); EXT adds the automation / analytics / data-driven terms that the original anchor paragraphs also contain. Applied with RE2 via DuckDB regexp_extract_all on lower-cased text; the bare acronym AI / A.I. is matched case-sensitively on the raw text."}} | |
| 22 | +Q = lambda s: "$q$" + s + "$q$" | |
| 23 | + | |
| 24 | +with Timer("section keyword counts"): | |
| 25 | + c.execute(f"""CREATE OR REPLACE TABLE sec_kw AS | |
| 26 | + SELECT s.section_id, s.accession_number, s.section_name, s.word_count, | |
| 27 | + len(regexp_extract_all(lower(s.text), {Q(CORE_RE)})) + len(regexp_extract_all(s.text, {Q(ACRONYM_RE)})) AS n_core, | |
| 28 | + len(regexp_extract_all(lower(s.text), {Q(EXT_RE)})) + len(regexp_extract_all(s.text, {Q(ACRONYM_RE)})) AS n_ext, | |
| 29 | + len(regexp_extract_all(s.text, {Q(ACRONYM_RE)})) AS n_acronym | |
| 30 | + FROM sections s""") | |
| 31 | + c.execute(f"COPY sec_kw TO '{RES}/sec_kw.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)") | |
| 32 | + out["sections"] = c.execute("""SELECT count(*) n, sum(n_core) core_hits, sum(n_ext) ext_hits, sum(n_acronym) acronym_hits, sum(word_count) words, | |
| 33 | + avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_sections_with_core, 1000.0*sum(n_core)/sum(word_count) core_per_1000w FROM sec_kw""").df().iloc[0].to_dict() | |
| 34 | + out["sections_by_name"] = c.execute("""SELECT section_name, count(*) n, 1000.0*sum(n_core)/sum(word_count) core_per_1000w, | |
| 35 | + avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_with_core FROM sec_kw GROUP BY 1 ORDER BY 1""").df().to_dict("records") | |
| 36 | + | |
| 37 | +with Timer("filing / firm-year keyword rates"): | |
| 38 | + W = weight_case("operational_measures", "k.section_name") | |
| 39 | + c.execute(f"""CREATE OR REPLACE TABLE filing_kw AS | |
| 40 | + SELECT k.accession_number, f.cik, f.ticker, f.form_type, f.filing_date, EXTRACT(YEAR FROM f.filing_date)::INT AS fiscal_year, | |
| 41 | + sum(k.n_core) n_core, sum(k.n_ext) n_ext, sum(k.word_count) words, | |
| 42 | + 1000.0*sum(k.n_core)/NULLIF(sum(k.word_count),0) AS core_per_1000w, | |
| 43 | + 1000.0*sum(k.n_ext)/NULLIF(sum(k.word_count),0) AS ext_per_1000w, | |
| 44 | + 1000.0*sum({W}*k.n_core)/NULLIF(sum({W}*k.word_count),0) AS core_w_per_1000w | |
| 45 | + FROM sec_kw k JOIN filings f USING (accession_number) GROUP BY 1,2,3,4,5,6""") | |
| 46 | + out["filings_by_form"] = c.execute("""SELECT form_type, count(*) n, avg(core_per_1000w) mean_core_per_1000w, median(core_per_1000w) med, | |
| 47 | + avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_any_core, 1000.0*sum(n_core)/sum(words) pooled_core_per_1000w FROM filing_kw GROUP BY 1 ORDER BY 1""").df().to_dict("records") | |
| 48 | + out["filings_by_year"] = c.execute("""SELECT fiscal_year, count(*) n, 1000.0*sum(n_core)/sum(words) pooled_core_per_1000w, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_any_core | |
| 49 | + FROM filing_kw GROUP BY 1 ORDER BY 1""").df().to_dict("records") | |
| 50 | + fy = c.execute("""SELECT cik, ANY_VALUE(ticker) ticker, fiscal_year, count(*) n_filings, sum(words) words, sum(n_core) n_core, sum(n_ext) n_ext, | |
| 51 | + avg(core_per_1000w) kw_core_mean, avg(core_w_per_1000w) kw_core_wmean, avg(ext_per_1000w) kw_ext_mean, | |
| 52 | + 1000.0*sum(n_core)/NULLIF(sum(words),0) kw_core_pooled, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) any_core_share, | |
| 53 | + avg(CASE WHEN form_type='10-K' THEN core_per_1000w END) kw_core_10k, avg(CASE WHEN form_type='10-Q' THEN core_per_1000w END) kw_core_10q, | |
| 54 | + avg(CASE WHEN form_type='8-K' THEN core_per_1000w END) kw_core_8k | |
| 55 | + FROM filing_kw GROUP BY cik, fiscal_year""").df() | |
| 56 | + af = c.execute("SELECT cik, ticker, fiscal_year, sector, filing_count, total_chunks, ai_adoption_wmean, ai_infrastructure_wmean, ai_disruption_risk_wmean, ai_adoption_mean, ai_adoption_top5pct FROM annual_final").df() | |
| 57 | + m = af.merge(fy.drop(columns=["ticker"]), on=["cik", "fiscal_year"], how="left") | |
| 58 | + out["firm_years_annual_final"] = int(len(af)); out["firm_years_matched_kw"] = int(m.kw_core_mean.notna().sum()) | |
| 59 | + m = m[m.kw_core_mean.notna() & (m.fiscal_year.between(2010, 2024))].copy() | |
| 60 | + m.to_csv(f"{RES}/keyword_firm_year.csv", index=False, float_format="%.6g") | |
| 61 | + | |
| 62 | +def z(g): return (g - g.mean()) / g.std(ddof=0) | |
| 63 | +for col in ["ai_adoption_wmean", "ai_infrastructure_wmean", "kw_core_mean", "kw_core_wmean", "kw_ext_mean", "kw_core_pooled", "kw_core_10k"]: | |
| 64 | + m[f"z_{col}"] = m.groupby("fiscal_year")[col].transform(z) | |
| 65 | +m["z_compos"] = m[["z_ai_adoption_wmean", "z_ai_infrastructure_wmean"]].mean(axis=1) | |
| 66 | +m["n_fy"] = 1 | |
| 67 | + | |
| 68 | +def agreement(d, a, b): | |
| 69 | + d = d[[a, b, "fiscal_year"]].dropna() | |
| 70 | + res = {"n": int(len(d)), "spearman": float(spearmanr(d[a], d[b]).correlation), "pearson": float(pearsonr(d[a], d[b])[0]), | |
| 71 | + "share_zero_b": float((d[b] == 0).mean())} | |
| 72 | + # within-year quintiles (ties broken by average rank -> qcut on rank with method first for a, 'average' for b to keep zeros together) | |
| 73 | + qa = d.groupby("fiscal_year")[a].transform(lambda s: pd.qcut(s.rank(method="first"), 5, labels=False) + 1) | |
| 74 | + qb = d.groupby("fiscal_year")[b].transform(lambda s: pd.qcut(s.rank(method="first"), 5, labels=False) + 1) | |
| 75 | + res["share_same_quintile"] = float((qa == qb).mean()) | |
| 76 | + res["share_a_Q5_in_b_Q5"] = float((qb[qa == 5] == 5).mean()) | |
| 77 | + res["share_a_Q5_in_b_Q45"] = float((qb[qa == 5] >= 4).mean()) | |
| 78 | + res["share_a_Q1_in_b_Q1"] = float((qb[qa == 1] == 1).mean()) | |
| 79 | + res["mean_b_by_a_quintile"] = d.groupby(qa)[b].mean().round(4).to_dict() | |
| 80 | + res["kendall_tau"] = float(d[[a, b]].corr(method="kendall").iloc[0, 1]) | |
| 81 | + return res | |
| 82 | + | |
| 83 | +with Timer("firm-year agreement"): | |
| 84 | + out["firm_year"] = { | |
| 85 | + "adopt_vs_kw_core_mean": agreement(m, "ai_adoption_wmean", "kw_core_mean"), | |
| 86 | + "adopt_vs_kw_core_wmean": agreement(m, "ai_adoption_wmean", "kw_core_wmean"), | |
| 87 | + "adopt_vs_kw_ext_mean": agreement(m, "ai_adoption_wmean", "kw_ext_mean"), | |
| 88 | + "infra_vs_kw_core_mean": agreement(m, "ai_infrastructure_wmean", "kw_core_mean"), | |
| 89 | + "compos_vs_kw_core_mean": agreement(m, "z_compos", "z_kw_core_mean"), | |
| 90 | + "compos_vs_kw_core_pooled": agreement(m, "z_compos", "z_kw_core_pooled"), | |
| 91 | + "compos_vs_kw_ext_mean": agreement(m, "z_compos", "z_kw_ext_mean"), | |
| 92 | + "adopt_top5pct_vs_kw_core_mean": agreement(m, "ai_adoption_top5pct", "kw_core_mean"), | |
| 93 | + "adopt_10k_vs_kw_core_10k": agreement(m[m.kw_core_10k.notna()], "ai_adoption_wmean", "kw_core_10k"), | |
| 94 | + } | |
| 95 | + by = [] | |
| 96 | + for y, g in m.groupby("fiscal_year"): | |
| 97 | + by.append({"fiscal_year": int(y), "n": int(len(g)), "share_zero_kw": float((g.kw_core_mean == 0).mean()), | |
| 98 | + "mean_kw_core_per_1000w": float(g.kw_core_mean.mean()), "mean_adopt_wmean": float(g.ai_adoption_wmean.mean()), | |
| 99 | + "spearman_adopt_kw": float(spearmanr(g.ai_adoption_wmean, g.kw_core_mean).correlation), | |
| 100 | + "spearman_infra_kw": float(spearmanr(g.ai_infrastructure_wmean, g.kw_core_mean).correlation), | |
| 101 | + "spearman_compos_kw": float(spearmanr(g.z_compos, g.kw_core_mean).correlation), | |
| 102 | + "spearman_compos_kwext": float(spearmanr(g.z_compos, g.kw_ext_mean).correlation), | |
| 103 | + "pearson_compos_kw": float(pearsonr(g.z_compos, g.kw_core_mean)[0])}) | |
| 104 | + pd.DataFrame(by).to_csv(f"{RES}/keyword_corr_by_year.csv", index=False, float_format="%.4f") | |
| 105 | + # sector view: mean composite and keyword rate, and the Utilities / Energy anomaly | |
| 106 | + sec = m.groupby("sector").agg(n=("n_fy", "sum"), mean_z_compos=("z_compos", "mean"), mean_kw_core=("kw_core_mean", "mean"), | |
| 107 | + share_zero_kw=("kw_core_mean", lambda s: float((s == 0).mean())), | |
| 108 | + spearman=("z_compos", lambda s: float(spearmanr(s, m.loc[s.index, "kw_core_mean"]).correlation))).reset_index() | |
| 109 | + out["by_sector"] = sec.round(4).to_dict("records") | |
| 110 | + # firms ranked by composite (>= 5 years) and their keyword rank | |
| 111 | + tf = m.groupby(["ticker", "sector"]).agg(z=("z_compos", "mean"), kw=("kw_core_mean", "mean"), yrs=("fiscal_year", "count")).reset_index() | |
| 112 | + tf = tf[tf.yrs >= 5].copy(); tf["rank_z"] = tf.z.rank(ascending=False); tf["rank_kw"] = tf.kw.rank(ascending=False) | |
| 113 | + top30 = tf.nsmallest(30, "rank_z") | |
| 114 | + out["top30_firms_by_composite"] = top30[["ticker", "sector", "z", "kw", "rank_z", "rank_kw"]].round(3).to_dict("records") | |
| 115 | + out["top30_firms_kw_rank_median"] = float(top30.rank_kw.median()); out["n_firms_ranked"] = int(len(tf)) | |
| 116 | + out["firm_level_spearman_z_vs_kw"] = float(spearmanr(tf.z, tf.kw).correlation) | |
| 117 | + out["top30_by_kw"] = tf.nsmallest(30, "rank_kw")[["ticker", "sector", "z", "kw", "rank_z", "rank_kw"]].round(3).to_dict("records") | |
| 118 | + | |
| 119 | +with Timer("chunk keyword counts (10.7 M chunks)"): | |
| 120 | + c.execute(f"""CREATE OR REPLACE TABLE chunk_kw AS | |
| 121 | + SELECT chunk_id, len(regexp_extract_all(lower(chunk_text), {Q(CORE_RE)})) + len(regexp_extract_all(chunk_text, {Q(ACRONYM_RE)})) AS n_core, | |
| 122 | + len(regexp_extract_all(lower(chunk_text), {Q(EXT_RE)})) + len(regexp_extract_all(chunk_text, {Q(ACRONYM_RE)})) AS n_ext | |
| 123 | + FROM chunks""") | |
| 124 | + c.execute(f"COPY chunk_kw TO '{RES}/chunk_kw.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)") | |
| 125 | + out["chunks"] = c.execute("SELECT count(*) n, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_core, avg(CASE WHEN n_ext>0 THEN 1.0 ELSE 0 END) share_ext, sum(n_core) core_hits FROM chunk_kw").df().iloc[0].to_dict() | |
| 126 | + | |
| 127 | +with Timer("chunk-level agreement"): | |
| 128 | + c.execute("""CREATE OR REPLACE TABLE cj AS | |
| 129 | + SELECT cs.chunk_id, cs.form_type, cs.fiscal_year, cs.section_name, cs.score_ai_adoption a, cs.score_ai_infrastructure i, | |
| 130 | + (cs.score_ai_adoption + cs.score_ai_infrastructure)/2 comp, k.n_core, k.n_ext, | |
| 131 | + ntile(10) OVER (ORDER BY cs.score_ai_adoption) dec_a, ntile(10) OVER (ORDER BY cs.score_ai_infrastructure) dec_i, | |
| 132 | + ntile(100) OVER (ORDER BY (cs.score_ai_adoption + cs.score_ai_infrastructure)/2) pct_comp | |
| 133 | + FROM chunk_scores cs JOIN chunk_kw k USING (chunk_id)""") | |
| 134 | + dec = c.execute("""SELECT dec_a AS decile, count(*) n, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_core_hit, avg(CASE WHEN n_ext>0 THEN 1.0 ELSE 0 END) share_ext_hit, | |
| 135 | + avg(n_core) mean_core_hits, min(a) min_score, max(a) max_score FROM cj GROUP BY 1 ORDER BY 1""").df() | |
| 136 | + dec.to_csv(f"{RES}/keyword_chunk_deciles.csv", index=False, float_format="%.5f") | |
| 137 | + out["chunk_adopt_deciles"] = dec.round(5).to_dict("records") | |
| 138 | + out["chunk_infra_deciles"] = c.execute("""SELECT dec_i AS decile, count(*) n, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_core_hit FROM cj GROUP BY 1 ORDER BY 1""").df().round(5).to_dict("records") | |
| 139 | + out["chunk_comp_top_percentiles"] = c.execute("""SELECT pct_comp AS percentile, count(*) n, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_core_hit, avg(CASE WHEN n_ext>0 THEN 1.0 ELSE 0 END) share_ext_hit FROM cj WHERE pct_comp >= 90 GROUP BY 1 ORDER BY 1""").df().round(5).to_dict("records") | |
| 140 | + out["chunk_score_by_hit_bucket"] = c.execute("""SELECT CASE WHEN n_core=0 THEN '0' WHEN n_core=1 THEN '1' WHEN n_core<=4 THEN '2-4' ELSE '5+' END bucket, count(*) n, | |
| 141 | + avg(a) mean_adopt, median(a) med_adopt, quantile_cont(a, 0.9) p90_adopt, avg(i) mean_infra, median(i) med_infra FROM cj GROUP BY 1 ORDER BY 1""").df().round(5).to_dict("records") | |
| 142 | + # AUC (Mann-Whitney) of the score for hit vs no-hit chunks, and Spearman via ranks | |
| 143 | + out["chunk_auc_adopt_core_hit"] = c.execute("""WITH r AS (SELECT n_core>0 h, rank() OVER (ORDER BY a) rk FROM cj) | |
| 144 | + SELECT (sum(CASE WHEN h THEN rk END) - (sum(CASE WHEN h THEN 1 END)*(sum(CASE WHEN h THEN 1 END)+1))/2.0) / (sum(CASE WHEN h THEN 1 END)*sum(CASE WHEN NOT h THEN 1 END)) FROM r""").fetchone()[0] | |
| 145 | + out["chunk_auc_infra_core_hit"] = c.execute("""WITH r AS (SELECT n_core>0 h, rank() OVER (ORDER BY i) rk FROM cj) | |
| 146 | + SELECT (sum(CASE WHEN h THEN rk END) - (sum(CASE WHEN h THEN 1 END)*(sum(CASE WHEN h THEN 1 END)+1))/2.0) / (sum(CASE WHEN h THEN 1 END)*sum(CASE WHEN NOT h THEN 1 END)) FROM r""").fetchone()[0] | |
| 147 | + out["chunk_spearman_adopt_ncore"] = c.execute("WITH r AS (SELECT rank() OVER (ORDER BY a) ra, rank() OVER (ORDER BY n_core) rk FROM cj) SELECT corr(ra, rk) FROM r").fetchone()[0] | |
| 148 | + out["chunk_spearman_adopt_next"] = c.execute("WITH r AS (SELECT rank() OVER (ORDER BY a) ra, rank() OVER (ORDER BY n_ext) rk FROM cj) SELECT corr(ra, rk) FROM r").fetchone()[0] | |
| 149 | + out["chunk_pearson_adopt_ncore"] = c.execute("SELECT corr(a, n_core) FROM cj").fetchone()[0] | |
| 150 | + out["chunk_by_form_section_top_decile_hit_share"] = c.execute("""SELECT form_type, section_name, count(*) n, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_core_hit | |
| 151 | + FROM cj WHERE dec_a = 10 GROUP BY 1,2 ORDER BY 1,2""").df().round(4).to_dict("records") | |
| 152 | + out["chunk_top_decile_hit_share_by_year"] = c.execute("""SELECT fiscal_year, count(*) n, avg(CASE WHEN n_core>0 THEN 1.0 ELSE 0 END) share_core_hit | |
| 153 | + FROM cj WHERE dec_a = 10 GROUP BY 1 ORDER BY 1""").df().round(4).to_dict("records") | |
| 154 | + | |
| 155 | +dump_json(out, f"{RES}/keyword_baseline.json") | |
| 156 | +print(json.dumps({k: out["firm_year"][k] for k in ["adopt_vs_kw_core_mean", "compos_vs_kw_core_mean", "compos_vs_kw_ext_mean"]}, indent=1, default=str)) | |
| 157 | +print("chunk AUC adopt", out["chunk_auc_adopt_core_hit"], "spearman", out["chunk_spearman_adopt_ncore"]) | |
| 158 | +print("deciles", dec.to_string()) | |
added
code/n1/n1_04_validation_samples.py
+123 −0
@@ -0,0 +1,123 @@ | ||
| 1 | +"""n1_04_validation_samples.py — human-validation sampling frames (AI_C-07, DATA-11, THEO-14, EDIT-05). | |
| 2 | + | |
| 3 | +Chunk composite = mean of corpus-standardized score_ai_adoption and score_ai_infrastructure (chunk-level analogue of | |
| 4 | +the paper's AI Compos). Deterministic selection by hash(chunk_id || seed), seed 20261105. Text clipped to 2,000 chars. | |
| 5 | +Frames: | |
| 6 | + A stratified : cohorts 2012 and 2024 x within-year composite decile x 20 chunks = 400 (blind coding sheet + key) | |
| 7 | + B top_firms : 20 top-composite chunks of each of the 30 top firms of Table 4 (>= 5 years), IDA and HE flagged | |
| 8 | + C median_firms: 20 top-composite chunks of each of the 30 firms around the median of the same ranking | |
| 9 | + D top_by_year: 25 top-composite chunks of 2010, 2014, 2018, 2022 | |
| 10 | +Outputs: results/validation_*.csv, validation_frames_summary.json, validation_codebook.md | |
| 11 | +""" | |
| 12 | +import os, sys, json | |
| 13 | +import numpy as np, pandas as pd | |
| 14 | +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | |
| 15 | +from n1_common import * # noqa | |
| 16 | + | |
| 17 | +c = connect(); views(c) | |
| 18 | +HERE = os.path.dirname(os.path.abspath(__file__)) | |
| 19 | +lists = json.load(open(f"{HERE}/firm_lists_t4.json")) | |
| 20 | +top30 = [r["ticker"] for r in lists["top30"]]; med30 = [r["ticker"] for r in lists["median30"]] | |
| 21 | +kw_path = f"{RES}/chunk_kw.parquet" | |
| 22 | +has_kw = os.path.exists(kw_path) | |
| 23 | +if has_kw: | |
| 24 | + c.execute(f"CREATE OR REPLACE VIEW chunk_kw AS SELECT * FROM read_parquet('{kw_path}')") | |
| 25 | +else: | |
| 26 | + c.execute("CREATE OR REPLACE VIEW chunk_kw AS SELECT chunk_id, NULL::INT n_core, NULL::INT n_ext FROM chunk_scores LIMIT 0") | |
| 27 | + | |
| 28 | +with Timer("base table"): | |
| 29 | + mu_a, sd_a, mu_i, sd_i = c.execute("SELECT avg(score_ai_adoption), stddev_pop(score_ai_adoption), avg(score_ai_infrastructure), stddev_pop(score_ai_infrastructure) FROM chunk_scores").fetchone() | |
| 30 | + c.execute(f"""CREATE OR REPLACE TABLE base AS | |
| 31 | + SELECT cs.chunk_id, cs.accession_number, cs.cik, cs.ticker, cs.form_type, cs.filing_date, cs.fiscal_year, cs.section_name, | |
| 32 | + cs.score_ai_adoption, cs.score_ai_infrastructure, cs.score_ai_disruption_risk, cs.score_cybersecurity, cs.score_automation_labor, | |
| 33 | + ((cs.score_ai_adoption - {mu_a})/{sd_a} + (cs.score_ai_infrastructure - {mu_i})/{sd_i})/2 AS chunk_composite, | |
| 34 | + k.n_core, k.n_ext, hash(cs.chunk_id || '{SEED}') AS h | |
| 35 | + FROM chunk_scores cs LEFT JOIN chunk_kw k USING (chunk_id) | |
| 36 | + WHERE cs.fiscal_year BETWEEN 2010 AND 2024""") | |
| 37 | + n_base = c.execute("SELECT count(*) FROM base").fetchone()[0] | |
| 38 | + | |
| 39 | +with Timer("frames"): | |
| 40 | + c.execute("""CREATE OR REPLACE TABLE fa AS | |
| 41 | + SELECT *, 'A_stratified' AS frame, fiscal_year::VARCHAR || '-D' || lpad(decile::VARCHAR, 2, '0') AS stratum FROM ( | |
| 42 | + SELECT *, ntile(10) OVER (PARTITION BY fiscal_year ORDER BY chunk_composite) AS decile FROM base WHERE fiscal_year IN (2012, 2024)) | |
| 43 | + QUALIFY row_number() OVER (PARTITION BY fiscal_year, decile ORDER BY h) <= 20""") | |
| 44 | + cut = c.execute("""SELECT fiscal_year, decile, min(chunk_composite) lo, max(chunk_composite) hi, count(*) n FROM ( | |
| 45 | + SELECT fiscal_year, chunk_composite, ntile(10) OVER (PARTITION BY fiscal_year ORDER BY chunk_composite) AS decile FROM base WHERE fiscal_year IN (2012, 2024)) | |
| 46 | + GROUP BY 1,2 ORDER BY 1,2""").df() | |
| 47 | + tl = ",".join(f"'{t}'" for t in top30); ml = ",".join(f"'{t}'" for t in med30) | |
| 48 | + c.execute(f"""CREATE OR REPLACE TABLE fb AS SELECT *, 0 AS decile, 'B_top_firms' AS frame, ticker AS stratum FROM base WHERE ticker IN ({tl}) | |
| 49 | + QUALIFY row_number() OVER (PARTITION BY ticker ORDER BY chunk_composite DESC, h) <= 20""") | |
| 50 | + c.execute(f"""CREATE OR REPLACE TABLE fc AS SELECT *, 0 AS decile, 'C_median_firms' AS frame, ticker AS stratum FROM base WHERE ticker IN ({ml}) | |
| 51 | + QUALIFY row_number() OVER (PARTITION BY ticker ORDER BY chunk_composite DESC, h) <= 20""") | |
| 52 | + c.execute("""CREATE OR REPLACE TABLE fd AS SELECT *, 0 AS decile, 'D_top_by_year' AS frame, fiscal_year::VARCHAR AS stratum FROM base WHERE fiscal_year IN (2010, 2014, 2018, 2022) | |
| 53 | + QUALIFY row_number() OVER (PARTITION BY fiscal_year ORDER BY chunk_composite DESC, h) <= 25""") | |
| 54 | + sel = c.execute("""SELECT s.*, ch.chunk_order, ch.word_count, ch.char_count, ch.chunk_text, co.company_name, co.sector, co.sub_industry | |
| 55 | + FROM (SELECT * FROM fa UNION ALL SELECT * FROM fb UNION ALL SELECT * FROM fc UNION ALL SELECT * FROM fd) s | |
| 56 | + JOIN chunks ch USING (chunk_id) LEFT JOIN companies co ON co.cik = s.cik""").df() | |
| 57 | + | |
| 58 | +sel["text"] = sel["chunk_text"].str.slice(0, 2000); sel["text_truncated_2000"] = sel["char_count"] > 2000 | |
| 59 | +sel = sel.drop(columns=["chunk_text", "h"]) | |
| 60 | +sel["special"] = np.where(sel.ticker.isin(["IDA", "HE"]), "utility_in_top12", "") | |
| 61 | +sel = sel.sort_values(["frame", "stratum", "chunk_composite"], ascending=[True, True, False]).reset_index(drop=True) | |
| 62 | +sel.insert(0, "sample_id", [f"N1-{i+1:05d}" for i in range(len(sel))]) | |
| 63 | +for col in ["label_ai_substantive", "label_ai_mention", "label_boilerplate", "label_topic", "reader", "notes"]: | |
| 64 | + sel[col] = "" | |
| 65 | +meta_cols = ["sample_id", "frame", "stratum", "decile", "chunk_id", "accession_number", "cik", "ticker", "company_name", "sector", "sub_industry", "form_type", | |
| 66 | + "filing_date", "fiscal_year", "section_name", "chunk_order", "word_count", "char_count", "text_truncated_2000", | |
| 67 | + "score_ai_adoption", "score_ai_infrastructure", "score_ai_disruption_risk", "score_cybersecurity", "score_automation_labor", | |
| 68 | + "chunk_composite", "n_core", "n_ext", "special"] | |
| 69 | +label_cols = ["label_ai_substantive", "label_ai_mention", "label_boilerplate", "label_topic", "reader", "notes"] | |
| 70 | +for fr, fname in [("A_stratified", "validation_stratified_key.csv"), ("B_top_firms", "validation_top_firms.csv"), | |
| 71 | + ("C_median_firms", "validation_median_firms.csv"), ("D_top_by_year", "validation_top_by_year.csv")]: | |
| 72 | + sel[sel.frame == fr][meta_cols + ["text"] + label_cols].to_csv(f"{RES}/{fname}", index=False, float_format="%.5f") | |
| 73 | +# blind coding sheet: stratified + top-by-year, shuffled, without scores / decile / keyword counts | |
| 74 | +rng = np.random.default_rng(SEED) | |
| 75 | +blind = sel[sel.frame.isin(["A_stratified", "D_top_by_year"])].copy() | |
| 76 | +blind = blind.iloc[rng.permutation(len(blind))] | |
| 77 | +blind[["sample_id", "form_type", "fiscal_year", "section_name", "sector", "text"] + label_cols].to_csv(f"{RES}/validation_coding_sheet_blind.csv", index=False) | |
| 78 | + | |
| 79 | +summ = {"seed": SEED, "n_base_chunks_2010_2024": int(n_base), "composite_moments": {"mu_adopt": mu_a, "sd_adopt": sd_a, "mu_infra": mu_i, "sd_infra": sd_i}, | |
| 80 | + "frames": sel.groupby("frame").agg(n=("sample_id", "count"), strata=("stratum", "nunique"), mean_composite=("chunk_composite", "mean"), | |
| 81 | + share_core_hit=("n_core", lambda s: float((s.fillna(0) > 0).mean())), | |
| 82 | + share_text_truncated=("text_truncated_2000", "mean")).round(4).to_dict("index"), | |
| 83 | + "stratified_by_decile": sel[sel.frame == "A_stratified"].groupby(["fiscal_year", "decile"]).agg(n=("sample_id", "count"), mean_composite=("chunk_composite", "mean"), | |
| 84 | + mean_adopt=("score_ai_adoption", "mean"), share_core_hit=("n_core", lambda s: float((s.fillna(0) > 0).mean())), | |
| 85 | + share_md_and_a=("section_name", lambda s: float((s == "md_and_a").mean())), | |
| 86 | + share_financial_statements=("section_name", lambda s: float((s == "financial_statements").mean()))).round(4).reset_index().to_dict("records"), | |
| 87 | + "decile_cutpoints": cut.round(4).to_dict("records"), | |
| 88 | + "top_firms_frame_by_firm": sel[sel.frame == "B_top_firms"].groupby(["ticker", "sector"]).agg(n=("sample_id", "count"), mean_composite=("chunk_composite", "mean"), | |
| 89 | + share_core_hit=("n_core", lambda s: float((s.fillna(0) > 0).mean())), top_section=("section_name", lambda s: s.mode().iloc[0]), | |
| 90 | + share_8k=("form_type", lambda s: float((s == "8-K").mean()))).round(4).reset_index().to_dict("records"), | |
| 91 | + "median_firms_frame_by_firm": sel[sel.frame == "C_median_firms"].groupby(["ticker", "sector"]).agg(n=("sample_id", "count"), mean_composite=("chunk_composite", "mean"), | |
| 92 | + share_core_hit=("n_core", lambda s: float((s.fillna(0) > 0).mean()))).round(4).reset_index().to_dict("records"), | |
| 93 | + "top_by_year_frame": sel[sel.frame == "D_top_by_year"].groupby("fiscal_year").agg(n=("sample_id", "count"), mean_composite=("chunk_composite", "mean"), | |
| 94 | + share_core_hit=("n_core", lambda s: float((s.fillna(0) > 0).mean())), firms=("ticker", "nunique"), | |
| 95 | + top_sector=("sector", lambda s: s.mode().iloc[0])).round(4).reset_index().to_dict("records"), | |
| 96 | + "ida_he_top_chunks": sel[(sel.frame == "B_top_firms") & sel.ticker.isin(["IDA", "HE"])][["sample_id", "ticker", "fiscal_year", "form_type", "section_name", "chunk_composite", "n_core"]].round(4).to_dict("records")} | |
| 97 | +dump_json(summ, f"{RES}/validation_frames_summary.json") | |
| 98 | + | |
| 99 | +codebook = """# Codebook — FIRC-0002 N1 human-validation frames (seed 20261105) | |
| 100 | + | |
| 101 | +Files: `validation_coding_sheet_blind.csv` (A stratified 400 + D top-by-year 100, shuffled, no scores shown — code this one), | |
| 102 | +`validation_stratified_key.csv` (same A rows with decile, scores and keyword counts — for the analyst, not the coders), | |
| 103 | +`validation_top_firms.csv` (B: 20 top chunks x 30 top firms of Table 4, IDA / HE flagged in `special`), | |
| 104 | +`validation_median_firms.csv` (C: 20 top chunks x 30 median firms), `validation_top_by_year.csv` (D with scores). | |
| 105 | + | |
| 106 | +Columns to fill (two readers, independently; kappa computed on `label_ai_substantive`): | |
| 107 | +- `label_ai_substantive`: 1 if the passage describes the FIRM'S OWN adoption, deployment, investment, product or | |
| 108 | + infrastructure in artificial intelligence / machine learning (including generative AI); 0 otherwise. Generic risk-factor | |
| 109 | + boilerplate, third-party / customer AI, pure automation without AI, cybersecurity without AI, generic data centres or | |
| 110 | + cloud capacity without AI purpose are 0. | |
| 111 | +- `label_ai_mention`: 1 if AI / machine learning is mentioned at all (any register), 0 otherwise. | |
| 112 | +- `label_boilerplate`: 1 if the passage is standardized language repeated across filings (risk factors, safe-harbor | |
| 113 | + statements, exhibit lists, accounting policies), 0 if firm-specific narrative. | |
| 114 | +- `label_topic`: one of AI | automation | cybersecurity | data_center_or_cloud | IT_generic | analytics_or_data | | |
| 115 | + utility_load_or_energy | financial_boilerplate | other. | |
| 116 | +- `reader`: initials. `notes`: free text. | |
| 117 | + | |
| 118 | +Precision by decile = mean of `label_ai_substantive` within each 2012 / 2024 decile (join on `sample_id` to the key file); | |
| 119 | +precision by sector from the `sector` column; the B/C frames give the top-firm precision (IDA / HE in particular). | |
| 120 | +Text is clipped at 2,000 characters (`text_truncated_2000` in the key files); the full chunk is in `filing_chunks` by `chunk_id`. | |
| 121 | +""" | |
| 122 | +open(f"{RES}/validation_codebook.md", "w").write(codebook) | |
| 123 | +print(json.dumps(summ["frames"], indent=1)); print(len(sel), "rows") | |
added
code/n1/n1_05_alt_anchors.py
+296 −0
@@ -0,0 +1,296 @@ | ||
| 1 | +"""n1_05_alt_anchors.py — X2 alternative-anchor test and X3 negation / boilerplate probe (AR-FIRC0002-2, AUTHORISED). | |
| 2 | + | |
| 3 | +Embeds ONLY the new anchor paragraphs (and ~40 probe sentences) locally with sentence-transformers/all-MiniLM-L6-v2 | |
| 4 | +(cached, offline, CPU) and scores them against the STORED chunk embeddings (read-only parquet copy of M3U96b). | |
| 5 | +No corpus re-embedding, no LLM call. | |
| 6 | +Steps: (1) re-embed the 8 original anchors locally and compare with the stored anchor vectors; | |
| 7 | + (2) stream the 10.7 M stored chunk vectors, dot products with [stored 8 | local 8 | alt k] anchors -> chunk_scores_alt.parquet; | |
| 8 | + (3) check that the stored-anchor recomputation reproduces chunk_scores to float precision; | |
| 9 | + (4) filing-level section-weighted means (pipeline weights) and firm-year means (eq. annual) for every column, | |
| 10 | + check against filing_scores / annual_final; firm-year rank correlations, composite quintile transitions, top-30 overlap; | |
| 11 | + (5) X3 probe sentences vs stored and alternative anchors. | |
| 12 | +Outputs: results/alt_anchor_results.json, alt_firm_year.csv, alt_corr_by_year.csv, alt_anchor_texts.json, probe_sentences.csv | |
| 13 | +""" | |
| 14 | +import os, sys, json, glob, time | |
| 15 | +import numpy as np, pandas as pd | |
| 16 | +import pyarrow as pa, pyarrow.parquet as pq, pyarrow.dataset as ds | |
| 17 | +from scipy.stats import spearmanr, pearsonr | |
| 18 | +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | |
| 19 | +from n1_common import * # noqa | |
| 20 | +os.environ.setdefault("HF_HUB_OFFLINE", "1"); os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") | |
| 21 | +import torch; torch.set_num_threads(8) | |
| 22 | +from sentence_transformers import SentenceTransformer | |
| 23 | + | |
| 24 | +# ----------------------------------------------------------------------------- original anchors (verbatim, src/factors/anchors.py) | |
| 25 | +ORIG = { | |
| 26 | + "ai_adoption": [ | |
| 27 | + "The company internally uses artificial intelligence, machine learning, generative AI, large language models, automation, predictive analytics, intelligent agents, and data-driven systems to improve productivity, operations, efficiency, decision-making, customer service, risk management, compliance, cybersecurity, supply chain, software development, finance, human resources, and other business processes.", | |
| 28 | + "We have implemented generative AI, large language models, and natural language processing capabilities across our enterprise. Our AI strategy includes deploying intelligent automation, computer vision, and predictive analytics systems to transform operations.", | |
| 29 | + "Investment in artificial intelligence research and development, including partnerships with AI technology providers, hiring of machine learning engineers, and deployment of AI infrastructure including GPU computing clusters for internal business applications."], | |
| 30 | + "ai_disruption_risk": [ | |
| 31 | + "Artificial intelligence may reduce the scarcity, defensibility, differentiation, pricing power, or competitive advantage of the company's products or services by enabling rapid replication, automation, synthetic generation, low-cost substitution, or commoditization of capabilities that were previously difficult, expensive, or specialized.", | |
| 32 | + "The rapid advancement of AI technologies may render our current offerings less competitive. New market entrants leveraging AI could fundamentally alter industry dynamics, reduce barriers to entry, and erode our market position through AI-native products and services.", | |
| 33 | + "Risks associated with artificial intelligence include potential competitive displacement by AI-powered alternatives, obsolescence of our existing products, regulatory uncertainty around AI governance, algorithmic bias, intellectual property challenges, and emergence of AI-native competitors."], | |
| 34 | + "ai_infrastructure": [ | |
| 35 | + "Investment in AI computing infrastructure including data centers, GPU and TPU clusters, cloud computing resources, semiconductor chips, accelerator hardware, and high-performance computing systems dedicated to artificial intelligence and machine learning workloads.", | |
| 36 | + "Capital expenditures for AI infrastructure, including networking equipment for distributed AI training, specialized computing platforms for large-scale model development, edge computing for AI inference, and data pipeline infrastructure for machine learning operations.", | |
| 37 | + "Cloud infrastructure spending for AI workloads, including partnerships with hyperscale cloud providers, deployment of AI-optimized hardware, GPU supply chain management, and investment in next-generation computing architectures for artificial intelligence applications."], | |
| 38 | + "esg_commitment": [ | |
| 39 | + "Environmental, social, and governance initiatives including sustainability reporting, corporate social responsibility programs, diversity equity and inclusion efforts, stakeholder engagement, and adherence to ESG frameworks and standards such as SASB, GRI, and TCFD.", | |
| 40 | + "The company is committed to ESG principles including reducing carbon emissions, improving energy efficiency, promoting workforce diversity, ensuring supply chain responsibility, ethical governance practices, and transparent reporting of environmental and social performance metrics.", | |
| 41 | + "Board-level oversight of environmental and social risk management, executive compensation tied to sustainability metrics, corporate governance improvements, and integration of ESG factors into strategic decision-making and capital allocation processes."], | |
| 42 | + "climate_risk": [ | |
| 43 | + "Climate change poses material risks to our operations including physical risks from extreme weather events, transition risks from evolving regulations, carbon pricing mechanisms, and shifting consumer preferences toward sustainable alternatives and low-carbon products.", | |
| 44 | + "Exposure to climate-related financial risks including greenhouse gas emission regulations, carbon taxes, stranded asset risk, water scarcity, supply chain disruption from natural disasters, and reputational risk from insufficient climate action and environmental degradation.", | |
| 45 | + "Climate risk assessment and scenario analysis aligned with TCFD recommendations, including evaluation of physical and transition risks under different warming scenarios, Scope 1, 2, and 3 greenhouse gas emissions measurement and reduction targets."], | |
| 46 | + "cybersecurity": [ | |
| 47 | + "Cybersecurity risks including data breaches, ransomware attacks, unauthorized access to systems, intellectual property theft, and operational disruption from cyber incidents. Investment in information security controls, threat detection, and incident response capabilities.", | |
| 48 | + "Data privacy and security compliance including GDPR, CCPA, and other data protection regulations. Cybersecurity governance, risk management frameworks, security operations centers, and employee security awareness training programs to protect critical infrastructure.", | |
| 49 | + "Critical infrastructure protection, network security, endpoint security, identity and access management, zero-trust architecture implementation, penetration testing, cyber insurance, and board-level oversight of cybersecurity risk management and AI-powered threat detection."], | |
| 50 | + "automation_labor": [ | |
| 51 | + "Automation of business processes, robotic process automation, intelligent automation, and workforce transformation. Impact of automation on labor force, reskilling and upskilling programs, and the balance between human workers and automated systems in manufacturing and services.", | |
| 52 | + "Workforce restructuring driven by technological automation, including reduction in manual processes, deployment of automated manufacturing and logistics systems, and investment in human-machine collaboration technologies and digital transformation of work.", | |
| 53 | + "Labor market dynamics including talent acquisition challenges, employee retention, workforce planning in the context of automation, gig economy participation, remote work infrastructure, and the future of work transformation driven by AI and robotics."], | |
| 54 | + "macro_risk": [ | |
| 55 | + "Macroeconomic risks including inflation, interest rate fluctuations, currency exchange rate volatility, recession risk, geopolitical tensions, trade policy uncertainty, tariffs, and supply chain disruptions affecting business performance and revenue.", | |
| 56 | + "Economic uncertainty and market volatility, including sensitivity to GDP growth, consumer spending patterns, credit market conditions, commodity price fluctuations, and the impact of monetary and fiscal policy changes on business operations and profitability.", | |
| 57 | + "Global economic risks including sovereign debt concerns, emerging market instability, trade war escalation, sanctions, energy price shocks, pandemic impacts on economic activity, and systemic financial market risks affecting corporate investment and capital markets."], | |
| 58 | +} | |
| 59 | +# ----------------------------------------------------------------------------- alternative anchors (X2) | |
| 60 | +ALT = { | |
| 61 | + # (a) AI-adoption anchor stripped of the automation / cybersecurity / data-driven / analytics terms shared with the placebo anchors | |
| 62 | + "ai_adoption_strip": {"profile": "operational_measures", "edit": "original 3 paragraphs minus 'automation', 'intelligent automation', 'cybersecurity', 'data-driven systems', 'predictive analytics', 'intelligent agents' and the generic business-function list", "texts": [ | |
| 63 | + "The company internally uses artificial intelligence, machine learning, generative AI, and large language models to improve productivity, operations, efficiency, decision-making, and customer service.", | |
| 64 | + "We have implemented generative AI, large language models, and natural language processing capabilities across our enterprise. Our AI strategy includes deploying machine learning models, computer vision, and neural networks to transform operations.", | |
| 65 | + "Investment in artificial intelligence research and development, including partnerships with AI technology providers, hiring of machine learning engineers, and deployment of AI infrastructure including GPU computing clusters for internal business applications."]}, | |
| 66 | + # (b) a narrow, single-register AI anchor (no infrastructure, no business-function list) | |
| 67 | + "ai_adoption_core": {"profile": "operational_measures", "edit": "new narrow paragraphs: only unambiguous AI / machine-learning terms, no automation, analytics, cybersecurity, hardware or business-function vocabulary", "texts": [ | |
| 68 | + "We use artificial intelligence and machine learning models in our products and operations.", | |
| 69 | + "The company has deployed generative artificial intelligence, large language models, and neural network models developed by our data science teams.", | |
| 70 | + "Our machine learning and artificial intelligence capabilities, including natural language processing and computer vision, are embedded in our services."]}, | |
| 71 | + # (c) AI-infrastructure anchor stripped of generic data-centre / cloud / semiconductor / networking vocabulary | |
| 72 | + "ai_infrastructure_strip": {"profile": "operational_measures", "edit": "original 3 paragraphs minus standalone 'data centers', 'cloud computing resources', 'semiconductor chips', 'high-performance computing systems', 'networking equipment', 'edge computing', 'hyperscale cloud providers', 'GPU supply chain management'; every remaining hardware term is tied to AI / machine learning", "texts": [ | |
| 73 | + "Investment in computing infrastructure dedicated to artificial intelligence and machine learning workloads, including GPU and TPU clusters and AI accelerator hardware for model training.", | |
| 74 | + "Capital expenditures for AI infrastructure, including specialized computing platforms for large-scale machine learning model development, hardware for AI inference, and data pipeline infrastructure for machine learning operations.", | |
| 75 | + "Spending on AI-optimized hardware and next-generation computing architectures for artificial intelligence applications, including partnerships with providers of AI model training capacity."]}, | |
| 76 | + # (d) placebo anchors stripped of their AI terms | |
| 77 | + "cybersecurity_strip": {"profile": "risk_measures", "edit": "original 3 paragraphs minus 'and AI-powered threat detection'", "texts": [ | |
| 78 | + ORIG["cybersecurity"][0], ORIG["cybersecurity"][1], | |
| 79 | + "Critical infrastructure protection, network security, endpoint security, identity and access management, zero-trust architecture implementation, penetration testing, cyber insurance, and board-level oversight of cybersecurity risk management."]}, | |
| 80 | + "automation_labor_strip": {"profile": "operational_measures", "edit": "original 3 paragraphs minus 'intelligent automation' and 'driven by AI and robotics' -> 'driven by robotics'", "texts": [ | |
| 81 | + "Automation of business processes, robotic process automation, and workforce transformation. Impact of automation on labor force, reskilling and upskilling programs, and the balance between human workers and automated systems in manufacturing and services.", | |
| 82 | + ORIG["automation_labor"][1], | |
| 83 | + "Labor market dynamics including talent acquisition challenges, employee retention, workforce planning in the context of automation, gig economy participation, remote work infrastructure, and the future of work transformation driven by robotics."]}, | |
| 84 | +} | |
| 85 | +# ----------------------------------------------------------------------------- X3 probe sentences | |
| 86 | +PROBES = [ | |
| 87 | + ("affirmative_ai", "We use artificial intelligence and machine learning to improve our operations and customer service."), | |
| 88 | + ("affirmative_ai", "We are an artificial-intelligence leader and have deployed generative AI across our enterprise."), | |
| 89 | + ("affirmative_ai", "During the year we implemented machine learning models to forecast demand and optimize pricing."), | |
| 90 | + ("affirmative_ai", "Our products incorporate large language models and natural language processing capabilities."), | |
| 91 | + ("affirmative_ai", "We invested in GPU clusters to train our proprietary machine learning models."), | |
| 92 | + ("affirmative_ai", "The company has hired machine learning engineers and formed an internal AI center of excellence."), | |
| 93 | + ("negated_ai", "We do not currently use artificial intelligence in our operations."), | |
| 94 | + ("negated_ai", "We do not use machine learning or generative AI in our products or services."), | |
| 95 | + ("negated_ai", "The company has not deployed artificial intelligence and has no plans to do so."), | |
| 96 | + ("negated_ai", "We have not implemented large language models or natural language processing capabilities."), | |
| 97 | + ("negated_ai", "Our business does not rely on artificial intelligence or machine learning technologies."), | |
| 98 | + ("negated_ai", "We have no investments in artificial intelligence infrastructure or GPU computing clusters."), | |
| 99 | + ("ai_risk_boilerplate", "The use of artificial intelligence by us or by third parties may expose us to regulatory, reputational and legal risks."), | |
| 100 | + ("ai_risk_boilerplate", "Developments in artificial intelligence could render our products obsolete or less competitive."), | |
| 101 | + ("ai_risk_boilerplate", "Issues in the development and use of AI, including generative AI, may result in reputational harm or liability."), | |
| 102 | + ("ai_risk_boilerplate", "Our competitors may adopt artificial intelligence more quickly than we do, which could adversely affect our results."), | |
| 103 | + ("paraphrase_no_keyword", "We deploy neural networks to forecast demand and route inventory in our warehouses."), | |
| 104 | + ("paraphrase_no_keyword", "Our platform learns from customer data to personalize recommendations automatically."), | |
| 105 | + ("paraphrase_no_keyword", "We trained a transformer model on our proprietary data to classify documents."), | |
| 106 | + ("paraphrase_no_keyword", "Our software uses predictive models that improve as more data is collected."), | |
| 107 | + ("automation_no_ai", "We automated our invoice processing with robotic process automation software."), | |
| 108 | + ("automation_no_ai", "The plant installed automated assembly lines and reduced manual labor by 15 percent."), | |
| 109 | + ("automation_no_ai", "Workforce restructuring reflects the deployment of automated logistics systems."), | |
| 110 | + ("automation_no_ai", "We continue to invest in process automation and digital transformation of our back office."), | |
| 111 | + ("cyber_no_ai", "We experienced a ransomware attack that disrupted our operations for several days."), | |
| 112 | + ("cyber_no_ai", "Cybersecurity incidents, data breaches and unauthorized access to our systems could harm our business."), | |
| 113 | + ("cyber_no_ai", "We maintain a security operations center and conduct annual penetration testing."), | |
| 114 | + ("utility_data_center_load", "Load growth from new data center customers is expected to increase our retail energy sales."), | |
| 115 | + ("utility_data_center_load", "We signed agreements to serve several large data centers, which require substantial new generation and transmission capacity."), | |
| 116 | + ("utility_data_center_load", "Our service territory is attracting data center and semiconductor manufacturing customers with significant electricity demand."), | |
| 117 | + ("utility_data_center_load", "The utility filed a rate case to recover the cost of infrastructure serving large load customers, including data centers."), | |
| 118 | + ("it_generic", "We migrated our enterprise resource planning system to the cloud during the year."), | |
| 119 | + ("it_generic", "Our information technology systems support order processing, inventory management and financial reporting."), | |
| 120 | + ("it_generic", "We invested in network equipment and data storage to support our growing customer base."), | |
| 121 | + ("unrelated", "Net sales increased 4 percent due to higher volumes in our beverage segment."), | |
| 122 | + ("unrelated", "The company declared a quarterly dividend of 0.25 dollars per share payable in March."), | |
| 123 | + ("unrelated", "We lease our corporate headquarters under an agreement that expires in 2029."), | |
| 124 | + ("unrelated", "Interest expense decreased because of lower average borrowings under the credit facility."), | |
| 125 | + ("unrelated", "Our effective tax rate was 21.4 percent compared with 22.1 percent in the prior year."), | |
| 126 | +] | |
| 127 | + | |
| 128 | +t_all = time.time() | |
| 129 | +out = {"model": MODEL_ID, "device": "cpu", "seed": SEED} | |
| 130 | +with Timer("load model + embed anchors (local, anchors only)"): | |
| 131 | + model = SentenceTransformer(MODEL_ID, device="cpu") | |
| 132 | + out["max_seq_length"] = int(model.max_seq_length) | |
| 133 | + def centroid(texts): | |
| 134 | + v = model.encode(texts, normalize_embeddings=True, batch_size=len(texts), show_progress_bar=False) | |
| 135 | + cvec = v.mean(axis=0); return (cvec / np.linalg.norm(cvec)).astype(np.float32) | |
| 136 | + stored = pd.read_parquet(f"{DATA}/anchor_embeddings/anchor_embeddings.parquet") | |
| 137 | + S = {r.anchor_id: np.asarray(r.vector, dtype=np.float32) for r in stored.itertuples()} | |
| 138 | + L = {k: centroid(v) for k, v in ORIG.items()} | |
| 139 | + A = {k: centroid(v["texts"]) for k, v in ALT.items()} | |
| 140 | + out["stored_anchor_norms"] = {k: float(np.linalg.norm(v)) for k, v in S.items()} | |
| 141 | + out["local_vs_stored_anchor_cosine"] = {k: float(L[k] @ S[k] / np.linalg.norm(S[k])) for k in ORIG} | |
| 142 | + out["alt_vs_original_anchor_cosine"] = {k: {o: float(A[k] @ S[o] / np.linalg.norm(S[o])) for o in ORIG} for k in ALT} | |
| 143 | + out["original_anchor_cosine_matrix"] = {k: {o: float(S[k] @ S[o] / np.linalg.norm(S[k]) / np.linalg.norm(S[o])) for o in ORIG} for k in ORIG} | |
| 144 | + cols = [f"s_{k}" for k in ORIG] + [f"l_{k}" for k in ORIG] + [f"alt_{k}" for k in ALT] | |
| 145 | + M = np.stack([S[k] / np.linalg.norm(S[k]) for k in ORIG] + [L[k] for k in ORIG] + [A[k] for k in ALT]).astype(np.float32) # (21, 384) | |
| 146 | + json.dump({"original": ORIG, "alternative": ALT, "columns": cols}, open(f"{RES}/alt_anchor_texts.json", "w"), indent=1) | |
| 147 | + print(json.dumps(out["local_vs_stored_anchor_cosine"], indent=1)) | |
| 148 | + | |
| 149 | +ALT_PATH = f"{RES}/chunk_scores_alt.parquet" | |
| 150 | +if not os.path.exists(ALT_PATH + ".done"): | |
| 151 | + with Timer("score 10.7 M stored chunk vectors against 21 anchors (streaming)"): | |
| 152 | + dataset = ds.dataset(f"{DATA}/embeddings", format="parquet", partitioning="hive") | |
| 153 | + scanner = dataset.scanner(columns=["source_id", "vector"], batch_size=20000) | |
| 154 | + schema = pa.schema([pa.field("chunk_id", pa.string())] + [pa.field(cn, pa.float32()) for cn in cols]) | |
| 155 | + writer = pq.ParquetWriter(ALT_PATH + ".tmp", schema, compression="zstd") | |
| 156 | + n = 0; norm_stats = []; first = True | |
| 157 | + for batch in scanner.to_batches(): | |
| 158 | + if batch.num_rows == 0: continue | |
| 159 | + vec = batch.column("vector") | |
| 160 | + if isinstance(vec, pa.ChunkedArray): vec = vec.combine_chunks() | |
| 161 | + arr = np.asarray(vec.flatten().to_numpy(zero_copy_only=False), dtype=np.float32).reshape(len(vec), -1) | |
| 162 | + norms = np.linalg.norm(arr, axis=1) | |
| 163 | + if first: | |
| 164 | + out["vector_dim"] = int(arr.shape[1]); out["first_batch_norm_mean"] = float(norms.mean()); first = False | |
| 165 | + norm_stats.append((float(norms.mean()), float(norms.min()), float(norms.max()), len(norms))) | |
| 166 | + if not (0.99 <= norms.mean() <= 1.01): # the pipeline normalizes only if the first sample is not pre-normalized; we mirror it per batch | |
| 167 | + arr = arr / np.clip(norms, 1e-12, None)[:, None] | |
| 168 | + sc = arr @ M.T | |
| 169 | + tbl = pa.table({"chunk_id": batch.column("source_id"), **{cn: pa.array(sc[:, j], type=pa.float32()) for j, cn in enumerate(cols)}}) | |
| 170 | + writer.write_table(tbl); n += len(vec) | |
| 171 | + if n % 1_000_000 < 20000: print(f" {n:,} vectors", flush=True) | |
| 172 | + writer.close(); os.replace(ALT_PATH + ".tmp", ALT_PATH); open(ALT_PATH + ".done", "w").write(str(n)) | |
| 173 | + ns = np.array([[a, b, c_, d] for a, b, c_, d in norm_stats]) | |
| 174 | + out["vectors_scored"] = int(n); out["vector_norm"] = {"mean": float((ns[:, 0] * ns[:, 3]).sum() / ns[:, 3].sum()), "min": float(ns[:, 1].min()), "max": float(ns[:, 2].max())} | |
| 175 | +else: | |
| 176 | + out["vectors_scored"] = int(open(ALT_PATH + ".done").read()) | |
| 177 | + | |
| 178 | +c = connect(); views(c) | |
| 179 | +with Timer("validate recomputation vs stored chunk_scores"): | |
| 180 | + c.execute(f"CREATE OR REPLACE TABLE alt AS SELECT * FROM read_parquet('{ALT_PATH}') QUALIFY row_number() OVER (PARTITION BY chunk_id ORDER BY chunk_id) = 1") | |
| 181 | + out["alt_rows_dedup"] = c.execute("SELECT count(*) FROM alt").fetchone()[0] | |
| 182 | + out["alt_join_chunk_scores"] = c.execute("SELECT count(*) FROM alt JOIN chunk_scores USING (chunk_id)").fetchone()[0] | |
| 183 | + exprs = ", ".join(f"max(abs(cs.score_{k} - a.s_{k})) AS maxdiff_s_{k}, corr(cs.score_{k}, a.s_{k}) AS corr_s_{k}, max(abs(cs.score_{k} - a.l_{k})) AS maxdiff_l_{k}, corr(cs.score_{k}, a.l_{k}) AS corr_l_{k}" for k in ORIG) | |
| 184 | + out["chunk_level_validation"] = c.execute(f"SELECT {exprs} FROM chunk_scores cs JOIN alt a USING (chunk_id)").df().iloc[0].to_dict() | |
| 185 | + out["chunk_level_alt_corr"] = c.execute("SELECT " + ", ".join(f"corr(cs.score_ai_adoption, a.alt_{k}) AS adopt_vs_{k}, corr(cs.score_ai_infrastructure, a.alt_{k}) AS infra_vs_{k}" for k in ALT) + " FROM chunk_scores cs JOIN alt a USING (chunk_id)").df().iloc[0].to_dict() | |
| 186 | + # Spearman at chunk level on a 2 M hash sample | |
| 187 | + out["chunk_level_alt_spearman_2Msample"] = c.execute("WITH s AS (SELECT cs.score_ai_adoption a0, cs.score_ai_infrastructure i0, a.* FROM chunk_scores cs JOIN alt a USING (chunk_id) WHERE hash(chunk_id) % 5 = 0), r AS (SELECT rank() OVER (ORDER BY a0) ra0, rank() OVER (ORDER BY i0) ri0, " + ", ".join(f"rank() OVER (ORDER BY alt_{k}) r_{k}" for k in ALT) + " FROM s) SELECT " + ", ".join(f"corr(ra0, r_{k}) AS adopt_vs_{k}, corr(ri0, r_{k}) AS infra_vs_{k}" for k in ALT) + " FROM r").df().iloc[0].to_dict() | |
| 188 | + | |
| 189 | +with Timer("filing-level and firm-year aggregation (pipeline weights)"): | |
| 190 | + wop = weight_case("operational_measures", "cs.section_name"); wrk = weight_case("risk_measures", "cs.section_name") | |
| 191 | + prof = {**{f"s_{k}": ANCHOR_PROFILE[k] for k in ORIG}, **{f"l_{k}": ANCHOR_PROFILE[k] for k in ORIG}, **{f"alt_{k}": ALT[k]["profile"] for k in ALT}} | |
| 192 | + stats = ", ".join(f"sum(a.{cn} * {wop if p == 'operational_measures' else wrk}) / NULLIF(sum({wop if p == 'operational_measures' else wrk}), 0) AS {cn}_wmean, avg(a.{cn}) AS {cn}_mean, max(a.{cn}) AS {cn}_max" for cn, p in prof.items()) | |
| 193 | + c.execute(f"""CREATE OR REPLACE TABLE filing_alt AS SELECT cs.accession_number, any_value(cs.cik) cik, any_value(cs.ticker) ticker, any_value(cs.form_type) form_type, | |
| 194 | + any_value(cs.fiscal_year) fiscal_year, count(*) chunk_count, {stats} FROM chunk_scores cs JOIN alt a USING (chunk_id) GROUP BY 1""") | |
| 195 | + out["filing_level_validation"] = c.execute("SELECT count(*) n, max(abs(f.ai_adoption_wmean - x.s_ai_adoption_wmean)) maxdiff_adopt_wmean, corr(f.ai_adoption_wmean, x.s_ai_adoption_wmean) corr_adopt_wmean, max(abs(f.ai_infrastructure_wmean - x.s_ai_infrastructure_wmean)) maxdiff_infra_wmean, max(abs(f.chunk_count - x.chunk_count)) maxdiff_chunks FROM filing_scores f JOIN filing_alt x USING (accession_number)").df().iloc[0].to_dict() | |
| 196 | + wcols = [f"{cn}_wmean" for cn in prof] | |
| 197 | + fy = c.execute("SELECT cik, any_value(ticker) ticker, fiscal_year, count(*) n_filings, " + ", ".join(f"avg({w}) AS {w}" for w in wcols) + ", " + ", ".join(f"avg(CASE WHEN form_type='10-K' THEN {w} END) AS {w}_10k" for w in wcols if w.startswith("alt_") or w in ("s_ai_adoption_wmean", "s_ai_infrastructure_wmean")) + " FROM filing_alt GROUP BY cik, fiscal_year").df() | |
| 198 | + af = c.execute("SELECT cik, ticker, fiscal_year, sector, ai_adoption_wmean, ai_infrastructure_wmean, ai_disruption_risk_wmean, cybersecurity_wmean, automation_labor_wmean FROM annual_final").df() | |
| 199 | + m = af.merge(fy.drop(columns=["ticker"]), on=["cik", "fiscal_year"], how="inner") | |
| 200 | + m = m[m.fiscal_year.between(2010, 2024)].copy() | |
| 201 | + out["firm_year_rows"] = int(len(m)); out["annual_final_rows_2010_2024"] = int(af.fiscal_year.between(2010, 2024).sum()) | |
| 202 | + out["firm_year_validation"] = {"maxdiff_adopt_wmean_vs_annual_final": float((m.ai_adoption_wmean - m.s_ai_adoption_wmean).abs().max()), | |
| 203 | + "corr_adopt_wmean_vs_annual_final": float(pearsonr(m.ai_adoption_wmean, m.s_ai_adoption_wmean)[0]), | |
| 204 | + "maxdiff_infra_wmean_vs_annual_final": float((m.ai_infrastructure_wmean - m.s_ai_infrastructure_wmean).abs().max()), | |
| 205 | + "maxdiff_adopt_local_vs_stored_anchor": float((m.l_ai_adoption_wmean - m.s_ai_adoption_wmean).abs().max())} | |
| 206 | + m.to_csv(f"{RES}/alt_firm_year.csv", index=False, float_format="%.6g") | |
| 207 | + | |
| 208 | +def z(g): return (g - g.mean()) / g.std(ddof=0) | |
| 209 | +with Timer("firm-year comparisons"): | |
| 210 | + for col in ["ai_adoption_wmean", "ai_infrastructure_wmean", "l_ai_adoption_wmean", "l_ai_infrastructure_wmean"] + [f"alt_{k}_wmean" for k in ALT] + ["alt_ai_adoption_strip_wmean_10k", "alt_ai_infrastructure_strip_wmean_10k", "s_ai_adoption_wmean_10k", "s_ai_infrastructure_wmean_10k"]: | |
| 211 | + m[f"z_{col}"] = m.groupby("fiscal_year")[col].transform(z) | |
| 212 | + m["z_compos"] = m[["z_ai_adoption_wmean", "z_ai_infrastructure_wmean"]].mean(axis=1) | |
| 213 | + m["z_compos_local"] = m[["z_l_ai_adoption_wmean", "z_l_ai_infrastructure_wmean"]].mean(axis=1) | |
| 214 | + m["z_compos_strip"] = m[["z_alt_ai_adoption_strip_wmean", "z_alt_ai_infrastructure_strip_wmean"]].mean(axis=1) | |
| 215 | + m["z_compos_core_strip"] = m[["z_alt_ai_adoption_core_wmean", "z_alt_ai_infrastructure_strip_wmean"]].mean(axis=1) | |
| 216 | + m["z_compos_strip_10k"] = m[["z_alt_ai_adoption_strip_wmean_10k", "z_alt_ai_infrastructure_strip_wmean_10k"]].mean(axis=1) | |
| 217 | + m["z_compos_10k"] = m[["z_s_ai_adoption_wmean_10k", "z_s_ai_infrastructure_wmean_10k"]].mean(axis=1) | |
| 218 | + def comp(a, b): | |
| 219 | + d = m[[a, b, "fiscal_year", "ticker"]].dropna() | |
| 220 | + qa = d.groupby("fiscal_year")[a].transform(lambda s: pd.qcut(s.rank(method="first"), 5, labels=False) + 1) | |
| 221 | + qb = d.groupby("fiscal_year")[b].transform(lambda s: pd.qcut(s.rank(method="first"), 5, labels=False) + 1) | |
| 222 | + tr = pd.crosstab(qa, qb, normalize="index").round(3) | |
| 223 | + return {"n": int(len(d)), "spearman": float(spearmanr(d[a], d[b]).correlation), "pearson": float(pearsonr(d[a], d[b])[0]), | |
| 224 | + "share_same_quintile": float((qa == qb).mean()), "share_Q5_stays_Q5": float((qb[qa == 5] == 5).mean()), "share_Q1_stays_Q1": float((qb[qa == 1] == 1).mean()), | |
| 225 | + "share_Q5_to_Q45": float((qb[qa == 5] >= 4).mean()), "share_Q5_to_Q12": float((qb[qa == 5] <= 2).mean()), | |
| 226 | + "quintile_transition_rows_a_cols_b": {int(i): {int(j): float(tr.loc[i, j]) for j in tr.columns} for i in tr.index}, | |
| 227 | + "spearman_by_year": {int(y): round(float(spearmanr(g[a], g[b]).correlation), 4) for y, g in d.groupby("fiscal_year")}} | |
| 228 | + out["firm_year"] = { | |
| 229 | + "adopt_vs_adopt_strip": comp("ai_adoption_wmean", "alt_ai_adoption_strip_wmean"), | |
| 230 | + "adopt_vs_adopt_core": comp("ai_adoption_wmean", "alt_ai_adoption_core_wmean"), | |
| 231 | + "infra_vs_infra_strip": comp("ai_infrastructure_wmean", "alt_ai_infrastructure_strip_wmean"), | |
| 232 | + "cyber_vs_cyber_strip": comp("cybersecurity_wmean", "alt_cybersecurity_strip_wmean"), | |
| 233 | + "autom_vs_autom_strip": comp("automation_labor_wmean", "alt_automation_labor_strip_wmean"), | |
| 234 | + "compos_vs_compos_strip": comp("z_compos", "z_compos_strip"), | |
| 235 | + "compos_vs_compos_core_strip": comp("z_compos", "z_compos_core_strip"), | |
| 236 | + "compos_vs_compos_local_reembedded_anchors": comp("z_compos", "z_compos_local"), | |
| 237 | + "compos_10k_vs_compos_strip_10k": comp("z_compos_10k", "z_compos_strip_10k"), | |
| 238 | + "adopt_vs_cyber_original": comp("ai_adoption_wmean", "cybersecurity_wmean"), | |
| 239 | + "adopt_strip_vs_cyber_strip": comp("alt_ai_adoption_strip_wmean", "alt_cybersecurity_strip_wmean"), | |
| 240 | + "adopt_vs_autom_original": comp("ai_adoption_wmean", "automation_labor_wmean"), | |
| 241 | + "adopt_strip_vs_autom_strip": comp("alt_ai_adoption_strip_wmean", "alt_automation_labor_strip_wmean"), | |
| 242 | + "adopt_core_vs_cyber_strip": comp("alt_ai_adoption_core_wmean", "alt_cybersecurity_strip_wmean"), | |
| 243 | + "adopt_core_vs_autom_strip": comp("alt_ai_adoption_core_wmean", "alt_automation_labor_strip_wmean"), | |
| 244 | + "adopt_vs_infra_original": comp("ai_adoption_wmean", "ai_infrastructure_wmean"), | |
| 245 | + "adopt_strip_vs_infra_strip": comp("alt_ai_adoption_strip_wmean", "alt_ai_infrastructure_strip_wmean"), | |
| 246 | + } | |
| 247 | + by = pd.DataFrame([{"fiscal_year": int(y), "n": int(len(g)), | |
| 248 | + "sp_adopt_strip": spearmanr(g.ai_adoption_wmean, g.alt_ai_adoption_strip_wmean).correlation, | |
| 249 | + "sp_adopt_core": spearmanr(g.ai_adoption_wmean, g.alt_ai_adoption_core_wmean).correlation, | |
| 250 | + "sp_infra_strip": spearmanr(g.ai_infrastructure_wmean, g.alt_ai_infrastructure_strip_wmean).correlation, | |
| 251 | + "sp_compos_strip": spearmanr(g.z_compos, g.z_compos_strip).correlation, | |
| 252 | + "sp_compos_core_strip": spearmanr(g.z_compos, g.z_compos_core_strip).correlation, | |
| 253 | + "sp_adopt_cyber_orig": spearmanr(g.ai_adoption_wmean, g.cybersecurity_wmean).correlation, | |
| 254 | + "sp_adoptstrip_cyberstrip": spearmanr(g.alt_ai_adoption_strip_wmean, g.alt_cybersecurity_strip_wmean).correlation, | |
| 255 | + "sp_adopt_autom_orig": spearmanr(g.ai_adoption_wmean, g.automation_labor_wmean).correlation, | |
| 256 | + "sp_adoptstrip_automstrip": spearmanr(g.alt_ai_adoption_strip_wmean, g.alt_automation_labor_strip_wmean).correlation, | |
| 257 | + "mean_adopt": g.ai_adoption_wmean.mean(), "mean_adopt_strip": g.alt_ai_adoption_strip_wmean.mean(), "mean_adopt_core": g.alt_ai_adoption_core_wmean.mean()} | |
| 258 | + for y, g in m.groupby("fiscal_year")]) | |
| 259 | + by.to_csv(f"{RES}/alt_corr_by_year.csv", index=False, float_format="%.4f") | |
| 260 | + # top-30 firms (>= 5 years) under each composite | |
| 261 | + def top(colz): | |
| 262 | + t = m.groupby(["ticker", "sector"])[colz].agg(["mean", "count"]).reset_index(); t = t[t["count"] >= 5].nlargest(30, "mean") | |
| 263 | + return t | |
| 264 | + t0 = top("z_compos"); t1 = top("z_compos_strip"); t2 = top("z_compos_core_strip") | |
| 265 | + out["top30_original"] = t0[["ticker", "sector", "mean"]].round(3).to_dict("records") | |
| 266 | + out["top30_strip"] = t1[["ticker", "sector", "mean"]].round(3).to_dict("records") | |
| 267 | + out["top30_core_strip"] = t2[["ticker", "sector", "mean"]].round(3).to_dict("records") | |
| 268 | + out["top30_overlap_strip"] = int(len(set(t0.ticker) & set(t1.ticker))); out["top30_overlap_core_strip"] = int(len(set(t0.ticker) & set(t2.ticker))) | |
| 269 | + out["top30_utilities_original"] = int((t0.sector == "Utilities").sum()); out["top30_utilities_strip"] = int((t1.sector == "Utilities").sum()); out["top30_utilities_core_strip"] = int((t2.sector == "Utilities").sum()) | |
| 270 | + rk = m.groupby(["ticker", "sector"]).agg(z=("z_compos", "mean"), zs=("z_compos_strip", "mean"), zc=("z_compos_core_strip", "mean"), yrs=("fiscal_year", "count")).reset_index() | |
| 271 | + rk = rk[rk.yrs >= 5]; rk["r"] = rk.z.rank(ascending=False); rk["rs"] = rk.zs.rank(ascending=False); rk["rc"] = rk.zc.rank(ascending=False) | |
| 272 | + out["firm_level_spearman_compos_vs_strip"] = float(spearmanr(rk.z, rk.zs).correlation); out["firm_level_spearman_compos_vs_core_strip"] = float(spearmanr(rk.z, rk.zc).correlation) | |
| 273 | + out["ida_he_ranks"] = rk[rk.ticker.isin(["IDA", "HE"])][["ticker", "z", "r", "zs", "rs", "zc", "rc"]].round(3).to_dict("records") | |
| 274 | + out["sector_mean_z"] = m.groupby("sector")[["z_compos", "z_compos_strip", "z_compos_core_strip"]].mean().round(3).to_dict("index") | |
| 275 | + | |
| 276 | +with Timer("X3 probe sentences"): | |
| 277 | + texts = [t for _, t in PROBES]; grp = [g for g, _ in PROBES] | |
| 278 | + E = model.encode(texts, normalize_embeddings=True, batch_size=64, show_progress_bar=False) | |
| 279 | + P = E @ M.T | |
| 280 | + pr = pd.DataFrame(P, columns=cols); pr.insert(0, "group", grp); pr.insert(1, "sentence", texts) | |
| 281 | + pr.to_csv(f"{RES}/probe_sentences.csv", index=False, float_format="%.4f") | |
| 282 | + keep = ["s_ai_adoption", "s_ai_infrastructure", "s_ai_disruption_risk", "s_cybersecurity", "s_automation_labor", "alt_ai_adoption_strip", "alt_ai_adoption_core", "alt_ai_infrastructure_strip"] | |
| 283 | + out["probe_group_means"] = pr.groupby("group")[keep].mean().round(4).to_dict("index") | |
| 284 | + aff = pr[pr.group == "affirmative_ai"]; neg = pr[pr.group == "negated_ai"] | |
| 285 | + out["probe_negation"] = {k: {"mean_affirmative": float(aff[k].mean()), "mean_negated": float(neg[k].mean()), "diff": float(aff[k].mean() - neg[k].mean()), | |
| 286 | + "share_pairs_aff_gt_neg": float((aff[k].values[:, None] > neg[k].values[None, :]).mean()), | |
| 287 | + "min_affirmative": float(aff[k].min()), "max_negated": float(neg[k].max())} for k in ["s_ai_adoption", "alt_ai_adoption_strip", "alt_ai_adoption_core"]} | |
| 288 | + # where would the probe sentences fall in the corpus distribution of score_ai_adoption? | |
| 289 | + qs = c.execute("SELECT quantile_cont(score_ai_adoption, [0.5, 0.9, 0.95, 0.99, 0.999]) FROM chunk_scores").fetchone()[0] | |
| 290 | + out["corpus_adopt_quantiles_50_90_95_99_999"] = [float(x) for x in qs] | |
| 291 | + out["probe_pct_rank_in_corpus"] = {g: float(c.execute(f"SELECT avg(CASE WHEN score_ai_adoption < {v} THEN 1.0 ELSE 0 END) FROM chunk_scores").fetchone()[0]) for g, v in pr.groupby("group")["s_ai_adoption"].mean().items()} | |
| 292 | + | |
| 293 | +out["runtime_s"] = time.time() - t_all | |
| 294 | +dump_json(out, f"{RES}/alt_anchor_results.json") | |
| 295 | +print(json.dumps({k: {kk: vv for kk, vv in v.items() if kk not in ("quintile_transition_rows_a_cols_b", "spearman_by_year")} for k, v in out["firm_year"].items()}, indent=1)) | |
| 296 | +print("probe", json.dumps(out["probe_negation"], indent=1)); print("top30 overlap", out["top30_overlap_strip"], out["top30_overlap_core_strip"], "utilities", out["top30_utilities_original"], out["top30_utilities_strip"], out["top30_utilities_core_strip"]) | |
added
code/n1/n1_06_alt_anchor_sorts.py
+117 −0
@@ -0,0 +1,117 @@ | ||
| 1 | +"""n1_06_alt_anchor_sorts.py — laptop step of X2 / N1: T5 / T7 / T9 rows for the stripped-anchor composites and the | |
| 2 | +keyword baseline, on the author's analysis panel (read-only; baseline returns as committed — rerun on the L2/L3 v2 panel | |
| 3 | +by passing its path as argv[1]). | |
| 4 | + | |
| 5 | +Replicates sort_table / nw_t / fama_macbeth / spread_sub of scripts/r3k_03_analysis.py (same code, same options) and | |
| 6 | +first reproduces the committed AI_COMPOS rows as a check. Signals: AI_COMPOS (original), ALT_COMPOS_STRIP | |
| 7 | +(½[z(adopt_strip)+z(infra_strip)]), ALT_COMPOS_CORE_STRIP, ALT_ADOPT_STRIP, ALT_ADOPT_CORE, ALT_INFRA_STRIP, | |
| 8 | +KW_CORE (regex hits per 1,000 words, firm-year mean over filings), KW_EXT. | |
| 9 | +Outputs: results/n1/alt_anchor_sorts.json, alt_anchor_sorts_rows.csv | |
| 10 | +usage: python -I n1_06_alt_anchor_sorts.py [analysis_panel.parquet] [alt_firm_year.csv] [keyword_firm_year.csv] [out_dir] | |
| 11 | +""" | |
| 12 | +import sys, os, json, warnings | |
| 13 | +import numpy as np, pandas as pd, statsmodels.api as sm | |
| 14 | +from statsmodels.stats.sandwich_covariance import cov_hac | |
| 15 | +warnings.filterwarnings("ignore") | |
| 16 | +HERE = os.path.dirname(os.path.abspath(__file__)); WT = os.path.abspath(os.path.join(HERE, "..", "..")) | |
| 17 | +PANEL = sys.argv[1] if len(sys.argv) > 1 else os.path.expanduser("~/Desktop/UQO/Recherche/Recherche-UQO/finance/ai-disclosure-stock-returns/data/processed_r3k/analysis_panel.parquet") | |
| 18 | +ALT = sys.argv[2] if len(sys.argv) > 2 else f"{WT}/results/n1/alt_firm_year.csv" | |
| 19 | +KW = sys.argv[3] if len(sys.argv) > 3 else f"{WT}/results/n1/keyword_firm_year.csv" | |
| 20 | +OUT = sys.argv[4] if len(sys.argv) > 4 else f"{WT}/results/n1" | |
| 21 | +PRE, POST = range(2010, 2022), range(2022, 2026) | |
| 22 | + | |
| 23 | +df = pd.read_parquet(PANEL) | |
| 24 | +df = df[df.fwd_ret.notna() & (df.fiscal_year <= 2024)].copy() | |
| 25 | +df["cik"] = df["cik"].astype(str).str.zfill(10) | |
| 26 | +alt = pd.read_csv(ALT, dtype={"cik": str}); alt["cik"] = alt["cik"].str.zfill(10) | |
| 27 | +kw = pd.read_csv(KW, dtype={"cik": str}); kw["cik"] = kw["cik"].str.zfill(10) | |
| 28 | +acols = ["alt_ai_adoption_strip_wmean", "alt_ai_adoption_core_wmean", "alt_ai_infrastructure_strip_wmean", "alt_cybersecurity_strip_wmean", "alt_automation_labor_strip_wmean", "s_ai_adoption_wmean", "s_ai_infrastructure_wmean"] | |
| 29 | +df = df.merge(alt[["cik", "fiscal_year"] + acols], on=["cik", "fiscal_year"], how="left").merge(kw[["cik", "fiscal_year", "kw_core_mean", "kw_ext_mean", "kw_core_pooled"]], on=["cik", "fiscal_year"], how="left") | |
| 30 | +print("panel", len(df), "| alt matched", int(df.alt_ai_adoption_strip_wmean.notna().sum()), "| kw matched", int(df.kw_core_mean.notna().sum())) | |
| 31 | +if "fwd_w" not in df.columns: | |
| 32 | + def wins(s, lo=.01, hi=.99): return s.clip(s.quantile(lo), s.quantile(hi)) | |
| 33 | + df["fwd_w"] = wins(df["fwd_ret"]) | |
| 34 | +def zscore(g): return (g - g.mean()) / g.std(ddof=0) | |
| 35 | +for col in acols + ["kw_core_mean", "kw_ext_mean", "kw_core_pooled"]: | |
| 36 | + df[f"z_{col}"] = df.groupby("fiscal_year")[col].transform(zscore) | |
| 37 | +df["z_ALT_COMPOS_STRIP"] = df[["z_alt_ai_adoption_strip_wmean", "z_alt_ai_infrastructure_strip_wmean"]].mean(axis=1) | |
| 38 | +df["z_ALT_COMPOS_CORE_STRIP"] = df[["z_alt_ai_adoption_core_wmean", "z_alt_ai_infrastructure_strip_wmean"]].mean(axis=1) | |
| 39 | +df["z_ALT_ADOPT_STRIP"] = df["z_alt_ai_adoption_strip_wmean"]; df["z_ALT_ADOPT_CORE"] = df["z_alt_ai_adoption_core_wmean"]; df["z_ALT_INFRA_STRIP"] = df["z_alt_ai_infrastructure_strip_wmean"] | |
| 40 | +df["z_CYBER_STRIP"] = df["z_alt_cybersecurity_strip_wmean"]; df["z_AUTOM_STRIP"] = df["z_alt_automation_labor_strip_wmean"] | |
| 41 | +df["z_KW_CORE"] = df["z_kw_core_mean"]; df["z_KW_EXT"] = df["z_kw_ext_mean"]; df["z_KW_CORE_POOLED"] = df["z_kw_core_pooled"] | |
| 42 | +df["z_COMPOS_RECOMP"] = df[["z_s_ai_adoption_wmean", "z_s_ai_infrastructure_wmean"]].mean(axis=1) # recomputed from stored anchors (check) | |
| 43 | +if "z_AI_COMPOS" not in df.columns: | |
| 44 | + for k, col in {"AI_ADOPT": "ai_adoption_wmean", "AI_INFRA": "ai_infrastructure_wmean"}.items(): df[f"z_{k}"] = df.groupby("fiscal_year")[col].transform(zscore) | |
| 45 | + df["z_AI_COMPOS"] = df[["z_AI_ADOPT", "z_AI_INFRA"]].mean(axis=1) | |
| 46 | + | |
| 47 | +def nw_t(x, lags=3): | |
| 48 | + x = pd.Series(x).dropna().values | |
| 49 | + if len(x) < 3: return (float(x.mean()) if len(x) else np.nan), np.nan | |
| 50 | + r = sm.OLS(x, np.ones((len(x), 1))).fit(); return float(x.mean()), float(x.mean() / np.sqrt(cov_hac(r, nlags=lags)[0, 0])) | |
| 51 | +def sort_table(d, z, weight=None, oc="fwd_ret", minn=40): | |
| 52 | + ann = {q: [] for q in range(1, 6)}; ls = [] | |
| 53 | + for yr, g in d.groupby("fiscal_year"): | |
| 54 | + g = g.dropna(subset=[z, oc]).copy() | |
| 55 | + if len(g) < minn: continue | |
| 56 | + g["q"] = pd.qcut(g[z].rank(method="first"), 5, labels=False) + 1 | |
| 57 | + for q in range(1, 6): | |
| 58 | + s = g[g.q == q] | |
| 59 | + if s.empty: continue | |
| 60 | + if weight is None: ann[q].append(s[oc].mean()) | |
| 61 | + else: | |
| 62 | + w = s[weight].clip(lower=0).fillna(0); ann[q].append(np.average(s[oc], weights=w) if w.sum() > 0 else s[oc].mean()) | |
| 63 | + if ann[5] and ann[1]: ls.append(ann[5][-1] - ann[1][-1]) | |
| 64 | + out = {f"Q{q}": nw_t(ann[q])[0] for q in range(1, 6)} | |
| 65 | + mu, t = nw_t(ls); out["Q5_minus_Q1"] = mu; out["t_stat"] = t; out["n_years"] = len(ls); return out | |
| 66 | +def fama_macbeth(d, xvars, minn=40, oc="fwd_ret"): | |
| 67 | + coefs = {v: [] for v in ["const"] + xvars} | |
| 68 | + for yr, g in d.groupby("fiscal_year"): | |
| 69 | + g = g.dropna(subset=[oc] + xvars) | |
| 70 | + if len(g) < minn: continue | |
| 71 | + r = sm.OLS(g[oc], sm.add_constant(g[xvars])).fit() | |
| 72 | + for v in coefs: coefs[v].append(r.params.get(v, np.nan)) | |
| 73 | + out = {v: dict(zip(["mean_coef", "t"], [round(x, 4) for x in nw_t(coefs[v])])) for v in coefs}; out["n_years"] = len(coefs["const"]); return out | |
| 74 | +def spread_sub(d, z, yrs, oc="fwd_ret", minn=40): | |
| 75 | + sp = {} | |
| 76 | + for yr, g in d[d.fiscal_year.isin(list(yrs))].groupby("fiscal_year"): | |
| 77 | + g = g.dropna(subset=[z, oc]).copy() | |
| 78 | + if len(g) < minn: continue | |
| 79 | + g["q"] = pd.qcut(g[z].rank(method="first"), 5, labels=False) + 1; sp[int(yr)] = float(g[g.q == 5][oc].mean() - g[g.q == 1][oc].mean()) | |
| 80 | + mu, t = nw_t(list(sp.values())); return {"mean": round(mu, 4), "t": round(t, 2) if t == t else None, "n": len(sp), "by_cohort": {k: round(v, 4) for k, v in sp.items()}} | |
| 81 | + | |
| 82 | +SIG = ["AI_COMPOS", "COMPOS_RECOMP", "ALT_COMPOS_STRIP", "ALT_COMPOS_CORE_STRIP", "ALT_ADOPT_STRIP", "ALT_ADOPT_CORE", "ALT_INFRA_STRIP", "CYBER_STRIP", "AUTOM_STRIP", "KW_CORE", "KW_EXT", "KW_CORE_POOLED"] | |
| 83 | +RES = {"panel": PANEL, "n_obs": int(len(df)), "n_firms": int(df.ticker.nunique()), "n_alt_matched": int(df.alt_ai_adoption_strip_wmean.notna().sum()), "n_kw_matched": int(df.kw_core_mean.notna().sum()), | |
| 84 | + "share_zero_kw_core": float((df.kw_core_mean == 0).mean()), "sorts": {}, "fm": {}, "prepost": {}, "sorts_sp500": {}, "prepost_sp500": {}, "fm_sp500": {}} | |
| 85 | +sp = df[df.in_sp500 == True] if "in_sp500" in df.columns else df.iloc[0:0] | |
| 86 | +for k in SIG: | |
| 87 | + z = f"z_{k}" | |
| 88 | + RES["sorts"][k] = {"EW": sort_table(df, z), "VW": sort_table(df, z, "mktcap"), "EW_madj": sort_table(df, z, None, "fwd_ret_mktadj"), "EW_wins": sort_table(df, z, None, "fwd_w")} | |
| 89 | + RES["fm"][k] = fama_macbeth(df, [z, "z_size", "z_mom", "z_vol"]) | |
| 90 | + RES["prepost"][k] = {"pre": spread_sub(df, z, PRE), "post": spread_sub(df, z, POST)} | |
| 91 | + if len(sp): | |
| 92 | + RES["sorts_sp500"][k] = {"EW": sort_table(sp, z), "VW": sort_table(sp, z, "mktcap")} | |
| 93 | + RES["prepost_sp500"][k] = {"pre": spread_sub(sp, z, PRE), "post": spread_sub(sp, z, POST)} | |
| 94 | + RES["fm_sp500"][k] = fama_macbeth(sp, [z, "z_size", "z_mom", "z_vol"]) | |
| 95 | +# composite with keyword control: does the semantic composite survive a keyword-rate control (FM)? | |
| 96 | +RES["fm_compos_with_kw_control"] = fama_macbeth(df, ["z_AI_COMPOS", "z_KW_CORE", "z_size", "z_mom", "z_vol"]) | |
| 97 | +RES["fm_strip_with_kw_control"] = fama_macbeth(df, ["z_ALT_COMPOS_STRIP", "z_KW_CORE", "z_size", "z_mom", "z_vol"]) | |
| 98 | +RES["fm_compos_vs_strip_horse_race"] = fama_macbeth(df, ["z_AI_COMPOS", "z_ALT_COMPOS_STRIP", "z_size", "z_mom", "z_vol"]) | |
| 99 | +RES["corr_in_panel"] = df[["z_AI_COMPOS", "z_COMPOS_RECOMP", "z_ALT_COMPOS_STRIP", "z_ALT_COMPOS_CORE_STRIP", "z_KW_CORE", "z_KW_EXT"]].corr(method="spearman").round(4).to_dict() | |
| 100 | +os.makedirs(OUT, exist_ok=True) | |
| 101 | +json.dump(RES, open(f"{OUT}/alt_anchor_sorts.json", "w"), indent=1, default=float) | |
| 102 | +rows = [] | |
| 103 | +for k in SIG: | |
| 104 | + for w in ["EW", "VW", "EW_madj", "EW_wins"]: | |
| 105 | + s = RES["sorts"][k][w]; rows.append({"signal": k, "sample": "Russell 3000", "weighting": w, **{q: round(100 * s[q], 1) for q in ["Q1", "Q2", "Q3", "Q4", "Q5", "Q5_minus_Q1"]}, "t": round(s["t_stat"], 2), "years": s["n_years"], | |
| 106 | + "fm_slope_pct_per_sd": round(100 * RES["fm"][k][f"z_{k}"]["mean_coef"], 2), "fm_t": round(RES["fm"][k][f"z_{k}"]["t"], 2), | |
| 107 | + "pre_spread": round(100 * RES["prepost"][k]["pre"]["mean"], 1), "pre_t": RES["prepost"][k]["pre"]["t"], "post_spread": round(100 * RES["prepost"][k]["post"]["mean"], 1), "post_t": RES["prepost"][k]["post"]["t"], | |
| 108 | + "post_cohorts": " / ".join(f"{100*v:+.1f}" for v in RES["prepost"][k]["post"]["by_cohort"].values())}) | |
| 109 | + if len(sp): | |
| 110 | + for w in ["EW", "VW"]: | |
| 111 | + s = RES["sorts_sp500"][k][w]; rows.append({"signal": k, "sample": "S&P 500", "weighting": w, **{q: round(100 * s[q], 1) for q in ["Q1", "Q2", "Q3", "Q4", "Q5", "Q5_minus_Q1"]}, "t": round(s["t_stat"], 2), "years": s["n_years"], | |
| 112 | + "fm_slope_pct_per_sd": round(100 * RES["fm_sp500"][k][f"z_{k}"]["mean_coef"], 2), "fm_t": round(RES["fm_sp500"][k][f"z_{k}"]["t"], 2), | |
| 113 | + "pre_spread": round(100 * RES["prepost_sp500"][k]["pre"]["mean"], 1), "pre_t": RES["prepost_sp500"][k]["pre"]["t"], "post_spread": round(100 * RES["prepost_sp500"][k]["post"]["mean"], 1), "post_t": RES["prepost_sp500"][k]["post"]["t"], | |
| 114 | + "post_cohorts": " / ".join(f"{100*v:+.1f}" for v in RES["prepost_sp500"][k]["post"]["by_cohort"].values())}) | |
| 115 | +pd.DataFrame(rows).to_csv(f"{OUT}/alt_anchor_sorts_rows.csv", index=False) | |
| 116 | +print(pd.DataFrame(rows)[["signal", "sample", "weighting", "Q5_minus_Q1", "t", "fm_slope_pct_per_sd", "fm_t", "pre_spread", "pre_t", "post_spread", "post_t"]].to_string()) | |
| 117 | +print("horse race", RES["fm_compos_vs_strip_horse_race"]); print("kw control", RES["fm_compos_with_kw_control"]) | |
added
code/n1/n1_common.py
+117 −0
@@ -0,0 +1,117 @@ | ||
| 1 | +"""n1_common.py — shared settings for the FIRC-0002 Stage D node batch N1 (M4M64a). | |
| 2 | + | |
| 3 | +Data: read-only rsync copy of M3U96b:~/UQO_RESEARCH_DB/data/parquet/ in ~/firc0002_review/data/ | |
| 4 | +(chmod a-w after the copy). Every query runs on M4M64a; nothing is ever written to M3U96b. | |
| 5 | +DuckDB 2.0.0, threads <= 8, memory_limit 24 GB, seed 20261105. | |
| 6 | +""" | |
| 7 | +from __future__ import annotations | |
| 8 | +import os, json, time, re | |
| 9 | +import duckdb | |
| 10 | + | |
| 11 | +BASE = os.path.expanduser("~/firc0002_review") | |
| 12 | +DATA = f"{BASE}/data" | |
| 13 | +RES = f"{BASE}/results" | |
| 14 | +LOGS = f"{BASE}/logs" | |
| 15 | +TMP = f"{BASE}/tmp" | |
| 16 | +SEED = 20261105 | |
| 17 | +THREADS = int(os.environ.get("N1_THREADS", "8")) | |
| 18 | +MEM = os.environ.get("N1_MEM", "24GB") | |
| 19 | +MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2" | |
| 20 | + | |
| 21 | +# Acceptance targets quoted in the manuscript / plan | |
| 22 | +TARGET_CHUNKS = 10_720_890 | |
| 23 | +TARGET_SECTIONS = 1_342_283 | |
| 24 | +TARGET_FILINGS = 527_548 | |
| 25 | + | |
| 26 | +# Section weights of the pipeline (config/factors.yaml, UQO_RESEARCH_DB) — reproduced verbatim | |
| 27 | +SECTION_WEIGHTS = { | |
| 28 | + "risk_measures": {"risk_factors": 2.0, "md_and_a": 1.5, "legal_proceedings": 1.3, "full_body": 1.0, | |
| 29 | + "controls_and_procedures": 0.8, "quantitative_disclosures": 0.8, | |
| 30 | + "financial_statements": 0.5, "unregistered_sales": 0.5}, | |
| 31 | + "operational_measures": {"md_and_a": 2.0, "full_body": 1.0, "risk_factors": 1.2, | |
| 32 | + "controls_and_procedures": 0.8, "quantitative_disclosures": 0.8, | |
| 33 | + "financial_statements": 0.5, "legal_proceedings": 0.5, "unregistered_sales": 0.5}, | |
| 34 | +} | |
| 35 | +ANCHOR_PROFILE = {"ai_adoption": "operational_measures", "ai_disruption_risk": "risk_measures", | |
| 36 | + "ai_infrastructure": "operational_measures", "esg_commitment": "operational_measures", | |
| 37 | + "climate_risk": "risk_measures", "cybersecurity": "risk_measures", | |
| 38 | + "automation_labor": "operational_measures", "macro_risk": "risk_measures"} | |
| 39 | +ORIG_ANCHORS = ["ai_adoption", "ai_disruption_risk", "ai_infrastructure", "esg_commitment", | |
| 40 | + "climate_risk", "cybersecurity", "automation_labor", "macro_risk"] | |
| 41 | + | |
| 42 | + | |
| 43 | +def weight_case(profile: str, col: str = "section_name") -> str: | |
| 44 | + w = SECTION_WEIGHTS[profile] | |
| 45 | + cases = " ".join(f"WHEN '{k}' THEN {float(v)}" for k, v in w.items()) | |
| 46 | + return f"CASE {col} {cases} ELSE 1.0 END" | |
| 47 | + | |
| 48 | + | |
| 49 | +# --------------------------------------------------------------------------- | |
| 50 | +# AI keyword lexicon for the regex baseline (reviewer-assembled; see README in results) | |
| 51 | +# CORE = unambiguous AI terms shared by the published 10-K / job-posting AI dictionaries | |
| 52 | +# (Babina, Fedyk, He & Hodson 2024 JFE core terms; Eisfeldt, Schubert & Zhang 2023 generative-AI terms). | |
| 53 | +# EXT = CORE + the broader "digital / analytics / automation" terms that the original anchor | |
| 54 | +# paragraphs also contain (the overlap the referees worry about). | |
| 55 | +# Patterns are applied to lower-cased text (RE2 syntax, DuckDB regexp_extract_all); the bare | |
| 56 | +# acronym is matched case-sensitively on the raw text (\bAI\b, \bA\.I\.) to avoid French/other noise. | |
| 57 | +CORE_TERMS = [ | |
| 58 | + r"artificial intelligence", r"machine[ -]learning", r"deep[ -]learning", r"neural network", | |
| 59 | + r"natural[ -]language processing", r"large[ -]language model", r"generative ai\b", r"\bgen ?ai\b", | |
| 60 | + r"chatgpt", r"\bopenai\b", r"computer vision", r"reinforcement learning", r"\bllms?\b", | |
| 61 | + r"\bai[ -](?:driven|powered|enabled|based|related|assisted)", | |
| 62 | +] | |
| 63 | +EXT_EXTRA_TERMS = [ | |
| 64 | + r"\bautomation\b", r"robotic process automation", r"predictive analytics", r"\bbig data\b", | |
| 65 | + r"data analytics", r"\balgorithms?\b", r"\bchatbots?\b", r"\bdata science\b", r"\bautonomous\b", | |
| 66 | + r"intelligent automation", r"cloud computing", r"\bdata[ -]driven\b", | |
| 67 | +] | |
| 68 | +CORE_RE = "(" + "|".join(CORE_TERMS) + ")" | |
| 69 | +EXT_RE = "(" + "|".join(CORE_TERMS + EXT_EXTRA_TERMS) + ")" | |
| 70 | +ACRONYM_RE = r"\bA\.?I\.?\b" # case-sensitive on raw text: AI, A.I. | |
| 71 | + | |
| 72 | + | |
| 73 | +def connect(threads: int = THREADS, mem: str = MEM) -> duckdb.DuckDBPyConnection: | |
| 74 | + os.makedirs(TMP, exist_ok=True) | |
| 75 | + c = duckdb.connect() | |
| 76 | + c.execute(f"SET threads={threads}") | |
| 77 | + c.execute(f"SET memory_limit='{mem}'") | |
| 78 | + c.execute(f"SET temp_directory='{TMP}'") | |
| 79 | + c.execute("SET preserve_insertion_order=false") | |
| 80 | + return c | |
| 81 | + | |
| 82 | + | |
| 83 | +def views(c: duckdb.DuckDBPyConnection) -> None: | |
| 84 | + c.execute(f"CREATE OR REPLACE VIEW filings AS SELECT * FROM read_parquet('{DATA}/filings/*.parquet')") | |
| 85 | + c.execute(f"CREATE OR REPLACE VIEW companies AS SELECT * FROM read_parquet('{DATA}/companies/*.parquet')") | |
| 86 | + c.execute(f"CREATE OR REPLACE VIEW chunks AS SELECT * FROM read_parquet('{DATA}/filing_chunks/*.parquet')") | |
| 87 | + c.execute(f"CREATE OR REPLACE VIEW sections AS SELECT * FROM read_parquet('{DATA}/filing_sections/*.parquet')") | |
| 88 | + c.execute(f"CREATE OR REPLACE VIEW chunk_scores AS SELECT * FROM read_parquet('{DATA}/chunk_scores/*/*.parquet', hive_partitioning=true)") | |
| 89 | + c.execute(f"CREATE OR REPLACE VIEW filing_scores AS SELECT * FROM read_parquet('{DATA}/filing_scores/*/*.parquet', hive_partitioning=true)") | |
| 90 | + c.execute(f"CREATE OR REPLACE VIEW annual_final AS SELECT * FROM read_parquet('{DATA}/annual_final/*.parquet')") | |
| 91 | + c.execute(f"CREATE OR REPLACE VIEW anchor_embeddings AS SELECT * FROM read_parquet('{DATA}/anchor_embeddings/*.parquet')") | |
| 92 | + | |
| 93 | + | |
| 94 | +def dump_json(obj, path: str) -> None: | |
| 95 | + os.makedirs(os.path.dirname(path), exist_ok=True) | |
| 96 | + with open(path, "w") as f: | |
| 97 | + json.dump(obj, f, indent=1, default=_default) | |
| 98 | + | |
| 99 | + | |
| 100 | +def _default(o): | |
| 101 | + import numpy as np | |
| 102 | + if isinstance(o, (np.integer,)): | |
| 103 | + return int(o) | |
| 104 | + if isinstance(o, (np.floating,)): | |
| 105 | + return float(o) | |
| 106 | + if isinstance(o, (np.ndarray,)): | |
| 107 | + return o.tolist() | |
| 108 | + return str(o) | |
| 109 | + | |
| 110 | + | |
| 111 | +class Timer: | |
| 112 | + def __init__(self, label: str): | |
| 113 | + self.label = label | |
| 114 | + def __enter__(self): | |
| 115 | + self.t0 = time.time(); print(f"[{time.strftime('%H:%M:%S')}] {self.label} ...", flush=True); return self | |
| 116 | + def __exit__(self, *a): | |
| 117 | + print(f"[{time.strftime('%H:%M:%S')}] {self.label} done in {time.time()-self.t0:,.1f}s", flush=True) | |
added
code/n1/run_n1.sh
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
results/n1/alt_anchor_results.json
+2010 −0
Large diff (2,011 lines) is not loaded by default to keep the page fast.Load diff
added
results/n1/alt_anchor_sorts.json
+2215 −0
Large diff (2,216 lines) is not loaded by default to keep the page fast.Load diff
added
results/n1/alt_anchor_sorts_rows.csv
+73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added
results/n1/alt_anchor_texts.json
+114 −0
Large diff (115 lines) is not loaded by default to keep the page fast.Load diff
added
results/n1/alt_corr_by_year.csv
+16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added
results/n1/alt_firm_year.csv
+14004 −0
Large diff (14,004 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.