SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: Part III data & protocol (Agent L)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent 75471a7 Browse files

2 changed files +271 −0

2 changed files
Unified Split
modified manuscript/sections/part02_evolution.tex +28 −0
@@ -70,3 +70,31 @@ A further difficulty is specific to computer science and concerns what the disci
70 70
71 71 The post-2022 period is not the first time computer science has been reorganised around a new technology. Three earlier transformations are directly relevant because they established the publication practices, the infrastructure, and the conceptual vocabulary within which the present one unfolds: the internet and web era, which created the systems and the industrial research laboratories; the deep-learning wave of 2012--2017, which created the benchmark culture and the compute economy; and the transformer and pretraining era of 2017--2022, which created the models themselves. For each, the account below is deliberately narrow. It does not attempt a history of computing; it asks what changed in how research was done and published, and how the change propagated from the subfield in which it began to the others. The answers suggest a way of reading the post-2022 data that the empirical parts of this paper then test.
72 72
73 +\subsection{The internet and web era}
74 +\label{sec:p2-web}
75 +
76 +The World-Wide Web was presented to the computing community in 1994 as an information system built on three modest conventions, a naming scheme, a transfer protocol and a markup language, that happened to compose \citep{bernerslee1994www}. Within five years the problem of finding things on the web had produced two results that still structure information retrieval and network science: the use of the hyperlink graph as a source of authority, through PageRank in the Google prototype \citep{brin1998anatomy} and through the hubs-and-authorities formulation of \citet{kleinberg1999hubs}. Within another five, the engineering problem of running such systems at scale had produced the Google File System and MapReduce \citep{ghemawat2003gfs,dean2008mapreduce}, and within a decade the commoditisation of that infrastructure as ``cloud computing'' had been analysed as a shift in the economics of computing rather than in its technology \citep{armbrust2010cloud}. Apache Spark, which generalised MapReduce into a unified engine for iterative and interactive workloads, marks the maturation of the era \citep{zaharia2016spark}.
77 +
78 +Three features of this transformation matter for what followed. First, the decisive results came from industrial laboratories or from academic groups that immediately founded companies, and they were published in the discipline's selective conferences (SOSP, OSDI, WWW, SIGCOMM) rather than in journals; the practice by which an industrial systems paper describes a deployed artefact, with measurements but without a released implementation, dates from this period and persists in the large-model reports discussed below. Second, the web made open-source software the ordinary form of a systems research artefact: Hadoop, the open re-implementation of MapReduce, propagated the method more effectively than the paper did, and the later machine-learning frameworks follow the same pattern. Third, the era created the data. Web-scale text, images, logs and graphs were the precondition for statistical methods in language, vision and recommendation, and the subfields that had access to such data (information retrieval, speech, machine translation) were the first to adopt statistical learning, a decade before the deep-learning wave made it general.
79 +
80 +The propagation pattern was \emph{infrastructural}. The web changed the objects of study in networks, databases, distributed systems and security, and it created new subfields (web search, social computing, cloud computing), but it did not change the methods of theory, programming languages, graphics or human-computer interaction except by supplying them with new problems. The discipline's map was extended rather than redrawn: the ACM classification of 2012 could still accommodate every web-era subfield under its existing top-level nodes \citep{rous2012ccs}.
81 +
82 +\subsection{The deep-learning wave, 2012--2017}
83 +\label{sec:p2-dl}
84 +
85 +The second transformation is usually dated to 2012, when a convolutional network trained on two GPUs reduced the top-5 error on the ImageNet Large Scale Visual Recognition Challenge by a margin that no incremental improvement in hand-designed features could match \citep{krizhevsky2017imagenet,deng2009imagenet,russakovsky2015imagenet}, and when deep networks replaced Gaussian mixtures in the acoustic models of the major speech laboratories at roughly the same time \citep{hinton2012speech}. The methods were not new: the long short-term memory cell dates from 1997 \citep{hochreiter1997lstm}, convolutional networks from the 1980s, and the retrospective of \citet{lecun2015deep} presents the decade's results as the delayed payoff of a thirty-year programme. What was new was the conjunction of labelled data at the scale of ImageNet, general-purpose graphics processors, and a benchmark with a public leaderboard.
86 +
87 +The subsequent five years produced the architectural vocabulary that the transformer era would inherit. Distributed word representations trained by simple predictive objectives \citep{mikolov2013word2vec} made it natural to treat language as a sequence of dense vectors; the variational autoencoder \citep{kingma2014vae} and the generative adversarial network \citep{goodfellow2014gan} established deep generative modelling as a research programme; sequence-to-sequence learning with recurrent encoders and decoders \citep{sutskever2014seq2seq} and, crucially, the attention mechanism that let a decoder consult every encoder state \citep{bahdanau2015attention} reframed machine translation and then every conditional generation task; residual connections \citep{he2016resnet} removed the practical ceiling on depth. In reinforcement learning, deep Q-networks learned Atari games from pixels \citep{mnih2015dqn}, and the AlphaGo lineage combined deep networks with tree search to defeat professional Go players, first with human games as training data and then without them \citep{silver2016alphago,silver2017alphagozero}, generalising to chess and shogi with a single algorithm \citep{silver2018alphazero} and, in a different formulation, to real-time strategy games \citep{vinyals2019alphastar}.
88 +
89 +The changes in publication practice were as consequential as the methods. Four deserve to be stated because the post-2022 data will be read against them.
90 +
91 +The first is the \emph{benchmark as the unit of progress}. ImageNet established the pattern in which a public dataset, a fixed metric and a leaderboard define a subfield's agenda; the pattern was reproduced in language understanding, question answering and reinforcement learning, and it made progress legible to outsiders at the cost of concentrating effort on whatever the benchmark measured. The critiques of that culture, which argue that benchmarks are chosen for convenience rather than validity and that leaderboard competition rewards narrow optimisation \citep{dehghani2021lottery,raji2021benchmark}, were written in the transformer era but describe a practice born in this one.
92 +
93 +The second is the \emph{preprint-first, conference-second} publication cycle. Machine-learning results began to appear on arXiv months before, or instead of, their conference versions; the practice spread from a minority to a norm within the decade, and it is the reason the arXiv \texttt{cs} harvest is indispensable to this corpus. The accompanying costs were catalogued at the end of the period: speculation presented as explanation, mathematics as decoration, suggestive language and the conflation of explanation with description \citep{lipton2019troubling}. Concerns about reproducibility were raised at the same time, both in audits of what published papers actually report \citep{gundersen2018reproducibility} and in the organised reproducibility programmes that the major conferences introduced in response \citep{pineau2021reproducibility}.
94 +
95 +The third is the \emph{open-source framework as a shared instrument}. TensorFlow and PyTorch \citep{abadi2016tensorflow,paszke2019pytorch}, released by Google and Facebook respectively, lowered the cost of entry to deep learning to the point where a method published on arXiv could be reproduced and extended within days; they also standardised the computational substrate on NVIDIA GPUs and, later, on tensor processing units \citep{jouppi2017tpu}, which made compute a measurable input to research. The retrospective analysis of \citet{sevilla2022compute} identifies this period as the start of a ``deep learning era'' in which the training compute of notable systems grew with a doubling time of about six months, far faster than before 2010; \citet{thompson2020limits} argued that such growth is economically and environmentally unsustainable, a theme to which the sustainable-computing category of this study returns.
96 +
97 +The fourth is the \emph{entry of industry as a producer rather than a consumer of research}. The laboratories of Google, DeepMind, Facebook, Microsoft and Baidu published the decisive results of the period in the academic venues and on arXiv, and their access to data and compute began to separate them from universities. \citet{ahmed2020dedemocratization} documented the growing presence of large technology firms and elite universities at the major AI conferences and the relative decline of other institutions; the later and broader analysis of \citet{ahmed2023industry} found that industry produced an increasing share of the largest models, the most-cited papers and the leading benchmarks. Whether that concentration intensified after 2022 is hypothesis H6 of the protocol.
98 +
99 +The propagation pattern of this wave was \emph{methodological} and sequential. Deep learning entered computer vision and speech first, natural language processing second (roughly 2014--2016, with the sequence-to-sequence results), reinforcement learning and robotics third, and then, more slowly and with more resistance, the subfields whose objects are not perceptual: program analysis \citep{allamanis2018survey}, systems (learned index structures and learned scheduling), databases, and scientific computing \citep{raissi2019pinn}. In each case the receiving subfield first treated learning as a tool for a pre-existing problem, and only later, if at all, reorganised its problems around it. That two-stage pattern, tool adoption followed by problem redefinition, is the template against which Part~IX reads the post-2022 movement of methods across the forty categories.
100 +
added reports/audit/CODE_AUDIT_2026-11-03.md +243 −0
@@ -0,0 +1,243 @@
1 +# WP44 — Code audit (Agent N, Reproducibility Auditor), 2026-11-03
2 +
3 +Scope: read-only audit of the 14 pipeline scripts listed below against `CLAUDE.md` §1/§6, `RESEARCH_PLAN.md` and
4 +`protocol/PROTOCOL.md` (v1.0 + amendments A-000…A-004). No file outside `reports/audit/` was modified; the compute node
5 +was not touched. Formula and logic checks were replicated on synthetic data on the laptop (DuckDB 1.5.2, numpy 2.4,
6 +pandas 3.0, statsmodels 0.14, networkx 3.x); scratch scripts live in `/tmp/wp44_audit/` and are not part of the repo.
7 +Machine-readable list: `reports/audit/code_issues.csv` (67 issues: 2 blocking, 20 major, 45 minor).
8 +
9 +Severity: **blocking** = produces wrong or missing results silently / crashes a pre-registered analysis;
10 +**major** = biases an estimand, contradicts the protocol, or loses data in a way that must be fixed before Phase 4
11 +results enter the manuscript; **minor** = hygiene, documentation, fragility, provenance.
12 +
13 +## 0. Verification checks performed (synthetic data)
14 +
15 +| Check | Result |
16 +|---|---|
17 +| Eligibility SQL of `define_corpus.py` with NULL `type` / NULL `source_name` | `eligible` becomes NULL → rows vanish from `WHERE eligible` **and** from every exclusion bucket; PRISMA sequential sum 4 of 6 |
18 +| `JOIN … USING (year,type,sf)` with NULL `sf` (`03_citation_impact.py`) | 1 of 3 rows survive |
19 +| Gini SQL (`03_citation_impact.py` L35) vs reference `2Σ i·x_(i)/(nΣx) − (n+1)/n`, with ties | identical (0.575 = 0.575); NaN when Σ = 0 |
20 +| Directed weighted modularity (`02_category_flows.py`) vs `networkx.community.modularity` | identical to 1e-6 |
21 +| Bai–Perron DP (`structural_change.py`) vs brute force, n = 30, m = 2, h = 5 | identical SSR and break indices; BIC and LWZ select m = 2 |
22 +| ITS design matrix | `post`, `t_post` (0 at event, +1/month), 11 month dummies — correct |
23 +| Placebo candidate months (default window 2018-01 → 2026-06) | 66 candidates, **30 after the true event**; placebo p = 0.015 vs 0.000 with baseline-only placebos |
24 +| `features_at` (emerging detection) invariance to columns > T | invariant (look-ahead-safe); trailing z-window = 25 months, not 24 |
25 +| Cutoff lists (emerging detection) | train 2020-01…2021-10, test 2023-01…2024-10; h = 24 → last test label month 2026-10, last train label month 2023-10 |
26 +| `series_categories` reindex on the boundary month (`run_structural_change.py`) | `ValueError: cannot reindex on an axis with duplicate labels` |
27 +| BOCPD update equations (NIG / Student-t) | correct by inspection; unit tests exist |
28 +| Embedding normalisation assumed by `semantic_atlas.py` | `embed_corpus.py` uses `normalize_embeddings=True` → cosine via dot product valid |
29 +
30 +Two documented facts in the repository corroborate findings: `data/metadata/linkage_report.json` shows
31 +`dblp_with_arxiv: 0` (dead linkage path), and no script ingests `api_increment/pages/` (window end mismatch).
32 +
33 +## 1. Per-file findings
34 +
35 +### ingestion/openalex_snapshot.py — no blocking/major issue
36 +
37 +| Line | Sev. | Finding | Fix |
38 +|---|---|---|---|
39 +| 148–157 | minor | A partial download larger than `expected_len` is resumed with `curl -C -` forever; the part stays in `error` on every run | delete `raw_path` on size mismatch before retrying |
40 +| 160–164 | minor | SHA-256 recorded but never compared (manifest has no hash) | capture the S3 ETag and compare; label the hash as provenance |
41 +| 188–189 | minor | works with NULL `publication_year` excluded from the CS subset without being counted | ledger column `n_null_year` (PRISMA S0) |
42 +| 236–239 | minor | `runs` table lacks git commit / environment hash | add both |
43 +
44 +Design is sound: resumable SQLite ledger, atomic `.part` renames, bad-line counter, slim table for all 465 M works.
45 +
46 +### ingestion/build_corpus.py — no blocking/major issue
47 +
48 +| Line | Sev. | Finding | Fix |
49 +|---|---|---|---|
50 +| 69–76 | minor | `ids["arxiv"]` is not in the OpenAlex schema (dead branch); DOI branch keeps `v\d+` while the URL branch strips it | remove dead branch, strip version everywhere |
51 +| 144 | minor | `is_primary` by URL equality → 0 or several primaries when URLs are NULL | compare source id + URLs |
52 +| 173–176 | minor | no Parquet metadata (snapshot release, commit) | write key/value metadata |
53 +
54 +### ingestion/link_sources.py
55 +
56 +| Line | Sev. | Finding | Fix |
57 +|---|---|---|---|
58 +| 36, 50–53 | **major** | DBLP→arXiv path yields **0** links (`dblp_with_arxiv: 0`): DBLP CoRR `ee` is now `https://doi.org/10.48550/arXiv.<id>`; the DBLP–arXiv version relation is unavailable to the merge step | derive `arxiv_id` from the DOI `10.48550/arxiv.<id>` too |
59 +| 54–60, 67–72, 81–83 | **major** | title+year linkage without the first-author surname required by plan S1 (180 k DBLP, 2.1 k arXiv, 14.9 k ACL links) | add first-author check; report precision by method in the H-1 sample |
60 +| 52–83 | minor | `NOT IN (SELECT …)` fragile to NULL keys | `NOT EXISTS` |
61 +| 85–118 | minor | report has no `pass` field; committed file is the pass-1 state | add pass + input hashes |
62 +
63 +### systematic_review/define_corpus.py
64 +
65 +| Line | Sev. | Finding | Fix |
66 +|---|---|---|---|
67 +| 52, 55, 65–66 | **BLOCKING** | NULL `type` or NULL `source_name` (non-DBLP, non-arXiv) → `e2_type`/`e7_generic_deposit` NULL → **`eligible` NULL**: silently dropped by every `WHERE eligible` and by every PRISMA bucket; sequential counts do not sum (verified) | `coalesce(type,'')`, `coalesce(source_name,'')`; assert `count(*) WHERE eligible IS NULL = 0`; PRISMA identity test |
68 +| 38–39 | major | `pub` CTE: `NOT is_arxiv_doi` is NULL when DOI is NULL → published records without DOI (common for ML conference venues in OpenAlex) can never absorb their preprint → double counting of the most AI-central population | `coalesce(is_arxiv_doi, FALSE)` |
69 +| 58, 63–66 | major | preprint flagged `e5_duplicate` even when the matched published record is itself excluded (E1/E2/E7/NULL) → the work disappears entirely | match only against published records passing e1–e4/e7 |
70 +| 36–43 | major | merge by exact normalised title only; arXiv-id relation not used; `version_relationship` (plan S2) not produced; retitled VoRs stay duplicated | add arXiv-id pairs + fuzzy fallback; produce `version_relationship`; report duplicate rate |
71 +| 50, 56 | major | `first_date = least(...)` lets a year-precision `YYYY-01-01` beat a precise later arXiv date; `date_precision` is defined on `publication_date`, not on the date used for `first_month` | treat year-precision dates as NULL in `least` when a precise date exists; define precision on `first_date` |
72 +| 16, 51 | major | `WINDOW_END = 2026-11-02` but data = snapshot 2026-10-14; the API increment is never ingested | ingest the increment or amend the protocol with the effective cut-off |
73 +| 51, 84 | minor | NULL `publication_date` counted as E1 | separate reason code |
74 +| 33 | minor | `any_value()` nondeterministic for multi-arXiv works | `arg_min(..., created)` |
75 +| 65–69 | minor | eligible works with `period='pre'` in yearly but not monthly tables | decide and document |
76 +| 67–68 | minor | boundary 2022-11-30 (day) vs ITS event 2022-12-01 (month) | state it |
77 +
78 +### bibliometrics/01_volume_series.py
79 +
80 +| Line | Sev. | Finding | Fix |
81 +|---|---|---|---|
82 +| 22–27 | major | indexing-lag table produced but no last-complete-month / completeness correction derived (plan §6); driver hard-codes `--last-complete 2026-06` | estimate completeness from the lag profile; emit `last_complete_month` JSON used by econometrics and emerging detection |
83 +| 22–27 | minor | lag computed on year-precision dates too | restrict to `date_precision='month'` |
84 +| 13 | minor | `first_month` vs `date_precision` inconsistency (upstream) | — |
85 +
86 +### bibliometrics/02_collaboration.py — no formula error found
87 +
88 +| Line | Sev. | Finding | Fix |
89 +|---|---|---|---|
90 +| 19–20 | minor | fractional counting = 1/#distinct countries per work, not author-fractional | state the rule |
91 +| 21, 24–25 | minor | `share_intl`, `share_company` conditional on affiliation coverage, which drifts by year | add coverage columns |
92 +| 14 | minor | retracted and `period='pre'` included | document |
93 +
94 +### bibliometrics/03_citation_impact.py
95 +
96 +| Line | Sev. | Finding | Fix |
97 +|---|---|---|---|
98 +| 26–29 | major | `JOIN USING (year,type,sf)` drops all works with NULL subfield (verified) | `IS NOT DISTINCT FROM` / coalesce |
99 +| 4–5, 20–22, 35 | major | `c12` = calendar years y..y+1 (13–24 months), `c36` = y..y+2 (25–36 months); H5 prescribes a 24-month Gini, code uses c36 | rename to calendar-year windows, use y..y+1 for H5, state the publication-month dependence |
100 +| 19, 32–37 | major | truncated windows (c36: 2024–2026; c12: 2025–2026) reported without flag | `window_complete` column; NULL when incomplete |
101 +| 19 | major | retracted works included (protocol E6 excludes them from impact analyses) | `AND NOT e6_retracted` |
102 +| 20–23 | major | citations of the merged preprint version not added to the published node (plan S2 one node per work) | aggregate `counts_by_year` over {id, preprint_work_id} |
103 +| 28 | minor | no minimum cell size for NCS normalisation | threshold + fallback |
104 +| 35 | — | Gini formula **correct** (verified) | — |
105 +
106 +### knowledge_graph/01_citation_graph.py
107 +
108 +| Line | Sev. | Finding | Fix |
109 +|---|---|---|---|
110 +| 19–26 | major | citations to the merged preprint become external edges → merged node loses inbound citations; propagates to 02 and emerging detection | remap `ref_id` via `duplicate_of` |
111 +| 23–26 | minor | dangling references (ids not in slim, e.g. merged ids) dropped silently; `merged_ids` not used | remap + report `n_dangling` |
112 +| 31–32 | minor | negative citation ages retained | flag/clip |
113 +
114 +### knowledge_graph/02_category_flows.py
115 +
116 +| Line | Sev. | Finding | Fix |
117 +|---|---|---|---|
118 +| 28–30 | major | inner join on `work_primary` drops unclassified works; shares conditional on classification | `unclassified` pseudo-category + share per year |
119 +| 13–17 | — | modularity formula **correct** (directed, weighted; verified) | — |
120 +| 52 | minor | no bootstrap CI (P6) | edge-resampling CIs |
121 +| 40–41 | minor | `n_month_precision` vs `first_month` inconsistency (upstream) | — |
122 +
123 +### econometrics/structural_change.py
124 +
125 +| Line | Sev. | Finding | Fix |
126 +|---|---|---|---|
127 +| 43–49 | major | placebo months span the whole sample (30/66 post-event); protocol: baseline only; pre-event placebos estimated on a sample containing the true break | restrict to baseline; fit on pre-event data (or control for the true event) |
128 +| 61–101 | major | Bai–Perron: DP + BIC/LWZ correct, but no supF/UDmax/sequential tests, no critical values, no seasonal regressors → H3 "5 % level" untestable | add BP tests/critical values or bootstrap p-values; add month dummies |
129 +| 32–36 | minor | fixed NW `maxlags=6` (auto rule → 4); design verified | document; lag sensitivity |
130 +| 64, 92–100 | minor | break index convention (first of new regime); `h = max(8, 0.15n)` | document |
131 +| 104–109 | minor | PELT penalty uses full-sample variance (inflated by the break) | robust scale |
132 +| 112–135 | minor | BOCPD data-dependent prior (first 6 obs), fixed hazard; equations correct | document; hazard sensitivity |
133 +
134 +### econometrics/run_structural_change.py
135 +
136 +| Line | Sev. | Finding | Fix |
137 +|---|---|---|---|
138 +| 23–28 | **BLOCKING** | `series_categories` sets `first_month` as index without aggregating; the boundary month 2022-11 has `baseline` and `post` rows → duplicate index → `reindex` raises `ValueError` → **all category-level runs (H3) fail** | `g.groupby("first_month")[col].sum()` |
139 +| 36, 44 | minor | `log(n+1)` and Bai–Perron/PELT on unadjusted log series for sparse category series | Poisson/NB ITS robustness; deseasonalise |
140 +| 53 | minor | `--first 2018-02` unexplained | document or use 2018-01 |
141 +| 63 | minor | `sens_tmp` expression always true; sensitivity runs overwrite each other's `placebo_total.csv`/`fit_total.csv` | name outputs by cut-off |
142 +| 56–67 | minor | no provenance stamp in `its_results.json` | add hashes/commit |
143 +
144 +### taxonomy/classify.py — deterministic; no formula error (tests pass by construction)
145 +
146 +| Line | Sev. | Finding | Fix |
147 +|---|---|---|---|
148 +| 108 | minor | `row_number() OVER ()` without `ORDER BY` → chunk composition not reproducible | `ORDER BY id` |
149 +| 120–129, 142–143 | minor | shard mode writes no report; outputs not version-stamped (docstring claims it) | stamp ontology version + sha256 |
150 +| 146–157 | minor | stale `input_named` symlinks across `--prepare` re-runs | clear in `prepare()` |
151 +| 28, 55–57 | minor | regexes compiled without flags; 273/277 patterns carry `(?i)` | compile with `re.I` or lint |
152 +
153 +### experiments/emerging_detection.py
154 +
155 +| Line | Sev. | Finding | Fix |
156 +|---|---|---|---|
157 +| 27–28, 22–23 | major | `KNOWN_SET` empty; H8 lead-time evaluation (`known_set_lead_times.csv`) not implemented | fill `known_set.csv`; implement lead time |
158 +| 78, 83–86, 66–74 | major | panel to 2026-10 (incomplete) while econometrics stops at 2026-06: h = 24 test labels truncated; last train label window (2023-10) overlaps test features (from 2023-01) | `T+h ≤ last_complete`; `last_train+h ≤ first_test` |
159 +| 31–38 | major (declare) | units = OpenAlex topics (2024 taxonomy that knows which topics emerged); cutoff-specific semantic clusters of the plan not implemented. Features are look-ahead-safe (verified) | implement cutoff-specific clusters; declare the unit-definition look-ahead |
160 +| 40–41 | minor | universe selected with full-period counts | per-cutoff selection |
161 +| 56–57 | minor | z-window 25 months, docstring 24 | align |
162 +| 85 | minor | train cutoffs 2020-01…2021-10, docstring ≤ 2021-12 | align |
163 +| 98, 108–113 | minor | unscaled logistic features; pooled AUC; no CIs | scaler pipeline; per-cutoff bootstrap |
164 +
165 +Label definition (top decile of `log1p(post) − log1p(pre)` among topics with ≥ 20 works in the last 12 months, and
166 +post ≥ 2·pre) matches the docstring and uses only (T−h, T] and (T, T+h] (verified).
167 +
168 +### embeddings/semantic_atlas.py — no blocking/major issue
169 +
170 +| Line | Sev. | Finding | Fix |
171 +|---|---|---|---|
172 +| 4, 66 | minor | "stratified sample" is a simple random sample | stratify or fix docstring |
173 +| 81–87 | minor | split-half ARI inside one UMAP (not bootstrap; UMAP not refitted) | refit per half |
174 +| 49–61 | minor | novelty centroids include year-precision months | restrict to month precision |
175 +| 30 | minor | duplicate ids across shards silently overwritten | assert uniqueness |
176 +| 41 | minor | baseline/post by calendar year instead of `period` | use `period` |
177 +| 67–69, 88 | minor | no version stamp (umap/numba/pynndescent) | record versions |
178 +
179 +## 2. Cross-cutting observations
180 +
181 +1. **Preprint–publication merge is only half-implemented.** `define_corpus.py` marks the preprint as a duplicate but no
182 + downstream script (citation graph, citation windows, category flows, emerging detection) re-attributes the preprint's
183 + references/citations/`counts_by_year` to the surviving node. Under plan S2 ("one node per work") this undercounts
184 + early citations and inter-category flows exactly where CS preprints dominate (AI/ML). This is the single most
185 + consequential conceptual gap after the two blocking bugs.
186 +2. **Date semantics are split across three columns** (`publication_date`, `first_date/first_month`, `date_precision`):
187 + monthly series use `first_month` but the precision flag is defined on `publication_date`; yearly tables use
188 + `publication_year`. The same work can therefore be counted in different years in yearly vs monthly outputs.
189 +3. **Right truncation is handled by a fixed cut-off (2026-06) in econometrics only**; citation windows, emerging-detection
190 + labels and the semantic atlas use the data up to the snapshot without a completeness flag. The plan's
191 + "indexing-delay profile correction" is not implemented.
192 +4. **Effective data end is 2026-10-14**, not 2026-11-02: the API increment is harvested but never ingested.
193 +5. **Retracted works** are excluded from impact analyses by the protocol but not by the code.
194 +6. **Provenance**: no script stamps the git commit, input hashes or package versions into its outputs; `tests/` covers the
195 + pure-python toolkit and the classifier but no DuckDB fixture tests exist for the SQL stages (which is where both
196 + blocking bugs sit). Seeds are set where randomness exists (GBM 44, UMAP/HDBSCAN/sample 44); `row_number() OVER ()`
197 + and `any_value()` are the only nondeterministic constructs.
198 +7. **Inference**: protocol P6 promises bootstrap/HAC uncertainty on all estimates; ITS has HAC, but Bai–Perron,
199 + modularity, cross-area shares, Gini and emerging-detection metrics have no intervals.
200 +
201 +## 3. Prioritised fix list
202 +
203 +**P0 — before any Phase 4 number is produced**
204 +1. `define_corpus.py` L52/L55: `coalesce()` on `type` and `source_name`; add `eligible IS NULL` assertion and a PRISMA
205 + identity test (identified = Σ sequential exclusions + after_exclusions). Re-run `define_corpus.py` and every
206 + downstream script; compare `prisma_counts.json` before/after.
207 +2. `run_structural_change.py` L28: aggregate by `first_month` (`groupby().sum()`), then re-run.
208 +
209 +**P1 — before results enter the manuscript**
210 +3. Merge semantics: `coalesce(is_arxiv_doi, FALSE)` in the `pub` CTE; mark `e5_duplicate` only against eligible
211 + published records; add arXiv-id-based pairs and `version_relationship`; remap `ref_id` via `duplicate_of` in
212 + `01_citation_graph.py`; sum `counts_by_year` over both versions in `03_citation_impact.py`.
213 +4. `03_citation_impact.py`: NULL-safe normalisation join; `window_complete` flag; exclude `e6_retracted`; rename
214 + windows to calendar-year semantics and compute H5 on the y..y+1 window.
215 +5. `structural_change.py`: baseline-only placebos (fit on pre-event data); add BP supF/sequential tests or bootstrap
216 + p-values and month dummies so that H3 can be tested at the stated level.
217 +6. Right truncation: derive `last_complete_month` from the lag profile in `01_volume_series.py` and consume it in
218 + `run_structural_change.py` and `emerging_detection.py` (labels require `T+h ≤ last_complete`; enforce
219 + `last_train+h ≤ first_test`).
220 +7. Window end: ingest the API increment or amend the protocol to the effective cut-off 2026-10-14.
221 +8. `first_date`/`date_precision` coherence (year-precision dates must not win in `least()`; precision defined on the
222 + chosen date); then re-derive `period`, `first_month`.
223 +9. `link_sources.py`: arXiv id from the DBLP DOI; first-author check for title+year links; report precision by method.
224 +10. `emerging_detection.py`: fill `KNOWN_SET` and implement lead times (H8); implement (or formally drop, by
225 + amendment) cutoff-specific semantic clusters and declare the topic-unit look-ahead.
226 +11. `02_category_flows.py`: `unclassified` pseudo-category; bootstrap CIs.
227 +
228 +**P2 — hygiene / reproducibility**
229 +12. Provenance stamps (git commit, input sha256, package versions) in every output; `ORDER BY id` in `classify.py`
230 + chunking; version stamp in classification outputs; DuckDB fixture tests for `define_corpus.py` and
231 + `03_citation_impact.py`; docstring/code alignments (z-window 25 vs 24, cutoffs, "stratified", "bootstrap");
232 + `sens_tmp` naming; `--first 2018-02`; NW lag sensitivity; PELT robust penalty.
233 +
234 +## 4. Summary
235 +
236 +Two blocking defects: (i) NULL propagation in the eligibility flags of `define_corpus.py` silently removes works with
237 +NULL `type` or NULL `source_name` from the corpus and from the PRISMA counts; (ii) `run_structural_change.py` crashes
238 +on every category series because the boundary month appears twice in `category_counts_monthly.csv`. Twenty major
239 +issues cluster around four themes: the preprint–publication merge is not propagated to citations/flows; citation windows
240 +mislabel calendar-year windows, ignore right truncation and include retracted works; the placebo design includes
241 +post-event months and Bai–Perron has no test at the 5 % level; the emerging-detection labels for h = 24 use incomplete
242 +months and the preregistered known-emergence evaluation is absent. Verified correct: Gini formula, directed modularity,
243 +Bai–Perron dynamic programme and BIC/LWZ penalties, ITS design matrix, BOCPD updates, feature look-ahead safety.
244
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere