# WP44 — Code audit (Agent N, Reproducibility Auditor), 2026-11-03

Scope: read-only audit of the 14 pipeline scripts listed below against `CLAUDE.md` §1/§6, `RESEARCH_PLAN.md` and
`protocol/PROTOCOL.md` (v1.0 + amendments A-000…A-004). No file outside `reports/audit/` was modified; the compute node
was not touched. Formula and logic checks were replicated on synthetic data on the laptop (DuckDB 1.5.2, numpy 2.4,
pandas 3.0, statsmodels 0.14, networkx 3.x); scratch scripts live in `/tmp/wp44_audit/` and are not part of the repo.
Machine-readable list: `reports/audit/code_issues.csv` (67 issues: 2 blocking, 20 major, 45 minor).

Severity: **blocking** = produces wrong or missing results silently / crashes a pre-registered analysis;
**major** = biases an estimand, contradicts the protocol, or loses data in a way that must be fixed before Phase 4
results enter the manuscript; **minor** = hygiene, documentation, fragility, provenance.

## 0. Verification checks performed (synthetic data)

| Check | Result |
|---|---|
| Eligibility SQL of `define_corpus.py` with NULL `type` / NULL `source_name` | `eligible` becomes NULL → rows vanish from `WHERE eligible` **and** from every exclusion bucket; PRISMA sequential sum 4 of 6 |
| `JOIN … USING (year,type,sf)` with NULL `sf` (`03_citation_impact.py`) | 1 of 3 rows survive |
| Gini SQL (`03_citation_impact.py` L35) vs reference `2Σ i·x_(i)/(nΣx) − (n+1)/n`, with ties | identical (0.575 = 0.575); NaN when Σ = 0 |
| Directed weighted modularity (`02_category_flows.py`) vs `networkx.community.modularity` | identical to 1e-6 |
| Bai–Perron DP (`structural_change.py`) vs brute force, n = 30, m = 2, h = 5 | identical SSR and break indices; BIC and LWZ select m = 2 |
| ITS design matrix | `post`, `t_post` (0 at event, +1/month), 11 month dummies — correct |
| Placebo candidate months (default window 2018-01 → 2026-06) | 66 candidates, **30 after the true event**; placebo p = 0.015 vs 0.000 with baseline-only placebos |
| `features_at` (emerging detection) invariance to columns > T | invariant (look-ahead-safe); trailing z-window = 25 months, not 24 |
| Cutoff lists (emerging detection) | train 2020-01…2021-10, test 2023-01…2024-10; h = 24 → last test label month 2026-10, last train label month 2023-10 |
| `series_categories` reindex on the boundary month (`run_structural_change.py`) | `ValueError: cannot reindex on an axis with duplicate labels` |
| BOCPD update equations (NIG / Student-t) | correct by inspection; unit tests exist |
| Embedding normalisation assumed by `semantic_atlas.py` | `embed_corpus.py` uses `normalize_embeddings=True` → cosine via dot product valid |

Two documented facts in the repository corroborate findings: `data/metadata/linkage_report.json` shows
`dblp_with_arxiv: 0` (dead linkage path), and no script ingests `api_increment/pages/` (window end mismatch).

## 1. Per-file findings

### ingestion/openalex_snapshot.py — no blocking/major issue

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 148–157 | minor | A partial download larger than `expected_len` is resumed with `curl -C -` forever; the part stays in `error` on every run | delete `raw_path` on size mismatch before retrying |
| 160–164 | minor | SHA-256 recorded but never compared (manifest has no hash) | capture the S3 ETag and compare; label the hash as provenance |
| 188–189 | minor | works with NULL `publication_year` excluded from the CS subset without being counted | ledger column `n_null_year` (PRISMA S0) |
| 236–239 | minor | `runs` table lacks git commit / environment hash | add both |

Design is sound: resumable SQLite ledger, atomic `.part` renames, bad-line counter, slim table for all 465 M works.

### ingestion/build_corpus.py — no blocking/major issue

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 69–76 | minor | `ids["arxiv"]` is not in the OpenAlex schema (dead branch); DOI branch keeps `v\d+` while the URL branch strips it | remove dead branch, strip version everywhere |
| 144 | minor | `is_primary` by URL equality → 0 or several primaries when URLs are NULL | compare source id + URLs |
| 173–176 | minor | no Parquet metadata (snapshot release, commit) | write key/value metadata |

### ingestion/link_sources.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 36, 50–53 | **major** | DBLP→arXiv path yields **0** links (`dblp_with_arxiv: 0`): DBLP CoRR `ee` is now `https://doi.org/10.48550/arXiv.<id>`; the DBLP–arXiv version relation is unavailable to the merge step | derive `arxiv_id` from the DOI `10.48550/arxiv.<id>` too |
| 54–60, 67–72, 81–83 | **major** | title+year linkage without the first-author surname required by plan S1 (180 k DBLP, 2.1 k arXiv, 14.9 k ACL links) | add first-author check; report precision by method in the H-1 sample |
| 52–83 | minor | `NOT IN (SELECT …)` fragile to NULL keys | `NOT EXISTS` |
| 85–118 | minor | report has no `pass` field; committed file is the pass-1 state | add pass + input hashes |

### systematic_review/define_corpus.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 52, 55, 65–66 | **BLOCKING** | NULL `type` or NULL `source_name` (non-DBLP, non-arXiv) → `e2_type`/`e7_generic_deposit` NULL → **`eligible` NULL**: silently dropped by every `WHERE eligible` and by every PRISMA bucket; sequential counts do not sum (verified) | `coalesce(type,'')`, `coalesce(source_name,'')`; assert `count(*) WHERE eligible IS NULL = 0`; PRISMA identity test |
| 38–39 | major | `pub` CTE: `NOT is_arxiv_doi` is NULL when DOI is NULL → published records without DOI (common for ML conference venues in OpenAlex) can never absorb their preprint → double counting of the most AI-central population | `coalesce(is_arxiv_doi, FALSE)` |
| 58, 63–66 | major | preprint flagged `e5_duplicate` even when the matched published record is itself excluded (E1/E2/E7/NULL) → the work disappears entirely | match only against published records passing e1–e4/e7 |
| 36–43 | major | merge by exact normalised title only; arXiv-id relation not used; `version_relationship` (plan S2) not produced; retitled VoRs stay duplicated | add arXiv-id pairs + fuzzy fallback; produce `version_relationship`; report duplicate rate |
| 50, 56 | major | `first_date = least(...)` lets a year-precision `YYYY-01-01` beat a precise later arXiv date; `date_precision` is defined on `publication_date`, not on the date used for `first_month` | treat year-precision dates as NULL in `least` when a precise date exists; define precision on `first_date` |
| 16, 51 | major | `WINDOW_END = 2026-11-02` but data = snapshot 2026-10-14; the API increment is never ingested | ingest the increment or amend the protocol with the effective cut-off |
| 51, 84 | minor | NULL `publication_date` counted as E1 | separate reason code |
| 33 | minor | `any_value()` nondeterministic for multi-arXiv works | `arg_min(..., created)` |
| 65–69 | minor | eligible works with `period='pre'` in yearly but not monthly tables | decide and document |
| 67–68 | minor | boundary 2022-11-30 (day) vs ITS event 2022-12-01 (month) | state it |

### bibliometrics/01_volume_series.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 22–27 | major | indexing-lag table produced but no last-complete-month / completeness correction derived (plan §6); driver hard-codes `--last-complete 2026-06` | estimate completeness from the lag profile; emit `last_complete_month` JSON used by econometrics and emerging detection |
| 22–27 | minor | lag computed on year-precision dates too | restrict to `date_precision='month'` |
| 13 | minor | `first_month` vs `date_precision` inconsistency (upstream) | — |

### bibliometrics/02_collaboration.py — no formula error found

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 19–20 | minor | fractional counting = 1/#distinct countries per work, not author-fractional | state the rule |
| 21, 24–25 | minor | `share_intl`, `share_company` conditional on affiliation coverage, which drifts by year | add coverage columns |
| 14 | minor | retracted and `period='pre'` included | document |

### bibliometrics/03_citation_impact.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 26–29 | major | `JOIN USING (year,type,sf)` drops all works with NULL subfield (verified) | `IS NOT DISTINCT FROM` / coalesce |
| 4–5, 20–22, 35 | major | `c12` = calendar years y..y+1 (13–24 months), `c36` = y..y+2 (25–36 months); H5 prescribes a 24-month Gini, code uses c36 | rename to calendar-year windows, use y..y+1 for H5, state the publication-month dependence |
| 19, 32–37 | major | truncated windows (c36: 2024–2026; c12: 2025–2026) reported without flag | `window_complete` column; NULL when incomplete |
| 19 | major | retracted works included (protocol E6 excludes them from impact analyses) | `AND NOT e6_retracted` |
| 20–23 | major | citations of the merged preprint version not added to the published node (plan S2 one node per work) | aggregate `counts_by_year` over {id, preprint_work_id} |
| 28 | minor | no minimum cell size for NCS normalisation | threshold + fallback |
| 35 | — | Gini formula **correct** (verified) | — |

### knowledge_graph/01_citation_graph.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 19–26 | major | citations to the merged preprint become external edges → merged node loses inbound citations; propagates to 02 and emerging detection | remap `ref_id` via `duplicate_of` |
| 23–26 | minor | dangling references (ids not in slim, e.g. merged ids) dropped silently; `merged_ids` not used | remap + report `n_dangling` |
| 31–32 | minor | negative citation ages retained | flag/clip |

### knowledge_graph/02_category_flows.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 28–30 | major | inner join on `work_primary` drops unclassified works; shares conditional on classification | `unclassified` pseudo-category + share per year |
| 13–17 | — | modularity formula **correct** (directed, weighted; verified) | — |
| 52 | minor | no bootstrap CI (P6) | edge-resampling CIs |
| 40–41 | minor | `n_month_precision` vs `first_month` inconsistency (upstream) | — |

### econometrics/structural_change.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 43–49 | major | placebo months span the whole sample (30/66 post-event); protocol: baseline only; pre-event placebos estimated on a sample containing the true break | restrict to baseline; fit on pre-event data (or control for the true event) |
| 61–101 | major | Bai–Perron: DP + BIC/LWZ correct, but no supF/UDmax/sequential tests, no critical values, no seasonal regressors → H3 "5 % level" untestable | add BP tests/critical values or bootstrap p-values; add month dummies |
| 32–36 | minor | fixed NW `maxlags=6` (auto rule → 4); design verified | document; lag sensitivity |
| 64, 92–100 | minor | break index convention (first of new regime); `h = max(8, 0.15n)` | document |
| 104–109 | minor | PELT penalty uses full-sample variance (inflated by the break) | robust scale |
| 112–135 | minor | BOCPD data-dependent prior (first 6 obs), fixed hazard; equations correct | document; hazard sensitivity |

### econometrics/run_structural_change.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 23–28 | **BLOCKING** | `series_categories` sets `first_month` as index without aggregating; the boundary month 2022-11 has `baseline` and `post` rows → duplicate index → `reindex` raises `ValueError` → **all category-level runs (H3) fail** | `g.groupby("first_month")[col].sum()` |
| 36, 44 | minor | `log(n+1)` and Bai–Perron/PELT on unadjusted log series for sparse category series | Poisson/NB ITS robustness; deseasonalise |
| 53 | minor | `--first 2018-02` unexplained | document or use 2018-01 |
| 63 | minor | `sens_tmp` expression always true; sensitivity runs overwrite each other's `placebo_total.csv`/`fit_total.csv` | name outputs by cut-off |
| 56–67 | minor | no provenance stamp in `its_results.json` | add hashes/commit |

### taxonomy/classify.py — deterministic; no formula error (tests pass by construction)

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 108 | minor | `row_number() OVER ()` without `ORDER BY` → chunk composition not reproducible | `ORDER BY id` |
| 120–129, 142–143 | minor | shard mode writes no report; outputs not version-stamped (docstring claims it) | stamp ontology version + sha256 |
| 146–157 | minor | stale `input_named` symlinks across `--prepare` re-runs | clear in `prepare()` |
| 28, 55–57 | minor | regexes compiled without flags; 273/277 patterns carry `(?i)` | compile with `re.I` or lint |

### experiments/emerging_detection.py

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 27–28, 22–23 | major | `KNOWN_SET` empty; H8 lead-time evaluation (`known_set_lead_times.csv`) not implemented | fill `known_set.csv`; implement lead time |
| 78, 83–86, 66–74 | major | panel to 2026-10 (incomplete) while econometrics stops at 2026-06: h = 24 test labels truncated; last train label window (2023-10) overlaps test features (from 2023-01) | `T+h ≤ last_complete`; `last_train+h ≤ first_test` |
| 31–38 | major (declare) | units = OpenAlex topics (2024 taxonomy that knows which topics emerged); cutoff-specific semantic clusters of the plan not implemented. Features are look-ahead-safe (verified) | implement cutoff-specific clusters; declare the unit-definition look-ahead |
| 40–41 | minor | universe selected with full-period counts | per-cutoff selection |
| 56–57 | minor | z-window 25 months, docstring 24 | align |
| 85 | minor | train cutoffs 2020-01…2021-10, docstring ≤ 2021-12 | align |
| 98, 108–113 | minor | unscaled logistic features; pooled AUC; no CIs | scaler pipeline; per-cutoff bootstrap |

Label definition (top decile of `log1p(post) − log1p(pre)` among topics with ≥ 20 works in the last 12 months, and
post ≥ 2·pre) matches the docstring and uses only (T−h, T] and (T, T+h] (verified).

### embeddings/semantic_atlas.py — no blocking/major issue

| Line | Sev. | Finding | Fix |
|---|---|---|---|
| 4, 66 | minor | "stratified sample" is a simple random sample | stratify or fix docstring |
| 81–87 | minor | split-half ARI inside one UMAP (not bootstrap; UMAP not refitted) | refit per half |
| 49–61 | minor | novelty centroids include year-precision months | restrict to month precision |
| 30 | minor | duplicate ids across shards silently overwritten | assert uniqueness |
| 41 | minor | baseline/post by calendar year instead of `period` | use `period` |
| 67–69, 88 | minor | no version stamp (umap/numba/pynndescent) | record versions |

## 2. Cross-cutting observations

1. **Preprint–publication merge is only half-implemented.** `define_corpus.py` marks the preprint as a duplicate but no
   downstream script (citation graph, citation windows, category flows, emerging detection) re-attributes the preprint's
   references/citations/`counts_by_year` to the surviving node. Under plan S2 ("one node per work") this undercounts
   early citations and inter-category flows exactly where CS preprints dominate (AI/ML). This is the single most
   consequential conceptual gap after the two blocking bugs.
2. **Date semantics are split across three columns** (`publication_date`, `first_date/first_month`, `date_precision`):
   monthly series use `first_month` but the precision flag is defined on `publication_date`; yearly tables use
   `publication_year`. The same work can therefore be counted in different years in yearly vs monthly outputs.
3. **Right truncation is handled by a fixed cut-off (2026-06) in econometrics only**; citation windows, emerging-detection
   labels and the semantic atlas use the data up to the snapshot without a completeness flag. The plan's
   "indexing-delay profile correction" is not implemented.
4. **Effective data end is 2026-10-14**, not 2026-11-02: the API increment is harvested but never ingested.
5. **Retracted works** are excluded from impact analyses by the protocol but not by the code.
6. **Provenance**: no script stamps the git commit, input hashes or package versions into its outputs; `tests/` covers the
   pure-python toolkit and the classifier but no DuckDB fixture tests exist for the SQL stages (which is where both
   blocking bugs sit). Seeds are set where randomness exists (GBM 44, UMAP/HDBSCAN/sample 44); `row_number() OVER ()`
   and `any_value()` are the only nondeterministic constructs.
7. **Inference**: protocol P6 promises bootstrap/HAC uncertainty on all estimates; ITS has HAC, but Bai–Perron,
   modularity, cross-area shares, Gini and emerging-detection metrics have no intervals.

## 3. Prioritised fix list

**P0 — before any Phase 4 number is produced**
1. `define_corpus.py` L52/L55: `coalesce()` on `type` and `source_name`; add `eligible IS NULL` assertion and a PRISMA
   identity test (identified = Σ sequential exclusions + after_exclusions). Re-run `define_corpus.py` and every
   downstream script; compare `prisma_counts.json` before/after.
2. `run_structural_change.py` L28: aggregate by `first_month` (`groupby().sum()`), then re-run.

**P1 — before results enter the manuscript**
3. Merge semantics: `coalesce(is_arxiv_doi, FALSE)` in the `pub` CTE; mark `e5_duplicate` only against eligible
   published records; add arXiv-id-based pairs and `version_relationship`; remap `ref_id` via `duplicate_of` in
   `01_citation_graph.py`; sum `counts_by_year` over both versions in `03_citation_impact.py`.
4. `03_citation_impact.py`: NULL-safe normalisation join; `window_complete` flag; exclude `e6_retracted`; rename
   windows to calendar-year semantics and compute H5 on the y..y+1 window.
5. `structural_change.py`: baseline-only placebos (fit on pre-event data); add BP supF/sequential tests or bootstrap
   p-values and month dummies so that H3 can be tested at the stated level.
6. Right truncation: derive `last_complete_month` from the lag profile in `01_volume_series.py` and consume it in
   `run_structural_change.py` and `emerging_detection.py` (labels require `T+h ≤ last_complete`; enforce
   `last_train+h ≤ first_test`).
7. Window end: ingest the API increment or amend the protocol to the effective cut-off 2026-10-14.
8. `first_date`/`date_precision` coherence (year-precision dates must not win in `least()`; precision defined on the
   chosen date); then re-derive `period`, `first_month`.
9. `link_sources.py`: arXiv id from the DBLP DOI; first-author check for title+year links; report precision by method.
10. `emerging_detection.py`: fill `KNOWN_SET` and implement lead times (H8); implement (or formally drop, by
    amendment) cutoff-specific semantic clusters and declare the topic-unit look-ahead.
11. `02_category_flows.py`: `unclassified` pseudo-category; bootstrap CIs.

**P2 — hygiene / reproducibility**
12. Provenance stamps (git commit, input sha256, package versions) in every output; `ORDER BY id` in `classify.py`
    chunking; version stamp in classification outputs; DuckDB fixture tests for `define_corpus.py` and
    `03_citation_impact.py`; docstring/code alignments (z-window 25 vs 24, cutoffs, "stratified", "bootstrap");
    `sens_tmp` naming; `--first 2018-02`; NW lag sensitivity; PELT robust penalty.

## 4. Summary

Two blocking defects: (i) NULL propagation in the eligibility flags of `define_corpus.py` silently removes works with
NULL `type` or NULL `source_name` from the corpus and from the PRISMA counts; (ii) `run_structural_change.py` crashes
on every category series because the boundary month appears twice in `category_counts_monthly.csv`. Twenty major
issues cluster around four themes: the preprint–publication merge is not propagated to citations/flows; citation windows
mislabel calendar-year windows, ignore right truncation and include retracted works; the placebo design includes
post-event months and Bai–Perron has no test at the 5 % level; the emerging-detection labels for h = 24 use incomplete
months and the preregistered known-emergence evaluation is absent. Verified correct: Gini formula, directed modularity,
Bai–Perron dynamic programme and BIC/LWZ penalties, ITS design matrix, BOCPD updates, feature look-ahead safety.