# WP44 — Code audit (Agent N, Reproducibility Auditor), 2026-11-03
Scope: read-only audit of the 14 pipeline scripts listed below against CLAUDE.md §1/§6, RESEARCH_PLAN.md and
protocol/PROTOCOL.md (v1.0 + amendments A-000…A-004). No file outside reports/audit/ was modified; the compute node
was not touched. Formula and logic checks were replicated on synthetic data on the laptop (DuckDB 1.5.2, numpy 2.4,
pandas 3.0, statsmodels 0.14, networkx 3.x); scratch scripts live in /tmp/wp44_audit/ and are not part of the repo.
Machine-readable list: reports/audit/code_issues.csv (67 issues: 2 blocking, 20 major, 45 minor).
Severity: blocking = produces wrong or missing results silently / crashes a pre-registered analysis;
major = biases an estimand, contradicts the protocol, or loses data in a way that must be fixed before Phase 4
results enter the manuscript; minor = hygiene, documentation, fragility, provenance.
| Check |
Result |
Eligibility SQL of define_corpus.py with NULL type / NULL source_name |
eligible becomes NULL → rows vanish from WHERE eligible and from every exclusion bucket; PRISMA sequential sum 4 of 6 |
JOIN … USING (year,type,sf) with NULL sf (03_citation_impact.py) |
1 of 3 rows survive |
Gini SQL (03_citation_impact.py L35) vs reference 2Σ i·x_(i)/(nΣx) − (n+1)/n, with ties |
identical (0.575 = 0.575); NaN when Σ = 0 |
Directed weighted modularity (02_category_flows.py) vs networkx.community.modularity |
identical to 1e-6 |
Bai–Perron DP (structural_change.py) vs brute force, n = 30, m = 2, h = 5 |
identical SSR and break indices; BIC and LWZ select m = 2 |
| ITS design matrix |
post, t_post (0 at event, +1/month), 11 month dummies — correct |
| Placebo candidate months (default window 2018-01 → 2026-06) |
66 candidates, 30 after the true event; placebo p = 0.015 vs 0.000 with baseline-only placebos |
features_at (emerging detection) invariance to columns > T |
invariant (look-ahead-safe); trailing z-window = 25 months, not 24 |
| Cutoff lists (emerging detection) |
train 2020-01…2021-10, test 2023-01…2024-10; h = 24 → last test label month 2026-10, last train label month 2023-10 |
series_categories reindex on the boundary month (run_structural_change.py) |
ValueError: cannot reindex on an axis with duplicate labels |
| BOCPD update equations (NIG / Student-t) |
correct by inspection; unit tests exist |
Embedding normalisation assumed by semantic_atlas.py |
embed_corpus.py uses normalize_embeddings=True → cosine via dot product valid |
Two documented facts in the repository corroborate findings: data/metadata/linkage_report.json shows
dblp_with_arxiv: 0 (dead linkage path), and no script ingests api_increment/pages/ (window end mismatch).
# 1. Per-file findings
# ingestion/openalex_snapshot.py — no blocking/major issue
| Line |
Sev. |
Finding |
Fix |
| 148–157 |
minor |
A partial download larger than expected_len is resumed with curl -C - forever; the part stays in error on every run |
delete raw_path on size mismatch before retrying |
| 160–164 |
minor |
SHA-256 recorded but never compared (manifest has no hash) |
capture the S3 ETag and compare; label the hash as provenance |
| 188–189 |
minor |
works with NULL publication_year excluded from the CS subset without being counted |
ledger column n_null_year (PRISMA S0) |
| 236–239 |
minor |
runs table lacks git commit / environment hash |
add both |
Design is sound: resumable SQLite ledger, atomic .part renames, bad-line counter, slim table for all 465 M works.
# ingestion/build_corpus.py — no blocking/major issue
| Line |
Sev. |
Finding |
Fix |
| 69–76 |
minor |
ids["arxiv"] is not in the OpenAlex schema (dead branch); DOI branch keeps v\d+ while the URL branch strips it |
remove dead branch, strip version everywhere |
| 144 |
minor |
is_primary by URL equality → 0 or several primaries when URLs are NULL |
compare source id + URLs |
| 173–176 |
minor |
no Parquet metadata (snapshot release, commit) |
write key/value metadata |
# ingestion/link_sources.py
| Line |
Sev. |
Finding |
Fix |
| 36, 50–53 |
major |
DBLP→arXiv path yields 0 links (dblp_with_arxiv: 0): DBLP CoRR ee is now https://doi.org/10.48550/arXiv.<id>; the DBLP–arXiv version relation is unavailable to the merge step |
derive arxiv_id from the DOI 10.48550/arxiv.<id> too |
| 54–60, 67–72, 81–83 |
major |
title+year linkage without the first-author surname required by plan S1 (180 k DBLP, 2.1 k arXiv, 14.9 k ACL links) |
add first-author check; report precision by method in the H-1 sample |
| 52–83 |
minor |
NOT IN (SELECT …) fragile to NULL keys |
NOT EXISTS |
| 85–118 |
minor |
report has no pass field; committed file is the pass-1 state |
add pass + input hashes |
# systematic_review/define_corpus.py
| Line |
Sev. |
Finding |
Fix |
| 52, 55, 65–66 |
BLOCKING |
NULL type or NULL source_name (non-DBLP, non-arXiv) → e2_type/e7_generic_deposit NULL → eligible NULL: silently dropped by every WHERE eligible and by every PRISMA bucket; sequential counts do not sum (verified) |
coalesce(type,''), coalesce(source_name,''); assert count(*) WHERE eligible IS NULL = 0; PRISMA identity test |
| 38–39 |
major |
pub CTE: NOT is_arxiv_doi is NULL when DOI is NULL → published records without DOI (common for ML conference venues in OpenAlex) can never absorb their preprint → double counting of the most AI-central population |
coalesce(is_arxiv_doi, FALSE) |
| 58, 63–66 |
major |
preprint flagged e5_duplicate even when the matched published record is itself excluded (E1/E2/E7/NULL) → the work disappears entirely |
match only against published records passing e1–e4/e7 |
| 36–43 |
major |
merge by exact normalised title only; arXiv-id relation not used; version_relationship (plan S2) not produced; retitled VoRs stay duplicated |
add arXiv-id pairs + fuzzy fallback; produce version_relationship; report duplicate rate |
| 50, 56 |
major |
first_date = least(...) lets a year-precision YYYY-01-01 beat a precise later arXiv date; date_precision is defined on publication_date, not on the date used for first_month |
treat year-precision dates as NULL in least when a precise date exists; define precision on first_date |
| 16, 51 |
major |
WINDOW_END = 2026-11-02 but data = snapshot 2026-10-14; the API increment is never ingested |
ingest the increment or amend the protocol with the effective cut-off |
| 51, 84 |
minor |
NULL publication_date counted as E1 |
separate reason code |
| 33 |
minor |
any_value() nondeterministic for multi-arXiv works |
arg_min(..., created) |
| 65–69 |
minor |
eligible works with period='pre' in yearly but not monthly tables |
decide and document |
| 67–68 |
minor |
boundary 2022-11-30 (day) vs ITS event 2022-12-01 (month) |
state it |
# bibliometrics/01_volume_series.py
| Line |
Sev. |
Finding |
Fix |
| 22–27 |
major |
indexing-lag table produced but no last-complete-month / completeness correction derived (plan §6); driver hard-codes --last-complete 2026-06 |
estimate completeness from the lag profile; emit last_complete_month JSON used by econometrics and emerging detection |
| 22–27 |
minor |
lag computed on year-precision dates too |
restrict to date_precision='month' |
| 13 |
minor |
first_month vs date_precision inconsistency (upstream) |
— |
| Line |
Sev. |
Finding |
Fix |
| 19–20 |
minor |
fractional counting = 1/#distinct countries per work, not author-fractional |
state the rule |
| 21, 24–25 |
minor |
share_intl, share_company conditional on affiliation coverage, which drifts by year |
add coverage columns |
| 14 |
minor |
retracted and period='pre' included |
document |
# bibliometrics/03_citation_impact.py
| Line |
Sev. |
Finding |
Fix |
| 26–29 |
major |
JOIN USING (year,type,sf) drops all works with NULL subfield (verified) |
IS NOT DISTINCT FROM / coalesce |
| 4–5, 20–22, 35 |
major |
c12 = calendar years y..y+1 (13–24 months), c36 = y..y+2 (25–36 months); H5 prescribes a 24-month Gini, code uses c36 |
rename to calendar-year windows, use y..y+1 for H5, state the publication-month dependence |
| 19, 32–37 |
major |
truncated windows (c36: 2024–2026; c12: 2025–2026) reported without flag |
window_complete column; NULL when incomplete |
| 19 |
major |
retracted works included (protocol E6 excludes them from impact analyses) |
AND NOT e6_retracted |
| 20–23 |
major |
citations of the merged preprint version not added to the published node (plan S2 one node per work) |
aggregate counts_by_year over {id, preprint_work_id} |
| 28 |
minor |
no minimum cell size for NCS normalisation |
threshold + fallback |
| 35 |
— |
Gini formula correct (verified) |
— |
# knowledge_graph/01_citation_graph.py
| Line |
Sev. |
Finding |
Fix |
| 19–26 |
major |
citations to the merged preprint become external edges → merged node loses inbound citations; propagates to 02 and emerging detection |
remap ref_id via duplicate_of |
| 23–26 |
minor |
dangling references (ids not in slim, e.g. merged ids) dropped silently; merged_ids not used |
remap + report n_dangling |
| 31–32 |
minor |
negative citation ages retained |
flag/clip |
# knowledge_graph/02_category_flows.py
| Line |
Sev. |
Finding |
Fix |
| 28–30 |
major |
inner join on work_primary drops unclassified works; shares conditional on classification |
unclassified pseudo-category + share per year |
| 13–17 |
— |
modularity formula correct (directed, weighted; verified) |
— |
| 52 |
minor |
no bootstrap CI (P6) |
edge-resampling CIs |
| 40–41 |
minor |
n_month_precision vs first_month inconsistency (upstream) |
— |
# econometrics/structural_change.py
| Line |
Sev. |
Finding |
Fix |
| 43–49 |
major |
placebo months span the whole sample (30/66 post-event); protocol: baseline only; pre-event placebos estimated on a sample containing the true break |
restrict to baseline; fit on pre-event data (or control for the true event) |
| 61–101 |
major |
Bai–Perron: DP + BIC/LWZ correct, but no supF/UDmax/sequential tests, no critical values, no seasonal regressors → H3 "5 % level" untestable |
add BP tests/critical values or bootstrap p-values; add month dummies |
| 32–36 |
minor |
fixed NW maxlags=6 (auto rule → 4); design verified |
document; lag sensitivity |
| 64, 92–100 |
minor |
break index convention (first of new regime); h = max(8, 0.15n) |
document |
| 104–109 |
minor |
PELT penalty uses full-sample variance (inflated by the break) |
robust scale |
| 112–135 |
minor |
BOCPD data-dependent prior (first 6 obs), fixed hazard; equations correct |
document; hazard sensitivity |
# econometrics/run_structural_change.py
| Line |
Sev. |
Finding |
Fix |
| 23–28 |
BLOCKING |
series_categories sets first_month as index without aggregating; the boundary month 2022-11 has baseline and post rows → duplicate index → reindex raises ValueError → all category-level runs (H3) fail |
g.groupby("first_month")[col].sum() |
| 36, 44 |
minor |
log(n+1) and Bai–Perron/PELT on unadjusted log series for sparse category series |
Poisson/NB ITS robustness; deseasonalise |
| 53 |
minor |
--first 2018-02 unexplained |
document or use 2018-01 |
| 63 |
minor |
sens_tmp expression always true; sensitivity runs overwrite each other's placebo_total.csv/fit_total.csv |
name outputs by cut-off |
| 56–67 |
minor |
no provenance stamp in its_results.json |
add hashes/commit |
| Line |
Sev. |
Finding |
Fix |
| 108 |
minor |
row_number() OVER () without ORDER BY → chunk composition not reproducible |
ORDER BY id |
| 120–129, 142–143 |
minor |
shard mode writes no report; outputs not version-stamped (docstring claims it) |
stamp ontology version + sha256 |
| 146–157 |
minor |
stale input_named symlinks across --prepare re-runs |
clear in prepare() |
| 28, 55–57 |
minor |
regexes compiled without flags; 273/277 patterns carry (?i) |
compile with re.I or lint |
# experiments/emerging_detection.py
| Line |
Sev. |
Finding |
Fix |
| 27–28, 22–23 |
major |
KNOWN_SET empty; H8 lead-time evaluation (known_set_lead_times.csv) not implemented |
fill known_set.csv; implement lead time |
| 78, 83–86, 66–74 |
major |
panel to 2026-10 (incomplete) while econometrics stops at 2026-06: h = 24 test labels truncated; last train label window (2023-10) overlaps test features (from 2023-01) |
T+h ≤ last_complete; last_train+h ≤ first_test |
| 31–38 |
major (declare) |
units = OpenAlex topics (2024 taxonomy that knows which topics emerged); cutoff-specific semantic clusters of the plan not implemented. Features are look-ahead-safe (verified) |
implement cutoff-specific clusters; declare the unit-definition look-ahead |
| 40–41 |
minor |
universe selected with full-period counts |
per-cutoff selection |
| 56–57 |
minor |
z-window 25 months, docstring 24 |
align |
| 85 |
minor |
train cutoffs 2020-01…2021-10, docstring ≤ 2021-12 |
align |
| 98, 108–113 |
minor |
unscaled logistic features; pooled AUC; no CIs |
scaler pipeline; per-cutoff bootstrap |
Label definition (top decile of log1p(post) − log1p(pre) among topics with ≥ 20 works in the last 12 months, and
post ≥ 2·pre) matches the docstring and uses only (T−h, T] and (T, T+h] (verified).
# embeddings/semantic_atlas.py — no blocking/major issue
| Line |
Sev. |
Finding |
Fix |
| 4, 66 |
minor |
"stratified sample" is a simple random sample |
stratify or fix docstring |
| 81–87 |
minor |
split-half ARI inside one UMAP (not bootstrap; UMAP not refitted) |
refit per half |
| 49–61 |
minor |
novelty centroids include year-precision months |
restrict to month precision |
| 30 |
minor |
duplicate ids across shards silently overwritten |
assert uniqueness |
| 41 |
minor |
baseline/post by calendar year instead of period |
use period |
| 67–69, 88 |
minor |
no version stamp (umap/numba/pynndescent) |
record versions |
# 2. Cross-cutting observations
- Preprint–publication merge is only half-implemented.
define_corpus.py marks the preprint as a duplicate but no
downstream script (citation graph, citation windows, category flows, emerging detection) re-attributes the preprint's
references/citations/counts_by_year to the surviving node. Under plan S2 ("one node per work") this undercounts
early citations and inter-category flows exactly where CS preprints dominate (AI/ML). This is the single most
consequential conceptual gap after the two blocking bugs.
- Date semantics are split across three columns (
publication_date, first_date/first_month, date_precision):
monthly series use first_month but the precision flag is defined on publication_date; yearly tables use
publication_year. The same work can therefore be counted in different years in yearly vs monthly outputs.
- Right truncation is handled by a fixed cut-off (2026-06) in econometrics only; citation windows, emerging-detection
labels and the semantic atlas use the data up to the snapshot without a completeness flag. The plan's
"indexing-delay profile correction" is not implemented.
- Effective data end is 2026-10-14, not 2026-11-02: the API increment is harvested but never ingested.
- Retracted works are excluded from impact analyses by the protocol but not by the code.
- Provenance: no script stamps the git commit, input hashes or package versions into its outputs;
tests/ covers the
pure-python toolkit and the classifier but no DuckDB fixture tests exist for the SQL stages (which is where both
blocking bugs sit). Seeds are set where randomness exists (GBM 44, UMAP/HDBSCAN/sample 44); row_number() OVER ()
and any_value() are the only nondeterministic constructs.
- Inference: protocol P6 promises bootstrap/HAC uncertainty on all estimates; ITS has HAC, but Bai–Perron,
modularity, cross-area shares, Gini and emerging-detection metrics have no intervals.
# 3. Prioritised fix list
P0 — before any Phase 4 number is produced
define_corpus.py L52/L55: coalesce() on type and source_name; add eligible IS NULL assertion and a PRISMA
identity test (identified = Σ sequential exclusions + after_exclusions). Re-run define_corpus.py and every
downstream script; compare prisma_counts.json before/after.run_structural_change.py L28: aggregate by first_month (groupby().sum()), then re-run.
P1 — before results enter the manuscript
3. Merge semantics: coalesce(is_arxiv_doi, FALSE) in the pub CTE; mark e5_duplicate only against eligible
published records; add arXiv-id-based pairs and version_relationship; remap ref_id via duplicate_of in
01_citation_graph.py; sum counts_by_year over both versions in 03_citation_impact.py.
4. 03_citation_impact.py: NULL-safe normalisation join; window_complete flag; exclude e6_retracted; rename
windows to calendar-year semantics and compute H5 on the y..y+1 window.
5. structural_change.py: baseline-only placebos (fit on pre-event data); add BP supF/sequential tests or bootstrap
p-values and month dummies so that H3 can be tested at the stated level.
6. Right truncation: derive last_complete_month from the lag profile in 01_volume_series.py and consume it in
run_structural_change.py and emerging_detection.py (labels require T+h ≤ last_complete; enforce
last_train+h ≤ first_test).
7. Window end: ingest the API increment or amend the protocol to the effective cut-off 2026-10-14.
8. first_date/date_precision coherence (year-precision dates must not win in least(); precision defined on the
chosen date); then re-derive period, first_month.
9. link_sources.py: arXiv id from the DBLP DOI; first-author check for title+year links; report precision by method.
10. emerging_detection.py: fill KNOWN_SET and implement lead times (H8); implement (or formally drop, by
amendment) cutoff-specific semantic clusters and declare the topic-unit look-ahead.
11. 02_category_flows.py: unclassified pseudo-category; bootstrap CIs.
P2 — hygiene / reproducibility
12. Provenance stamps (git commit, input sha256, package versions) in every output; ORDER BY id in classify.py
chunking; version stamp in classification outputs; DuckDB fixture tests for define_corpus.py and
03_citation_impact.py; docstring/code alignments (z-window 25 vs 24, cutoffs, "stratified", "bootstrap");
sens_tmp naming; --first 2018-02; NW lag sensitivity; PELT robust penalty.
# 4. Summary
Two blocking defects: (i) NULL propagation in the eligibility flags of define_corpus.py silently removes works with
NULL type or NULL source_name from the corpus and from the PRISMA counts; (ii) run_structural_change.py crashes
on every category series because the boundary month appears twice in category_counts_monthly.csv. Twenty major
issues cluster around four themes: the preprint–publication merge is not propagated to citations/flows; citation windows
mislabel calendar-year windows, ignore right truncation and include retracted works; the placebo design includes
post-event months and Bai–Perron has no test at the 5 % level; the emerging-detection labels for h = 24 use incomplete
months and the preregistered known-emergence evaluation is absent. Verified correct: Gini formula, directed modularity,
Bai–Perron dynamic programme and BIC/LWZ penalties, ITS design matrix, BOCPD updates, feature look-ahead safety.