# DATA_AUDIT.md — Stage A preliminary audit

Audit date: 2026-07-13 Auditor: automated pipeline (scripts/01_pilot_audit.py) + targeted follow-ups (logs in logs/audit/) Status: Stage A PASSED — no stop condition of CLAUDE.md §31 triggered. Proceed to Stage B (full download) after infrastructure review.

# 1. Dataset identity and provenance

⚠️ Licensing flag: the card names Finnhub as the original source but carries no license. Redistribution/derived-research permissions are therefore unknown. This must be resolved (contact author, check Finnhub terms) before public release of derived data; research use with citation is standard practice but should be documented in docs/DECISION_LOG.md. This is a caution, not a §31 stop: the analysis outputs (aggregates, estimates) do not redistribute raw data.

# 2. Pilot sample

Three months downloaded with SHA-256 checksums (data/manifests/pilot_downloads.json):

# 3. Physical schema (identical across all three pilot files)

timestamp  Datetime(ns, tz=UTC)   start of the bar minute (per card; consistent with data)
open       Float64                USD
high       Float64                USD
low        Float64                USD
close      Float64                USD
volume     Float64                shares (integer-valued in all 58.8M pilot rows)
ticker     String                 exchange ticker, incl. class suffixes ("BRK.A", "HEI.A")

No nulls in any column in any pilot file.

# 4. Timestamp convention — empirically established

• All timestamps have seconds == 0 (pure minute grid).
• 1992-06 (EDT, UTC−4): first bar 13:30 UTC = 09:30 ET; regular close maps to 16:00 ET.
• 2025-12 (EST, UTC−5): regular session maps to 14:30–21:00 UTC = 09:30–16:00 ET.
• Conclusion: timestamps are true UTC; DST is handled correctly by the source. Convert with America/New_York for session logic.
• No weekend rows in any pilot month.
• Calendar: trading-day counts match the NYSE calendar in all three months. The 2025-12-24 early close is respected in the data: market-wide 13:00–16:00 ET volume is 7.3% of morning volume (vs 60.4% on the adjacent normal day); residual post-13:00 bars are sparse low-volume prints (off-exchange). Cleaning rule: truncate sessions using an exchange calendar, do not rely on bar presence.

# 5. Session composition

Extended-hours coverage exists but grows over time — extended-hours analyses cannot be compared across decades on equal footing. Baseline must be RTH-only.

# 6. Data quality results (pilot)

• The 14 duplicate pairs (28 rows) are all in illiquid OTC names (AVRW, DSHKP, NVNXF, OTRKQ, UNICY); same minute and price, usually different volume — consistent with venue/feed corrections. Resolution rule (to be logged in DECISION_LOG): aggregate duplicates by volume-sum with max(high)/min(low), and report sensitivity.
• Zero-volume bars do not exist → a missing minute means "no reported trade." The panel is trade-conditional, not a filled grid. All liquidity, staleness, and zero-return measures must treat missing minutes explicitly (cf. CLAUDE.md §5.7).

# 7. Bar density and units

• Liquid names are complete: AAPL, MSFT, SPY, NVDA each have exactly 390 RTH bars/day (median) in 2025-12; 1992 mega-caps (IBM 334, KO 368, T 332, AAPL 330, GE 291) are near-complete.
• Cross-section is highly skewed: in 2025-12, per-ticker median RTH bars/day has p10=1, median=5.5, p90=230; only 827 of 23,047 tickers have ≥350 bars/day. The 2025 file includes vast OTC/foreign coverage (F/Y-suffixed tickers, sub-penny names).
• 1992-06 overall median bars/symbol-day = 9 (sparse small caps) vs 89 in 2009-06.
• Volume units = shares: SPY mean daily volume 2025-12 = 75.6M (matches public data); IBM 1992-06 = 2.0M (plausible pre-split).
• Prices are raw (not split-adjusted): BRK.A trades at 771,021 USD in 2025-12 at correct level; corporate-action detection (CLAUDE.md §5.4) is mandatory before return construction.
• Sub-penny closes (1e-06) exist in OTC names; min price filters needed for baseline universe.

# 8. Survivorship assessment (preliminary)

Ticker overlap: 1992∩2009 = 25.8% of 1992 tickers; 1992∩2025 = 22.1%; 2009∩2025 = 39.3%. Massive non-overlap implies delisted/disappeared securities are present in historical files — the data are not a backfilled survivor universe. Formal entry/exit curves require the full download (Stage C). Ticker reuse across decades is a real risk (a 1992 "X" and a 2025 "X" need not be the same firm): analyses spanning identity must use ticker-spell intervals (CLAUDE.md §5.5).

# 9. Storage and compute estimate

• Raw compressed: 87.7 GB (4.8 GB 1990s / 25.3 GB 2000s / 30.7 GB 2010s / 27.0 GB 2020s incl. 2026Q1).
• Estimated rows: ~7–9 billion (pilot bytes/row ≈ 7–13 B compressed).
• Working estimate: raw 88 GB + cleaned RTH panel ≈ 60–80 GB + features ≈ 100–150 GB + outputs ⇒ ~300–350 GB total; local disk has ~1.0 TB free — sufficient. MacLustr cluster (284 cores) available for node-sharded download/processing per CLAUDE.md §27.
• Download at observed throughput (~11–21 MB/s anonymous): ~1.5–2.5 h serial; minutes when sharded across nodes. Set HF_TOKEN for faster authenticated transfers.

# 10. Open questions carried to Stage B/C

1. Whether bars aggregate consolidated-tape trades or a single venue (affects early-close prints and volume interpretation) — probe via known halt/auction episodes.
2. Point-in-time correctness of tickers (renames, e.g., FB→META) — build spell map from full panel + external reference.
3. Security-type classification (common stock vs ETF vs ADR vs preferred/warrant/OTC) — needs external security master; ticker-suffix heuristics as fallback.
4. 1992–1999 depth: whether small-cap sparseness is a data-coverage artifact or true trading sparsity — compare against known daily volumes (external validation).
5. License clarification (see §1).
6. Outlier contamination of moments: in the pilot summary table (top-500 universe, no corporate-action screens), the SD of minute log returns is 255 bps (1992-06) and 1,173 bps (2009-06) against p1–p99 spreads of only ±146 and ±38 bps — second moments are dominated by a handful of extreme prints/split-like jumps. Confirms that split detection and print screens (§5.4) are prerequisites for any variance-based measure.

# STAGE C — Full-sample raw audit (2026-07-13)

Scope: all 411 months, 7,686,275,524 bars, 81,306 distinct tickers. Scripts: 04_stage_c_monthly_audit.py (per-month + daily ticker-day panel in data/metadata/daily/), 05_stage_c_cross_audit.py (cross-month). Outputs in outputs/diagnostics/stage_c/.

# C.1 Calendar — PERFECT MATCH

Observed trading dates vs official XNYS calendar 1992-01-02 → 2026-03-31: 0 missing sessions, 0 dates outside the calendar, 0 weekend rows.

# C.2 Timezone / DST — CONFIRMED CLEAN

Mean RTH share by calendar month is flat (0.9675–0.9709) with no level shift in March/November; a DST mishandling would displace ~60 min of the 390-minute session (~15% of rows). UTC + correct DST holds across all 34 years. RTH share per month ranges 0.892–0.988, declining secularly with extended-hours growth (expected).

# C.3 Integrity — 7.7 billion rows

# C.4 Corporate-action candidates

442,251 ticker-days with overnight close→open ratios within 2% of rational split ratios (2,3,4,5,6,7,8,10,20,3/2,4/3,5/4 and inverses), split_candidates.parquet. This is a deliberately over-inclusive screen (illiquid-name overnight moves hit 4/3 and 5/4 by chance); Stage D refines with volume/price-level context before building exclusion windows. Never delete without logging (§5.4).

# C.5 Identity and survivorship

• 81,306 tickers; 221,957 continuous spells (gap > 30 calendar days ⇒ new spell); 29,112 tickers (36%) have multiple spells → ticker-spell, not ticker, is the identity unit for long-run analyses. Median spell 6 days (sparse OTC tail), p90 ≈ 1,084 days.
• Yearly universe: ~10.0k (1992) → ~14.4k (1996) → ~12.2–12.8k (2004–2016) → 22.9k in 2020 → 28.6k in 2024. Entries jump to 10,965 in 2020: a source- coverage regime change (OTC/foreign expansion), not a market event. Exits run ~800–1,600/yr throughout ⇒ delisted names are present continuously (limited survivorship bias); 2026 exit spike is right-censoring.
• Consequence (falsification §16): all secular results must hold within a fixed-rule universe (top-N by lagged dollar volume); the 2020 coverage break is a mandatory robustness axis.