TeX 53.6%
Python 46.2%
1# DATA_AUDIT.md — Stage A preliminary audit23**Audit date:** 2026-07-134**Auditor:** automated pipeline (`scripts/01_pilot_audit.py`) + targeted follow-ups (logs in `logs/audit/`)5**Status:** Stage A PASSED — no stop condition of CLAUDE.md §31 triggered. Proceed to Stage B (full download) after infrastructure review.67---89## 1. Dataset identity and provenance1011| Property | Value | Verification |12|---|---|---|13| Dataset | `mito0o852/OHLCV-1m` (Hugging Face) | HF API |14| Pinned revision | `776328445b7ac6e7815ef3a483e9c8ded1eb6d56` | HF API, recorded in all manifests |15| Last modified | 2026-05-03T16:22:10Z | HF API |16| Claimed source | Finnhub.io, repackaged from monthly `.tar` archives | dataset card (unverified upstream) |17| Files | 411 monthly Parquet files, `data/ohlcv_YYYY-MM.parquet` | inventory `data/manifests/repo_inventory.json` |18| Coverage | 1992-01 through 2026-03, **no missing months** | continuity check on inventory |19| Total compressed size | 87.72 GB | HF file metadata |20| Access | public, not gated, anonymous download works | verified empirically |21| **License** | **NOT SPECIFIED on the dataset card** | HF API `cardData` has no license field |2223⚠️ **Licensing flag:** the card names Finnhub as the original source but carries no license. Redistribution/derived-research permissions are therefore **unknown**. This must be resolved (contact author, check Finnhub terms) before public release of derived data; research use with citation is standard practice but should be documented in `docs/DECISION_LOG.md`. This is a *caution*, not a §31 stop: the analysis outputs (aggregates, estimates) do not redistribute raw data.2425## 2. Pilot sample2627Three months downloaded with SHA-256 checksums (`data/manifests/pilot_downloads.json`):2829| Month | Bytes | Rows | Unique tickers | Trading days |30|---|---|---|---|---|31| 1992-06 | 24,052,854 | 3,392,433 | 8,267 | 22 (correct) |32| 2009-06 | 270,431,130 | 24,638,699 | 8,050 | 22 (correct) |33| 2025-12 | 401,427,461 | 30,792,037 | 23,095 | 22 (correct, incl. Dec 24 half-day; Dec 25 closed) |3435## 3. Physical schema (identical across all three pilot files)3637```38timestamp Datetime(ns, tz=UTC) start of the bar minute (per card; consistent with data)39open Float64 USD40high Float64 USD41low Float64 USD42close Float64 USD43volume Float64 shares (integer-valued in all 58.8M pilot rows)44ticker String exchange ticker, incl. class suffixes ("BRK.A", "HEI.A")45```4647No nulls in any column in any pilot file.4849## 4. Timestamp convention — empirically established5051- All timestamps have `seconds == 0` (pure minute grid).52- 1992-06 (EDT, UTC−4): first bar 13:30 UTC = **09:30 ET**; regular close maps to 16:00 ET.53- 2025-12 (EST, UTC−5): regular session maps to 14:30–21:00 UTC = **09:30–16:00 ET**.54- **Conclusion: timestamps are true UTC; DST is handled correctly by the source.** Convert with `America/New_York` for session logic.55- No weekend rows in any pilot month.56- Calendar: trading-day counts match the NYSE calendar in all three months. The 2025-12-24 early close is respected in the data: market-wide 13:00–16:00 ET volume is 7.3% of morning volume (vs 60.4% on the adjacent normal day); residual post-13:00 bars are sparse low-volume prints (off-exchange). **Cleaning rule: truncate sessions using an exchange calendar, do not rely on bar presence.**5758## 5. Session composition5960| Month | RTH share (09:30–16:00 ET) | Pre-market rows | After-hours rows |61|---|---|---|---|62| 1992-06 | 97.4% | 0 | 89,364 |63| 2009-06 | 97.7% | 274,855 | 298,811 |64| 2025-12 | 92.0% | 1,498,056 | 951,715 |6566Extended-hours coverage exists but grows over time — **extended-hours analyses cannot be compared across decades on equal footing.** Baseline must be RTH-only.6768## 6. Data quality results (pilot)6970| Check | 1992-06 | 2009-06 | 2025-12 |71|---|---|---|---|72| OHLC violations (Low>min(O,C), High<max(O,C), Low>High) | 0 | 0 | 0 |73| Duplicate (ticker,timestamp) keys | 0 | 0 | 14 pairs |74| Non-positive prices | 0 | 0 | 0 |75| Negative volume | 0 | 0 | 0 |76| Zero-volume bars | 0 | 0 | 0 |77| Non-integer volume | 0 | 0 | 0 |78| Nulls | 0 | 0 | 0 |7980- The 14 duplicate pairs (28 rows) are all in illiquid OTC names (AVRW, DSHKP, NVNXF, OTRKQ, UNICY); same minute and price, usually different volume — consistent with venue/feed corrections. **Resolution rule (to be logged in DECISION_LOG): aggregate duplicates by volume-sum with max(high)/min(low), and report sensitivity.**81- **Zero-volume bars do not exist → a missing minute means "no reported trade."** The panel is trade-conditional, not a filled grid. All liquidity, staleness, and zero-return measures must treat missing minutes explicitly (cf. CLAUDE.md §5.7).8283## 7. Bar density and units8485- Liquid names are complete: AAPL, MSFT, SPY, NVDA each have exactly 390 RTH bars/day (median) in 2025-12; 1992 mega-caps (IBM 334, KO 368, T 332, AAPL 330, GE 291) are near-complete.86- Cross-section is highly skewed: in 2025-12, per-ticker median RTH bars/day has p10=1, median=5.5, p90=230; only 827 of 23,047 tickers have ≥350 bars/day. The 2025 file includes vast OTC/foreign coverage (F/Y-suffixed tickers, sub-penny names).87- 1992-06 overall median bars/symbol-day = 9 (sparse small caps) vs 89 in 2009-06.88- **Volume units = shares:** SPY mean daily volume 2025-12 = 75.6M (matches public data); IBM 1992-06 = 2.0M (plausible pre-split).89- **Prices are raw (not split-adjusted):** BRK.A trades at 771,021 USD in 2025-12 at correct level; corporate-action detection (CLAUDE.md §5.4) is mandatory before return construction.90- Sub-penny closes (1e-06) exist in OTC names; min price filters needed for baseline universe.9192## 8. Survivorship assessment (preliminary)9394Ticker overlap: 1992∩2009 = 25.8% of 1992 tickers; 1992∩2025 = 22.1%; 2009∩2025 = 39.3%.95Massive non-overlap implies **delisted/disappeared securities are present in historical files** — the data are not a backfilled survivor universe. Formal entry/exit curves require the full download (Stage C). Ticker reuse across decades is a real risk (a 1992 "X" and a 2025 "X" need not be the same firm): analyses spanning identity must use ticker-spell intervals (CLAUDE.md §5.5).9697## 9. Storage and compute estimate9899- Raw compressed: **87.7 GB** (4.8 GB 1990s / 25.3 GB 2000s / 30.7 GB 2010s / 27.0 GB 2020s incl. 2026Q1).100- Estimated rows: ~7–9 billion (pilot bytes/row ≈ 7–13 B compressed).101- Working estimate: raw 88 GB + cleaned RTH panel ≈ 60–80 GB + features ≈ 100–150 GB + outputs ⇒ **~300–350 GB total**; local disk has ~1.0 TB free — sufficient. MacLustr cluster (284 cores) available for node-sharded download/processing per CLAUDE.md §27.102- Download at observed throughput (~11–21 MB/s anonymous): ~1.5–2.5 h serial; minutes when sharded across nodes. Set `HF_TOKEN` for faster authenticated transfers.103104## 10. Open questions carried to Stage B/C1051061. Whether bars aggregate consolidated-tape trades or a single venue (affects early-close prints and volume interpretation) — probe via known halt/auction episodes.1072. Point-in-time correctness of tickers (renames, e.g., FB→META) — build spell map from full panel + external reference.1083. Security-type classification (common stock vs ETF vs ADR vs preferred/warrant/OTC) — needs external security master; ticker-suffix heuristics as fallback.1094. 1992–1999 depth: whether small-cap sparseness is a data-coverage artifact or true trading sparsity — compare against known daily volumes (external validation).1105. License clarification (see §1).1116. Outlier contamination of moments: in the pilot summary table (top-500 universe, no112 corporate-action screens), the SD of minute log returns is 255 bps (1992-06) and113 1,173 bps (2009-06) against p1–p99 spreads of only ±146 and ±38 bps — second moments114 are dominated by a handful of extreme prints/split-like jumps. Confirms that split115 detection and print screens (§5.4) are prerequisites for any variance-based measure.116117---118119# STAGE C — Full-sample raw audit (2026-07-13)120121Scope: all 411 months, 7,686,275,524 bars, 81,306 distinct tickers. Scripts:122`04_stage_c_monthly_audit.py` (per-month + daily ticker-day panel in123`data/metadata/daily/`), `05_stage_c_cross_audit.py` (cross-month). Outputs in124`outputs/diagnostics/stage_c/`.125126## C.1 Calendar — PERFECT MATCH127Observed trading dates vs official XNYS calendar 1992-01-02 → 2026-03-31:128**0 missing sessions, 0 dates outside the calendar, 0 weekend rows.**129130## C.2 Timezone / DST — CONFIRMED CLEAN131Mean RTH share by calendar month is flat (0.9675–0.9709) with **no level shift in132March/November**; a DST mishandling would displace ~60 min of the 390-minute session133(~15% of rows). UTC + correct DST holds across all 34 years. RTH share per month134ranges 0.892–0.988, declining secularly with extended-hours growth (expected).135136## C.3 Integrity — 7.7 billion rows137| Check | Count | Rate |138|---|---|---|139| OHLC violations | 100 | 1.3 per 10^8 — concentrated 1993–1995 (max 7/month); flag & exclude |140| Non-positive prices | 0 | — |141| Zero / negative volume | 0 | — |142| Duplicate (ticker,minute) keys | 10,128 | 1.3 per 10^6 |143| Weekend rows | 0 | — |144145## C.4 Corporate-action candidates146442,251 ticker-days with overnight close→open ratios within 2% of rational split147ratios (2,3,4,5,6,7,8,10,20,3/2,4/3,5/4 and inverses), `split_candidates.parquet`.148This is a deliberately over-inclusive screen (illiquid-name overnight moves hit1494/3 and 5/4 by chance); Stage D refines with volume/price-level context before150building exclusion windows. Never delete without logging (§5.4).151152## C.5 Identity and survivorship153- 81,306 tickers; 221,957 continuous spells (gap > 30 calendar days ⇒ new spell);154 **29,112 tickers (36%) have multiple spells** → ticker-spell, not ticker, is the155 identity unit for long-run analyses. Median spell 6 days (sparse OTC tail),156 p90 ≈ 1,084 days.157- Yearly universe: ~10.0k (1992) → ~14.4k (1996) → ~12.2–12.8k (2004–2016) →158 **22.9k in 2020 → 28.6k in 2024**. Entries jump to 10,965 in 2020: a **source-159 coverage regime change** (OTC/foreign expansion), not a market event. Exits run160 ~800–1,600/yr throughout ⇒ delisted names are present continuously (limited161 survivorship bias); 2026 exit spike is right-censoring.162- Consequence (falsification §16): all secular results must hold within a163 fixed-rule universe (top-N by lagged dollar volume); the 2020 coverage break is164 a mandatory robustness axis.165