--- project: anomaly-atlas document: Data source profile author: Simon-Pierre Boucher contact: contact@spboucher.ai data_source: hfmarketdata.io created: 2026-08-12 modified: 2026-08-12 status: reviewed --- # Data source profile — hfmarketdata.io **Phase 0.5 output (Experiment A).** Everything below was established *empirically* against the live API on 2026-08-12, through the single client (`src/anomaly_atlas/data/hf_client.py`); every probe is cached and recorded in `data_manifest/index.jsonl`. Raw evidence: `results/expA_data_reality/20260812T054515Z/results.json`. ## 1. API shape OpenAPI-documented FastAPI service, open (no auth), JSON/CSV: | Endpoint | Purpose | |---|---| | `GET /v1/status` | full dataset inventory (assets × timeframes × adjustments × ticker counts) | | `GET /v1/{asset}/tickers` | ticker lists (`search`, `limit`) | | `GET /v1/bars/{asset}/{ticker}` | OHLCV bars (`timeframe`, `adjustment`, `start`, `end`, `order`, `limit`, `format`) | | `GET /v1/bars/{asset}` | same, multi-ticker (`tickers=A,B,C`) | | `GET /v1/snapshot/{asset}` | last bar ≤ instant `at` for each ticker (cross-section) | | `GET /v1/options/{quarters,tickers,chain,expirations,history}` | daily options chains | Bar row schema: `{ticker, datetime, open, high, low, close, volume}` — **index bars have no `volume` field**. Errors: 404 unknown ticker/asset, 400 invalid timeframe. `end` is an **exclusive instant** (a bare date means midnight — `start=D&end=D` returns nothing; use `end = D+1 day`). ## 2. Coverage and history depth (verified per class) | Asset | Tickers (1min) | Earliest 1min | Latest seen | Adjustments | |---|---|---|---|---| | stock | 7 670 | 2000-01-04 09:30 (AAPL) | 2026-08-07 19:59 | UNADJUSTED, adj_split, adj_splitdiv | | etf | 5 161 | 2000-01-03 09:31 (SPY) | 2026-08-07 19:59 | UNADJUSTED, adj_split, adj_splitdiv | | futures (continuous) | 131 | 2008-01-02 06:00 (ES) | 2026-08-06 23:59 | contin_UNadj, contin_adj_absolute, contin_adj_ratio | | futures_contracts | ~14.5k archive + 2.5k live | — | — | archive / update | | index | 123 | 2008-01-02 09:30 (SPX; 1day back to 2000-11) | 2026-08-07 16:50 | none | | fx | 79 | 2010-01-03 17:00 (EURUSD) | 2026-08-07 16:59 | none | | crypto | 74 | 2013-04-01 (BTC) | 2026-08-09 23:59 | none | | options | ~6 000 underlyings/quarter | 2010_q1 | 2026_q3_partial (67 quarters) | n/a | Timeframes: **1min, 5min, 30min, 1hour, 1day** — 1-minute is genuinely served (verified), it is the finest granularity. Note: UNADJUSTED intraday exists only at 1min and 1day for stock/etf; 5min/30min/1hour exist only adjusted. **Data lag: this is a periodically refreshed archive, not a live feed.** On Wednesday 2026-08-12, equities/fx/index ended 2026-08-07 (previous Friday), futures 2026-08-06, crypto 2026-08-09. Design experiments accordingly — no same-day data. ## 3. Timestamp semantics (critical for lead-lag work) * Timestamps are **US/Eastern wall-clock strings with no timezone marker** (`YYYY-MM-DD HH:MM:SS`). Evidence: FX week runs Sunday 17:00 → Friday 16:59 (the classic ET convention); equity sessions run 04:00–19:59. * Bars are **bar-start labeled** (first RTH bar 09:30, last extended bar 19:59). * Session windows on the probe day (Thu 2026-08-06): stock/etf 04:00–19:59 (pre + RTH + post), futures ≈ 24 h, fx 24 h (ET week), crypto 24/7, **index SPX 09:30–16:20** (settlement prints after the close). * These are vendor-consolidated last-trade bars at minute resolution. Assume vendor time ≈ exchange time, but **cross-asset closes are non-synchronous by construction** (16:00 auction vs 16:20 index prints vs 24 h sessions) — a built-in source of spurious lead-lag (see artifact taxonomy). ## 4. Missing data: bars exist only where trades occurred * **No zero-volume placeholder bars anywhere** (0 found on probe day). * Liquid names are near-complete in RTH (AAPL & SPY: 390/390 RTH minutes) but sparse pre/post (AAPL: 904 of 960 extended minutes). * Illiquid names are sparse even in RTH: AIZN printed **38 bars in the whole day**. A last-observation-carried-forward join makes such series look autocorrelated and cross-predictable — this is the **stale-price artifact**, to be neutralized explicitly (Experiment B). ## 5. Daily-bar semantics (do NOT mix carelessly with 1min) For AAPL on 2026-08-06 (`adj_splitdiv`): * Daily bar = **RTH-only OHLC with the official auction close**: daily close 312.41 vs 312.49 for the last 1min RTH bar — the closing auction is not in the 1min series. * Daily **volume includes consolidated/auction volume absent from 1min bars**: daily 46.14 M vs 34.67 M (sum of all extended-hours 1min) vs 25.69 M (RTH 1min only). Any volume-based signal must pick one convention and stick to it. ## 6. Corporate actions / adjustments (verified on AAPL 4:1, 2020-08-31) | Series | 2020-08-28 close | 2020-08-31 close | |---|---|---| | UNADJUSTED | 499.23 | 129.04 | | adj_split | 124.8075 (= 499.23/4) | 129.04 | | adj_splitdiv | 121.06 | 125.1654 | * Split arithmetic is exact. * **`adj_splitdiv` re-bases the entire history to the dataset build date** — even the 2020-08-31 close differs from its traded price. Adjusted series are therefore *not point-in-time stable*: they change whenever a new dividend occurs. For any experiment sensitive to this, use UNADJUSTED + explicit adjustment, or freeze the cache (which our client does by design). ## 7. Options Daily granularity: one row per contract per `trade_date`, 17 columns — `bid/ask`, `last_price`, `bid_iv/ask_iv`, `delta/gamma/vega/theta/rho`, `open_interest`, `volume`, `strike`, `expiry`, `call_put`. 67 quarters (2010_q1 → 2026_q3_partial). SPY had 35 listed expirations on 2026-06-15. Old quarters may carry null Greeks (documented server-side NaN handling). This is the only bid/ask information anywhere in the source — options spreads may inform equity cost models (Experiment G), with care. ## 8. Limits, performance, credits * **Hard cap: 50 000 rows per JSON response** (requested 1 000 000, got 50 000). The client paginates on the last datetime; verified on a full year of SPY 1min: 218 006 rows, 0 duplicates, strictly ascending. * No rate-limit headers, no 429 observed at gentle sequential rates (~5–7 req/s). Latency: 0.08–0.22 s small requests, ~1.7 s per 50 000-row page. There is no credit system — but the client throttles anyway (min 0.15 s between requests) and caches everything; the cache is the reproducibility anchor. * `format=csv` exists (not exercised in Experiment A; JSON + local parquet cache is our path). ## 9. Consequences for the research design 1. **Universe**: ample for Q1–Q3 — thousands of stock/etf tickers × 26 years × 1min. Holdout years are affordable. 2. **Artifact taxonomy seeds** (→ `artifact_taxonomy.md`): stale/missing minutes; non-synchronous session ends across assets; auction close absent from 1min bars; daily-vs-intraday volume mismatch; rolling adjustment anchor; continuous-futures splicing method (3 variants exposed — good: the choice is testable). 3. **No bid/ask on bars**: spreads must be *estimated* (Roll model et al.) — the bounce null (Experiment B) is mandatory, not optional. 4. **Minute data is the floor**: intraday lead-lag finer than 1 minute is untestable here; hypotheses must respect that (Epps effect at 1min scale). 5. Data ends ~the previous Friday: "recent regime" claims are bounded.