spb/anomaly-atlas Public License

1---2project: anomaly-atlas3document: Data source profile4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6data_source: hfmarketdata.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Data source profile — hfmarketdata.io1314**Phase 0.5 output (Experiment A).** Everything below was established15*empirically* against the live API on 2026-08-12, through the single client16(`src/anomaly_atlas/data/hf_client.py`); every probe is cached and recorded in17`data_manifest/index.jsonl`. Raw evidence:18`results/expA_data_reality/20260812T054515Z/results.json`.1920## 1. API shape2122OpenAPI-documented FastAPI service, open (no auth), JSON/CSV:2324| Endpoint | Purpose |25|---|---|26| `GET /v1/status` | full dataset inventory (assets × timeframes × adjustments × ticker counts) |27| `GET /v1/{asset}/tickers` | ticker lists (`search`, `limit`) |28| `GET /v1/bars/{asset}/{ticker}` | OHLCV bars (`timeframe`, `adjustment`, `start`, `end`, `order`, `limit`, `format`) |29| `GET /v1/bars/{asset}` | same, multi-ticker (`tickers=A,B,C`) |30| `GET /v1/snapshot/{asset}` | last bar ≤ instant `at` for each ticker (cross-section) |31| `GET /v1/options/{quarters,tickers,chain,expirations,history}` | daily options chains |3233Bar row schema: `{ticker, datetime, open, high, low, close, volume}` —34**index bars have no `volume` field**. Errors: 404 unknown ticker/asset,35400 invalid timeframe. `end` is an **exclusive instant** (a bare date means36midnight — `start=D&end=D` returns nothing; use `end = D+1 day`).3738## 2. Coverage and history depth (verified per class)3940| Asset | Tickers (1min) | Earliest 1min | Latest seen | Adjustments |41|---|---|---|---|---|42| stock | 7 670 | 2000-01-04 09:30 (AAPL) | 2026-08-07 19:59 | UNADJUSTED, adj_split, adj_splitdiv |43| etf | 5 161 | 2000-01-03 09:31 (SPY) | 2026-08-07 19:59 | UNADJUSTED, adj_split, adj_splitdiv |44| futures (continuous) | 131 | 2008-01-02 06:00 (ES) | 2026-08-06 23:59 | contin_UNadj, contin_adj_absolute, contin_adj_ratio |45| futures_contracts | ~14.5k archive + 2.5k live | — | — | archive / update |46| index | 123 | 2008-01-02 09:30 (SPX; 1day back to 2000-11) | 2026-08-07 16:50 | none |47| fx | 79 | 2010-01-03 17:00 (EURUSD) | 2026-08-07 16:59 | none |48| crypto | 74 | 2013-04-01 (BTC) | 2026-08-09 23:59 | none |49| options | ~6 000 underlyings/quarter | 2010_q1 | 2026_q3_partial (67 quarters) | n/a |5051Timeframes: **1min, 5min, 30min, 1hour, 1day** — 1-minute is genuinely served52(verified), it is the finest granularity. Note: UNADJUSTED intraday exists only53at 1min and 1day for stock/etf; 5min/30min/1hour exist only adjusted.5455**Data lag: this is a periodically refreshed archive, not a live feed.**56On Wednesday 2026-08-12, equities/fx/index ended 2026-08-07 (previous Friday),57futures 2026-08-06, crypto 2026-08-09. Design experiments accordingly — no58same-day data.5960## 3. Timestamp semantics (critical for lead-lag work)6162* Timestamps are **US/Eastern wall-clock strings with no timezone marker**63  (`YYYY-MM-DD HH:MM:SS`). Evidence: FX week runs Sunday 17:00 → Friday 16:5964  (the classic ET convention); equity sessions run 04:00–19:59.65* Bars are **bar-start labeled** (first RTH bar 09:30, last extended bar 19:59).66* Session windows on the probe day (Thu 2026-08-06):67  stock/etf 04:00–19:59 (pre + RTH + post), futures ≈ 24 h, fx 24 h (ET week),68  crypto 24/7, **index SPX 09:30–16:20** (settlement prints after the close).69* These are vendor-consolidated last-trade bars at minute resolution. Assume70  vendor time ≈ exchange time, but **cross-asset closes are non-synchronous by71  construction** (16:00 auction vs 16:20 index prints vs 24 h sessions) — a72  built-in source of spurious lead-lag (see artifact taxonomy).7374## 4. Missing data: bars exist only where trades occurred7576* **No zero-volume placeholder bars anywhere** (0 found on probe day).77* Liquid names are near-complete in RTH (AAPL & SPY: 390/390 RTH minutes) but78  sparse pre/post (AAPL: 904 of 960 extended minutes).79* Illiquid names are sparse even in RTH: AIZN printed **38 bars in the whole80  day**. A last-observation-carried-forward join makes such series look81  autocorrelated and cross-predictable — this is the **stale-price artifact**,82  to be neutralized explicitly (Experiment B).8384## 5. Daily-bar semantics (do NOT mix carelessly with 1min)8586For AAPL on 2026-08-06 (`adj_splitdiv`):8788* Daily bar = **RTH-only OHLC with the official auction close**: daily close89  312.41 vs 312.49 for the last 1min RTH bar — the closing auction is not in90  the 1min series.91* Daily **volume includes consolidated/auction volume absent from 1min bars**:92  daily 46.14 M vs 34.67 M (sum of all extended-hours 1min) vs 25.69 M (RTH93  1min only). Any volume-based signal must pick one convention and stick to it.9495## 6. Corporate actions / adjustments (verified on AAPL 4:1, 2020-08-31)9697| Series | 2020-08-28 close | 2020-08-31 close |98|---|---|---|99| UNADJUSTED | 499.23 | 129.04 |100| adj_split | 124.8075 (= 499.23/4) | 129.04 |101| adj_splitdiv | 121.06 | 125.1654 |102103* Split arithmetic is exact.104* **`adj_splitdiv` re-bases the entire history to the dataset build date** —105  even the 2020-08-31 close differs from its traded price. Adjusted series are106  therefore *not point-in-time stable*: they change whenever a new dividend107  occurs. For any experiment sensitive to this, use UNADJUSTED + explicit108  adjustment, or freeze the cache (which our client does by design).109110## 7. Options111112Daily granularity: one row per contract per `trade_date`, 17 columns —113`bid/ask`, `last_price`, `bid_iv/ask_iv`, `delta/gamma/vega/theta/rho`,114`open_interest`, `volume`, `strike`, `expiry`, `call_put`. 67 quarters115(2010_q1 → 2026_q3_partial). SPY had 35 listed expirations on 2026-06-15.116Old quarters may carry null Greeks (documented server-side NaN handling).117This is the only bid/ask information anywhere in the source — options spreads118may inform equity cost models (Experiment G), with care.119120## 8. Limits, performance, credits121122* **Hard cap: 50 000 rows per JSON response** (requested 1 000 000, got123  50 000). The client paginates on the last datetime; verified on a full year124  of SPY 1min: 218 006 rows, 0 duplicates, strictly ascending.125* No rate-limit headers, no 429 observed at gentle sequential rates126  (~5–7 req/s). Latency: 0.08–0.22 s small requests, ~1.7 s per 50 000-row127  page. There is no credit system — but the client throttles anyway128  (min 0.15 s between requests) and caches everything; the cache is the129  reproducibility anchor.130* `format=csv` exists (not exercised in Experiment A; JSON + local parquet131  cache is our path).132133## 9. Consequences for the research design1341351. **Universe**: ample for Q1–Q3 — thousands of stock/etf tickers × 26 years136   × 1min. Holdout years are affordable.1372. **Artifact taxonomy seeds** (→ `artifact_taxonomy.md`): stale/missing138   minutes; non-synchronous session ends across assets; auction close absent139   from 1min bars; daily-vs-intraday volume mismatch; rolling adjustment140   anchor; continuous-futures splicing method (3 variants exposed — good:141   the choice is testable).1423. **No bid/ask on bars**: spreads must be *estimated* (Roll model et al.) —143   the bounce null (Experiment B) is mandatory, not optional.1444. **Minute data is the floor**: intraday lead-lag finer than 1 minute is145   untestable here; hypotheses must respect that (Epps effect at 1min scale).1465. Data ends ~the previous Friday: "recent regime" claims are bounded.147