spb/market-atlas

1# Market Atlas — architecture (v0.1, 2026-09-12)23Market Atlas is a market **observation infrastructure**: a connector network observes public/official sources,4normalizes their outputs into one observation model, resolves instruments, builds a multi-source consensus,5derives events, keeps the history, and redistributes the result through one web app, a REST API and a stream.67```8 sources (WS / XHR / official APIs / RSS / XML / HTML / bulk files)9   └─ connectors/            ConnectorDefinition (metadata + rights + normalize())     [L0 raw archived, sampled]10        └─ ConnectorManager  lifecycle · adaptive polling · rate limits · circuit breaker · schema drift11             └─ Pipeline     validate → resolve instrument → Observation (L3) → ObservationWriter (Postgres, daily partitions)12                  ├─ ConsensusEngine   weighted median · source families · outliers · freshness → CanonicalQuote13                  │     └─ QuoteStore  hot cache + debounced upsert · bus "canonical.quote"14                  ├─ EventEngine       price change · session high/low · volatility · divergence · halts · filings · market open/close15                  ├─ BarAggregator     live 1m bars → SQL rollups 5m/15m/1h/1d (+ backfill bars from historical connectors)16                  └─ HealthEngine      state · latency · parse rate · reliability score (0–100) · minute snapshots17 MarketBus (in-process, topic names = spec) → API (Fastify): REST /v1 · WebSocket /v1/stream · SSE /v1/sse · Prometheus /v1/metrics18 apps/web (Next.js) consumes the API and the stream; in production the API process is the public edge and proxies to Next.19```2021## Packages2223| Path | Role |24|---|---|25| `packages/market-model` | Vocabularies (asset classes, fields, rights/real-time statuses, event types), types (Instrument, Observation, CanonicalQuote, MarketEvent…), stable id builders, zod schemas, Intl-only time-zone utilities. |26| `packages/connector-sdk` | `defineConnector`, `ConnectorContext`, polite `HttpClient` (per-host token buckets, ETag/If-Modified-Since, jittered retries, redacted URLs), `ManagedWebSocket` (reconnect, heartbeat, stale detection), parsers (RSS/Atom, XML, delimited, numbers, HTML tables, change-detection fingerprints), schema fingerprints, SSRF guard, offline test helpers. |27| `connectors/` | One folder per connector: `index.ts`, `fixtures/`, `README.md`; `sources.ts` = source registry with families; `connectors.test.ts`. |28| `apps/api` | Runtime: config, Postgres access + forward-only SQL migrations (`infra/migrations`), core engines, connector manager, scheduler, REST/WS/SSE, admin API, discovery prototype, CLI (`ma`). Bundled with esbuild to `dist/`. |29| `apps/web` | Next.js 16 app (see `docs/FRONTEND.md`). |3031## Data layers3233- **L0 raw** — `MA_DATA_DIR/raw/<day>/<connector>/<sha>.json.gz` (redacted, content-addressed). Polled/bulk payloads always; streaming frames sampled (`MA_RAW_SAMPLE_RATE`). `raw_ref` on observations points back. `ma replay <day>` re-normalizes offline.34- **L1/L2** — connector `normalize()` (pure, deterministic, no network) → `NormalizedObservation` with source symbol + instrument hint, rights & real-time status, timestamp trust.35- **L3** — `observations` (partitioned by `received_at` day; fingerprint dedupes reconnect replays; `normalizer_version`; REALTIME streaming ticks are persisted at most once per source/instrument/field per `MA_TICK_PERSIST_INTERVAL_MS` = 2 s while consensus/events see every tick). Retention `MA_OBSERVATION_RETENTION_DAYS` then gzip NDJSON archive in `MA_DATA_DIR/archive` (never silently deleted; `observation_archives` index).36- **L4** — `canonical_quotes`, `bars`, `market_events`, `filings`, `connector_health`, `fact_changes`, `document_snapshots`.3738## Source Mesh v2 (2026-09-13)3940**One instrument → many observers → one canonical market state.** Every observation carries an `observation_type`41(TRADE, MID, QUOTE, INDEX_VALUE, INDICATIVE, STABLECOIN_PROXY, DERIVED, OFFICIAL_FIX, REFERENCE_RATE, SETTLEMENT, EOD_CLOSE, NAV) that42maps to a comparability class (LIVE / FIX / EOD). Only comparable observations are compared: an ECB fixing is never measured against a live43market, last week's close is never "divergent" from yesterday's. Roles: real markets **vote**; stablecoin/derived proxies **confirm**44(reduced weight, never counted as families); restricted-rights sources (nasdaq.com) **validate** only (`validation_only`, never voting,45never redistributed). `core/coverage.ts` scores every quoted instrument against a tier target (A ≥ 3 families / 5 observations; B ≥ 2 / 3;46C 1 / 2; D official single source) and produces the source-expansion queue (`GET /v1/coverage`). `core/lineage.ts` infers shared upstreams47statistically (identical values within 2 s over ≥ 120 aligned samples → similarity ≥ 0.92 → one family, `source_lineage` table).48Live FX without keys: Kraken fiat markets (12 pairs, TRADE), Bitstamp EUR/USD & GBP/USD, Bitfinex EUR/GBP/JPY vs USD, plus stablecoin49proxies (EUR/USDT, USDC/EUR, USDT/BRL…) on Binance, OKX, Coinbase, Bybit, Gate, KuCoin.5051## Consensus (apps/api/src/core/consensus.ts)5253Per instrument and field, the newest observation of each source is kept (out-of-order values ignored). A value is *fresh* within a54window depending on its real-time class (REALTIME 15 s, DELAYED 30 min, INDICATIVE 1 h, END_OF_DAY 10 d; 4 d for live classes while the55venue is closed — the last session value is reported as AT_CLOSE, never as live). Comparability class first (LIVE from real markets > FIX >56EOD > live proxies alone), then temporal tolerance inside the class (live: 2 min; fixings/closes: same UTC day → `temporal_mismatch`).57Weight = source reliability × timestamp trust × real-time class × freshness decay × official bonus (×1.6 for rates). One vote per58**source family** (declared or statistically inferred). Crypto weights scale with the venue's share of 24 h notional volume.59Outliers (> 2 % from the weighted median with ≥ 3 candidates) are excluded. Output: weighted median, dispersion (bps),60independent-family count, freshness, confidence (agreement + redundancy + freshness + reliability, capped at 0.995), and the full61contribution list with inclusion reasons — exposed as **"Why this price?"** (`GET /v1/quotes/:id/provenance`).6263Rights guard: a canonical value built from any non-redistributable source is `INTERNAL_ONLY` and withheld from public responses64(metadata still visible). `PUBLICLY_REDISTRIBUTABLE` in `market-model` is the single source of truth.6566## Market hours (core/calendar.ts)6768Exchanges carry IANA time zone, sessions (regular/pre/post, multi-session for Asia, weekday sets for Gulf markets, `continuous` for69crypto venues) and holidays (seed + HTML connector). States PRE/OPEN/POST/CLOSED with DST handled by Intl. Used by adaptive polling,70consensus freshness, MARKET_OPEN/CLOSE events, session resets and the world map.7172## Events (core/events.ts)7374Derived from canonical quotes with per-instrument adaptive baselines (EWMA of tick moves): PRICE_CHANGE (≥ max(0.5 %, 8× typical75tick)), SESSION_HIGH/LOW (after history, ≥ 0.1 % improvement, cooldown), VOLATILITY_SPIKE (30-tick realized vol ≥ 4× baseline),76SOURCE_DIVERGENCE (> 100 bps between included sources, hourly per instrument). Connector-proposed events (halts, filings, document changes) and system events77(SOURCE_FAILURE, SCHEMA_DRIFT, MARKET_OPEN/CLOSE) go through the same dedupe (fingerprint, 24 h) and confirmation counting.7879## Resilience8081- Connector failures: jittered exponential backoff, FAILED after 3 consecutive poll failures (SOURCE_FAILURE event), auto-restart.82- Schema drift: structural fingerprint per payload kind; new shapes are recorded (`connector_schema_changes`); repeated shapes that83  normalize to nothing pause the connector (SCHEMA_DRIFT event) rather than emitting wrong values.84- Backpressure: bounded observation queue (200 k) — only REALTIME ticks are dropped under pressure, never events; batched inserts.85- Graceful shutdown flushes writers, quotes, bars, events, raw archive.86- Split roles are possible (`MA_ROLE=api|worker`) — the bus is in-process today; topic names match the spec for a future NATS/Redpanda.8788## Security8990Admin API behind `x-ma-admin-token` (timing-safe compare); public REST rate-limited per IP (20 req/s, burst 300; web pages and streams exempt); security headers; strict91parameter validation (zod); SSRF guard on discovery URLs (private ranges, localhost, cluster hostnames, DNS resolution check);92secrets redacted in logs and raw archives; no stack traces in responses.93