# Market Atlas — architecture (v0.1, 2026-09-12)

Market Atlas is a market observation infrastructure: a connector network observes public/official sources, normalizes their outputs into one observation model, resolves instruments, builds a multi-source consensus, derives events, keeps the history, and redistributes the result through one web app, a REST API and a stream.

 sources (WS / XHR / official APIs / RSS / XML / HTML / bulk files)
   └─ connectors/            ConnectorDefinition (metadata + rights + normalize())     [L0 raw archived, sampled]
        └─ ConnectorManager  lifecycle · adaptive polling · rate limits · circuit breaker · schema drift
             └─ Pipeline     validate → resolve instrument → Observation (L3) → ObservationWriter (Postgres, daily partitions)
                  ├─ ConsensusEngine   weighted median · source families · outliers · freshness → CanonicalQuote
                  │     └─ QuoteStore  hot cache + debounced upsert · bus "canonical.quote"
                  ├─ EventEngine       price change · session high/low · volatility · divergence · halts · filings · market open/close
                  ├─ BarAggregator     live 1m bars → SQL rollups 5m/15m/1h/1d (+ backfill bars from historical connectors)
                  └─ HealthEngine      state · latency · parse rate · reliability score (0–100) · minute snapshots
 MarketBus (in-process, topic names = spec) → API (Fastify): REST /v1 · WebSocket /v1/stream · SSE /v1/sse · Prometheus /v1/metrics
 apps/web (Next.js) consumes the API and the stream; in production the API process is the public edge and proxies to Next.

# Packages

# Data layers

• L0 raw — MA_DATA_DIR/raw/<day>/<connector>/<sha>.json.gz (redacted, content-addressed). Polled/bulk payloads always; streaming frames sampled (MA_RAW_SAMPLE_RATE). raw_ref on observations points back. ma replay <day> re-normalizes offline.
• L1/L2 — connector normalize() (pure, deterministic, no network) → NormalizedObservation with source symbol + instrument hint, rights & real-time status, timestamp trust.
• L3 — observations (partitioned by received_at day; fingerprint dedupes reconnect replays; normalizer_version; REALTIME streaming ticks are persisted at most once per source/instrument/field per MA_TICK_PERSIST_INTERVAL_MS = 2 s while consensus/events see every tick). Retention MA_OBSERVATION_RETENTION_DAYS then gzip NDJSON archive in MA_DATA_DIR/archive (never silently deleted; observation_archives index).
• L4 — canonical_quotes, bars, market_events, filings, connector_health, fact_changes, document_snapshots.

# Source Mesh v2 (2026-09-13)

One instrument → many observers → one canonical market state. Every observation carries an observation_type (TRADE, MID, QUOTE, INDEX_VALUE, INDICATIVE, STABLECOIN_PROXY, DERIVED, OFFICIAL_FIX, REFERENCE_RATE, SETTLEMENT, EOD_CLOSE, NAV) that maps to a comparability class (LIVE / FIX / EOD). Only comparable observations are compared: an ECB fixing is never measured against a live market, last week's close is never "divergent" from yesterday's. Roles: real markets vote; stablecoin/derived proxies confirm (reduced weight, never counted as families); restricted-rights sources (nasdaq.com) validate only (validation_only, never voting, never redistributed). core/coverage.ts scores every quoted instrument against a tier target (A ≥ 3 families / 5 observations; B ≥ 2 / 3; C 1 / 2; D official single source) and produces the source-expansion queue (GET /v1/coverage). core/lineage.ts infers shared upstreams statistically (identical values within 2 s over ≥ 120 aligned samples → similarity ≥ 0.92 → one family, source_lineage table). Live FX without keys: Kraken fiat markets (12 pairs, TRADE), Bitstamp EUR/USD & GBP/USD, Bitfinex EUR/GBP/JPY vs USD, plus stablecoin proxies (EUR/USDT, USDC/EUR, USDT/BRL…) on Binance, OKX, Coinbase, Bybit, Gate, KuCoin.

# Consensus (apps/api/src/core/consensus.ts)

Per instrument and field, the newest observation of each source is kept (out-of-order values ignored). A value is fresh within a window depending on its real-time class (REALTIME 15 s, DELAYED 30 min, INDICATIVE 1 h, END_OF_DAY 10 d; 4 d for live classes while the venue is closed — the last session value is reported as AT_CLOSE, never as live). Comparability class first (LIVE from real markets > FIX > EOD > live proxies alone), then temporal tolerance inside the class (live: 2 min; fixings/closes: same UTC day → temporal_mismatch). Weight = source reliability × timestamp trust × real-time class × freshness decay × official bonus (×1.6 for rates). One vote per source family (declared or statistically inferred). Crypto weights scale with the venue's share of 24 h notional volume. Outliers (> 2 % from the weighted median with ≥ 3 candidates) are excluded. Output: weighted median, dispersion (bps), independent-family count, freshness, confidence (agreement + redundancy + freshness + reliability, capped at 0.995), and the full contribution list with inclusion reasons — exposed as "Why this price?" (GET /v1/quotes/:id/provenance).

Rights guard: a canonical value built from any non-redistributable source is INTERNAL_ONLY and withheld from public responses (metadata still visible). PUBLICLY_REDISTRIBUTABLE in market-model is the single source of truth.

# Market hours (core/calendar.ts)

Exchanges carry IANA time zone, sessions (regular/pre/post, multi-session for Asia, weekday sets for Gulf markets, continuous for crypto venues) and holidays (seed + HTML connector). States PRE/OPEN/POST/CLOSED with DST handled by Intl. Used by adaptive polling, consensus freshness, MARKET_OPEN/CLOSE events, session resets and the world map.

# Events (core/events.ts)

Derived from canonical quotes with per-instrument adaptive baselines (EWMA of tick moves): PRICE_CHANGE (≥ max(0.5 %, 8× typical tick)), SESSION_HIGH/LOW (after history, ≥ 0.1 % improvement, cooldown), VOLATILITY_SPIKE (30-tick realized vol ≥ 4× baseline), SOURCE_DIVERGENCE (> 100 bps between included sources, hourly per instrument). Connector-proposed events (halts, filings, document changes) and system events (SOURCE_FAILURE, SCHEMA_DRIFT, MARKET_OPEN/CLOSE) go through the same dedupe (fingerprint, 24 h) and confirmation counting.

# Resilience

• Connector failures: jittered exponential backoff, FAILED after 3 consecutive poll failures (SOURCE_FAILURE event), auto-restart.
• Schema drift: structural fingerprint per payload kind; new shapes are recorded (connector_schema_changes); repeated shapes that normalize to nothing pause the connector (SCHEMA_DRIFT event) rather than emitting wrong values.
• Backpressure: bounded observation queue (200 k) — only REALTIME ticks are dropped under pressure, never events; batched inserts.
• Graceful shutdown flushes writers, quotes, bars, events, raw archive.
• Split roles are possible (MA_ROLE=api|worker) — the bus is in-process today; topic names match the spec for a future NATS/Redpanda.

# Security

Admin API behind x-ma-admin-token (timing-safe compare); public REST rate-limited per IP (20 req/s, burst 300; web pages and streams exempt); security headers; strict parameter validation (zod); SSRF guard on discovery URLs (private ranges, localhost, cluster hostnames, DNS resolution check); secrets redacted in logs and raw archives; no stack traces in responses.