SPB Git
spboucher.ai

spb/market-atlas

Public

Market Atlas v0.1.0 — connector infrastructure, consensus engine, API/stream, 14 connectors

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 12, 2026) Browse files

154 changed files +15,165 −0

154 changed files
Load all 154 files Unified Split
added .env.example +15 −0
@@ -0,0 +1,15 @@
1 +# Market Atlas — runtime configuration (copy to .env for local development)
2 +APP_ENV=development
3 +MA_SITE_URL=http://localhost:8390
4 +MA_API_HOST=127.0.0.1
5 +MA_API_PORT=8391 # production: web 8380 / api 8381
6 +MA_ROLE=all # all | api | worker
7 +DATABASE_URL=postgres://localhost:5432/market_atlas
8 +MA_DATA_DIR=./data # raw payload archive, backups, logs
9 +MA_ADMIN_TOKEN=change-me # x-ma-admin-token header for /v1/admin/*
10 +MA_LOG_JSON=0
11 +MA_CONTACT_EMAIL=contact@market-atlas.co # sent in User-Agent to official sources (SEC requires it)
12 +HFMD_API_KEY= # hfmarketdata.io key (own data lake) — optional, raises limits
13 +MA_DISABLED_CONNECTORS= # comma-separated connector ids to keep off
14 +MA_RAW_SAMPLE_RATE=0.02 # share of streaming frames archived at L0 (bulk/poll payloads are always archived)
15 +MA_OBSERVATION_RETENTION_DAYS=45
added .gitignore +19 −0
@@ -0,0 +1,19 @@
1 +node_modules/
2 +dist/
3 +.next/
4 +.next-*/
5 +apps/web/next-env.d.ts
6 +*.tsbuildinfo
7 +.env
8 +.env.*
9 +!.env.example
10 +data/
11 +/tmp/
12 +logs/
13 +.DS_Store
14 +.claude/
15 +deploy/.admin-token
16 +deploy/.hfmd-key
17 +deploy/rendered/
18 +apps/web/qa/screens/
19 +coverage/
added CLAUDE.md +209 −0
@@ -0,0 +1,209 @@
1 +# CLAUDE.md
2 +
3 +# Market Atlas
4 +
5 +**Production domain:** `https://www.market-atlas.co`
6 +**Product name:** Market Atlas
7 +**Project type:** Global financial market intelligence, market-data aggregation, live-feed reconstruction, source-observation, time-series, and market-event platform
8 +**Primary interface:** Web application
9 +**Secondary interfaces:** Public/internal APIs, WebSocket/SSE streaming endpoints, background ingestion workers, connector management console
10 +**Deployment target:** Self-hosted production infrastructure / cluster (MacLustr, node M2U64 via `mld` — see `docs/DEPLOY.md`)
11 +**Primary objective:** Build a global, continuously updating representation of financial markets using a heterogeneous network of market-data connectors, public web sources, official feeds where available, observable frontend data streams, filings, exchange publications, issuer pages, and additional legally usable sources.
12 +
13 +> Implementation state (v0.1.0, 2026-09-12): TypeScript monorepo — `packages/market-model`, `packages/connector-sdk`, `connectors/` (14 production connectors),
14 +> `apps/api` (engines, REST/WS/SSE, admin, CLI), `apps/web` (Next.js). Read `docs/ARCHITECTURE.md`, `docs/CONNECTORS.md`, `docs/API.md`, `docs/FRONTEND.md`, `docs/DEPLOY.md` before changing anything structural.
15 +
16 +---
17 +
18 +# 1. CORE VISION
19 +
20 +Market Atlas must become a live map of global financial markets. This is NOT simply a stock screener, a quote website, a Yahoo Finance / TradingView / Bloomberg clone, a static company database, or an API wrapper. Market Atlas should behave as an independent **market observation infrastructure**: it continuously observes thousands of market-related sources, normalizes their outputs, identifies financial entities, compares independent observations, detects changes and anomalies, creates its own canonical event stream, stores historical observations, and redistributes normalized information through the Market Atlas application.
21 +
22 +```text
23 +THE WEB / EXCHANGES / ISSUERS / DATA SOURCES → CONNECTOR NETWORK → RAW OBSERVATIONS → NORMALIZATION + RESOLUTION → CONSENSUS / VALIDATION → MARKET BUS
24 + ┌──────────────┼──────────────┐
25 + Live UI Historical DB APIs
26 + Alerts Analytics WebSocket
27 +```
28 +
29 +Market Atlas becomes more valuable as time passes: every captured observation adds to the historical dataset. Two simultaneous products: **the live market state** and **the historical market memory**.
30 +
31 +# 2. PRIMARY PRODUCT PRINCIPLE
32 +
33 +Market Atlas must not depend structurally on one external financial-data API. External APIs may be optional sources, fallbacks, validation layers, or licensed sources — never architectural single points of failure. Build a flexible **connector infrastructure** able to observe many source types (exchange websites, issuer IR sites, regulatory filings, public quote pages and dashboards, exchange status pages, official downloadable files, RSS/Atom, public JSON endpoints, XHR/fetch responses used by frontends, GraphQL, SSE, public WebSocket streams, HTML, embedded JSON, JavaScript state, CSV, XML, PDF, official FTP/SFTP where permitted, public bulk downloads and object storage, press-release feeds, market-status endpoints). Always retain where an observation came from. Never detach market information from provenance.
34 +
35 +# 3. LEGAL AND DATA-USAGE PRINCIPLE
36 +
37 +Technical accessibility does NOT imply redistribution rights. Market Atlas has a first-class data-rights and provenance system. Never bypass authentication, paywalls, CAPTCHAs or access controls; never impersonate private users, use leaked credentials, scrape private pages, access data requiring unauthorized sessions, reverse-engineer protected feeds for unauthorized redistribution, or intentionally evade source restrictions.
38 +
39 +Every connector declares its data status: `PUBLIC_OPEN, PUBLIC_ATTRIBUTED, PUBLIC_RESTRICTED_REDISTRIBUTION, OFFICIAL_OPEN_DATA, LICENSED, DELAYED, INDICATIVE, RESEARCH_ONLY, INTERNAL_ONLY, UNKNOWN, DISABLED`. Every field supports `realtime_status: REALTIME, DELAYED, END_OF_DAY, INDICATIVE, STALE, UNKNOWN`. Without redistribution rights a source may be used internally where legally appropriate (validation, quality analysis) but its raw data must not be exposed publicly. This constraint is built into the architecture (`PUBLICLY_REDISTRIBUTABLE`, `publicQuote` withholding), not treated as documentation.
40 +
41 +# 4. PRODUCT PHILOSOPHY
42 +
43 +Make financial markets feel alive: Bloomberg infrastructure + FlightRadar24-like global visibility + GitHub activity feed + financial search engine + real-time event bus + market observatory — visually modern, approachable, fast, cleaner than a terminal. Avoid old-fashioned terminal aesthetics, excessive neon, fake "AI" gimmicks. Communicate scale, motion, data density, reliability, precision, geography, connectivity, provenance, temporal change.
44 +
45 +# 5. MAJOR PRODUCT SURFACES
46 +
47 +`/ /live /markets /stocks /etfs /indices /crypto /forex /commodities /bonds /rates /futures /options /exchanges /companies /countries /sectors /events /filings /earnings /dividends /splits /ipos /halts /insiders /news /sources /connectors /data-health /atlas /search /compare /api /developers /methodology /status`. Not every section ships in V1; the architecture must support them.
48 +
49 +# 6. HOMEPAGE
50 +
51 +"The world's markets, observed live." Hero: logo, universal search, live market status, instruments tracked, sources active, observations today, exchanges online, event throughput — **never hard-coded; metrics come from system telemetry** (`/v1/stats`). Then: Global Market Pulse (indices, crypto, FX, commodities, rates, top moves), Live Tape (real internal stream), Live Events (canonical events), World Market Map (open/closed, direction, activity), Source Health.
52 +
53 +# 7. THE LIVE FEED (`/live`)
54 +
55 +Global activity stream of market changes and events, filterable by asset class, country, exchange, sector, company, ticker, event type, magnitude, source confidence, time. Items come from canonical normalized events, never frontend hacks. Each item shows confidence and independent source count.
56 +
57 +# 8. CONNECTOR ARCHITECTURE
58 +
59 +The connector system is the core intellectual infrastructure — never a pile of scraping scripts. Standard interface (`packages/connector-sdk`): `metadata()`, optional `discover`, `start/stop` or `poll`, `normalize(raw)`, `health`. Metadata: id, name, organization, sourceType (WEBSOCKET, SSE, XHR, FETCH, GRAPHQL, HTML, RSS, CSV, XML, PDF, OFFICIAL_API, BULK_FILE), jurisdiction, rightsStatus, realtimeStatus, expectedLatency, supportsStreaming/Historical, symbols/exchanges/assetClasses, homepage, enabled, sourceFamily.
60 +
61 +# 9. CONNECTOR DISCOVERY ENGINE
62 +
63 +Automated (browser-based, Playwright/Chromium) discovery understands **how a page receives market information**: network requests, fetch/XHR, WebSocket, SSE, GraphQL, JSON/protobuf payloads, DOM mutations, embedded script state (Next/Nuxt/React hydration), downloadable files. It answers: does the page contain an instrument? how is the price obtained? which request changes with the price? symbol/price/volume/bid/ask/timestamp fields? delayed? multiplexed? subscription message? reconnect? public? authentication? restrictions? stability? Never auto-activate: `DISCOVERED → ANALYZED → VALIDATED → RIGHTS_REVIEW → STAGING → PRODUCTION`. (v0.1 ships a network-level prototype at `POST /v1/admin/discovery`.)
64 +
65 +# 10. NETWORK OBSERVATION
66 +
67 +Capture request URL/method/status/content type/headers/payload size/timing/initiator/resource type, WebSocket URL and frames, EventSource messages, JSON structure, field changes. Never store secrets; automatically redact cookies, Authorization, session identifiers, tokens, personal identifiers, CSRF secrets. Do not reproduce private sessions; discovery runs unauthenticated.
68 +
69 +# 11. AUTOMATIC PAYLOAD UNDERSTANDING
70 +
71 +Payloads vary enormously; normalize them to one canonical schema. Deterministic mappings first. LLMs may propose mappings, classify endpoints, generate connectors/docs/tests — production normalization must be reproducible. Never require an LLM per live tick.
72 +
73 +# 12. CANONICAL INSTRUMENT MODEL
74 +
75 +Global instrument master. Ticker alone never identifies an instrument: consider instrument_id, ticker, exchange, MIC, ISIN, FIGI (where legally usable), CUSIP (licensed), SEDOL, issuer, currency, country, asset class, security type. Example `eq_us_xnas_aapl`. Support symbol aliases (`BRK.B`, `BRK-B`, `BRK/B`, `BRK B`).
76 +
77 +# 13. ASSET CLASSES
78 +
79 +`EQUITY ETF ETN INDEX CRYPTO FOREX COMMODITY FUTURE OPTION BOND TREASURY INTEREST_RATE MUTUAL_FUND ADR GDR REIT WARRANT CFD OTHER`. Not all fields apply to all classes.
80 +
81 +# 14. OBSERVATION MODEL
82 +
83 +Atomic unit = observation: observation_id, instrument_id, field, value, currency, source_id, connector_id, source_timestamp, received_at, latency_ms, rights_status, realtime_status, confidence, raw_ref. Fields: `LAST_PRICE OPEN HIGH LOW PREVIOUS_CLOSE BID ASK BID_SIZE ASK_SIZE VOLUME VWAP MARKET_CAP CHANGE CHANGE_PERCENT OPEN_INTEREST YIELD RATE NAV 52_WEEK_HIGH 52_WEEK_LOW`.
84 +
85 +# 15. RAW DATA RETENTION
86 +
87 +Never discard raw input immediately. Layers: L0 RAW · L1 PARSED · L2 NORMALIZED · L3 CANONICAL · L4 DERIVED — debuggable, reprocessable.
88 +
89 +# 16. MARKET CONSENSUS ENGINE
90 +
91 +Sources disagree; never overwrite one price with the newest request. Weighted consensus using freshness, historical reliability, observed delay, source type, official status, error/disconnect rates, cross-source agreement, timestamp quality, coverage, known delay. Never conceal that a canonical value is an aggregation; expose methodology.
92 +
93 +# 17. SOURCE RELIABILITY SCORE
94 +
95 +Dynamic 0–100 operational score per source (availability 24h/30d, latency p50/p95, freshness, message rate, parse success, schema-change rate, reconnects, outlier rate, consensus agreement, timestamp integrity, coverage). It measures operational reliability, not financial truth.
96 +
97 +# 18. SOURCE DIVERGENCE
98 +
99 +Detect disagreement, mark stale sources, emit `SOURCE_DIVERGENCE`; useful internally and through data-health tools.
100 +
101 +# 19. MARKET EVENT ENGINE
102 +
103 +Transform noisy ticks into meaningful events: `PRICE_CHANGE SESSION_HIGH SESSION_LOW PRICE_BREAKOUT PRICE_GAP VOLUME_SPIKE VOLATILITY_SPIKE SPREAD_WIDENING SPREAD_COMPRESSION LIQUIDITY_CHANGE MARKET_OPEN MARKET_CLOSE TRADING_HALT TRADING_RESUME CORPORATE_ACTION DIVIDEND_DECLARED DIVIDEND_CHANGED STOCK_SPLIT EARNINGS_RELEASE GUIDANCE_CHANGE FILING_PUBLISHED INSIDER_TRANSACTION IPO_STATUS_CHANGE INDEX_CONSTITUENT_CHANGE RATING_CHANGE SOURCE_FAILURE SOURCE_RECOVERY SOURCE_DIVERGENCE`. Events carry id, type, instrument_ids, entity_ids, timestamp, severity, confidence, source_count, supporting observations, metadata.
104 +
105 +# 20. EVENT DEDUPLICATION
106 +
107 +Eleven sources publishing one announcement = one canonical event (entity, type, time window, semantic similarity, identifiers, references); confirmations increment source count.
108 +
109 +# 21–23. TEMPORAL DATABASE, AGGREGATION, COMPRESSION
110 +
111 +PostgreSQL for canonical metadata (instruments, exchanges, connectors, sources, events, configuration); time-series storage fit for purpose (Timescale/ClickHouse at scale — v0.1 uses daily-partitioned Postgres tables with gzip archive tiers); Redis for hot cache/pub-sub when needed; object storage for raw payloads, bulk files, PDFs, cold partitions. Canonical OHLCV bars 1s…1mo, reproducible from source observations. Columnar/delta/dictionary/ZSTD compression, partitioning, HOT (7 d) / WARM (90 d) / COLD tiers. Downsample strategically; never blindly delete history.
112 +
113 +# 24. INTERNAL MARKET BUS
114 +
115 +All normalized events enter one bus (NATS JetStream / Kafka-Redpanda / Redis Streams for small deployments; in-process today with spec topic names). `CONNECTORS → INGEST BUS → NORMALIZERS → CANONICAL BUS → { DB writers, WebSocket gateway, event engine, analytics, anomaly detection, monitoring }`. No connector → frontend coupling.
116 +
117 +# 25–27. STREAMING AND REST
118 +
119 +Own normalized stream (`wss://…/v1/stream`): subscribe/unsubscribe, heartbeat, reconnect, sequence ids, resume tokens, backpressure, rate limits, compression, batching; SSE for simple clients; REST `/v1/instruments, /v1/instruments/{id}, /v1/quotes/{symbol}, /v1/history/{symbol}, /v1/events, /v1/exchanges, /v1/markets, /v1/sources, /v1/status`. Never expose restricted observations without rights.
120 +
121 +# 28–31. COMPANY, EXCHANGE, COUNTRY PAGES, WORLD MAP
122 +
123 +Company/instrument page: live quote, market state, chart, volume, spread, market cap, performance, corporate actions, filings, earnings, dividends, related instruments, source confidence, events, profile, plus the unique **Data Provenance** panel (expandable, not overwhelming). Exchange page: status, local time, country, timezone, MIC, instruments tracked, advancers/decliners, volume, volatility, largest moves, events, halts, source health. Country page: exchanges, indices, largest companies, market cap, currency, bond yields, rates, commodity exposure, events, sector breakdown, hours — an atlas. World map: exchanges/countries/market centers with hours, volatility, activity; time-aware (New York activates at 09:30, Asia lights up as Europe closes). Informative, not decorative.
124 +
125 +# 32–34. SEARCH, COMPARE, MARKET MEMORY
126 +
127 +Universal hybrid search (exact identifiers, prefix, aliases, fuzzy, semantic) grouped by stocks, companies, indices, crypto, FX, commodities, bonds, exchanges, countries, events. `/compare` across asset classes with normalized performance, correlation, volatility, drawdown, event overlays. Market Replay: reconstruct prices, events, direction, volatility, filings, news, exchange status at time T.
128 +
129 +# 35–37. FILINGS, CORPORATE ACTIONS, MARKET HOURS
130 +
131 +Official regulators (SEC, SEDAR+, others, exchange announcements) → company, form type, timestamp, URLs, period, structured fields; never hallucinate contents; link every fact to its document. Corporate actions (dividends, splits, mergers, spin-offs, symbol changes, delistings, listings, rights, buybacks, special distributions) must adjust history correctly. Market-hours engine: time zones, DST, holidays, half days, special closures, overnight/pre/post/futures sessions; states `PRE OPEN HALTED CLOSED POST AUCTION UNKNOWN`; never hard-code 09:30–16:00.
132 +
133 +# 38–42. FRESHNESS, LATENCY, HEALTH, SCHEMA DRIFT, SELF-HEALING
134 +
135 +Every visible quote has freshness ("Updated 320 ms ago" / "Last update 4m 12s ago — data may be delayed"); never present stale as live. Track source/connector/frontend timestamps → source, ingestion, processing, delivery latency. Connector heartbeats and states `HEALTHY DEGRADED STALE RECONNECTING FAILED PAUSED DISABLED`; exponential backoff with jitter; no request storms. Schema fingerprints detect `SCHEMA_DRIFT`; pause unsafe normalization instead of producing wrong prices; notify admin. Self-healing workflow (replay → detect → discovery → proposed mapping → tests → compare → candidate → approval); an LLM never silently deploys production code.
136 +
137 +# 43–50. SDK, FIXTURES, ETHICS, EFFICIENCY, LIMITS, POLLING, CHANGE DETECTION
138 +
139 +Reusable SDK (`defineConnector`, HTTP, browser, WebSocket, SSE, retry, rate limiting, normalization, symbol resolution, timestamp parsing, logging, metrics, fixtures, tests). Every connector stores sample payloads; tests never touch production sources. Respectful crawling: host rate limits, connection limits, identifying user agent, caching, ETag / If-Modified-Since, backoff, robots awareness. Never amplify requests (one stream or bulk endpoint → many instruments; never one socket per symbol when multiplexing exists). Central rate limiter (token/leaky bucket, adaptive) keyed by source/host/connector/route/credential; connectors never sleep on their own. Adaptive polling (market state, activity, change frequency, constraints, importance, ETag). Content fingerprints (DOM/section/semantic/document/field hashes) trigger processing only on change.
140 +
141 +# 51–57. DOCUMENTS, AI, DERIVED METRICS, ANOMALIES, QUALITY, PROVENANCE GRAPH
142 +
143 +Document pipeline: download → hash → store raw → parse → metadata → classify → resolve company → extract facts → verify → index; keep the original. AI enhances (discovery, schema interpretation, classification, summarization, entity resolution, explanations, natural-language search) but never invents prices, trades, filings, market cap, exchange status. Derived metrics (returns, volatility, moving averages, drawdown, z-scores, relative volume, spread, breadth, correlation, beta, momentum, ATR, VWAP) are labelled derived. Statistically grounded anomaly detection with historical baselines. Every canonical datapoint may carry quality/confidence/source_count/freshness/dispersion/validation flags. Long-term provenance graph: Instrument → Observation → Connector → Source → Organization → URL; Company → Filing → Regulator → Document.
144 +
145 +# 58–61. ADMIN CONSOLE, CONNECTOR PAGES, DISCOVERY UI, STATUS
146 +
147 +`/admin` (strongly secured): overview, connectors, sources, instruments, entity resolution, observations, data quality, divergence, failed jobs, schema changes, events, storage, workers, streams, API, security, legal/data rights, logs. Per connector: status, type, host, rights, real-time status, last message/error, rates, latency, coverage, schema fingerprint, version, health history; actions pause/resume/restart/test/discover/fixtures/schema/sanitized payloads; never expose secrets. Discovery UI: submit URL → observed requests/WebSockets/SSE/JSON endpoints → likely quote source + field mapping → "Generate connector candidate" (not production). Public `/status` without infrastructure details.
148 +
149 +# 62–75. TECHNOLOGY, SERVICES, DEPLOYMENT, PROXY, FAILOVER, OBSERVABILITY, LOGGING, SECURITY, PERFORMANCE, MOBILE
150 +
151 +Frontend: Next.js, React, TypeScript, Tailwind, light state, WebSocket client, MapLibre/D3 for maps, ECharts/lightweight-charts/D3 by need; performance-driven library choices. Backend: TypeScript/Node for application services, Python for analytics/AI, Go/Rust only for proven high-throughput needs, PostgreSQL, ClickHouse/Timescale, Redis, NATS/Redpanda, Playwright; no new languages without clear reason. Services (web, api, stream-gateway, connector-manager/workers, discovery, normalization, entity-resolution, event-engine, analytics, database-writer, document-worker, scheduler, admin-api, health-monitor) start as a modular monolith and split when load justifies. Containerize where possible, health checks, safe restarts, no hand-started shells (MacLustr: PM2 via `mld`). TLS everywhere via reverse proxy (Caddy on the MacLustr Tunnel); no internal ports exposed. Workers support leases/locks/heartbeats/reassignment. Observability: Prometheus/Grafana/OpenTelemetry, structured JSON logs (never passwords, tokens, cookies, keys), metrics for observations/s, events/s, connector states, WebSocket connections, stream lag, latencies, queue depth, resources, storage growth. Security: HTTPS, secure headers, strict CORS, validation, parameterized SQL, rate limits, secret manager, admin MFA, least privilege, segmentation, dependency scanning, audit logs; isolated browser workers (no secrets, no DB credentials, temporary profiles); SSRF protection (localhost, private ranges, internal hostnames, metadata endpoints, redirects); API keys/quotas/analytics/revocation; service auth internally. Web performance: fast first paint, streaming updates without re-render storms, virtualized feeds, efficient charts, edge caching, SSR for public pages, granular subscriptions. Mobile-first surfaces (search, watch view, overview, instrument page, live feed, events, chart, world markets): bottom navigation, swipe-friendly cards, compact tape, sticky quote header, responsive charts, accessible touch targets, no horizontal overflow.
152 +
153 +# 76–84. DESIGN SYSTEM, MODES, COLOR, CHARTS, LIVE UI, SEO, SITEMAPS, LICENSING, DISCLAIMER
154 +
155 +Premium, dense but breathable, institutional, modern, high-information; not boxy, cartoonish or crypto-casino; hierarchy via spacing, typography, dividers, alignment, subtle surfaces — not isolated rounded cards. Dark and light modes first-class; avoid pure black; never red/green as the only signal. Color semantics: positive, negative, neutral, stale, warning, failure (+ icons/text). Charts: 1D 5D 1M 3M 6M YTD 1Y 5Y MAX, live intraday, line/area/candlestick/volume/comparison, no fake interpolation. Live UI: subtle directional flash, batch updates at 10–20 refresh cycles/s max. SEO: crawlable public pages with title/description/canonical/OpenGraph/structured data; no thin pages; segmented sitemaps (stocks, companies, exchanges, countries, indices, crypto, commodities). `/methodology`, `/data-sources`, `/licensing` explain methodology, delays, provenance, derived calculations, limitations, not-investment-advice, venue differences, licensing. Disclaimer: "Market Atlas is an informational market-data and analytics platform. Information may be delayed, indicative, incomplete, or sourced from third parties. Market Atlas is not an exchange, broker, investment adviser, or execution venue." (final legal review before commercial launch).
156 +
157 +# 85–94. TESTING, LOAD, REPLAY, MIGRATIONS, BACKUPS, RETENTION, REPROCESSING, VERSIONING
158 +
159 +Tests: unit, integration, connector (valid/missing/null/unexpected schema/stale timestamp/invalid symbol/duplicate/disconnect/reconnect/rate-limit/server error), schema, normalization, entity resolution, WebSocket, API, database, frontend, end-to-end, load, security. Consensus scenarios: all agree, one stale, one outlier, two clusters, single source, differing timestamps, market closed. Load: 100K obs/s, 10K clients, 100K subscriptions, 1M cached instruments, large historical queries — scale progressively. Internal replay feeds stored observations through normalizer/consensus/events. Versioned forward-only migrations; never edit production tables by hand. Automated backups (Postgres, metadata, connector config, object storage replication) — a backup never restored is not trusted. Configurable retention by source/asset class/field/resolution/rights (raw WS frames 7 d; normalized ticks permanent compressed; 1s/1m/1d bars permanent). Reprocess raw → canonical v2 with lineage; canonical datasets carry `normalizer_version`, `consensus_version`.
160 +
161 +# 95–105. IDENTIFIERS, SOURCES DIRECTORY, DATA HEALTH, "WHY THIS PRICE?", COVERAGE, PRIORITY, TIERS, MULTI-SOURCE, CORRELATION, LINEAGE, DELAY DETECTION
162 +
163 +Stable public ids (`eq_us_xnas_aapl`, `crypto_btc_usd`, `fx_eur_usd`, `index_us_spx`), never auto-increment ids. `/sources` directory by category with status/coverage/frequency/latency/rights/last observation — no endpoint details. `/data-health`: sources online, global freshness, markets covered, consensus, incidents — real metrics only. **"Why this price?"** on every instrument: canonical price, independent observations, dispersion, newest/oldest included, weights, reasons. Market Atlas coverage score per country/exchange (observation coverage, not market quality). Connector priority engine (market cap represented, demand, coverage gap, redundancy, legal accessibility, reliability, cost). Tier 1: major exchanges, regulators, large-cap IR, major public quote sources, crypto exchanges with public feeds, central banks, government debt, commodity references; Tier 2: regional exchanges, ETF issuers, index providers, economic data, specialized sites; Tier 3: small exchanges, local portals, specialized classes, regional announcements. Target ≥ 3 independent observations for major instruments (5–10+ for the most important). Source families: websites sharing an upstream are not independent; weight independence; infer lineage from timestamps/identical payloads/rounding/documentation/attribution — never assert hidden commercial relationships without evidence. Estimate delay against trusted references (0–2 s, 15 s, 1 min, 15 min, 20 min, EOD) and label estimates as such.
164 +
165 +# 106–121. MARKET CLOCK, TOPICS, IDEMPOTENCY, ORDERING, CLOCKS, TIMESTAMP TRUST, ENTITY & EXCHANGE GRAPHS, CURRENCY, LATER FEATURES, DEVELOPER PORTAL, API CONVENTIONS
166 +
167 +Global market clock (New York, London, Tokyo, Hong Kong, Sydney) with states. Topics: `raw.{source} normalized.quote normalized.trade normalized.event canonical.quote market.event connector.health source.divergence filing.detected corporate_action.detected`. Idempotent ingestion via fingerprints (source + instrument + timestamp + value + type); no duplicate rows on reconnect replay. Out-of-order messages: keep source sequence/source timestamp/receive timestamp. NTP-synchronized clocks. Timestamp trust: exchange / source / connector / unknown → influences confidence. Company ≠ security (common stock, ADR, preferred, bonds, subsidiaries). Exchange graph (operator, venue, MIC, country, timezone, asset classes, sessions). Store original currency; converted values record FX source/time. Later: watchlists, saved searches, alerts (price, volume, volatility, filing, earnings, dividend, halt, divergence, highs/lows, events), dashboards, API keys, exports, notebooks — the public atlas must be useful first. No brokerage in V1 (no execution, custody, deposits, leverage). `/developers`: REST, WebSocket, SSE, schemas, examples, SDKs (JS, Python, Go), status, rate limits. Versioned `/v1/`, no silent breaking changes; envelopes `{data, meta{request_id, timestamp, data_status}}`; structured errors `{error{code, message}}`, no stack traces.
168 +
169 +# 122. PROJECT STRUCTURE
170 +
171 +```text
172 +market-atlas/
173 +├── apps/ web · api (engines, stream gateway, admin, worker roles)
174 +├── packages/ market-model · connector-sdk (+ future: schemas, database, streaming, logging, security, ui)
175 +├── connectors/ one directory per connector (index.ts, fixtures/, README.md)
176 +├── infra/ migrations (+ docker, monitoring, deployment as they come)
177 +├── deploy/ mld manifest + render script (MacLustr)
178 +├── docs/ ARCHITECTURE · CONNECTORS · API · FRONTEND · DEPLOY
179 +├── scripts/ tests/
180 +└── CLAUDE.md
181 +```
182 +
183 +# 123. CODING RULES FOR CLAUDE
184 +
185 +1. Read this file before major architecture changes. 2. Inspect existing code before creating duplicate systems. 3. Prefer reusable infrastructure over one-off scripts. 4. Do not break existing connectors. 5. No hard-coded secrets. 6. No hard-coded production IPs. 7. No fake market data. 8. Never silently replace real data with mock values. 9. Strict TypeScript. 10. Validate external payloads. 11. Treat all external content as untrusted. 12. Add meaningful tests. 13. Keep the UI responsive. 14. Preserve data provenance. 15. Never present unknown freshness as real time. 16. Never remove data-rights metadata. 17. Prefer graceful degradation. 18. Add monitoring to new ingestion systems. 19. Document unusual source behaviour. 20. Keep connector code isolated from core business logic.
186 +
187 +# 124–127. WHEN ADDING A CONNECTOR / SCHEMA / FIELD / AI
188 +
189 +Connector = implementation + metadata + rights classification + normalization mapping + fixtures + unit tests + health check + documentation (no undocumented connectors). Schema change = inspect database, API, stream schemas, frontend consumers, tests, historical processing — no silent breaking changes. New market field = canonical name, type, unit, applicable asset classes, nullability, source-timestamp handling, display formatting. Adding AI: if deterministic code can solve it, use deterministic code; AI only where interpretation is needed; no LLM dependencies for basic parsing.
190 +
191 +# 128–137. PERFORMANCE, QUERIES, CACHE, CONTRACTS, EVENT SCHEMA, DISASTER MODES, CIRCUIT BREAKERS, SHUTDOWN, BACKPRESSURE, PRIORITY
192 +
193 +Never solve scale only with bigger servers — first inspect duplication, query structure, indexes, cacheability, batching, serialization, compression, memory copies, re-renders. Heavy historical queries: partition pruning, indexes, no `SELECT *`, limits, pagination, downsampling. Cache instrument/exchange metadata, latest quotes, market status, popular pages, aggregates — never user-specific or stale critical values incorrectly. Strongly versioned contracts (zod / JSON Schema / Protobuf / Avro); explicit bus contracts; event schema `{version, id, type, instrument_id, timestamp, confidence, severity, data, sources}`. Define degraded behaviour when Redis/ClickHouse/source/gateway/worker/browser/network/schema fail (e.g. historical DB down → live stream continues, charts unavailable). Circuit breakers on failing sources. Graceful shutdown: stop new jobs, flush, commit, release leases, close, exit. Bounded queues, batch writes, disk-backed queues where needed, priority tiers, drop policy only for expendable data — never silently drop canonical events. Priorities: P0 halts/critical events, P1 canonical quotes, P2 ticks, P3 secondary analytics, P4 reprocessing.
194 +
195 +# 138–151. ONBOARDING PIPELINE, EXPANSION, HISTORY, ARCHIVE, KNOWLEDGE GRAPH, CONSTITUENTS, DIFFS, EXPLORER, "MARKET CHANGES", BREADTH, ACTIVITY INDEX, PRECISION, PREDICTIONS
196 +
197 +Automated onboarding (candidate → metadata → discovery → rights → fixtures → mapping → validation → comparison → staging → observation → production). Long-term source discovery jobs feed a review queue; no unlimited crawling of arbitrary domains. Store source change history (versions, schema, endpoints, availability, delay class, rights class) and allow historical source-quality queries. Market observation archive: "what did the observable financial web say at time T?". Knowledge graph (Company, Instrument, Exchange, Country, Sector, Executive, ETF, Index, Filing, Corporate Action, Event, Source; LISTED_ON, HEADQUARTERED_IN, MEMBER_OF_INDEX, HOLDS, ISSUED_BY, FILED, ANNOUNCED, OBSERVED_BY). Index membership with effective dates. Diff log for slowly changing facts (old/new/detected/effective/source). Future `/explore` data laboratory. "Market changes" product (what changed in 1 min / 5 min / 1 h / 1 d: instruments > 1 %, filings, halts, new highs, source outages). Global breadth engine (advancers, decliners, highs/lows, median/weighted return, sector/country breadth, regional scopes). Internal activity score for ranking what is happening now — never a prediction. No fake precision (no 99.999 % confidence without justification; transparent methodology). Core product = observation, aggregation, historical intelligence, analytics — not guaranteed predictions or recommendations.
198 +
199 +# 152–158. V1 DEFINITION, CONNECTOR GOAL, PHASES, SUCCESS METRICS, NORTH STAR
200 +
201 +V1: homepage, universal search, market overview, live feed, equities, crypto, major FX, major indices, instrument pages, charts, basic map, sources directory, data health, connector framework, 5–20 strong production connectors, health monitoring, canonical normalization, basic consensus, PostgreSQL, time-series storage, cache, WebSocket gateway, admin console. Prove diversity: WebSocket, XHR/fetch, official source, HTML change detection, regulatory/filing source, bulk structured source. Phase 2: 50–100 connectors, multiple exchanges, corporate actions, filings, world map, consensus, country pages, advanced events. Phase 3: hundreds/thousands of connectors, discovery automation, global instrument graph, replay, developer APIs, advanced analytics, institutional monitoring. Phase 4: answer *what is happening, where, how fast, which sources confirm it, how reliable, what happened before, what else moved*. Metrics: active connectors, uptime, instruments covered, observations/day, median freshness, consensus coverage, quality incidents, historical depth, stream latency; product: searches, views, live-feed engagement, return users, API usage. **North star: high-quality, provenance-backed market observations successfully processed.**
202 +
203 +# 159–166. BRAND, STANDARD, THE MOST IMPORTANT RULE, MINDSET, IMPLEMENTATION ORDER, FIRST ENTITIES, NO 1,000 SCRAPERS, EXPECTATION
204 +
205 +Brand: **Market Atlas**, `www.market-atlas.co` — "The live map of global markets." Every feature reinforces GLOBAL, LIVE, HISTORICAL, VERIFIABLE, MULTI-SOURCE, CONNECTED, SEARCHABLE, FAST; otherwise reconsider it. **The proprietary asset is not the website**: it is connectors + normalization + entity graph + source reliability + consensus + historical observations + market events; the frontend is the best interface to that system. Build for 100,000+ instruments, 1,000+ sources, hundreds of millions of observations/day, years of history, thousands of stream consumers — incrementally, without premature distributed complexity. Implementation order: audit → structure → canonical schemas → DB models → connector SDK → rights metadata → representative connectors → ingestion bus → normalization → resolver → current-state cache → canonical quotes → WebSocket gateway → historical persistence → health metrics → consensus → REST → homepage → instrument page → live feed → admin dashboard → discovery prototype → world map → observability → load test/harden. First entities: Instrument, Company, Exchange, Country, Currency, Source, Connector, ConnectorVersion, Observation, CanonicalQuote, MarketEvent, CorporateAction, Filing, SourceHealth, ConnectorHealth, SymbolAlias, TradingSession. Strategy: excellent connector infrastructure → validate with 10 heterogeneous sources → improve → 50 → automate onboarding → hundreds → thousands; twenty excellent connectors beat a thousand fragile scripts. Claude behaves as a senior systems/data/quant-data/security/product engineer: inspect what exists, preserve working behaviour, smallest coherent architectural improvement, implement rather than describe, test, verify production compatibility, update documentation, leave the repository deployable. Never substitute placeholders for requested real functionality unless an upstream dependency makes it unavoidable; when a source cannot legally or technically provide real-time information, label it accurately.
206 +
207 +# END GOAL
208 +
209 +Market Atlas should function like a live nervous system for financial markets: thousands of independent sources emit observations; Market Atlas listens, resolves what they refer to, measures their reliability, compares them, records them, detects what changed, remembers what happened, and exposes the resulting market state through one unified global interface — **https://www.market-atlas.co**.
added README.md +25 −0
@@ -0,0 +1,25 @@
1 +# Market Atlas
2 +
3 +**The live map of global markets** — https://www.market-atlas.co
4 +
5 +Market Atlas observes thousands of market-related sources (exchange feeds, official statistics, regulators, public quote pages,
6 +directories), normalizes every observation with its provenance and data rights, resolves it to a canonical instrument, compares
7 +independent sources into a consensus value, detects what changed, remembers everything, and redistributes the result through one
8 +web application, a REST API and a WebSocket/SSE stream.
9 +
10 +```
11 +pnpm install
12 +cp .env.example .env # Postgres URL, admin token, contact e-mail, optional HFMD_API_KEY
13 +pnpm migrate && pnpm seed # schema + reference data (countries, exchanges, holidays)
14 +pnpm dev:api # Fastify + workers + connectors on :8391 (roles: MA_ROLE=all|api|worker)
15 +pnpm dev:web # Next.js on :8390 (rewrites /v1 → API)
16 +pnpm test · pnpm typecheck # vitest (model, SDK, connectors, consensus, calendar) · tsc
17 +pnpm ma connectors | run <id> | replay <day> | status | archive | backup
18 +```
19 +
20 +Repository map: `packages/market-model` (canonical vocabulary & types) · `packages/connector-sdk` · `connectors/` (14 production
21 +connectors with fixtures and READMEs) · `apps/api` (engines, API, stream, admin, CLI) · `apps/web` (Next.js) · `infra/migrations` ·
22 +`docs/` (ARCHITECTURE, CONNECTORS, API, FRONTEND, DEPLOY) · `deploy/` (mld manifest) · `CLAUDE.md` (product & engineering charter).
23 +
24 +Market Atlas is an informational market-data and analytics platform. Information may be delayed, indicative, incomplete, or sourced
25 +from third parties. Market Atlas is not an exchange, broker, investment adviser, or execution venue.
added apps/api/build.mjs +22 −0
@@ -0,0 +1,22 @@
1 +// Bundles the API (and the workspace packages it imports) into dist/ with esbuild.
2 +// Third-party dependencies stay external (resolved from node_modules at runtime).
3 +import { build } from "esbuild";
4 +import { readFileSync } from "node:fs";
5 +
6 +const pkg = JSON.parse(readFileSync(new URL("./package.json", import.meta.url), "utf8"));
7 +const external = Object.keys(pkg.dependencies).filter((d) => !d.startsWith("@market-atlas/"));
8 +
9 +await build({
10 + entryPoints: ["src/main.ts", "src/cli.ts"],
11 + outdir: "dist",
12 + bundle: true,
13 + platform: "node",
14 + format: "esm",
15 + target: "node22",
16 + sourcemap: true,
17 + external,
18 + logLevel: "info",
19 + banner: {
20 + js: "import { createRequire as __createRequire } from 'node:module'; const require = __createRequire(import.meta.url);",
21 + },
22 +});
added apps/api/package.json +33 −0
@@ -0,0 +1,33 @@
1 +{
2 + "name": "@market-atlas/api",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "scripts": {
7 + "build": "node build.mjs",
8 + "dev": "node --env-file=../../.env --import tsx src/main.ts",
9 + "start": "node dist/main.js",
10 + "cli": "node --env-file-if-exists=../../.env --import tsx src/cli.ts",
11 + "migrate": "node --env-file-if-exists=../../.env --import tsx src/cli.ts migrate",
12 + "seed": "node --env-file-if-exists=../../.env --import tsx src/cli.ts seed"
13 + },
14 + "dependencies": {
15 + "@fastify/cors": "^11.0.1",
16 + "@fastify/http-proxy": "^11.1.2",
17 + "@fastify/websocket": "^11.1.0",
18 + "@market-atlas/connector-sdk": "workspace:*",
19 + "@market-atlas/connectors": "workspace:*",
20 + "@market-atlas/market-model": "workspace:*",
21 + "fastify": "^5.4.0",
22 + "pg": "^8.16.3",
23 + "pino": "^9.7.0",
24 + "ws": "^8.18.3",
25 + "zod": "^3.24.1"
26 + },
27 + "devDependencies": {
28 + "@types/pg": "^8.15.4",
29 + "@types/ws": "^8.18.1",
30 + "esbuild": "^0.25.9",
31 + "tsx": "^4.20.5"
32 + }
33 +}
added apps/api/src/api/public-view.ts +106 −0
@@ -0,0 +1,106 @@
1 +import type { CanonicalQuote, Instrument } from "@market-atlas/market-model";
2 +import { PUBLICLY_REDISTRIBUTABLE } from "@market-atlas/market-model";
3 +import { calendar } from "../core/calendar.js";
4 +
5 +/**
6 + * Public representation of a canonical quote. Values whose rights do not allow redistribution are
7 + * withheld (price nulled, `withheld: true`) while provenance/health metadata stays visible.
8 + */
9 +/**
10 + * @param marketState state of the instrument's venue (from the calendar) — when the venue is closed the
11 + * last session value is reported as AT_CLOSE instead of STALE.
12 + */
13 +export function publicQuote(q: CanonicalQuote | undefined, inst?: Instrument, marketState?: string | null) {
14 + if (!q) return null;
15 + const redistributable = PUBLICLY_REDISTRIBUTABLE.has(q.rightsStatus);
16 + const now = Date.now();
17 + const age = now - q.updatedAt;
18 + const staleFor = q.realtimeStatus === "REALTIME" ? 60_000 : q.realtimeStatus === "DELAYED" ? 45 * 60_000 : 4 * 86_400_000;
19 + if (marketState === undefined) {
20 + const ex = inst?.exchangeId ? calendar.exchange(inst.exchangeId) : undefined;
21 + marketState = ex && !ex.sessions.continuous ? calendar.state(ex.id, now) : null;
22 + }
23 + const closed = marketState != null && marketState !== "OPEN" && marketState !== "UNKNOWN";
24 + const recentSession = q.sourceTimestamp != null && now - q.sourceTimestamp < 4 * 86_400_000;
25 + let status: string;
26 + if (!redistributable) status = "WITHHELD";
27 + else if (q.realtimeStatus === "STALE") status = closed && q.price != null && recentSession ? "AT_CLOSE" : "STALE";
28 + else if (closed && recentSession && (q.realtimeStatus === "REALTIME" || q.realtimeStatus === "DELAYED")) status = "AT_CLOSE";
29 + else if (age > staleFor) status = "STALE";
30 + else status = q.realtimeStatus;
31 + const v = (x: number | null) => (redistributable ? x : null);
32 + return {
33 + instrument_id: q.instrumentId,
34 + symbol: q.symbol,
35 + name: inst?.name ?? null,
36 + asset_class: inst?.assetClass ?? null,
37 + exchange_id: inst?.exchangeId ?? null,
38 + country: inst?.country ?? null,
39 + price: v(q.price),
40 + open: v(q.open),
41 + high: v(q.high),
42 + low: v(q.low),
43 + previous_close: v(q.previousClose),
44 + change: v(q.change),
45 + change_percent: v(q.changePercent),
46 + volume: v(q.volume),
47 + bid: v(q.bid),
48 + ask: v(q.ask),
49 + currency: q.currency,
50 + source_count: q.sourceCount,
51 + dispersion_bps: q.dispersionBps,
52 + confidence: q.confidence,
53 + freshness_ms: q.freshnessMs == null ? null : q.freshnessMs + age,
54 + data_status: status,
55 + market_state: marketState ?? null,
56 + rights_status: q.rightsStatus,
57 + withheld: !redistributable,
58 + updated_at: new Date(q.updatedAt).toISOString(),
59 + source_timestamp: q.sourceTimestamp == null ? null : new Date(q.sourceTimestamp).toISOString(),
60 + session_high: v(q.sessionHigh),
61 + session_low: v(q.sessionLow),
62 + };
63 +}
64 +
65 +export type PublicQuote = NonNullable<ReturnType<typeof publicQuote>>;
66 +
67 +export function publicInstrument(i: Instrument) {
68 + return {
69 + id: i.id,
70 + symbol: i.symbol,
71 + name: i.name,
72 + asset_class: i.assetClass,
73 + exchange_id: i.exchangeId,
74 + mic: i.mic,
75 + currency: i.currency,
76 + country: i.country,
77 + company_id: i.companyId,
78 + security_type: i.securityType,
79 + base: i.base,
80 + quote: i.quote,
81 + is_active: i.isActive,
82 + metadata: i.metadata,
83 + };
84 +}
85 +
86 +/** Compact wire format for the stream ("quote" messages). */
87 +export function streamQuote(q: CanonicalQuote) {
88 + if (!PUBLICLY_REDISTRIBUTABLE.has(q.rightsStatus)) return null;
89 + return {
90 + type: "quote" as const,
91 + instrument_id: q.instrumentId,
92 + symbol: q.symbol,
93 + price: q.price,
94 + change: q.change,
95 + change_pct: q.changePercent,
96 + bid: q.bid,
97 + ask: q.ask,
98 + volume: q.volume,
99 + currency: q.currency,
100 + timestamp: q.sourceTimestamp ?? q.updatedAt,
101 + received: q.updatedAt,
102 + confidence: q.confidence,
103 + sources: q.sourceCount,
104 + status: q.realtimeStatus,
105 + };
106 +}
added apps/api/src/api/routes/admin.ts +155 −0
@@ -0,0 +1,155 @@
1 +import type { FastifyInstance, FastifyRequest } from "fastify";
2 +import { timingSafeEqual } from "node:crypto";
3 +import { config } from "../../config.js";
4 +import { pool } from "../../db/pool.js";
5 +import { connectorManager } from "../../core/connector-manager.js";
6 +import { health } from "../../core/health.js";
7 +import { rawArchive } from "../../core/raw-archive.js";
8 +import { telemetry } from "../../core/telemetry.js";
9 +import { observationWriter } from "../../core/observations.js";
10 +import { assertPublicUrl, redactObject } from "@market-atlas/connector-sdk";
11 +import { ApiError, envelope, notFound } from "../server.js";
12 +import { runDiscovery } from "../../discovery/probe.js";
13 +
14 +function requireAdmin(req: FastifyRequest) {
15 + const token = req.headers["x-ma-admin-token"];
16 + if (!config.adminToken || typeof token !== "string" || token.length !== config.adminToken.length || !timingSafeEqual(Buffer.from(token), Buffer.from(config.adminToken))) {
17 + telemetry.inc("admin_auth_failures_total");
18 + throw new ApiError(401, "UNAUTHORIZED", "admin token required");
19 + }
20 +}
21 +
22 +export async function registerAdminRoutes(app: FastifyInstance) {
23 + app.addHook("onRequest", async (req) => {
24 + if (req.url.startsWith("/v1/admin")) requireAdmin(req);
25 + });
26 +
27 + app.get("/v1/admin/overview", async () => {
28 + const [obs, evts, sizes] = await Promise.all([
29 + pool.query<{ n: number }>("select coalesce(sum(observations),0)::bigint as n from daily_stats"),
30 + pool.query<{ n: number }>("select count(*)::int as n from market_events"),
31 + pool.query<{ relname: string; bytes: number }>(
32 + `select c.relname, pg_total_relation_size(c.oid)::bigint as bytes from pg_class c join pg_namespace n on n.oid = c.relnamespace where n.nspname = 'public' and c.relkind in ('r','p') order by 2 desc limit 25`,
33 + ),
34 + ]);
35 + return envelope({
36 + version: config.version,
37 + role: config.role,
38 + uptime_s: Math.round(process.uptime()),
39 + memory: process.memoryUsage(),
40 + observations_total: obs.rows[0]?.n ?? 0,
41 + events_total: evts.rows[0]?.n ?? 0,
42 + queue_depth: observationWriter.depth(),
43 + rates: { observations_per_sec: telemetry.rate("observations_received_total"), events_per_min: telemetry.rate("events_total") * 60, http_per_min: telemetry.rate("http_requests_total") * 60 },
44 + tables: sizes.rows,
45 + rate_limiter: connectorManager.limiter.snapshot(),
46 + bus: telemetry.counter("canonical_quotes_total"),
47 + });
48 + });
49 +
50 + app.get("/v1/admin/connectors", async () => {
51 + return envelope(
52 + connectorManager.list().map((r) => ({
53 + metadata: r.def.metadata,
54 + health: health.snapshot(r.def.metadata.id),
55 + paused: r.paused,
56 + running: r.running,
57 + symbols: r.symbols,
58 + schedule: r.def.schedule ?? null,
59 + consecutive_failures: r.consecutiveFailures,
60 + drift_strikes: r.driftStrikes,
61 + sockets: r.sockets.map((s) => ({ url: s.url.replace(/\?.*$/, ""), open: s.isOpen(), reconnects: s.reconnects, connected_at: s.connectedAt })),
62 + })),
63 + );
64 + });
65 +
66 + app.get<{ Params: { id: string } }>("/v1/admin/connectors/:id", async (req) => {
67 + const rt = connectorManager.get(req.params.id);
68 + if (!rt) throw notFound("connector");
69 + const [changes, healthHistory, recent] = await Promise.all([
70 + pool.query("select id, kind, old_fingerprint, new_fingerprint, sample, detected_at, acknowledged from connector_schema_changes where connector_id = $1 order by detected_at desc limit 20", [rt.def.metadata.id]),
71 + pool.query("select ts, status, messages_1m, errors_1m, median_latency_ms, reliability_score from connector_health where connector_id = $1 and ts > now() - interval '24 hours' order by ts", [rt.def.metadata.id]),
72 + pool.query("select received_at, source_ts, instrument_id, field, value, latency_ms, raw_ref from observations where connector_id = $1 and received_at > now() - interval '1 day' order by received_at desc limit 25", [rt.def.metadata.id]),
73 + ]);
74 + return envelope({
75 + metadata: rt.def.metadata,
76 + health: health.snapshot(rt.def.metadata.id),
77 + paused: rt.paused,
78 + running: rt.running,
79 + symbols: rt.symbols,
80 + schedule: rt.def.schedule ?? null,
81 + state: redactObject(rt.stateCache),
82 + schema_changes: changes.rows,
83 + health_history: healthHistory.rows,
84 + recent_observations: recent.rows,
85 + fingerprints: health.fingerprints(rt.def.metadata.id),
86 + });
87 + });
88 +
89 + for (const action of ["pause", "resume", "restart"] as const) {
90 + app.post<{ Params: { id: string } }>(`/v1/admin/connectors/:id/${action}`, async (req) => {
91 + const rt = connectorManager.get(req.params.id);
92 + if (!rt) throw notFound("connector");
93 + await connectorManager[action](rt.def.metadata.id);
94 + return envelope({ id: rt.def.metadata.id, action, state: health.state(rt.def.metadata.id) });
95 + });
96 + }
97 +
98 + app.post<{ Params: { id: string } }>("/v1/admin/connectors/:id/test", async (req) => {
99 + const rt = connectorManager.get(req.params.id);
100 + if (!rt) throw notFound("connector");
101 + try {
102 + const res = await connectorManager.runOnce(rt.def.metadata.id);
103 + return envelope({ ok: true, ...res });
104 + } catch (err) {
105 + return envelope({ ok: false, error: err instanceof Error ? err.message : String(err) });
106 + }
107 + });
108 +
109 + app.post<{ Params: { id: string; changeId: string } }>("/v1/admin/connectors/:id/schema-changes/:changeId/ack", async (req) => {
110 + await pool.query("update connector_schema_changes set acknowledged = true where id = $1 and connector_id = $2", [Number(req.params.changeId), req.params.id]);
111 + return envelope({ ok: true });
112 + });
113 +
114 + app.get("/v1/admin/schema-changes", async () => {
115 + const r = await pool.query("select id, connector_id, kind, old_fingerprint, new_fingerprint, detected_at, acknowledged from connector_schema_changes where not acknowledged order by detected_at desc limit 200");
116 + return envelope(r.rows);
117 + });
118 +
119 + app.get<{ Querystring: { ref: string } }>("/v1/admin/raw", async (req) => {
120 + const ref = req.query.ref;
121 + if (!ref) throw new ApiError(400, "BAD_REQUEST", "ref required");
122 + try {
123 + return envelope(await rawArchive.read(ref));
124 + } catch {
125 + throw notFound("raw payload");
126 + }
127 + });
128 +
129 + app.get("/v1/admin/divergence", async () => {
130 + const r = await pool.query("select id, ts, instrument_ids, title, data from market_events where type = 'SOURCE_DIVERGENCE' order by ts desc limit 100");
131 + return envelope(r.rows);
132 + });
133 +
134 + app.get("/v1/admin/storage", async () => {
135 + const [parts, archives, db] = await Promise.all([
136 + pool.query(`select c.relname, pg_total_relation_size(c.oid)::bigint as bytes, (select reltuples::bigint from pg_class x where x.oid = c.oid) as est_rows from pg_inherits i join pg_class c on c.oid = i.inhrelid join pg_class p on p.oid = i.inhparent where p.relname = 'observations' order by 1 desc`),
137 + pool.query("select * from observation_archives order by day desc limit 60"),
138 + pool.query<{ bytes: number }>("select pg_database_size(current_database())::bigint as bytes"),
139 + ]);
140 + return envelope({ database_bytes: db.rows[0]?.bytes ?? 0, partitions: parts.rows, archives: archives.rows, retention_days: config.observationRetentionDays, data_dir: config.dataDir });
141 + });
142 +
143 + /** Discovery prototype: observe how a public page obtains market data (network-level, unauthenticated, SSRF-guarded). */
144 + app.post<{ Body: { url?: string } }>("/v1/admin/discovery", async (req) => {
145 + const url = req.body?.url;
146 + if (!url || typeof url !== "string") throw new ApiError(400, "BAD_REQUEST", "url required");
147 + let target;
148 + try {
149 + target = await assertPublicUrl(url);
150 + } catch (err) {
151 + throw new ApiError(400, "URL_REJECTED", err instanceof Error ? err.message : "invalid url");
152 + }
153 + return envelope(await runDiscovery(target.toString(), connectorManager.http));
154 + });
155 +}
added apps/api/src/api/routes/markets.ts +270 −0
@@ -0,0 +1,270 @@
1 +import type { FastifyInstance } from "fastify";
2 +import { z } from "zod";
3 +import { PUBLICLY_REDISTRIBUTABLE, type AssetClass, type Instrument } from "@market-atlas/market-model";
4 +import { pool } from "../../db/pool.js";
5 +import { calendar } from "../../core/calendar.js";
6 +import { instruments } from "../../core/instruments.js";
7 +import { quoteStore } from "../../core/quotes.js";
8 +import { badRequest, envelope, notFound } from "../server.js";
9 +import { publicInstrument, publicQuote } from "../public-view.js";
10 +import { rowToEvent, round } from "./public.js";
11 +
12 +const quoted = (pred: (i: Instrument) => boolean) =>
13 + instruments
14 + .all()
15 + .filter((i) => i.isActive && pred(i))
16 + .map((i) => ({ inst: i, q: quoteStore.get(i.id) }))
17 + .filter((x) => x.q?.price != null && PUBLICLY_REDISTRIBUTABLE.has(x.q!.rightsStatus));
18 +
19 +const view = (x: { inst: Instrument; q: ReturnType<typeof quoteStore.get> }) => ({ ...publicInstrument(x.inst), quote: publicQuote(x.q, x.inst) });
20 +
21 +/** Featured instruments per asset class, chosen by metadata.featured (seeded by connectors) then by activity. */
22 +function featured(cls: AssetClass, n: number) {
23 + const all = quoted((i) => i.assetClass === cls);
24 + all.sort((a, b) => Number(!!b.inst.metadata.featured) - Number(!!a.inst.metadata.featured) || (b.q!.volume ?? 0) - (a.q!.volume ?? 0) || a.inst.symbol.localeCompare(b.inst.symbol));
25 + return all.slice(0, n).map(view);
26 +}
27 +
28 +export async function registerMarketRoutes(app: FastifyInstance) {
29 + /** Global market overview for the homepage and /markets. */
30 + app.get("/v1/markets", async () => {
31 + const exchanges = calendar.list().map((e) => ({ ...e, status: calendar.status(e.id) }));
32 + const eq = quoted((i) => i.assetClass === "EQUITY" || i.assetClass === "ETF");
33 + const movers = (dir: 1 | -1) =>
34 + [...eq]
35 + .filter((x) => x.q!.changePercent != null && (x.q!.volume ?? 0) > 0)
36 + .sort((a, b) => dir * ((b.q!.changePercent ?? 0) - (a.q!.changePercent ?? 0)))
37 + .slice(0, 10)
38 + .map(view);
39 + return envelope({
40 + indices: featured("INDEX", 12),
41 + crypto: featured("CRYPTO", 12),
42 + forex: featured("FOREX", 12),
43 + rates: featured("TREASURY", 12).concat(featured("INTEREST_RATE", 4)),
44 + commodities: featured("COMMODITY", 8).concat(featured("FUTURE", 8)),
45 + equities: featured("EQUITY", 16),
46 + gainers: movers(1),
47 + losers: movers(-1),
48 + exchanges: exchanges.map((e) => ({ id: e.id, name: e.name, mic: e.mic, country: e.country, city: e.city, timezone: e.timezone, lat: e.lat, lon: e.lon, status: e.status, asset_classes: e.assetClasses })),
49 + breadth: breadth(eq),
50 + });
51 + });
52 +
53 + app.get("/v1/breadth", async (req) => {
54 + const p = z.object({ scope: z.string().default("world") }).parse(req.query);
55 + const scope = p.scope.toLowerCase();
56 + const list = quoted((i) => {
57 + if (i.assetClass !== "EQUITY" && i.assetClass !== "ETF") return false;
58 + if (scope === "world") return true;
59 + if (scope.length === 2) return i.country === scope.toUpperCase();
60 + return i.exchangeId === scope;
61 + });
62 + return envelope({ scope, ...breadth(list) });
63 + });
64 +
65 + /** "What changed" across markets in the last window. */
66 + app.get("/v1/changes", async (req) => {
67 + const p = z.object({ window: z.enum(["1m", "5m", "15m", "1h", "1d"]).default("1h") }).parse(req.query);
68 + const ms = { "1m": 60_000, "5m": 300_000, "15m": 900_000, "1h": 3_600_000, "1d": 86_400_000 }[p.window];
69 + const since = new Date(Date.now() - ms).toISOString();
70 + const [events, filings, byType, movers] = await Promise.all([
71 + pool.query<{ n: number }>("select count(*)::int as n from market_events where ts >= $1", [since]),
72 + pool.query<{ n: number }>("select count(*)::int as n from filings where filed_at >= $1", [since]),
73 + pool.query<{ type: string; n: number }>("select type, count(*)::int as n from market_events where ts >= $1 group by type order by n desc", [since]),
74 + pool.query<{ instrument_id: string; first: number; last: number }>(
75 + `select instrument_id, (array_agg(close order by ts))[1] as first, (array_agg(close order by ts desc))[1] as last from bars where resolution = '1m' and ts >= $1 group by instrument_id`,
76 + [since],
77 + ),
78 + ]);
79 + const moved = movers.rows
80 + .map((r) => ({ id: r.instrument_id, pct: r.first ? ((r.last - r.first) / r.first) * 100 : 0 }))
81 + .filter((m) => Number.isFinite(m.pct));
82 + const big = moved.filter((m) => Math.abs(m.pct) >= 1).length;
83 + const top = [...moved]
84 + .sort((a, b) => Math.abs(b.pct) - Math.abs(a.pct))
85 + .slice(0, 12)
86 + .map((m) => {
87 + const inst = instruments.get(m.id);
88 + return inst ? { ...publicInstrument(inst), window_change_percent: round(m.pct, 3), quote: publicQuote(quoteStore.get(inst.id), inst) } : null;
89 + })
90 + .filter(Boolean);
91 + return envelope({
92 + window: p.window,
93 + since,
94 + instruments_tracked: moved.length,
95 + instruments_moved_over_1pct: big,
96 + events: events.rows[0]?.n ?? 0,
97 + filings: filings.rows[0]?.n ?? 0,
98 + events_by_type: byType.rows,
99 + top_movers: top,
100 + });
101 + });
102 +
103 + app.get("/v1/exchanges", async () => {
104 + return envelope(
105 + calendar.list().map((e) => {
106 + const list = quoted((i) => i.exchangeId === e.id);
107 + return {
108 + ...pubExchange(e),
109 + status: calendar.status(e.id),
110 + instruments_tracked: instruments.all().filter((i) => i.exchangeId === e.id && i.isActive).length,
111 + instruments_quoted: list.length,
112 + breadth: list.length ? breadth(list) : null,
113 + };
114 + }),
115 + );
116 + });
117 +
118 + app.get<{ Params: { id: string } }>("/v1/exchanges/:id", async (req) => {
119 + const e = calendar.exchange(req.params.id.toLowerCase()) ?? calendar.list().find((x) => x.mic?.toLowerCase() === req.params.id.toLowerCase());
120 + if (!e) throw notFound("exchange");
121 + const list = quoted((i) => i.exchangeId === e.id);
122 + const sorted = [...list].filter((x) => x.q!.changePercent != null).sort((a, b) => (b.q!.changePercent ?? 0) - (a.q!.changePercent ?? 0));
123 + const events = await pool.query(
124 + `select id, type, ts, instrument_ids, severity, confidence, source_count, sources, title, summary, data, confirmed_at from market_events where (data->>'exchangeId' = $1 or instrument_ids && $2::text[]) order by ts desc limit 30`,
125 + [e.id, list.map((x) => x.inst.id)],
126 + );
127 + return envelope({
128 + exchange: pubExchange(e),
129 + status: calendar.status(e.id),
130 + holidays: calendar.holidaysFor(e.id).filter((h) => h.date >= new Date().toISOString().slice(0, 10)).slice(0, 12),
131 + instruments_tracked: instruments.all().filter((i) => i.exchangeId === e.id && i.isActive).length,
132 + instruments_quoted: list.length,
133 + breadth: breadth(list),
134 + gainers: sorted.slice(0, 10).map(view),
135 + losers: sorted.slice(-10).reverse().map(view),
136 + most_active: [...list].sort((a, b) => (b.q!.volume ?? 0) - (a.q!.volume ?? 0)).slice(0, 10).map(view),
137 + events: events.rows.map(rowToEvent),
138 + });
139 + });
140 +
141 + app.get("/v1/countries", async () => {
142 + const rows = await pool.query<{ code: string; name: string; region: string; currency: string | null }>("select code, name, region, currency from countries order by name");
143 + return envelope(
144 + rows.rows.map((c) => {
145 + const ex = calendar.list().filter((e) => e.country === c.code);
146 + return { ...c, exchanges: ex.map((e) => ({ id: e.id, name: e.name, status: calendar.status(e.id).state })), instruments_tracked: instruments.all().filter((i) => i.country === c.code && i.isActive).length };
147 + }),
148 + );
149 + });
150 +
151 + app.get<{ Params: { code: string } }>("/v1/countries/:code", async (req) => {
152 + const code = req.params.code.toUpperCase();
153 + if (!/^[A-Z]{2}$/.test(code)) throw badRequest("country code must be ISO-3166 alpha-2");
154 + const c = (await pool.query("select code, name, region, currency from countries where code = $1", [code])).rows[0];
155 + if (!c) throw notFound("country");
156 + const ex = calendar.list().filter((e) => e.country === code);
157 + const byClass = (cls: AssetClass[]) => quoted((i) => i.country === code && cls.includes(i.assetClass)).map(view);
158 + const fx = quoted((i) => i.assetClass === "FOREX" && (i.base === c.currency || i.quote === c.currency)).map(view);
159 + const eq = quoted((i) => i.country === code && (i.assetClass === "EQUITY" || i.assetClass === "ETF"));
160 + const events = await pool.query(
161 + `select id, type, ts, instrument_ids, severity, confidence, source_count, sources, title, summary, data, confirmed_at from market_events where instrument_ids && $1::text[] order by ts desc limit 30`,
162 + [instruments.all().filter((i) => i.country === code).map((i) => i.id)],
163 + );
164 + return envelope({
165 + country: c,
166 + exchanges: ex.map((e) => ({ ...pubExchange(e), status: calendar.status(e.id) })),
167 + indices: byClass(["INDEX"]),
168 + rates: byClass(["TREASURY", "INTEREST_RATE", "BOND"]),
169 + fx,
170 + equities: [...eq].sort((a, b) => (b.q!.volume ?? 0) - (a.q!.volume ?? 0)).slice(0, 20).map(view),
171 + breadth: eq.length ? breadth(eq) : null,
172 + events: events.rows.map(rowToEvent),
173 + instruments_tracked: instruments.all().filter((i) => i.country === code && i.isActive).length,
174 + });
175 + });
176 +
177 + /** Cross-asset comparison: normalized daily closes for up to 8 instruments. */
178 + app.get("/v1/compare", async (req) => {
179 + const p = z.object({ ids: z.string(), resolution: z.enum(["1m", "1h", "1d"]).default("1d"), limit: z.coerce.number().int().min(10).max(2000).default(365) }).parse(req.query);
180 + const ids = p.ids.split(",").map((s) => s.trim()).filter(Boolean).slice(0, 8);
181 + const series = [];
182 + for (const raw of ids) {
183 + const inst = instruments.get(raw) ?? instruments.bySymbolCandidates(raw)[0];
184 + if (!inst) continue;
185 + const r = await pool.query("select ts, close from bars where instrument_id = $1 and resolution = $2 order by ts desc limit $3", [inst.id, p.resolution, p.limit]);
186 + const pts = r.rows.reverse().map((b) => ({ t: Date.parse(b.ts), c: b.close as number }));
187 + const base = pts[0]?.c ?? null;
188 + const rets = pts.slice(1).map((pt, i) => (pts[i]!.c ? pt.c / pts[i]!.c - 1 : 0));
189 + const vol = rets.length > 1 ? Math.sqrt(rets.reduce((a, r) => a + r * r, 0) / rets.length) * Math.sqrt(p.resolution === "1d" ? 252 : p.resolution === "1h" ? 252 * 6.5 : 252 * 390) : null;
190 + let peak = -Infinity;
191 + let mdd = 0;
192 + for (const pt of pts) {
193 + peak = Math.max(peak, pt.c);
194 + mdd = Math.min(mdd, pt.c / peak - 1);
195 + }
196 + series.push({
197 + instrument: publicInstrument(inst),
198 + quote: publicQuote(quoteStore.get(inst.id), inst),
199 + points: pts.map((pt) => ({ t: pt.t, c: pt.c, n: base ? (pt.c / base) * 100 : null })),
200 + stats: { return_percent: base && pts.length ? round((pts[pts.length - 1]!.c / base - 1) * 100, 2) : null, annualized_volatility_percent: vol == null ? null : round(vol * 100, 2), max_drawdown_percent: round(mdd * 100, 2), points: pts.length },
201 + });
202 + }
203 + // Pairwise correlation of returns on aligned timestamps.
204 + const correlations: Array<{ a: string; b: string; rho: number | null }> = [];
205 + for (let i = 0; i < series.length; i++)
206 + for (let j = i + 1; j < series.length; j++) {
207 + correlations.push({ a: series[i]!.instrument.id, b: series[j]!.instrument.id, rho: correlation(series[i]!.points, series[j]!.points) });
208 + }
209 + return envelope({ resolution: p.resolution, series, correlations });
210 + });
211 +}
212 +
213 +function correlation(a: Array<{ t: number; c: number }>, b: Array<{ t: number; c: number }>): number | null {
214 + const mb = new Map(b.map((p) => [p.t, p.c]));
215 + const pairs: Array<[number, number]> = [];
216 + let prevA: number | null = null;
217 + let prevB: number | null = null;
218 + for (const p of a) {
219 + const bc = mb.get(p.t);
220 + if (bc == null) continue;
221 + if (prevA != null && prevB != null && prevA && prevB) pairs.push([p.c / prevA - 1, bc / prevB - 1]);
222 + prevA = p.c;
223 + prevB = bc;
224 + }
225 + if (pairs.length < 5) return null;
226 + const ma = pairs.reduce((s, x) => s + x[0], 0) / pairs.length;
227 + const mbb = pairs.reduce((s, x) => s + x[1], 0) / pairs.length;
228 + let num = 0,
229 + da = 0,
230 + db = 0;
231 + for (const [x, y] of pairs) {
232 + num += (x - ma) * (y - mbb);
233 + da += (x - ma) ** 2;
234 + db += (y - mbb) ** 2;
235 + }
236 + return da && db ? round(num / Math.sqrt(da * db), 3) : null;
237 +}
238 +
239 +function breadth(list: Array<{ inst: Instrument; q: ReturnType<typeof quoteStore.get> }>) {
240 + const ch = list.map((x) => x.q!.changePercent).filter((x): x is number => x != null);
241 + const adv = ch.filter((x) => x > 0).length;
242 + const dec = ch.filter((x) => x < 0).length;
243 + const sorted = [...ch].sort((a, b) => a - b);
244 + return {
245 + instruments: list.length,
246 + advancers: adv,
247 + decliners: dec,
248 + unchanged: ch.length - adv - dec,
249 + median_change_percent: sorted.length ? round(sorted[Math.floor(sorted.length / 2)]!, 3) : null,
250 + mean_change_percent: ch.length ? round(ch.reduce((a, b) => a + b, 0) / ch.length, 3) : null,
251 + new_session_highs: list.filter((x) => x.q!.sessionHigh != null && x.q!.price === x.q!.sessionHigh && x.q!.realtimeStatus === "REALTIME").length,
252 + new_session_lows: list.filter((x) => x.q!.sessionLow != null && x.q!.price === x.q!.sessionLow && x.q!.realtimeStatus === "REALTIME").length,
253 + };
254 +}
255 +
256 +const pubExchange = (e: ReturnType<typeof calendar.list>[number]) => ({
257 + id: e.id,
258 + mic: e.mic,
259 + name: e.name,
260 + operator: e.operator,
261 + country: e.country,
262 + city: e.city,
263 + timezone: e.timezone,
264 + currency: e.currency,
265 + website: e.website,
266 + sessions: e.sessions,
267 + lat: e.lat,
268 + lon: e.lon,
269 + asset_classes: e.assetClasses,
270 +});
added apps/api/src/api/routes/public.ts +484 −0
@@ -0,0 +1,484 @@
1 +import type { FastifyInstance } from "fastify";
2 +import { z } from "zod";
3 +import { ASSET_CLASSES, BAR_RESOLUTIONS, PUBLICLY_REDISTRIBUTABLE, canonicalSymbol } from "@market-atlas/market-model";
4 +import { pool, tsToMs } from "../../db/pool.js";
5 +import { config } from "../../config.js";
6 +import { instruments } from "../../core/instruments.js";
7 +import { quoteStore } from "../../core/quotes.js";
8 +import { health } from "../../core/health.js";
9 +import { connectorManager } from "../../core/connector-manager.js";
10 +import { telemetry } from "../../core/telemetry.js";
11 +import { calendar } from "../../core/calendar.js";
12 +import { pipeline } from "../../core/pipeline.js";
13 +import { badRequest, envelope, notFound } from "../server.js";
14 +import { publicInstrument, publicQuote } from "../public-view.js";
15 +import { SOURCES } from "@market-atlas/connectors";
16 +
17 +const q = <T extends z.ZodTypeAny>(schema: T, input: unknown): z.infer<T> => {
18 + const r = schema.safeParse(input);
19 + if (!r.success) throw badRequest(r.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`).join("; "));
20 + return r.data;
21 +};
22 +
23 +const ListQuery = z.object({
24 + asset_class: z.enum(ASSET_CLASSES).optional(),
25 + exchange: z.string().optional(),
26 + country: z.string().length(2).optional(),
27 + q: z.string().max(80).optional(),
28 + quoted: z.enum(["1", "0"]).optional(),
29 + limit: z.coerce.number().int().min(1).max(500).default(100),
30 + offset: z.coerce.number().int().min(0).default(0),
31 + sort: z.enum(["symbol", "change", "-change", "volume", "name"]).default("symbol"),
32 +});
33 +
34 +/** Resolve an id or a symbol (optionally EXCHANGE:SYMBOL) to an instrument; quoted instruments win for bare symbols. */
35 +export function resolveInstrument(idOrSymbol: string) {
36 + const direct = instruments.get(idOrSymbol);
37 + if (direct) return direct;
38 + let sym = idOrSymbol;
39 + let venue: string | null = null;
40 + const m = idOrSymbol.match(/^([A-Za-z]+):(.+)$/);
41 + if (m) {
42 + venue = m[1]!.toLowerCase();
43 + sym = m[2]!;
44 + }
45 + const candidates = instruments.bySymbolCandidates(sym).filter((i) => !venue || i.exchangeId === venue || i.mic?.toLowerCase() === venue);
46 + if (!candidates.length) return undefined;
47 + const quoted = candidates.filter((c) => quoteStore.get(c.id)?.price != null);
48 + const pool_ = quoted.length ? quoted : candidates;
49 + const rank = (c: (typeof candidates)[number]) => ({ EQUITY: 0, ETF: 1, INDEX: 2, CRYPTO: 3, FOREX: 4 } as Record<string, number>)[c.assetClass] ?? 9;
50 + return pool_.sort((a, b) => rank(a) - rank(b))[0];
51 +}
52 +
53 +export async function registerPublicRoutes(app: FastifyInstance) {
54 + app.get("/v1/health", async () => {
55 + const db = await pool.query("select 1").then(() => "ok").catch(() => "down");
56 + const healthy = connectorManager.list().filter((r) => health.state(r.def.metadata.id) === "HEALTHY").length;
57 + const status = db === "ok" ? "ok" : "degraded";
58 + return { status, version: config.version, role: config.role, db, connectors: { total: connectorManager.list().length, healthy }, quotes: quoteStore.size(), uptime_s: Math.round(process.uptime()) };
59 + });
60 +
61 + app.get("/v1/status", async () => {
62 + const list = connectorManager.list();
63 + const states = list.map((r) => health.state(r.def.metadata.id));
64 + const count = (s: string) => states.filter((x) => x === s).length;
65 + const db = await pool.query("select 1").then(() => "operational").catch(() => "outage");
66 + const streaming = states.some((s) => s === "HEALTHY") ? "operational" : "degraded";
67 + return envelope({
68 + components: [
69 + { id: "website", name: "Website", status: "operational" },
70 + { id: "api", name: "REST API", status: db === "operational" ? "operational" : "degraded" },
71 + { id: "stream", name: "Live stream", status: streaming },
72 + { id: "market-data", name: "Market data ingestion", status: count("FAILED") > list.length / 2 ? "degraded" : "operational" },
73 + { id: "historical", name: "Historical database", status: db },
74 + { id: "crypto", name: "Crypto sources", status: groupStatus(list, "CRYPTO") },
75 + { id: "equities", name: "Equity & index sources", status: groupStatus(list, "EQUITY") },
76 + { id: "official", name: "Official & regulatory sources", status: groupStatus(list, "TREASURY", "FOREX") },
77 + ],
78 + connectors: { total: list.length, healthy: count("HEALTHY"), degraded: count("DEGRADED") + count("STALE") + count("RECONNECTING"), failed: count("FAILED"), paused: count("PAUSED") },
79 + version: config.version,
80 + });
81 + });
82 +
83 + app.get("/v1/metrics", async (_req, reply) => reply.type("text/plain; version=0.0.4").send(telemetry.prometheus()));
84 +
85 + /** Homepage telemetry — every number comes from the running system. */
86 + app.get("/v1/stats", async () => {
87 + const [daily, filings, evts] = await Promise.all([
88 + pool.query<{ observations: number; events: number; filings: number }>("select observations, events, filings from daily_stats where day = current_date"),
89 + pool.query<{ n: number }>("select count(*)::int as n from filings where filed_at > now() - interval '24 hours'"),
90 + pool.query<{ n: number }>("select count(*)::int as n from market_events where ts > now() - interval '24 hours'"),
91 + ]);
92 + const list = connectorManager.list();
93 + const states = list.map((r) => health.state(r.def.metadata.id));
94 + const quoted = quoteStore.all().filter((x) => x.price != null && PUBLICLY_REDISTRIBUTABLE.has(x.rightsStatus));
95 + const exchanges = calendar.list();
96 + const open = exchanges.filter((e) => calendar.status(e.id).state === "OPEN").length;
97 + return envelope({
98 + instruments_total: instruments.count(),
99 + instruments_quoted: quoted.length,
100 + instruments_live: quoted.filter((x) => x.realtimeStatus === "REALTIME" && Date.now() - x.updatedAt < 60_000).length,
101 + exchanges_total: exchanges.length,
102 + exchanges_open: open,
103 + sources_total: SOURCES.length,
104 + connectors_total: list.length,
105 + connectors_healthy: states.filter((s) => s === "HEALTHY").length,
106 + observations_today: daily.rows[0]?.observations ?? 0,
107 + observations_per_sec: round(telemetry.rate("observations_received_total"), 1),
108 + observations_total: telemetry.counter("observations_written_total"),
109 + events_today: daily.rows[0]?.events ?? 0,
110 + events_24h: evts.rows[0]?.n ?? 0,
111 + events_per_min: round(telemetry.rate("events_total") * 60, 2),
112 + filings_24h: filings.rows[0]?.n ?? 0,
113 + median_freshness_ms: median(quoted.filter((x) => x.realtimeStatus === "REALTIME").map((x) => (x.freshnessMs ?? 0) + (Date.now() - x.updatedAt))),
114 + stream_clients: telemetry.counter("stream_clients_connected") - telemetry.counter("stream_clients_disconnected"),
115 + uptime_s: Math.round(process.uptime()),
116 + });
117 + });
118 +
119 + app.get("/v1/instruments", async (req) => {
120 + const p = q(ListQuery, req.query);
121 + let list = instruments.all().filter((i) => i.isActive);
122 + if (p.asset_class) list = list.filter((i) => i.assetClass === p.asset_class);
123 + if (p.exchange) list = list.filter((i) => i.exchangeId === p.exchange!.toLowerCase() || i.mic?.toLowerCase() === p.exchange!.toLowerCase());
124 + if (p.country) list = list.filter((i) => i.country === p.country!.toUpperCase());
125 + if (p.q) {
126 + const needle = p.q.toLowerCase();
127 + list = list.filter((i) => i.symbol.toLowerCase().includes(needle) || i.name.toLowerCase().includes(needle));
128 + }
129 + if (p.quoted === "1") list = list.filter((i) => quoteStore.get(i.id)?.price != null);
130 + const quote = (id: string) => quoteStore.get(id);
131 + list.sort((a, b) => {
132 + switch (p.sort) {
133 + case "change":
134 + return (quote(a.id)?.changePercent ?? -Infinity) - (quote(b.id)?.changePercent ?? -Infinity);
135 + case "-change":
136 + return (quote(b.id)?.changePercent ?? -Infinity) - (quote(a.id)?.changePercent ?? -Infinity);
137 + case "volume":
138 + return (quote(b.id)?.volume ?? -1) - (quote(a.id)?.volume ?? -1);
139 + case "name":
140 + return a.name.localeCompare(b.name);
141 + default:
142 + return a.symbol.localeCompare(b.symbol);
143 + }
144 + });
145 + const total = list.length;
146 + const page = list.slice(p.offset, p.offset + p.limit);
147 + return envelope(
148 + page.map((i) => ({ ...publicInstrument(i), quote: publicQuote(quoteStore.get(i.id), i) })),
149 + { total, limit: p.limit, offset: p.offset },
150 + );
151 + });
152 +
153 + app.get<{ Params: { id: string } }>("/v1/instruments/:id", async (req) => {
154 + const inst = resolveInstrument(req.params.id);
155 + if (!inst) throw notFound("instrument");
156 + const [company, exchange] = await Promise.all([inst.companyId ? instruments.getCompany(inst.companyId) : null, Promise.resolve(instruments.exchange(inst.exchangeId))]);
157 + const quote = quoteStore.get(inst.id);
158 + const related = instruments
159 + .all()
160 + .filter((i) => i.id !== inst.id && ((inst.companyId && i.companyId === inst.companyId) || (inst.base && i.base === inst.base)))
161 + .slice(0, 12)
162 + .map(publicInstrument);
163 + const sources = pipeline.consensus.sourcesFor(inst.id);
164 + return envelope({
165 + instrument: publicInstrument(inst),
166 + company,
167 + exchange: exchange ? { ...exchange, status: calendar.status(exchange.id) } : null,
168 + quote: publicQuote(quote, inst),
169 + aliases: instruments.aliasesOf(inst.id).map((a) => ({ alias: a.alias, source_id: a.sourceId })),
170 + sources,
171 + related,
172 + });
173 + });
174 +
175 + app.get<{ Params: { id: string } }>("/v1/quotes/:id", async (req) => {
176 + const inst = resolveInstrument(req.params.id);
177 + if (!inst) throw notFound("instrument");
178 + const quote = quoteStore.get(inst.id);
179 + return envelope(publicQuote(quote, inst), { data_status: quote?.realtimeStatus ?? "UNKNOWN" });
180 + });
181 +
182 + app.get("/v1/quotes", async (req) => {
183 + const p = q(z.object({ ids: z.string().max(4000) }), req.query);
184 + const ids = p.ids.split(",").map((s) => s.trim()).filter(Boolean).slice(0, 200);
185 + const out = ids.map((id) => {
186 + const inst = resolveInstrument(id);
187 + return inst ? publicQuote(quoteStore.get(inst.id), inst) : null;
188 + });
189 + return envelope(out);
190 + });
191 +
192 + /** "Why this price?" — every contribution, included or not, with the reason. */
193 + app.get<{ Params: { id: string } }>("/v1/quotes/:id/provenance", async (req) => {
194 + const inst = resolveInstrument(req.params.id);
195 + if (!inst) throw notFound("instrument");
196 + const quote = quoteStore.get(inst.id);
197 + if (!quote) throw notFound("quote");
198 + const redistributable = PUBLICLY_REDISTRIBUTABLE.has(quote.rightsStatus);
199 + const sourceMeta = Object.fromEntries(SOURCES.map((s) => [s.id, { name: s.name, organization: s.organization, family: s.family, rights_status: s.rightsStatus, realtime_status: s.realtimeStatus, category: s.category }]));
200 + return envelope({
201 + instrument_id: inst.id,
202 + symbol: inst.symbol,
203 + canonical_price: redistributable ? quote.price : null,
204 + currency: quote.currency,
205 + confidence: quote.confidence,
206 + dispersion_bps: quote.dispersionBps,
207 + independent_sources: quote.sourceCount,
208 + freshness_ms: quote.freshnessMs,
209 + realtime_status: quote.realtimeStatus,
210 + rights_status: quote.rightsStatus,
211 + consensus_version: quote.consensusVersion,
212 + computed_at: new Date(quote.updatedAt).toISOString(),
213 + method: "weighted median over fresh observations; one vote per upstream source family; outliers > 2% from the median excluded; weights = source reliability × timestamp quality × real-time class × freshness decay",
214 + contributions: quote.contributions.map((c) => ({
215 + source_id: c.sourceId,
216 + source: sourceMeta[c.sourceId] ?? null,
217 + connector_id: c.connectorId,
218 + value: PUBLICLY_REDISTRIBUTABLE.has(c.rightsStatus) ? c.value : null,
219 + source_timestamp: c.sourceTimestamp == null ? null : new Date(c.sourceTimestamp).toISOString(),
220 + received_at: new Date(c.receivedAt).toISOString(),
221 + age_ms: c.ageMs,
222 + weight: c.weight,
223 + included: c.included,
224 + reason: c.reason ?? null,
225 + realtime_status: c.realtimeStatus,
226 + rights_status: c.rightsStatus,
227 + reliability_score: health.sourceReliability(c.sourceId),
228 + })),
229 + });
230 + });
231 +
232 + const HistoryQuery = z.object({
233 + resolution: z.enum(BAR_RESOLUTIONS).default("1d"),
234 + from: z.string().optional(),
235 + to: z.string().optional(),
236 + limit: z.coerce.number().int().min(1).max(5000).default(500),
237 + });
238 + app.get<{ Params: { id: string } }>("/v1/history/:id", async (req) => {
239 + const inst = resolveInstrument(req.params.id);
240 + if (!inst) throw notFound("instrument");
241 + const p = q(HistoryQuery, req.query);
242 + const from = p.from ? new Date(p.from) : null;
243 + const to = p.to ? new Date(p.to) : null;
244 + if ((from && Number.isNaN(from.getTime())) || (to && Number.isNaN(to.getTime()))) throw badRequest("invalid from/to");
245 + const r = await pool.query(
246 + `select ts, open, high, low, close, volume, source_count, producer from bars where instrument_id = $1 and resolution = $2
247 + and ($3::timestamptz is null or ts >= $3) and ($4::timestamptz is null or ts <= $4) order by ts desc limit $5`,
248 + [inst.id, p.resolution, from?.toISOString() ?? null, to?.toISOString() ?? null, p.limit],
249 + );
250 + const bars = r.rows.reverse().map((b) => ({ t: tsToMs(b.ts), o: b.open, h: b.high, l: b.low, c: b.close, v: b.volume, n: b.source_count, p: b.producer }));
251 + const producers = [...new Set(bars.map((b) => b.p))];
252 + return envelope(bars, { instrument_id: inst.id, resolution: p.resolution, count: bars.length, producers, data_status: producers.includes("consensus") ? "DERIVED" : "END_OF_DAY" });
253 + });
254 +
255 + const EventsQuery = z.object({
256 + type: z.string().optional(),
257 + instrument: z.string().optional(),
258 + country: z.string().optional(),
259 + asset_class: z.enum(ASSET_CLASSES).optional(),
260 + severity: z.string().optional(),
261 + since: z.string().optional(),
262 + before: z.string().optional(),
263 + limit: z.coerce.number().int().min(1).max(500).default(100),
264 + });
265 + app.get("/v1/events", async (req) => {
266 + const p = q(EventsQuery, req.query);
267 + const types = p.type ? p.type.split(",").map((s) => s.trim().toUpperCase()) : null;
268 + let instIds: string[] | null = null;
269 + if (p.instrument) {
270 + const inst = resolveInstrument(p.instrument);
271 + instIds = inst ? [inst.id] : ["__none__"];
272 + } else if (p.country || p.asset_class) {
273 + instIds = instruments
274 + .all()
275 + .filter((i) => (!p.country || i.country === p.country!.toUpperCase()) && (!p.asset_class || i.assetClass === p.asset_class))
276 + .map((i) => i.id);
277 + if (!instIds.length) instIds = ["__none__"];
278 + }
279 + const r = await pool.query(
280 + `select id, type, ts, instrument_ids, severity, confidence, source_count, sources, title, summary, data, confirmed_at from market_events
281 + where ($1::text[] is null or type = any($1)) and ($2::text[] is null or instrument_ids && $2) and ($3::text is null or severity = $3)
282 + and ($4::timestamptz is null or ts >= $4) and ($5::timestamptz is null or ts < $5)
283 + order by ts desc limit $6`,
284 + [types, instIds, p.severity ?? null, p.since ?? null, p.before ?? null, p.limit],
285 + );
286 + return envelope(r.rows.map(rowToEvent));
287 + });
288 +
289 + app.get<{ Params: { id: string } }>("/v1/events/:id", async (req) => {
290 + const r = await pool.query("select * from market_events where id = $1", [req.params.id]);
291 + if (!r.rows[0]) throw notFound("event");
292 + return envelope(rowToEvent(r.rows[0]));
293 + });
294 +
295 + app.get("/v1/filings", async (req) => {
296 + const p = q(z.object({ form: z.string().optional(), cik: z.string().optional(), q: z.string().optional(), limit: z.coerce.number().int().min(1).max(500).default(100), offset: z.coerce.number().int().min(0).default(0) }), req.query);
297 + const forms = p.form ? p.form.split(",").map((s) => s.trim().toUpperCase()) : null;
298 + const r = await pool.query(
299 + `select id, source_id, cik, company_name, form_type, filed_at, url, instrument_ids, metadata from filings
300 + where ($1::text[] is null or upper(form_type) = any($1)) and ($2::text is null or cik = $2) and ($3::text is null or company_name ilike '%' || $3 || '%')
301 + order by filed_at desc limit $4 offset $5`,
302 + [forms, p.cik ? p.cik.replace(/^0+/, "") : null, p.q ?? null, p.limit, p.offset],
303 + );
304 + const total = await pool.query<{ n: number }>("select count(*)::int as n from filings where filed_at > now() - interval '7 days'");
305 + return envelope(
306 + r.rows.map((f) => ({ id: f.id, source_id: f.source_id, cik: f.cik, company_name: f.company_name, form_type: f.form_type, filed_at: f.filed_at, url: f.url, instrument_ids: f.instrument_ids, metadata: f.metadata })),
307 + { last_7_days: total.rows[0]?.n ?? 0 },
308 + );
309 + });
310 +
311 + app.get("/v1/search", async (req) => {
312 + const p = q(z.object({ q: z.string().min(1).max(80), limit: z.coerce.number().int().min(1).max(50).default(20) }), req.query);
313 + const needle = p.q.trim();
314 + const lower = needle.toLowerCase();
315 + const cs = canonicalSymbol(needle);
316 + const scored: Array<{ score: number; item: unknown; group: string }> = [];
317 + for (const i of instruments.all()) {
318 + if (!i.isActive) continue;
319 + const sym = i.symbol.toLowerCase();
320 + const name = i.name.toLowerCase();
321 + let score = 0;
322 + if (canonicalSymbol(i.symbol) === cs) score = 100;
323 + else if (sym.startsWith(lower)) score = 80 - (sym.length - lower.length);
324 + else if (name === lower) score = 75;
325 + else if (name.startsWith(lower)) score = 60;
326 + else if (name.includes(lower) || sym.includes(lower)) score = 40;
327 + else if (i.base && i.base.toLowerCase() === lower) score = 70;
328 + else if (i.id.includes(cs)) score = 30;
329 + if (!score) continue;
330 + const quote = quoteStore.get(i.id);
331 + if (quote?.price != null) score += 10;
332 + if (i.assetClass === "EQUITY" && quote?.price == null) score -= 5;
333 + scored.push({ score, group: i.assetClass, item: { ...publicInstrument(i), quote: publicQuote(quote, i) } });
334 + }
335 + for (const e of calendar.list()) {
336 + const hay = `${e.name} ${e.mic ?? ""} ${e.id} ${e.city ?? ""}`.toLowerCase();
337 + if (hay.includes(lower)) scored.push({ score: e.name.toLowerCase().startsWith(lower) || (e.mic ?? "").toLowerCase() === lower ? 85 : 45, group: "EXCHANGE", item: { id: e.id, name: e.name, mic: e.mic, country: e.country, status: calendar.status(e.id).state } });
338 + }
339 + const countries = await pool.query<{ code: string; name: string; region: string }>("select code, name, region from countries where lower(name) like $1 or lower(code) = $2 limit 5", [`%${lower}%`, lower]);
340 + for (const c of countries.rows) scored.push({ score: c.name.toLowerCase().startsWith(lower) ? 70 : 40, group: "COUNTRY", item: c });
341 + scored.sort((a, b) => b.score - a.score);
342 + const top = scored.slice(0, p.limit);
343 + const groups: Record<string, unknown[]> = {};
344 + for (const s of top) (groups[s.group] ??= []).push(s.item);
345 + return envelope({ query: needle, results: top.map((s) => ({ group: s.group, score: s.score, item: s.item })), groups });
346 + });
347 +
348 + app.get("/v1/sources", async () => {
349 + const rows = await pool.query("select id, name, organization, source_type, homepage, jurisdiction, rights_status, realtime_status, family, category, enabled from sources order by category, name");
350 + const byConnector = connectorManager.list();
351 + return envelope(
352 + rows.rows.map((s) => {
353 + const conns = byConnector.filter((r) => r.def.metadata.sourceId === s.id);
354 + const snaps = conns.map((c) => health.snapshot(c.def.metadata.id));
355 + const last = snaps.map((x) => x.lastMessageAt ?? 0).reduce((a, b) => Math.max(a, b), 0);
356 + return {
357 + ...s,
358 + connectors: conns.length,
359 + status: conns.length ? worst(snaps.map((x) => x.state)) : "NONE",
360 + coverage: snaps.reduce((a, x) => a + x.instrumentsCovered, 0),
361 + last_observation: last ? new Date(last).toISOString() : null,
362 + reliability_score: snaps.map((x) => x.reliabilityScore).filter((x): x is number => x != null).sort((a, b) => b - a)[0] ?? null,
363 + asset_classes: [...new Set(conns.flatMap((c) => c.def.metadata.assetClasses))],
364 + description: conns[0]?.def.metadata.description ?? null,
365 + rights_notes: conns[0]?.def.metadata.rightsNotes ?? null,
366 + terms_url: conns[0]?.def.metadata.termsUrl ?? null,
367 + };
368 + }),
369 + );
370 + });
371 +
372 + /** Public connector directory: operational facts only, no endpoints. */
373 + app.get("/v1/connectors", async () => {
374 + return envelope(
375 + connectorManager.list().map((r) => {
376 + const m = r.def.metadata;
377 + const s = health.snapshot(m.id);
378 + return {
379 + id: m.id,
380 + name: m.name,
381 + source_id: m.sourceId,
382 + organization: m.organization,
383 + source_type: m.sourceType,
384 + rights_status: m.rightsStatus,
385 + realtime_status: m.realtimeStatus,
386 + asset_classes: m.assetClasses,
387 + exchanges: m.exchanges,
388 + jurisdiction: m.jurisdiction,
389 + supports_streaming: m.supportsStreaming,
390 + supports_historical: m.supportsHistorical,
391 + expected_latency_ms: m.expectedLatencyMs,
392 + description: m.description,
393 + version: m.version,
394 + status: s.state,
395 + last_message_at: s.lastMessageAt ? new Date(s.lastMessageAt).toISOString() : null,
396 + messages_1m: s.messages1m,
397 + messages_total: s.messagesTotal,
398 + median_latency_ms: s.medianLatencyMs,
399 + p95_latency_ms: s.p95LatencyMs,
400 + parse_success_rate: s.parseSuccessRate,
401 + instruments_covered: s.instrumentsCovered,
402 + reconnects: s.reconnects,
403 + reliability_score: s.reliabilityScore,
404 + next_poll_at: s.nextPollAt ? new Date(s.nextPollAt).toISOString() : null,
405 + };
406 + }),
407 + );
408 + });
409 +
410 + app.get("/v1/data-health", async () => {
411 + const list = connectorManager.list();
412 + const snaps = list.map((r) => health.snapshot(r.def.metadata.id));
413 + const states = snaps.map((s) => s.state);
414 + const count = (...s: string[]) => states.filter((x) => s.includes(x)).length;
415 + const quotes = quoteStore.all().filter((x) => x.price != null && PUBLICLY_REDISTRIBUTABLE.has(x.rightsStatus));
416 + const now = Date.now();
417 + const realtime = quotes.filter((x) => x.realtimeStatus === "REALTIME");
418 + const incidents = await pool.query(
419 + `select id, type, ts, severity, title, data from market_events where type in ('SOURCE_FAILURE','SOURCE_RECOVERY','SCHEMA_DRIFT','SOURCE_DIVERGENCE') and ts > now() - interval '7 days' order by ts desc limit 50`,
420 + );
421 + const history = await pool.query(
422 + `select date_trunc('hour', ts) as hour, avg(case when status = 'HEALTHY' then 1.0 else 0.0 end) as healthy_ratio, sum(messages_1m) as messages, avg(median_latency_ms) as latency
423 + from connector_health where ts > now() - interval '48 hours' group by 1 order by 1`,
424 + );
425 + return envelope({
426 + connectors: { total: list.length, healthy: count("HEALTHY"), degraded: count("DEGRADED", "STALE"), recovering: count("RECONNECTING", "STARTING"), failed: count("FAILED"), paused: count("PAUSED", "DISABLED") },
427 + healthy_ratio: list.length ? count("HEALTHY") / list.length : null,
428 + quotes_public: quotes.length,
429 + quotes_realtime: realtime.length,
430 + median_freshness_ms: median(realtime.map((x) => (x.freshnessMs ?? 0) + (now - x.updatedAt))),
431 + p95_freshness_ms: quantile(realtime.map((x) => (x.freshnessMs ?? 0) + (now - x.updatedAt)), 0.95),
432 + multi_source_quotes: quotes.filter((x) => x.sourceCount >= 2).length,
433 + mean_confidence: quotes.length ? round(quotes.reduce((a, x) => a + x.confidence, 0) / quotes.length, 3) : null,
434 + mean_dispersion_bps: round(mean(quotes.map((x) => x.dispersionBps).filter((x): x is number => x != null)), 2),
435 + observations_per_sec: round(telemetry.rate("observations_received_total"), 1),
436 + queue_depth: telemetry.counter("observation_queue_depth"),
437 + by_connector: snaps.map((s) => ({ id: s.connectorId, state: s.state, messages_1m: s.messages1m, median_latency_ms: s.medianLatencyMs, reliability_score: s.reliabilityScore, instruments: s.instrumentsCovered, last_message_at: s.lastMessageAt ? new Date(s.lastMessageAt).toISOString() : null, last_error: s.lastError })),
438 + incidents: incidents.rows.map((r) => ({ id: r.id, type: r.type, ts: r.ts, severity: r.severity, title: r.title, data: r.data })),
439 + history: history.rows.map((r) => ({ hour: r.hour, healthy_ratio: Number(r.healthy_ratio), messages: Number(r.messages), latency_ms: r.latency == null ? null : Math.round(Number(r.latency)) })),
440 + });
441 + });
442 +}
443 +
444 +export const rowToEvent = (r: any) => ({
445 + id: r.id,
446 + type: r.type,
447 + timestamp: r.ts,
448 + instrument_ids: r.instrument_ids,
449 + severity: r.severity,
450 + confidence: r.confidence,
451 + source_count: r.source_count,
452 + sources: r.sources,
453 + title: r.title,
454 + summary: r.summary,
455 + data: r.data,
456 + confirmed_at: r.confirmed_at,
457 + instruments: (r.instrument_ids as string[]).map((id: string) => {
458 + const i = instruments.get(id);
459 + return i ? { id, symbol: i.symbol, name: i.name, asset_class: i.assetClass } : { id };
460 + }),
461 +});
462 +
463 +function groupStatus(list: ReturnType<typeof connectorManager.list>, ...classes: string[]) {
464 + const rel = list.filter((r) => r.def.metadata.assetClasses.some((c) => classes.includes(c)));
465 + if (!rel.length) return "unknown";
466 + const st = rel.map((r) => health.state(r.def.metadata.id));
467 + if (st.every((s) => s === "HEALTHY")) return "operational";
468 + if (st.some((s) => s === "HEALTHY")) return "degraded";
469 + return "outage";
470 +}
471 +
472 +function worst(states: string[]): string {
473 + const order = ["FAILED", "PAUSED", "DISABLED", "RECONNECTING", "STALE", "DEGRADED", "STARTING", "HEALTHY"];
474 + return states.sort((a, b) => order.indexOf(a) - order.indexOf(b))[0] ?? "NONE";
475 +}
476 +
477 +export const median = (xs: number[]) => quantile(xs, 0.5);
478 +export function quantile(xs: number[], qq: number): number | null {
479 + if (!xs.length) return null;
480 + const s = [...xs].sort((a, b) => a - b);
481 + return s[Math.min(s.length - 1, Math.floor(qq * s.length))] ?? null;
482 +}
483 +export const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : NaN);
484 +export const round = (n: number, d: number) => (Number.isFinite(n) ? Math.round(n * 10 ** d) / 10 ** d : null);
added apps/api/src/api/server.ts +103 −0
@@ -0,0 +1,103 @@
1 +import Fastify, { type FastifyInstance, type FastifyReply, type FastifyRequest } from "fastify";
2 +import cors from "@fastify/cors";
3 +import websocket from "@fastify/websocket";
4 +import { randomUUID } from "node:crypto";
5 +import { config } from "../config.js";
6 +import { logger } from "../logger.js";
7 +import { telemetry } from "../core/telemetry.js";
8 +import { registerPublicRoutes } from "./routes/public.js";
9 +import { registerMarketRoutes } from "./routes/markets.js";
10 +import { registerAdminRoutes } from "./routes/admin.js";
11 +import { registerStream } from "./stream.js";
12 +
13 +export class ApiError extends Error {
14 + constructor(
15 + public status: number,
16 + public code: string,
17 + message: string,
18 + ) {
19 + super(message);
20 + }
21 +}
22 +
23 +export const notFound = (what: string) => new ApiError(404, `${what.toUpperCase()}_NOT_FOUND`, `${what} not found`);
24 +export const badRequest = (msg: string) => new ApiError(400, "BAD_REQUEST", msg);
25 +
26 +/** Standard envelope: { data, meta } — meta carries request id, timestamp and the data status of the payload. */
27 +export function envelope<T>(data: T, meta: Record<string, unknown> = {}) {
28 + return { data, meta: { request_id: randomUUID(), timestamp: new Date().toISOString(), ...meta } };
29 +}
30 +
31 +export async function buildServer(): Promise<FastifyInstance> {
32 + const app = Fastify({
33 + logger: false,
34 + trustProxy: true,
35 + bodyLimit: 256 * 1024,
36 + disableRequestLogging: true,
37 + });
38 + await app.register(cors, { origin: true, methods: ["GET", "POST", "OPTIONS"], allowedHeaders: ["content-type", "x-ma-admin-token"] });
39 + await app.register(websocket, { options: { maxPayload: 64 * 1024 } });
40 +
41 + // Simple per-IP token bucket (public REST). 240 req/min sustained, 60 burst.
42 + const buckets = new Map<string, { tokens: number; last: number }>();
43 + app.addHook("onRequest", async (req, reply) => {
44 + (req as FastifyRequest & { startedAt: number }).startedAt = Date.now();
45 + if (req.url.startsWith("/v1/stream") || req.url.startsWith("/v1/sse")) return;
46 + const ip = req.ip;
47 + const now = Date.now();
48 + let b = buckets.get(ip);
49 + if (!b) {
50 + b = { tokens: 60, last: now };
51 + buckets.set(ip, b);
52 + }
53 + b.tokens = Math.min(60, b.tokens + ((now - b.last) / 1000) * 4);
54 + b.last = now;
55 + if (b.tokens < 1) {
56 + telemetry.inc("api_rate_limited_total");
57 + reply.header("retry-after", "5");
58 + return reply.code(429).send({ error: { code: "RATE_LIMITED", message: "too many requests" } });
59 + }
60 + b.tokens -= 1;
61 + if (buckets.size > 50_000) for (const [k, v] of buckets) if (now - v.last > 600_000) buckets.delete(k);
62 + });
63 + app.addHook("onSend", async (req, reply, payload) => {
64 + reply.header("x-content-type-options", "nosniff");
65 + reply.header("referrer-policy", "strict-origin-when-cross-origin");
66 + reply.header("x-frame-options", "DENY");
67 + if (config.isProd) reply.header("strict-transport-security", "max-age=31536000; includeSubDomains");
68 + const ms = Date.now() - ((req as FastifyRequest & { startedAt?: number }).startedAt ?? Date.now());
69 + telemetry.inc("api_requests_total", 1, { status: String(reply.statusCode) });
70 + telemetry.observe("api_latency_ms", ms);
71 + return payload;
72 + });
73 + app.setErrorHandler((err: unknown, req: FastifyRequest, reply: FastifyReply) => {
74 + if (err instanceof ApiError) return reply.code(err.status).send({ error: { code: err.code, message: err.message } });
75 + const e = err as { statusCode?: number; validation?: unknown; message?: string };
76 + if (e.validation) return reply.code(400).send({ error: { code: "VALIDATION_ERROR", message: e.message } });
77 + if (e.statusCode && e.statusCode < 500) return reply.code(e.statusCode).send({ error: { code: "REQUEST_ERROR", message: e.message } });
78 + logger.error({ err, url: req.url }, "unhandled API error");
79 + telemetry.inc("api_errors_total");
80 + return reply.code(500).send({ error: { code: "INTERNAL_ERROR", message: "internal error" } });
81 + });
82 + app.setNotFoundHandler((_req, reply) => reply.code(404).send({ error: { code: "NOT_FOUND", message: "route not found" } }));
83 +
84 + await registerPublicRoutes(app);
85 + await registerMarketRoutes(app);
86 + await registerAdminRoutes(app);
87 + await registerStream(app);
88 +
89 + // Edge mode: one public port. /v1/* (REST, WebSocket, SSE) is served here; every other path goes to the web app.
90 + if (config.webUpstream) {
91 + const { default: proxy } = await import("@fastify/http-proxy");
92 + await app.register(proxy, {
93 + upstream: config.webUpstream,
94 + prefix: "/",
95 + rewritePrefix: "/",
96 + websocket: false,
97 + http2: false,
98 + replyOptions: { rewriteRequestHeaders: (req, headers) => ({ ...headers, "x-forwarded-host": req.headers.host ?? "", "x-forwarded-proto": (req.headers["x-forwarded-proto"] as string) ?? "https" }) },
99 + });
100 + logger.info({ upstream: config.webUpstream }, "edge mode: proxying non-API paths to the web app");
101 + }
102 + return app;
103 +}
added apps/api/src/api/stream.ts +210 −0
@@ -0,0 +1,210 @@
1 +import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
2 +import type { WebSocket } from "ws";
3 +import type { CanonicalQuote, MarketEvent } from "@market-atlas/market-model";
4 +import { StreamClientMessageSchema } from "@market-atlas/market-model";
5 +import { bus } from "../core/bus.js";
6 +import { instruments } from "../core/instruments.js";
7 +import { telemetry } from "../core/telemetry.js";
8 +import { streamQuote } from "./public-view.js";
9 +import { resolveInstrument } from "./routes/public.js";
10 +import { logger } from "../logger.js";
11 +
12 +/**
13 + * Market Atlas stream gateway (WebSocket + SSE). Channels:
14 + * quotes:<SYMBOL|id> canonical quotes for one instrument
15 + * quotes:* every canonical quote (throttled per instrument to 2/s)
16 + * events:* every canonical market event
17 + * events:<COUNTRY> events touching instruments of a country (ISO-2)
18 + * events:type:<TYPE> events of one type
19 + * market:<exchange> market state changes
20 + * tape compact tape of price changes (batched at 10 Hz)
21 + * Messages carry `seq` so clients can detect gaps; the gateway batches at 10–20 Hz per client.
22 + */
23 +interface Client {
24 + send(data: string): void;
25 + channels: Set<string>;
26 + pending: unknown[];
27 + timer: NodeJS.Timeout | null;
28 + seq: number;
29 + lastQuoteAt: Map<string, number>;
30 +}
31 +
32 +const clients = new Set<Client>();
33 +let globalSeq = 0;
34 +
35 +function channelsFor(msg: { kind: "quote"; q: CanonicalQuote } | { kind: "event"; e: MarketEvent } | { kind: "market"; exchangeId: string }): string[] {
36 + if (msg.kind === "quote") {
37 + const inst = instruments.get(msg.q.instrumentId);
38 + const out = [`quotes:${msg.q.instrumentId}`, `quotes:${msg.q.symbol.toUpperCase()}`, "quotes:*", "tape"];
39 + if (inst?.assetClass) out.push(`quotes:class:${inst.assetClass}`);
40 + return out;
41 + }
42 + if (msg.kind === "event") {
43 + const out = ["events:*", `events:type:${msg.e.type}`];
44 + for (const id of msg.e.instrumentIds) {
45 + out.push(`events:${id}`);
46 + const inst = instruments.get(id);
47 + if (inst?.country) out.push(`events:${inst.country}`);
48 + if (inst?.symbol) out.push(`events:${inst.symbol.toUpperCase()}`);
49 + }
50 + return out;
51 + }
52 + return [`market:${msg.exchangeId}`, "market:*"];
53 +}
54 +
55 +function enqueue(c: Client, payload: unknown) {
56 + c.pending.push(payload);
57 + if (c.pending.length > 2000) c.pending.splice(0, c.pending.length - 2000); // backpressure: drop oldest
58 + if (!c.timer) {
59 + c.timer = setTimeout(() => {
60 + c.timer = null;
61 + const batch = c.pending.splice(0);
62 + if (!batch.length) return;
63 + c.seq++;
64 + try {
65 + c.send(JSON.stringify({ type: "batch", seq: c.seq, ts: Date.now(), messages: batch }));
66 + } catch {
67 + /* closed */
68 + }
69 + }, 100);
70 + }
71 +}
72 +
73 +function broadcast(channels: string[], payload: unknown, perInstrumentKey?: string) {
74 + globalSeq++;
75 + for (const c of clients) {
76 + let hit = false;
77 + for (const ch of channels) if (c.channels.has(ch)) hit = true;
78 + if (!hit) continue;
79 + if (perInstrumentKey && c.channels.has("quotes:*") && !c.channels.has(`quotes:${perInstrumentKey}`)) {
80 + const last = c.lastQuoteAt.get(perInstrumentKey) ?? 0;
81 + const now = Date.now();
82 + if (now - last < 500) continue;
83 + c.lastQuoteAt.set(perInstrumentKey, now);
84 + }
85 + enqueue(c, payload);
86 + }
87 +}
88 +
89 +bus.subscribe("canonical.quote", (q) => {
90 + if (!clients.size) return;
91 + const w = streamQuote(q);
92 + if (!w) return;
93 + broadcast(channelsFor({ kind: "quote", q }), w, q.instrumentId);
94 +});
95 +bus.subscribe("market.event", (e) => {
96 + if (!clients.size) return;
97 + broadcast(channelsFor({ kind: "event", e }), { type: "event", event: publicEvent(e) });
98 +});
99 +bus.subscribe("market.state", (m) => {
100 + if (!clients.size) return;
101 + broadcast(channelsFor({ kind: "market", exchangeId: m.exchangeId }), { type: "market_state", ...m });
102 +});
103 +
104 +export function publicEvent(e: MarketEvent) {
105 + return {
106 + id: e.id,
107 + type: e.type,
108 + timestamp: e.timestamp,
109 + instrument_ids: e.instrumentIds,
110 + instruments: e.instrumentIds.map((id) => {
111 + const i = instruments.get(id);
112 + return i ? { id, symbol: i.symbol, name: i.name, asset_class: i.assetClass, country: i.country } : { id };
113 + }),
114 + severity: e.severity,
115 + confidence: e.confidence,
116 + source_count: e.sourceCount,
117 + sources: e.sources,
118 + title: e.title,
119 + summary: e.summary,
120 + data: e.data,
121 + };
122 +}
123 +
124 +function normalizeChannels(chs: string[]): string[] {
125 + return chs.map((ch) => {
126 + const m = ch.match(/^quotes:(.+)$/);
127 + if (m && m[1] !== "*" && !m[1]!.startsWith("class:")) {
128 + const inst = resolveInstrument(m[1]!);
129 + return inst ? `quotes:${inst.id}` : `quotes:${m[1]!.toUpperCase()}`;
130 + }
131 + const e = ch.match(/^events:(.+)$/);
132 + if (e && e[1] !== "*" && !e[1]!.startsWith("type:")) {
133 + const inst = resolveInstrument(e[1]!);
134 + if (inst) return `events:${inst.id}`;
135 + return `events:${e[1]!.toUpperCase()}`;
136 + }
137 + return ch;
138 + });
139 +}
140 +
141 +export async function registerStream(app: FastifyInstance) {
142 + app.get("/v1/stream", { websocket: true }, (socket: WebSocket, req: FastifyRequest) => {
143 + const client: Client = { send: (d) => socket.send(d), channels: new Set(), pending: [], timer: null, seq: 0, lastQuoteAt: new Map() };
144 + clients.add(client);
145 + telemetry.inc("stream_clients_connected");
146 + telemetry.gauge("stream_clients", clients.size);
147 + socket.send(JSON.stringify({ type: "hello", version: 1, server_seq: globalSeq, ts: Date.now(), channels_hint: ["quotes:AAPL", "quotes:*", "events:*", "events:US", "market:xnas", "tape"] }));
148 + const hb = setInterval(() => {
149 + try {
150 + socket.send(JSON.stringify({ type: "heartbeat", ts: Date.now(), seq: client.seq }));
151 + } catch {
152 + /* ignore */
153 + }
154 + }, 25_000);
155 + socket.on("message", (raw) => {
156 + let msg;
157 + try {
158 + msg = StreamClientMessageSchema.parse(JSON.parse(raw.toString()));
159 + } catch {
160 + socket.send(JSON.stringify({ type: "error", code: "BAD_MESSAGE", message: "expected {action: subscribe|unsubscribe|ping, channels: []}" }));
161 + return;
162 + }
163 + if (msg.action === "ping") return socket.send(JSON.stringify({ type: "pong", id: msg.id ?? null, ts: Date.now() }));
164 + const chs = normalizeChannels(msg.channels);
165 + if (msg.action === "subscribe") {
166 + for (const ch of chs) if (client.channels.size < 500) client.channels.add(ch);
167 + } else for (const ch of chs) client.channels.delete(ch);
168 + socket.send(JSON.stringify({ type: "subscriptions", channels: [...client.channels] }));
169 + });
170 + socket.on("close", () => {
171 + clients.delete(client);
172 + if (client.timer) clearTimeout(client.timer);
173 + clearInterval(hb);
174 + telemetry.inc("stream_clients_disconnected");
175 + telemetry.gauge("stream_clients", clients.size);
176 + });
177 + socket.on("error", (err) => logger.debug({ err: err.message, ip: req.ip }, "stream socket error"));
178 + });
179 +
180 + /** SSE: /v1/sse?channels=quotes:AAPL,events:* */
181 + app.get<{ Querystring: { channels?: string } }>("/v1/sse", async (req: FastifyRequest<{ Querystring: { channels?: string } }>, reply: FastifyReply) => {
182 + const chs = normalizeChannels((req.query.channels ?? "events:*").split(",").map((s) => s.trim()).filter(Boolean).slice(0, 100));
183 + reply.raw.writeHead(200, {
184 + "content-type": "text/event-stream",
185 + "cache-control": "no-cache, no-transform",
186 + connection: "keep-alive",
187 + "x-accel-buffering": "no",
188 + "access-control-allow-origin": "*",
189 + });
190 + reply.raw.write(`event: hello\ndata: ${JSON.stringify({ version: 1, channels: chs, ts: Date.now() })}\n\n`);
191 + const client: Client = {
192 + send: (d) => reply.raw.write(`event: batch\nid: ${++globalSeq}\ndata: ${d}\n\n`),
193 + channels: new Set(chs),
194 + pending: [],
195 + timer: null,
196 + seq: 0,
197 + lastQuoteAt: new Map(),
198 + };
199 + clients.add(client);
200 + telemetry.inc("stream_clients_connected");
201 + const hb = setInterval(() => reply.raw.write(`: keepalive ${Date.now()}\n\n`), 20_000);
202 + req.raw.on("close", () => {
203 + clients.delete(client);
204 + clearInterval(hb);
205 + if (client.timer) clearTimeout(client.timer);
206 + telemetry.inc("stream_clients_disconnected");
207 + });
208 + await new Promise(() => {}); // keep the handler alive until the socket closes
209 + });
210 +}
added apps/api/src/cli.ts +123 −0
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/config.ts +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/bars.ts +107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/bus.ts +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/calendar.test.ts +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/calendar.ts +135 −0
Large diff (135 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/connector-manager.ts +371 −0
Large diff (371 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/consensus.test.ts +132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/consensus.ts +279 −0
Large diff (279 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/events.ts +247 −0
Large diff (247 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/health.ts +233 −0
Large diff (233 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/instruments.ts +242 −0
Large diff (242 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/jobs.ts +155 −0
Large diff (155 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/observations.ts +117 −0
Large diff (117 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/pipeline.ts +300 −0
Large diff (300 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/quotes.ts +110 −0
Large diff (110 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/raw-archive.ts +71 −0
Large diff (71 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/registry.ts +47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/core/telemetry.ts +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/db/migrate.ts +66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/db/pool.ts +49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/discovery/probe.ts +128 −0
Large diff (128 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/logger.ts +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/main.ts +86 −0
Large diff (86 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/seed/countries.json +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/seed/exchanges.json +63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/seed/holidays.json +136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added apps/api/src/seed/index.ts +61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/next.config.ts +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/package.json +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/postcss.config.mjs +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/globals.css +289 −0
Large diff (289 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/icon.svg +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/layout.tsx +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/logo.tsx +29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/market-clock.tsx +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/mobile-tab-bar.tsx +46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/search-dialog.tsx +172 −0
Large diff (172 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/site-footer.tsx +45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/site-header.tsx +53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/stream-indicator.tsx +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/layout/theme-toggle.tsx +28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/ui/confidence.tsx +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/ui/freshness.tsx +60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/ui/price.tsx +52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/ui/status-badge.tsx +68 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/api.ts +58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/client-api.ts +14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/fonts.ts +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/format.ts +190 −0
Large diff (190 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/prepaint.ts +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/site.ts +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/stream.ts +283 −0
Large diff (283 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/types.ts +331 −0
Large diff (331 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/tsconfig.json +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added connectors/package.json +13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/_shared/crypto.ts +102 −0
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/_shared/us-universe.ts +104 −0
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/bank-of-canada/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/bank-of-canada/fixtures/fx.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/bank-of-canada/fixtures/rates.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/bank-of-canada/index.ts +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/binance-ws/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/binance-ws/fixtures/miniTicker.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/binance-ws/index.ts +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/cboe-delayed-quotes/README.md +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/cboe-delayed-quotes/fixtures/closed_market.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/cboe-delayed-quotes/fixtures/index.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/cboe-delayed-quotes/fixtures/quote.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/cboe-delayed-quotes/index.ts +105 −0
Large diff (105 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/coinbase-ws/README.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/coinbase-ws/fixtures/malformed.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/coinbase-ws/fixtures/ticker.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/coinbase-ws/index.ts +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/connectors.test.ts +247 −0
Large diff (247 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/ecb-frankfurter/README.md +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/ecb-frankfurter/fixtures/timeseries.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/ecb-frankfurter/index.ts +96 −0
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/hfmarketdata/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/hfmarketdata/fixtures/bars.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/hfmarketdata/fixtures/continuous.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/hfmarketdata/index.ts +141 −0
Large diff (141 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/index.ts +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/kraken-ws/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/kraken-ws/fixtures/ticker.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/kraken-ws/index.ts +73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-market-calendar/README.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-market-calendar/fixtures/page.html +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-market-calendar/index.ts +98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-symbol-directory/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-symbol-directory/fixtures/nasdaqlisted.txt +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-symbol-directory/fixtures/otherlisted.txt +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-symbol-directory/index.ts +106 −0
Large diff (106 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-trade-halts/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-trade-halts/fixtures/rss.xml +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/nasdaq-trade-halts/index.ts +120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/okx-ws/README.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/okx-ws/fixtures/tickers.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/okx-ws/index.ts +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sec-company-tickers/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sec-company-tickers/fixtures/directory.json +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sec-company-tickers/index.ts +77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sec-edgar-filings/README.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sec-edgar-filings/fixtures/atom.xml +25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sec-edgar-filings/index.ts +73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/sources.ts +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/us-treasury-yield-curve/README.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/us-treasury-yield-curve/fixtures/month.xml +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added connectors/src/us-treasury-yield-curve/index.ts +106 −0
Large diff (106 lines) is not loaded by default to keep the page fast.Load diff
added deploy/market-atlas.mld.json +114 −0
Large diff (114 lines) is not loaded by default to keep the page fast.Load diff
added deploy/render-manifest.sh +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added docs/API.md +44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added docs/ARCHITECTURE.md +77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added docs/CONNECTORS.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added docs/DEPLOY.md +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added infra/migrations/0001_init.sql +346 −0
Large diff (346 lines) is not loaded by default to keep the page fast.Load diff
added package.json +28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/package.json +18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/define.ts +21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/fingerprint.ts +48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/http.ts +198 −0
Large diff (198 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/index.ts +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/parse.ts +181 −0
Large diff (181 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/ratelimit.ts +87 −0
Large diff (87 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/redact.ts +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/sdk.test.ts +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/testing.ts +91 −0
Large diff (91 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/types.ts +133 −0
Large diff (133 lines) is not loaded by default to keep the page fast.Load diff
added packages/connector-sdk/src/ws.ts +146 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/package.json +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/enums.ts +170 −0
Large diff (170 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/ids.ts +72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/index.ts +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/model.test.ts +44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/schemas.ts +91 −0
Large diff (91 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/time.ts +113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added packages/market-model/src/types.ts +331 −0
Large diff (331 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-lock.yaml +3122 −0
Large diff (3,122 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-workspace.yaml +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added tsconfig.base.json +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added tsconfig.json +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added vitest.config.ts +24 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere