SPB Git forge

spb/market-atlas

Public
12commits 1branches 0releases
1.1 MBsize
maindefault branch
10 days agolast push
TypeScript 96.7% SQL 1.6% CSS 0.8% JavaScript 0.5%

Market Atlas v0.1.0 — connector infrastructure, consensus engine, API/stream, 14 connectors

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026)

154 changed files +15,165 −0

added .env.example +15 −0
@@ -0,0 +1,15 @@
1 +# Market Atlas — runtime configuration (copy to .env for local development)
2 +APP_ENV=development
3 +MA_SITE_URL=http://localhost:8390
4 +MA_API_HOST=127.0.0.1
5 +MA_API_PORT=8391 # production: web 8380 / api 8381
6 +MA_ROLE=all # all | api | worker
7 +DATABASE_URL=postgres://localhost:5432/market_atlas
8 +MA_DATA_DIR=./data # raw payload archive, backups, logs
9 +MA_ADMIN_TOKEN=change-me # x-ma-admin-token header for /v1/admin/*
10 +MA_LOG_JSON=0
11 +MA_CONTACT_EMAIL=contact@market-atlas.co # sent in User-Agent to official sources (SEC requires it)
12 +HFMD_API_KEY= # hfmarketdata.io key (own data lake) — optional, raises limits
13 +MA_DISABLED_CONNECTORS= # comma-separated connector ids to keep off
14 +MA_RAW_SAMPLE_RATE=0.02 # share of streaming frames archived at L0 (bulk/poll payloads are always archived)
15 +MA_OBSERVATION_RETENTION_DAYS=45
added .gitignore +19 −0
@@ -0,0 +1,19 @@
1 +node_modules/
2 +dist/
3 +.next/
4 +.next-*/
5 +apps/web/next-env.d.ts
6 +*.tsbuildinfo
7 +.env
8 +.env.*
9 +!.env.example
10 +data/
11 +/tmp/
12 +logs/
13 +.DS_Store
14 +.claude/
15 +deploy/.admin-token
16 +deploy/.hfmd-key
17 +deploy/rendered/
18 +apps/web/qa/screens/
19 +coverage/
added CLAUDE.md +209 −0
@@ -0,0 +1,209 @@
1 +# CLAUDE.md
2 +
3 +# Market Atlas
4 +
5 +**Production domain:** `https://www.market-atlas.co`
6 +**Product name:** Market Atlas
7 +**Project type:** Global financial market intelligence, market-data aggregation, live-feed reconstruction, source-observation, time-series, and market-event platform
8 +**Primary interface:** Web application
9 +**Secondary interfaces:** Public/internal APIs, WebSocket/SSE streaming endpoints, background ingestion workers, connector management console
10 +**Deployment target:** Self-hosted production infrastructure / cluster (MacLustr, node M2U64 via `mld` — see `docs/DEPLOY.md`)
11 +**Primary objective:** Build a global, continuously updating representation of financial markets using a heterogeneous network of market-data connectors, public web sources, official feeds where available, observable frontend data streams, filings, exchange publications, issuer pages, and additional legally usable sources.
12 +
13 +> Implementation state (v0.1.0, 2026-09-12): TypeScript monorepo — `packages/market-model`, `packages/connector-sdk`, `connectors/` (14 production connectors),
14 +> `apps/api` (engines, REST/WS/SSE, admin, CLI), `apps/web` (Next.js). Read `docs/ARCHITECTURE.md`, `docs/CONNECTORS.md`, `docs/API.md`, `docs/FRONTEND.md`, `docs/DEPLOY.md` before changing anything structural.
15 +
16 +---
17 +
18 +# 1. CORE VISION
19 +
20 +Market Atlas must become a live map of global financial markets. This is NOT simply a stock screener, a quote website, a Yahoo Finance / TradingView / Bloomberg clone, a static company database, or an API wrapper. Market Atlas should behave as an independent **market observation infrastructure**: it continuously observes thousands of market-related sources, normalizes their outputs, identifies financial entities, compares independent observations, detects changes and anomalies, creates its own canonical event stream, stores historical observations, and redistributes normalized information through the Market Atlas application.
21 +
22 +```text
23 +THE WEB / EXCHANGES / ISSUERS / DATA SOURCES → CONNECTOR NETWORK → RAW OBSERVATIONS → NORMALIZATION + RESOLUTION → CONSENSUS / VALIDATION → MARKET BUS
24 + ┌──────────────┼──────────────┐
25 + Live UI Historical DB APIs
26 + Alerts Analytics WebSocket
27 +```
28 +
29 +Market Atlas becomes more valuable as time passes: every captured observation adds to the historical dataset. Two simultaneous products: **the live market state** and **the historical market memory**.
30 +
31 +# 2. PRIMARY PRODUCT PRINCIPLE
32 +
33 +Market Atlas must not depend structurally on one external financial-data API. External APIs may be optional sources, fallbacks, validation layers, or licensed sources — never architectural single points of failure. Build a flexible **connector infrastructure** able to observe many source types (exchange websites, issuer IR sites, regulatory filings, public quote pages and dashboards, exchange status pages, official downloadable files, RSS/Atom, public JSON endpoints, XHR/fetch responses used by frontends, GraphQL, SSE, public WebSocket streams, HTML, embedded JSON, JavaScript state, CSV, XML, PDF, official FTP/SFTP where permitted, public bulk downloads and object storage, press-release feeds, market-status endpoints). Always retain where an observation came from. Never detach market information from provenance.
34 +
35 +# 3. LEGAL AND DATA-USAGE PRINCIPLE
36 +
37 +Technical accessibility does NOT imply redistribution rights. Market Atlas has a first-class data-rights and provenance system. Never bypass authentication, paywalls, CAPTCHAs or access controls; never impersonate private users, use leaked credentials, scrape private pages, access data requiring unauthorized sessions, reverse-engineer protected feeds for unauthorized redistribution, or intentionally evade source restrictions.
38 +
39 +Every connector declares its data status: `PUBLIC_OPEN, PUBLIC_ATTRIBUTED, PUBLIC_RESTRICTED_REDISTRIBUTION, OFFICIAL_OPEN_DATA, LICENSED, DELAYED, INDICATIVE, RESEARCH_ONLY, INTERNAL_ONLY, UNKNOWN, DISABLED`. Every field supports `realtime_status: REALTIME, DELAYED, END_OF_DAY, INDICATIVE, STALE, UNKNOWN`. Without redistribution rights a source may be used internally where legally appropriate (validation, quality analysis) but its raw data must not be exposed publicly. This constraint is built into the architecture (`PUBLICLY_REDISTRIBUTABLE`, `publicQuote` withholding), not treated as documentation.
40 +
41 +# 4. PRODUCT PHILOSOPHY
42 +
43 +Make financial markets feel alive: Bloomberg infrastructure + FlightRadar24-like global visibility + GitHub activity feed + financial search engine + real-time event bus + market observatory — visually modern, approachable, fast, cleaner than a terminal. Avoid old-fashioned terminal aesthetics, excessive neon, fake "AI" gimmicks. Communicate scale, motion, data density, reliability, precision, geography, connectivity, provenance, temporal change.
44 +
45 +# 5. MAJOR PRODUCT SURFACES
46 +
47 +`/ /live /markets /stocks /etfs /indices /crypto /forex /commodities /bonds /rates /futures /options /exchanges /companies /countries /sectors /events /filings /earnings /dividends /splits /ipos /halts /insiders /news /sources /connectors /data-health /atlas /search /compare /api /developers /methodology /status`. Not every section ships in V1; the architecture must support them.
48 +
49 +# 6. HOMEPAGE
50 +
51 +"The world's markets, observed live." Hero: logo, universal search, live market status, instruments tracked, sources active, observations today, exchanges online, event throughput — **never hard-coded; metrics come from system telemetry** (`/v1/stats`). Then: Global Market Pulse (indices, crypto, FX, commodities, rates, top moves), Live Tape (real internal stream), Live Events (canonical events), World Market Map (open/closed, direction, activity), Source Health.
52 +
53 +# 7. THE LIVE FEED (`/live`)
54 +
55 +Global activity stream of market changes and events, filterable by asset class, country, exchange, sector, company, ticker, event type, magnitude, source confidence, time. Items come from canonical normalized events, never frontend hacks. Each item shows confidence and independent source count.
56 +
57 +# 8. CONNECTOR ARCHITECTURE
58 +
59 +The connector system is the core intellectual infrastructure — never a pile of scraping scripts. Standard interface (`packages/connector-sdk`): `metadata()`, optional `discover`, `start/stop` or `poll`, `normalize(raw)`, `health`. Metadata: id, name, organization, sourceType (WEBSOCKET, SSE, XHR, FETCH, GRAPHQL, HTML, RSS, CSV, XML, PDF, OFFICIAL_API, BULK_FILE), jurisdiction, rightsStatus, realtimeStatus, expectedLatency, supportsStreaming/Historical, symbols/exchanges/assetClasses, homepage, enabled, sourceFamily.
60 +
61 +# 9. CONNECTOR DISCOVERY ENGINE
62 +
63 +Automated (browser-based, Playwright/Chromium) discovery understands **how a page receives market information**: network requests, fetch/XHR, WebSocket, SSE, GraphQL, JSON/protobuf payloads, DOM mutations, embedded script state (Next/Nuxt/React hydration), downloadable files. It answers: does the page contain an instrument? how is the price obtained? which request changes with the price? symbol/price/volume/bid/ask/timestamp fields? delayed? multiplexed? subscription message? reconnect? public? authentication? restrictions? stability? Never auto-activate: `DISCOVERED → ANALYZED → VALIDATED → RIGHTS_REVIEW → STAGING → PRODUCTION`. (v0.1 ships a network-level prototype at `POST /v1/admin/discovery`.)
64 +
65 +# 10. NETWORK OBSERVATION
66 +
67 +Capture request URL/method/status/content type/headers/payload size/timing/initiator/resource type, WebSocket URL and frames, EventSource messages, JSON structure, field changes. Never store secrets; automatically redact cookies, Authorization, session identifiers, tokens, personal identifiers, CSRF secrets. Do not reproduce private sessions; discovery runs unauthenticated.
68 +
69 +# 11. AUTOMATIC PAYLOAD UNDERSTANDING
70 +
71 +Payloads vary enormously; normalize them to one canonical schema. Deterministic mappings first. LLMs may propose mappings, classify endpoints, generate connectors/docs/tests — production normalization must be reproducible. Never require an LLM per live tick.
72 +
73 +# 12. CANONICAL INSTRUMENT MODEL
74 +
75 +Global instrument master. Ticker alone never identifies an instrument: consider instrument_id, ticker, exchange, MIC, ISIN, FIGI (where legally usable), CUSIP (licensed), SEDOL, issuer, currency, country, asset class, security type. Example `eq_us_xnas_aapl`. Support symbol aliases (`BRK.B`, `BRK-B`, `BRK/B`, `BRK B`).
76 +
77 +# 13. ASSET CLASSES
78 +
79 +`EQUITY ETF ETN INDEX CRYPTO FOREX COMMODITY FUTURE OPTION BOND TREASURY INTEREST_RATE MUTUAL_FUND ADR GDR REIT WARRANT CFD OTHER`. Not all fields apply to all classes.
80 +
81 +# 14. OBSERVATION MODEL
82 +
83 +Atomic unit = observation: observation_id, instrument_id, field, value, currency, source_id, connector_id, source_timestamp, received_at, latency_ms, rights_status, realtime_status, confidence, raw_ref. Fields: `LAST_PRICE OPEN HIGH LOW PREVIOUS_CLOSE BID ASK BID_SIZE ASK_SIZE VOLUME VWAP MARKET_CAP CHANGE CHANGE_PERCENT OPEN_INTEREST YIELD RATE NAV 52_WEEK_HIGH 52_WEEK_LOW`.
84 +
85 +# 15. RAW DATA RETENTION
86 +
87 +Never discard raw input immediately. Layers: L0 RAW · L1 PARSED · L2 NORMALIZED · L3 CANONICAL · L4 DERIVED — debuggable, reprocessable.
88 +
89 +# 16. MARKET CONSENSUS ENGINE
90 +
91 +Sources disagree; never overwrite one price with the newest request. Weighted consensus using freshness, historical reliability, observed delay, source type, official status, error/disconnect rates, cross-source agreement, timestamp quality, coverage, known delay. Never conceal that a canonical value is an aggregation; expose methodology.
92 +
93 +# 17. SOURCE RELIABILITY SCORE
94 +
95 +Dynamic 0–100 operational score per source (availability 24h/30d, latency p50/p95, freshness, message rate, parse success, schema-change rate, reconnects, outlier rate, consensus agreement, timestamp integrity, coverage). It measures operational reliability, not financial truth.
96 +
97 +# 18. SOURCE DIVERGENCE
98 +
99 +Detect disagreement, mark stale sources, emit `SOURCE_DIVERGENCE`; useful internally and through data-health tools.
100 +
101 +# 19. MARKET EVENT ENGINE
102 +
103 +Transform noisy ticks into meaningful events: `PRICE_CHANGE SESSION_HIGH SESSION_LOW PRICE_BREAKOUT PRICE_GAP VOLUME_SPIKE VOLATILITY_SPIKE SPREAD_WIDENING SPREAD_COMPRESSION LIQUIDITY_CHANGE MARKET_OPEN MARKET_CLOSE TRADING_HALT TRADING_RESUME CORPORATE_ACTION DIVIDEND_DECLARED DIVIDEND_CHANGED STOCK_SPLIT EARNINGS_RELEASE GUIDANCE_CHANGE FILING_PUBLISHED INSIDER_TRANSACTION IPO_STATUS_CHANGE INDEX_CONSTITUENT_CHANGE RATING_CHANGE SOURCE_FAILURE SOURCE_RECOVERY SOURCE_DIVERGENCE`. Events carry id, type, instrument_ids, entity_ids, timestamp, severity, confidence, source_count, supporting observations, metadata.
104 +
105 +# 20. EVENT DEDUPLICATION
106 +
107 +Eleven sources publishing one announcement = one canonical event (entity, type, time window, semantic similarity, identifiers, references); confirmations increment source count.
108 +
109 +# 21–23. TEMPORAL DATABASE, AGGREGATION, COMPRESSION
110 +
111 +PostgreSQL for canonical metadata (instruments, exchanges, connectors, sources, events, configuration); time-series storage fit for purpose (Timescale/ClickHouse at scale — v0.1 uses daily-partitioned Postgres tables with gzip archive tiers); Redis for hot cache/pub-sub when needed; object storage for raw payloads, bulk files, PDFs, cold partitions. Canonical OHLCV bars 1s…1mo, reproducible from source observations. Columnar/delta/dictionary/ZSTD compression, partitioning, HOT (7 d) / WARM (90 d) / COLD tiers. Downsample strategically; never blindly delete history.
112 +
113 +# 24. INTERNAL MARKET BUS
114 +
115 +All normalized events enter one bus (NATS JetStream / Kafka-Redpanda / Redis Streams for small deployments; in-process today with spec topic names). `CONNECTORS → INGEST BUS → NORMALIZERS → CANONICAL BUS → { DB writers, WebSocket gateway, event engine, analytics, anomaly detection, monitoring }`. No connector → frontend coupling.
116 +
117 +# 25–27. STREAMING AND REST
118 +
119 +Own normalized stream (`wss://…/v1/stream`): subscribe/unsubscribe, heartbeat, reconnect, sequence ids, resume tokens, backpressure, rate limits, compression, batching; SSE for simple clients; REST `/v1/instruments, /v1/instruments/{id}, /v1/quotes/{symbol}, /v1/history/{symbol}, /v1/events, /v1/exchanges, /v1/markets, /v1/sources, /v1/status`. Never expose restricted observations without rights.
120 +
121 +# 28–31. COMPANY, EXCHANGE, COUNTRY PAGES, WORLD MAP
122 +
123 +Company/instrument page: live quote, market state, chart, volume, spread, market cap, performance, corporate actions, filings, earnings, dividends, related instruments, source confidence, events, profile, plus the unique **Data Provenance** panel (expandable, not overwhelming). Exchange page: status, local time, country, timezone, MIC, instruments tracked, advancers/decliners, volume, volatility, largest moves, events, halts, source health. Country page: exchanges, indices, largest companies, market cap, currency, bond yields, rates, commodity exposure, events, sector breakdown, hours — an atlas. World map: exchanges/countries/market centers with hours, volatility, activity; time-aware (New York activates at 09:30, Asia lights up as Europe closes). Informative, not decorative.
124 +
125 +# 32–34. SEARCH, COMPARE, MARKET MEMORY
126 +
127 +Universal hybrid search (exact identifiers, prefix, aliases, fuzzy, semantic) grouped by stocks, companies, indices, crypto, FX, commodities, bonds, exchanges, countries, events. `/compare` across asset classes with normalized performance, correlation, volatility, drawdown, event overlays. Market Replay: reconstruct prices, events, direction, volatility, filings, news, exchange status at time T.
128 +
129 +# 35–37. FILINGS, CORPORATE ACTIONS, MARKET HOURS
130 +
131 +Official regulators (SEC, SEDAR+, others, exchange announcements) → company, form type, timestamp, URLs, period, structured fields; never hallucinate contents; link every fact to its document. Corporate actions (dividends, splits, mergers, spin-offs, symbol changes, delistings, listings, rights, buybacks, special distributions) must adjust history correctly. Market-hours engine: time zones, DST, holidays, half days, special closures, overnight/pre/post/futures sessions; states `PRE OPEN HALTED CLOSED POST AUCTION UNKNOWN`; never hard-code 09:30–16:00.
132 +
133 +# 38–42. FRESHNESS, LATENCY, HEALTH, SCHEMA DRIFT, SELF-HEALING
134 +
135 +Every visible quote has freshness ("Updated 320 ms ago" / "Last update 4m 12s ago — data may be delayed"); never present stale as live. Track source/connector/frontend timestamps → source, ingestion, processing, delivery latency. Connector heartbeats and states `HEALTHY DEGRADED STALE RECONNECTING FAILED PAUSED DISABLED`; exponential backoff with jitter; no request storms. Schema fingerprints detect `SCHEMA_DRIFT`; pause unsafe normalization instead of producing wrong prices; notify admin. Self-healing workflow (replay → detect → discovery → proposed mapping → tests → compare → candidate → approval); an LLM never silently deploys production code.
136 +
137 +# 43–50. SDK, FIXTURES, ETHICS, EFFICIENCY, LIMITS, POLLING, CHANGE DETECTION
138 +
139 +Reusable SDK (`defineConnector`, HTTP, browser, WebSocket, SSE, retry, rate limiting, normalization, symbol resolution, timestamp parsing, logging, metrics, fixtures, tests). Every connector stores sample payloads; tests never touch production sources. Respectful crawling: host rate limits, connection limits, identifying user agent, caching, ETag / If-Modified-Since, backoff, robots awareness. Never amplify requests (one stream or bulk endpoint → many instruments; never one socket per symbol when multiplexing exists). Central rate limiter (token/leaky bucket, adaptive) keyed by source/host/connector/route/credential; connectors never sleep on their own. Adaptive polling (market state, activity, change frequency, constraints, importance, ETag). Content fingerprints (DOM/section/semantic/document/field hashes) trigger processing only on change.
140 +
141 +# 51–57. DOCUMENTS, AI, DERIVED METRICS, ANOMALIES, QUALITY, PROVENANCE GRAPH
142 +
143 +Document pipeline: download → hash → store raw → parse → metadata → classify → resolve company → extract facts → verify → index; keep the original. AI enhances (discovery, schema interpretation, classification, summarization, entity resolution, explanations, natural-language search) but never invents prices, trades, filings, market cap, exchange status. Derived metrics (returns, volatility, moving averages, drawdown, z-scores, relative volume, spread, breadth, correlation, beta, momentum, ATR, VWAP) are labelled derived. Statistically grounded anomaly detection with historical baselines. Every canonical datapoint may carry quality/confidence/source_count/freshness/dispersion/validation flags. Long-term provenance graph: Instrument → Observation → Connector → Source → Organization → URL; Company → Filing → Regulator → Document.
144 +
145 +# 58–61. ADMIN CONSOLE, CONNECTOR PAGES, DISCOVERY UI, STATUS
146 +
147 +`/admin` (strongly secured): overview, connectors, sources, instruments, entity resolution, observations, data quality, divergence, failed jobs, schema changes, events, storage, workers, streams, API, security, legal/data rights, logs. Per connector: status, type, host, rights, real-time status, last message/error, rates, latency, coverage, schema fingerprint, version, health history; actions pause/resume/restart/test/discover/fixtures/schema/sanitized payloads; never expose secrets. Discovery UI: submit URL → observed requests/WebSockets/SSE/JSON endpoints → likely quote source + field mapping → "Generate connector candidate" (not production). Public `/status` without infrastructure details.
148 +
149 +# 62–75. TECHNOLOGY, SERVICES, DEPLOYMENT, PROXY, FAILOVER, OBSERVABILITY, LOGGING, SECURITY, PERFORMANCE, MOBILE
150 +
151 +Frontend: Next.js, React, TypeScript, Tailwind, light state, WebSocket client, MapLibre/D3 for maps, ECharts/lightweight-charts/D3 by need; performance-driven library choices. Backend: TypeScript/Node for application services, Python for analytics/AI, Go/Rust only for proven high-throughput needs, PostgreSQL, ClickHouse/Timescale, Redis, NATS/Redpanda, Playwright; no new languages without clear reason. Services (web, api, stream-gateway, connector-manager/workers, discovery, normalization, entity-resolution, event-engine, analytics, database-writer, document-worker, scheduler, admin-api, health-monitor) start as a modular monolith and split when load justifies. Containerize where possible, health checks, safe restarts, no hand-started shells (MacLustr: PM2 via `mld`). TLS everywhere via reverse proxy (Caddy on the MacLustr Tunnel); no internal ports exposed. Workers support leases/locks/heartbeats/reassignment. Observability: Prometheus/Grafana/OpenTelemetry, structured JSON logs (never passwords, tokens, cookies, keys), metrics for observations/s, events/s, connector states, WebSocket connections, stream lag, latencies, queue depth, resources, storage growth. Security: HTTPS, secure headers, strict CORS, validation, parameterized SQL, rate limits, secret manager, admin MFA, least privilege, segmentation, dependency scanning, audit logs; isolated browser workers (no secrets, no DB credentials, temporary profiles); SSRF protection (localhost, private ranges, internal hostnames, metadata endpoints, redirects); API keys/quotas/analytics/revocation; service auth internally. Web performance: fast first paint, streaming updates without re-render storms, virtualized feeds, efficient charts, edge caching, SSR for public pages, granular subscriptions. Mobile-first surfaces (search, watch view, overview, instrument page, live feed, events, chart, world markets): bottom navigation, swipe-friendly cards, compact tape, sticky quote header, responsive charts, accessible touch targets, no horizontal overflow.
152 +
153 +# 76–84. DESIGN SYSTEM, MODES, COLOR, CHARTS, LIVE UI, SEO, SITEMAPS, LICENSING, DISCLAIMER
154 +
155 +Premium, dense but breathable, institutional, modern, high-information; not boxy, cartoonish or crypto-casino; hierarchy via spacing, typography, dividers, alignment, subtle surfaces — not isolated rounded cards. Dark and light modes first-class; avoid pure black; never red/green as the only signal. Color semantics: positive, negative, neutral, stale, warning, failure (+ icons/text). Charts: 1D 5D 1M 3M 6M YTD 1Y 5Y MAX, live intraday, line/area/candlestick/volume/comparison, no fake interpolation. Live UI: subtle directional flash, batch updates at 10–20 refresh cycles/s max. SEO: crawlable public pages with title/description/canonical/OpenGraph/structured data; no thin pages; segmented sitemaps (stocks, companies, exchanges, countries, indices, crypto, commodities). `/methodology`, `/data-sources`, `/licensing` explain methodology, delays, provenance, derived calculations, limitations, not-investment-advice, venue differences, licensing. Disclaimer: "Market Atlas is an informational market-data and analytics platform. Information may be delayed, indicative, incomplete, or sourced from third parties. Market Atlas is not an exchange, broker, investment adviser, or execution venue." (final legal review before commercial launch).
156 +
157 +# 85–94. TESTING, LOAD, REPLAY, MIGRATIONS, BACKUPS, RETENTION, REPROCESSING, VERSIONING
158 +
159 +Tests: unit, integration, connector (valid/missing/null/unexpected schema/stale timestamp/invalid symbol/duplicate/disconnect/reconnect/rate-limit/server error), schema, normalization, entity resolution, WebSocket, API, database, frontend, end-to-end, load, security. Consensus scenarios: all agree, one stale, one outlier, two clusters, single source, differing timestamps, market closed. Load: 100K obs/s, 10K clients, 100K subscriptions, 1M cached instruments, large historical queries — scale progressively. Internal replay feeds stored observations through normalizer/consensus/events. Versioned forward-only migrations; never edit production tables by hand. Automated backups (Postgres, metadata, connector config, object storage replication) — a backup never restored is not trusted. Configurable retention by source/asset class/field/resolution/rights (raw WS frames 7 d; normalized ticks permanent compressed; 1s/1m/1d bars permanent). Reprocess raw → canonical v2 with lineage; canonical datasets carry `normalizer_version`, `consensus_version`.
160 +
161 +# 95–105. IDENTIFIERS, SOURCES DIRECTORY, DATA HEALTH, "WHY THIS PRICE?", COVERAGE, PRIORITY, TIERS, MULTI-SOURCE, CORRELATION, LINEAGE, DELAY DETECTION
162 +
163 +Stable public ids (`eq_us_xnas_aapl`, `crypto_btc_usd`, `fx_eur_usd`, `index_us_spx`), never auto-increment ids. `/sources` directory by category with status/coverage/frequency/latency/rights/last observation — no endpoint details. `/data-health`: sources online, global freshness, markets covered, consensus, incidents — real metrics only. **"Why this price?"** on every instrument: canonical price, independent observations, dispersion, newest/oldest included, weights, reasons. Market Atlas coverage score per country/exchange (observation coverage, not market quality). Connector priority engine (market cap represented, demand, coverage gap, redundancy, legal accessibility, reliability, cost). Tier 1: major exchanges, regulators, large-cap IR, major public quote sources, crypto exchanges with public feeds, central banks, government debt, commodity references; Tier 2: regional exchanges, ETF issuers, index providers, economic data, specialized sites; Tier 3: small exchanges, local portals, specialized classes, regional announcements. Target ≥ 3 independent observations for major instruments (5–10+ for the most important). Source families: websites sharing an upstream are not independent; weight independence; infer lineage from timestamps/identical payloads/rounding/documentation/attribution — never assert hidden commercial relationships without evidence. Estimate delay against trusted references (0–2 s, 15 s, 1 min, 15 min, 20 min, EOD) and label estimates as such.
164 +
165 +# 106–121. MARKET CLOCK, TOPICS, IDEMPOTENCY, ORDERING, CLOCKS, TIMESTAMP TRUST, ENTITY & EXCHANGE GRAPHS, CURRENCY, LATER FEATURES, DEVELOPER PORTAL, API CONVENTIONS
166 +
167 +Global market clock (New York, London, Tokyo, Hong Kong, Sydney) with states. Topics: `raw.{source} normalized.quote normalized.trade normalized.event canonical.quote market.event connector.health source.divergence filing.detected corporate_action.detected`. Idempotent ingestion via fingerprints (source + instrument + timestamp + value + type); no duplicate rows on reconnect replay. Out-of-order messages: keep source sequence/source timestamp/receive timestamp. NTP-synchronized clocks. Timestamp trust: exchange / source / connector / unknown → influences confidence. Company ≠ security (common stock, ADR, preferred, bonds, subsidiaries). Exchange graph (operator, venue, MIC, country, timezone, asset classes, sessions). Store original currency; converted values record FX source/time. Later: watchlists, saved searches, alerts (price, volume, volatility, filing, earnings, dividend, halt, divergence, highs/lows, events), dashboards, API keys, exports, notebooks — the public atlas must be useful first. No brokerage in V1 (no execution, custody, deposits, leverage). `/developers`: REST, WebSocket, SSE, schemas, examples, SDKs (JS, Python, Go), status, rate limits. Versioned `/v1/`, no silent breaking changes; envelopes `{data, meta{request_id, timestamp, data_status}}`; structured errors `{error{code, message}}`, no stack traces.
168 +
169 +# 122. PROJECT STRUCTURE
170 +
171 +```text
172 +market-atlas/
173 +├── apps/ web · api (engines, stream gateway, admin, worker roles)
174 +├── packages/ market-model · connector-sdk (+ future: schemas, database, streaming, logging, security, ui)
175 +├── connectors/ one directory per connector (index.ts, fixtures/, README.md)
176 +├── infra/ migrations (+ docker, monitoring, deployment as they come)
177 +├── deploy/ mld manifest + render script (MacLustr)
178 +├── docs/ ARCHITECTURE · CONNECTORS · API · FRONTEND · DEPLOY
179 +├── scripts/ tests/
180 +└── CLAUDE.md
181 +```
182 +
183 +# 123. CODING RULES FOR CLAUDE
184 +
185 +1. Read this file before major architecture changes. 2. Inspect existing code before creating duplicate systems. 3. Prefer reusable infrastructure over one-off scripts. 4. Do not break existing connectors. 5. No hard-coded secrets. 6. No hard-coded production IPs. 7. No fake market data. 8. Never silently replace real data with mock values. 9. Strict TypeScript. 10. Validate external payloads. 11. Treat all external content as untrusted. 12. Add meaningful tests. 13. Keep the UI responsive. 14. Preserve data provenance. 15. Never present unknown freshness as real time. 16. Never remove data-rights metadata. 17. Prefer graceful degradation. 18. Add monitoring to new ingestion systems. 19. Document unusual source behaviour. 20. Keep connector code isolated from core business logic.
186 +
187 +# 124–127. WHEN ADDING A CONNECTOR / SCHEMA / FIELD / AI
188 +
189 +Connector = implementation + metadata + rights classification + normalization mapping + fixtures + unit tests + health check + documentation (no undocumented connectors). Schema change = inspect database, API, stream schemas, frontend consumers, tests, historical processing — no silent breaking changes. New market field = canonical name, type, unit, applicable asset classes, nullability, source-timestamp handling, display formatting. Adding AI: if deterministic code can solve it, use deterministic code; AI only where interpretation is needed; no LLM dependencies for basic parsing.
190 +
191 +# 128–137. PERFORMANCE, QUERIES, CACHE, CONTRACTS, EVENT SCHEMA, DISASTER MODES, CIRCUIT BREAKERS, SHUTDOWN, BACKPRESSURE, PRIORITY
192 +
193 +Never solve scale only with bigger servers — first inspect duplication, query structure, indexes, cacheability, batching, serialization, compression, memory copies, re-renders. Heavy historical queries: partition pruning, indexes, no `SELECT *`, limits, pagination, downsampling. Cache instrument/exchange metadata, latest quotes, market status, popular pages, aggregates — never user-specific or stale critical values incorrectly. Strongly versioned contracts (zod / JSON Schema / Protobuf / Avro); explicit bus contracts; event schema `{version, id, type, instrument_id, timestamp, confidence, severity, data, sources}`. Define degraded behaviour when Redis/ClickHouse/source/gateway/worker/browser/network/schema fail (e.g. historical DB down → live stream continues, charts unavailable). Circuit breakers on failing sources. Graceful shutdown: stop new jobs, flush, commit, release leases, close, exit. Bounded queues, batch writes, disk-backed queues where needed, priority tiers, drop policy only for expendable data — never silently drop canonical events. Priorities: P0 halts/critical events, P1 canonical quotes, P2 ticks, P3 secondary analytics, P4 reprocessing.
194 +
195 +# 138–151. ONBOARDING PIPELINE, EXPANSION, HISTORY, ARCHIVE, KNOWLEDGE GRAPH, CONSTITUENTS, DIFFS, EXPLORER, "MARKET CHANGES", BREADTH, ACTIVITY INDEX, PRECISION, PREDICTIONS
196 +
197 +Automated onboarding (candidate → metadata → discovery → rights → fixtures → mapping → validation → comparison → staging → observation → production). Long-term source discovery jobs feed a review queue; no unlimited crawling of arbitrary domains. Store source change history (versions, schema, endpoints, availability, delay class, rights class) and allow historical source-quality queries. Market observation archive: "what did the observable financial web say at time T?". Knowledge graph (Company, Instrument, Exchange, Country, Sector, Executive, ETF, Index, Filing, Corporate Action, Event, Source; LISTED_ON, HEADQUARTERED_IN, MEMBER_OF_INDEX, HOLDS, ISSUED_BY, FILED, ANNOUNCED, OBSERVED_BY). Index membership with effective dates. Diff log for slowly changing facts (old/new/detected/effective/source). Future `/explore` data laboratory. "Market changes" product (what changed in 1 min / 5 min / 1 h / 1 d: instruments > 1 %, filings, halts, new highs, source outages). Global breadth engine (advancers, decliners, highs/lows, median/weighted return, sector/country breadth, regional scopes). Internal activity score for ranking what is happening now — never a prediction. No fake precision (no 99.999 % confidence without justification; transparent methodology). Core product = observation, aggregation, historical intelligence, analytics — not guaranteed predictions or recommendations.
198 +
199 +# 152–158. V1 DEFINITION, CONNECTOR GOAL, PHASES, SUCCESS METRICS, NORTH STAR
200 +
201 +V1: homepage, universal search, market overview, live feed, equities, crypto, major FX, major indices, instrument pages, charts, basic map, sources directory, data health, connector framework, 5–20 strong production connectors, health monitoring, canonical normalization, basic consensus, PostgreSQL, time-series storage, cache, WebSocket gateway, admin console. Prove diversity: WebSocket, XHR/fetch, official source, HTML change detection, regulatory/filing source, bulk structured source. Phase 2: 50–100 connectors, multiple exchanges, corporate actions, filings, world map, consensus, country pages, advanced events. Phase 3: hundreds/thousands of connectors, discovery automation, global instrument graph, replay, developer APIs, advanced analytics, institutional monitoring. Phase 4: answer *what is happening, where, how fast, which sources confirm it, how reliable, what happened before, what else moved*. Metrics: active connectors, uptime, instruments covered, observations/day, median freshness, consensus coverage, quality incidents, historical depth, stream latency; product: searches, views, live-feed engagement, return users, API usage. **North star: high-quality, provenance-backed market observations successfully processed.**
202 +
203 +# 159–166. BRAND, STANDARD, THE MOST IMPORTANT RULE, MINDSET, IMPLEMENTATION ORDER, FIRST ENTITIES, NO 1,000 SCRAPERS, EXPECTATION
204 +
205 +Brand: **Market Atlas**, `www.market-atlas.co` — "The live map of global markets." Every feature reinforces GLOBAL, LIVE, HISTORICAL, VERIFIABLE, MULTI-SOURCE, CONNECTED, SEARCHABLE, FAST; otherwise reconsider it. **The proprietary asset is not the website**: it is connectors + normalization + entity graph + source reliability + consensus + historical observations + market events; the frontend is the best interface to that system. Build for 100,000+ instruments, 1,000+ sources, hundreds of millions of observations/day, years of history, thousands of stream consumers — incrementally, without premature distributed complexity. Implementation order: audit → structure → canonical schemas → DB models → connector SDK → rights metadata → representative connectors → ingestion bus → normalization → resolver → current-state cache → canonical quotes → WebSocket gateway → historical persistence → health metrics → consensus → REST → homepage → instrument page → live feed → admin dashboard → discovery prototype → world map → observability → load test/harden. First entities: Instrument, Company, Exchange, Country, Currency, Source, Connector, ConnectorVersion, Observation, CanonicalQuote, MarketEvent, CorporateAction, Filing, SourceHealth, ConnectorHealth, SymbolAlias, TradingSession. Strategy: excellent connector infrastructure → validate with 10 heterogeneous sources → improve → 50 → automate onboarding → hundreds → thousands; twenty excellent connectors beat a thousand fragile scripts. Claude behaves as a senior systems/data/quant-data/security/product engineer: inspect what exists, preserve working behaviour, smallest coherent architectural improvement, implement rather than describe, test, verify production compatibility, update documentation, leave the repository deployable. Never substitute placeholders for requested real functionality unless an upstream dependency makes it unavoidable; when a source cannot legally or technically provide real-time information, label it accurately.
206 +
207 +# END GOAL
208 +
209 +Market Atlas should function like a live nervous system for financial markets: thousands of independent sources emit observations; Market Atlas listens, resolves what they refer to, measures their reliability, compares them, records them, detects what changed, remembers what happened, and exposes the resulting market state through one unified global interface — **https://www.market-atlas.co**.
added README.md +25 −0
@@ -0,0 +1,25 @@
1 +# Market Atlas
2 +
3 +**The live map of global markets** — https://www.market-atlas.co
4 +
5 +Market Atlas observes thousands of market-related sources (exchange feeds, official statistics, regulators, public quote pages,
6 +directories), normalizes every observation with its provenance and data rights, resolves it to a canonical instrument, compares
7 +independent sources into a consensus value, detects what changed, remembers everything, and redistributes the result through one
8 +web application, a REST API and a WebSocket/SSE stream.
9 +
10 +```
11 +pnpm install
12 +cp .env.example .env # Postgres URL, admin token, contact e-mail, optional HFMD_API_KEY
13 +pnpm migrate && pnpm seed # schema + reference data (countries, exchanges, holidays)
14 +pnpm dev:api # Fastify + workers + connectors on :8391 (roles: MA_ROLE=all|api|worker)
15 +pnpm dev:web # Next.js on :8390 (rewrites /v1 → API)
16 +pnpm test · pnpm typecheck # vitest (model, SDK, connectors, consensus, calendar) · tsc
17 +pnpm ma connectors | run <id> | replay <day> | status | archive | backup
18 +```
19 +
20 +Repository map: `packages/market-model` (canonical vocabulary & types) · `packages/connector-sdk` · `connectors/` (14 production
21 +connectors with fixtures and READMEs) · `apps/api` (engines, API, stream, admin, CLI) · `apps/web` (Next.js) · `infra/migrations` ·
22 +`docs/` (ARCHITECTURE, CONNECTORS, API, FRONTEND, DEPLOY) · `deploy/` (mld manifest) · `CLAUDE.md` (product & engineering charter).
23 +
24 +Market Atlas is an informational market-data and analytics platform. Information may be delayed, indicative, incomplete, or sourced
25 +from third parties. Market Atlas is not an exchange, broker, investment adviser, or execution venue.
added apps/api/build.mjs +22 −0
@@ -0,0 +1,22 @@
1 +// Bundles the API (and the workspace packages it imports) into dist/ with esbuild.
2 +// Third-party dependencies stay external (resolved from node_modules at runtime).
3 +import { build } from "esbuild";
4 +import { readFileSync } from "node:fs";
5 +
6 +const pkg = JSON.parse(readFileSync(new URL("./package.json", import.meta.url), "utf8"));
7 +const external = Object.keys(pkg.dependencies).filter((d) => !d.startsWith("@market-atlas/"));
8 +
9 +await build({
10 + entryPoints: ["src/main.ts", "src/cli.ts"],
11 + outdir: "dist",
12 + bundle: true,
13 + platform: "node",
14 + format: "esm",
15 + target: "node22",
16 + sourcemap: true,
17 + external,
18 + logLevel: "info",
19 + banner: {
20 + js: "import { createRequire as __createRequire } from 'node:module'; const require = __createRequire(import.meta.url);",
21 + },
22 +});
added apps/api/package.json +33 −0
@@ -0,0 +1,33 @@
1 +{
2 + "name": "@market-atlas/api",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "scripts": {
7 + "build": "node build.mjs",
8 + "dev": "node --env-file=../../.env --import tsx src/main.ts",
9 + "start": "node dist/main.js",
10 + "cli": "node --env-file-if-exists=../../.env --import tsx src/cli.ts",
11 + "migrate": "node --env-file-if-exists=../../.env --import tsx src/cli.ts migrate",
12 + "seed": "node --env-file-if-exists=../../.env --import tsx src/cli.ts seed"
13 + },
14 + "dependencies": {
15 + "@fastify/cors": "^11.0.1",
16 + "@fastify/http-proxy": "^11.1.2",
17 + "@fastify/websocket": "^11.1.0",
18 + "@market-atlas/connector-sdk": "workspace:*",
19 + "@market-atlas/connectors": "workspace:*",
20 + "@market-atlas/market-model": "workspace:*",
21 + "fastify": "^5.4.0",
22 + "pg": "^8.16.3",
23 + "pino": "^9.7.0",
24 + "ws": "^8.18.3",
25 + "zod": "^3.24.1"
26 + },
27 + "devDependencies": {
28 + "@types/pg": "^8.15.4",
29 + "@types/ws": "^8.18.1",
30 + "esbuild": "^0.25.9",
31 + "tsx": "^4.20.5"
32 + }
33 +}
added apps/api/src/api/public-view.ts +106 −0
@@ -0,0 +1,106 @@
1 +import type { CanonicalQuote, Instrument } from "@market-atlas/market-model";
2 +import { PUBLICLY_REDISTRIBUTABLE } from "@market-atlas/market-model";
3 +import { calendar } from "../core/calendar.js";
4 +
5 +/**
6 + * Public representation of a canonical quote. Values whose rights do not allow redistribution are
7 + * withheld (price nulled, `withheld: true`) while provenance/health metadata stays visible.
8 + */
9 +/**
10 + * @param marketState state of the instrument's venue (from the calendar) — when the venue is closed the
11 + * last session value is reported as AT_CLOSE instead of STALE.
12 + */
13 +export function publicQuote(q: CanonicalQuote | undefined, inst?: Instrument, marketState?: string | null) {
14 + if (!q) return null;
15 + const redistributable = PUBLICLY_REDISTRIBUTABLE.has(q.rightsStatus);
16 + const now = Date.now();
17 + const age = now - q.updatedAt;
18 + const staleFor = q.realtimeStatus === "REALTIME" ? 60_000 : q.realtimeStatus === "DELAYED" ? 45 * 60_000 : 4 * 86_400_000;
19 + if (marketState === undefined) {
20 + const ex = inst?.exchangeId ? calendar.exchange(inst.exchangeId) : undefined;
21 + marketState = ex && !ex.sessions.continuous ? calendar.state(ex.id, now) : null;
22 + }
23 + const closed = marketState != null && marketState !== "OPEN" && marketState !== "UNKNOWN";
24 + const recentSession = q.sourceTimestamp != null && now - q.sourceTimestamp < 4 * 86_400_000;
25 + let status: string;
26 + if (!redistributable) status = "WITHHELD";
27 + else if (q.realtimeStatus === "STALE") status = closed && q.price != null && recentSession ? "AT_CLOSE" : "STALE";
28 + else if (closed && recentSession && (q.realtimeStatus === "REALTIME" || q.realtimeStatus === "DELAYED")) status = "AT_CLOSE";
29 + else if (age > staleFor) status = "STALE";
30 + else status = q.realtimeStatus;
31 + const v = (x: number | null) => (redistributable ? x : null);
32 + return {
33 + instrument_id: q.instrumentId,
34 + symbol: q.symbol,
35 + name: inst?.name ?? null,
36 + asset_class: inst?.assetClass ?? null,
37 + exchange_id: inst?.exchangeId ?? null,
38 + country: inst?.country ?? null,
39 + price: v(q.price),
40 + open: v(q.open),
41 + high: v(q.high),
42 + low: v(q.low),
43 + previous_close: v(q.previousClose),
44 + change: v(q.change),
45 + change_percent: v(q.changePercent),
46 + volume: v(q.volume),
47 + bid: v(q.bid),
48 + ask: v(q.ask),
49 + currency: q.currency,
50 + source_count: q.sourceCount,
51 + dispersion_bps: q.dispersionBps,
52 + confidence: q.confidence,
53 + freshness_ms: q.freshnessMs == null ? null : q.freshnessMs + age,
54 + data_status: status,
55 + market_state: marketState ?? null,
56 + rights_status: q.rightsStatus,
57 + withheld: !redistributable,
58 + updated_at: new Date(q.updatedAt).toISOString(),
59 + source_timestamp: q.sourceTimestamp == null ? null : new Date(q.sourceTimestamp).toISOString(),
60 + session_high: v(q.sessionHigh),
61 + session_low: v(q.sessionLow),
62 + };
63 +}
64 +
65 +export type PublicQuote = NonNullable<ReturnType<typeof publicQuote>>;
66 +
67 +export function publicInstrument(i: Instrument) {
68 + return {
69 + id: i.id,
70 + symbol: i.symbol,
71 + name: i.name,
72 + asset_class: i.assetClass,
73 + exchange_id: i.exchangeId,
74 + mic: i.mic,
75 + currency: i.currency,
76 + country: i.country,
77 + company_id: i.companyId,
78 + security_type: i.securityType,
79 + base: i.base,
80 + quote: i.quote,
81 + is_active: i.isActive,
82 + metadata: i.metadata,
83 + };
84 +}
85 +
86 +/** Compact wire format for the stream ("quote" messages). */
87 +export function streamQuote(q: CanonicalQuote) {
88 + if (!PUBLICLY_REDISTRIBUTABLE.has(q.rightsStatus)) return null;
89 + return {
90 + type: "quote" as const,
91 + instrument_id: q.instrumentId,
92 + symbol: q.symbol,
93 + price: q.price,
94 + change: q.change,
95 + change_pct: q.changePercent,
96 + bid: q.bid,
97 + ask: q.ask,
98 + volume: q.volume,
99 + currency: q.currency,
100 + timestamp: q.sourceTimestamp ?? q.updatedAt,
101 + received: q.updatedAt,
102 + confidence: q.confidence,
103 + sources: q.sourceCount,
104 + status: q.realtimeStatus,
105 + };
106 +}
added apps/api/src/api/routes/admin.ts +155 −0
@@ -0,0 +1,155 @@
1 +import type { FastifyInstance, FastifyRequest } from "fastify";
2 +import { timingSafeEqual } from "node:crypto";
3 +import { config } from "../../config.js";
4 +import { pool } from "../../db/pool.js";
5 +import { connectorManager } from "../../core/connector-manager.js";
6 +import { health } from "../../core/health.js";
7 +import { rawArchive } from "../../core/raw-archive.js";
8 +import { telemetry } from "../../core/telemetry.js";
9 +import { observationWriter } from "../../core/observations.js";
10 +import { assertPublicUrl, redactObject } from "@market-atlas/connector-sdk";
11 +import { ApiError, envelope, notFound } from "../server.js";
12 +import { runDiscovery } from "../../discovery/probe.js";
13 +
14 +function requireAdmin(req: FastifyRequest) {
15 + const token = req.headers["x-ma-admin-token"];
16 + if (!config.adminToken || typeof token !== "string" || token.length !== config.adminToken.length || !timingSafeEqual(Buffer.from(token), Buffer.from(config.adminToken))) {
17 + telemetry.inc("admin_auth_failures_total");
18 + throw new ApiError(401, "UNAUTHORIZED", "admin token required");
19 + }
20 +}
21 +
22 +export async function registerAdminRoutes(app: FastifyInstance) {
23 + app.addHook("onRequest", async (req) => {
24 + if (req.url.startsWith("/v1/admin")) requireAdmin(req);
25 + });
26 +
27 + app.get("/v1/admin/overview", async () => {
28 + const [obs, evts, sizes] = await Promise.all([
29 + pool.query<{ n: number }>("select coalesce(sum(observations),0)::bigint as n from daily_stats"),
30 + pool.query<{ n: number }>("select count(*)::int as n from market_events"),
31 + pool.query<{ relname: string; bytes: number }>(
32 + `select c.relname, pg_total_relation_size(c.oid)::bigint as bytes from pg_class c join pg_namespace n on n.oid = c.relnamespace where n.nspname = 'public' and c.relkind in ('r','p') order by 2 desc limit 25`,
33 + ),
34 + ]);
35 + return envelope({
36 + version: config.version,
37 + role: config.role,
38 + uptime_s: Math.round(process.uptime()),
39 + memory: process.memoryUsage(),
40 + observations_total: obs.rows[0]?.n ?? 0,
41 + events_total: evts.rows[0]?.n ?? 0,
42 + queue_depth: observationWriter.depth(),
43 + rates: { observations_per_sec: telemetry.rate("observations_received_total"), events_per_min: telemetry.rate("events_total") * 60, http_per_min: telemetry.rate("http_requests_total") * 60 },
44 + tables: sizes.rows,
45 + rate_limiter: connectorManager.limiter.snapshot(),
46 + bus: telemetry.counter("canonical_quotes_total"),
47 + });
48 + });
49 +
50 + app.get("/v1/admin/connectors", async () => {
51 + return envelope(
52 + connectorManager.list().map((r) => ({
53 + metadata: r.def.metadata,
54 + health: health.snapshot(r.def.metadata.id),
55 + paused: r.paused,
56 + running: r.running,
57 + symbols: r.symbols,
58 + schedule: r.def.schedule ?? null,
59 + consecutive_failures: r.consecutiveFailures,
60 + drift_strikes: r.driftStrikes,
61 + sockets: r.sockets.map((s) => ({ url: s.url.replace(/\?.*$/, ""), open: s.isOpen(), reconnects: s.reconnects, connected_at: s.connectedAt })),
62 + })),
63 + );
64 + });
65 +
66 + app.get<{ Params: { id: string } }>("/v1/admin/connectors/:id", async (req) => {
67 + const rt = connectorManager.get(req.params.id);
68 + if (!rt) throw notFound("connector");
69 + const [changes, healthHistory, recent] = await Promise.all([
70 + pool.query("select id, kind, old_fingerprint, new_fingerprint, sample, detected_at, acknowledged from connector_schema_changes where connector_id = $1 order by detected_at desc limit 20", [rt.def.metadata.id]),
71 + pool.query("select ts, status, messages_1m, errors_1m, median_latency_ms, reliability_score from connector_health where connector_id = $1 and ts > now() - interval '24 hours' order by ts", [rt.def.metadata.id]),
72 + pool.query("select received_at, source_ts, instrument_id, field, value, latency_ms, raw_ref from observations where connector_id = $1 and received_at > now() - interval '1 day' order by received_at desc limit 25", [rt.def.metadata.id]),
73 + ]);
74 + return envelope({
75 + metadata: rt.def.metadata,
76 + health: health.snapshot(rt.def.metadata.id),
77 + paused: rt.paused,
78 + running: rt.running,
79 + symbols: rt.symbols,
80 + schedule: rt.def.schedule ?? null,
81 + state: redactObject(rt.stateCache),
82 + schema_changes: changes.rows,
83 + health_history: healthHistory.rows,
84 + recent_observations: recent.rows,
85 + fingerprints: health.fingerprints(rt.def.metadata.id),
86 + });
87 + });
88 +
89 + for (const action of ["pause", "resume", "restart"] as const) {
90 + app.post<{ Params: { id: string } }>(`/v1/admin/connectors/:id/${action}`, async (req) => {
91 + const rt = connectorManager.get(req.params.id);
92 + if (!rt) throw notFound("connector");
93 + await connectorManager[action](rt.def.metadata.id);
94 + return envelope({ id: rt.def.metadata.id, action, state: health.state(rt.def.metadata.id) });
95 + });
96 + }
97 +
98 + app.post<{ Params: { id: string } }>("/v1/admin/connectors/:id/test", async (req) => {
99 + const rt = connectorManager.get(req.params.id);
100 + if (!rt) throw notFound("connector");
101 + try {
102 + const res = await connectorManager.runOnce(rt.def.metadata.id);
103 + return envelope({ ok: true, ...res });
104 + } catch (err) {
105 + return envelope({ ok: false, error: err instanceof Error ? err.message : String(err) });
106 + }
107 + });
108 +
109 + app.post<{ Params: { id: string; changeId: string } }>("/v1/admin/connectors/:id/schema-changes/:changeId/ack", async (req) => {
110 + await pool.query("update connector_schema_changes set acknowledged = true where id = $1 and connector_id = $2", [Number(req.params.changeId), req.params.id]);
111 + return envelope({ ok: true });
112 + });
113 +
114 + app.get("/v1/admin/schema-changes", async () => {
115 + const r = await pool.query("select id, connector_id, kind, old_fingerprint, new_fingerprint, detected_at, acknowledged from connector_schema_changes where not acknowledged order by detected_at desc limit 200");
116 + return envelope(r.rows);
117 + });
118 +
119 + app.get<{ Querystring: { ref: string } }>("/v1/admin/raw", async (req) => {
120 + const ref = req.query.ref;
121 + if (!ref) throw new ApiError(400, "BAD_REQUEST", "ref required");
122 + try {
123 + return envelope(await rawArchive.read(ref));
124 + } catch {
125 + throw notFound("raw payload");
126 + }
127 + });
128 +
129 + app.get("/v1/admin/divergence", async () => {
130 + const r = await pool.query("select id, ts, instrument_ids, title, data from market_events where type = 'SOURCE_DIVERGENCE' order by ts desc limit 100");
131 + return envelope(r.rows);
132 + });
133 +
134 + app.get("/v1/admin/storage", async () => {
135 + const [parts, archives, db] = await Promise.all([
136 + pool.query(`select c.relname, pg_total_relation_size(c.oid)::bigint as bytes, (select reltuples::bigint from pg_class x where x.oid = c.oid) as est_rows from pg_inherits i join pg_class c on c.oid = i.inhrelid join pg_class p on p.oid = i.inhparent where p.relname = 'observations' order by 1 desc`),
137 + pool.query("select * from observation_archives order by day desc limit 60"),
138 + pool.query<{ bytes: number }>("select pg_database_size(current_database())::bigint as bytes"),
139 + ]);
140 + return envelope({ database_bytes: db.rows[0]?.bytes ?? 0, partitions: parts.rows, archives: archives.rows, retention_days: config.observationRetentionDays, data_dir: config.dataDir });
141 + });
142 +
143 + /** Discovery prototype: observe how a public page obtains market data (network-level, unauthenticated, SSRF-guarded). */
144 + app.post<{ Body: { url?: string } }>("/v1/admin/discovery", async (req) => {
145 + const url = req.body?.url;
146 + if (!url || typeof url !== "string") throw new ApiError(400, "BAD_REQUEST", "url required");
147 + let target;
148 + try {
149 + target = await assertPublicUrl(url);
150 + } catch (err) {
151 + throw new ApiError(400, "URL_REJECTED", err instanceof Error ? err.message : "invalid url");
152 + }
153 + return envelope(await runDiscovery(target.toString(), connectorManager.http));
154 + });
155 +}
added apps/api/src/api/routes/markets.ts +270 −0
@@ -0,0 +1,270 @@
1 +import type { FastifyInstance } from "fastify";
2 +import { z } from "zod";
3 +import { PUBLICLY_REDISTRIBUTABLE, type AssetClass, type Instrument } from "@market-atlas/market-model";
4 +import { pool } from "../../db/pool.js";
5 +import { calendar } from "../../core/calendar.js";
6 +import { instruments } from "../../core/instruments.js";
7 +import { quoteStore } from "../../core/quotes.js";
8 +import { badRequest, envelope, notFound } from "../server.js";
9 +import { publicInstrument, publicQuote } from "../public-view.js";
10 +import { rowToEvent, round } from "./public.js";
11 +
12 +const quoted = (pred: (i: Instrument) => boolean) =>
13 + instruments
14 + .all()
15 + .filter((i) => i.isActive && pred(i))
16 + .map((i) => ({ inst: i, q: quoteStore.get(i.id) }))
17 + .filter((x) => x.q?.price != null && PUBLICLY_REDISTRIBUTABLE.has(x.q!.rightsStatus));
18 +
19 +const view = (x: { inst: Instrument; q: ReturnType<typeof quoteStore.get> }) => ({ ...publicInstrument(x.inst), quote: publicQuote(x.q, x.inst) });
20 +
21 +/** Featured instruments per asset class, chosen by metadata.featured (seeded by connectors) then by activity. */
22 +function featured(cls: AssetClass, n: number) {
23 + const all = quoted((i) => i.assetClass === cls);
24 + all.sort((a, b) => Number(!!b.inst.metadata.featured) - Number(!!a.inst.metadata.featured) || (b.q!.volume ?? 0) - (a.q!.volume ?? 0) || a.inst.symbol.localeCompare(b.inst.symbol));
25 + return all.slice(0, n).map(view);
26 +}
27 +
28 +export async function registerMarketRoutes(app: FastifyInstance) {
29 + /** Global market overview for the homepage and /markets. */
30 + app.get("/v1/markets", async () => {
31 + const exchanges = calendar.list().map((e) => ({ ...e, status: calendar.status(e.id) }));
32 + const eq = quoted((i) => i.assetClass === "EQUITY" || i.assetClass === "ETF");
33 + const movers = (dir: 1 | -1) =>
34 + [...eq]
35 + .filter((x) => x.q!.changePercent != null && (x.q!.volume ?? 0) > 0)
36 + .sort((a, b) => dir * ((b.q!.changePercent ?? 0) - (a.q!.changePercent ?? 0)))
37 + .slice(0, 10)
38 + .map(view);
39 + return envelope({
40 + indices: featured("INDEX", 12),
41 + crypto: featured("CRYPTO", 12),
42 + forex: featured("FOREX", 12),
43 + rates: featured("TREASURY", 12).concat(featured("INTEREST_RATE", 4)),
44 + commodities: featured("COMMODITY", 8).concat(featured("FUTURE", 8)),
45 + equities: featured("EQUITY", 16),
46 + gainers: movers(1),
47 + losers: movers(-1),
48 + exchanges: exchanges.map((e) => ({ id: e.id, name: e.name, mic: e.mic, country: e.country, city: e.city, timezone: e.timezone, lat: e.lat, lon: e.lon, status: e.status, asset_classes: e.assetClasses })),
49 + breadth: breadth(eq),
50 + });
51 + });
52 +
53 + app.get("/v1/breadth", async (req) => {
54 + const p = z.object({ scope: z.string().default("world") }).parse(req.query);
55 + const scope = p.scope.toLowerCase();
56 + const list = quoted((i) => {
57 + if (i.assetClass !== "EQUITY" && i.assetClass !== "ETF") return false;
58 + if (scope === "world") return true;
59 + if (scope.length === 2) return i.country === scope.toUpperCase();
60 + return i.exchangeId === scope;
61 + });
62 + return envelope({ scope, ...breadth(list) });
63 + });
64 +
65 + /** "What changed" across markets in the last window. */
66 + app.get("/v1/changes", async (req) => {
67 + const p = z.object({ window: z.enum(["1m", "5m", "15m", "1h", "1d"]).default("1h") }).parse(req.query);
68 + const ms = { "1m": 60_000, "5m": 300_000, "15m": 900_000, "1h": 3_600_000, "1d": 86_400_000 }[p.window];
69 + const since = new Date(Date.now() - ms).toISOString();
70 + const [events, filings, byType, movers] = await Promise.all([
71 + pool.query<{ n: number }>("select count(*)::int as n from market_events where ts >= $1", [since]),
72 + pool.query<{ n: number }>("select count(*)::int as n from filings where filed_at >= $1", [since]),
73 + pool.query<{ type: string; n: number }>("select type, count(*)::int as n from market_events where ts >= $1 group by type order by n desc", [since]),
74 + pool.query<{ instrument_id: string; first: number; last: number }>(
75 + `select instrument_id, (array_agg(close order by ts))[1] as first, (array_agg(close order by ts desc))[1] as last from bars where resolution = '1m' and ts >= $1 group by instrument_id`,
76 + [since],
77 + ),
78 + ]);
79 + const moved = movers.rows
80 + .map((r) => ({ id: r.instrument_id, pct: r.first ? ((r.last - r.first) / r.first) * 100 : 0 }))
81 + .filter((m) => Number.isFinite(m.pct));
82 + const big = moved.filter((m) => Math.abs(m.pct) >= 1).length;
83 + const top = [...moved]
84 + .sort((a, b) => Math.abs(b.pct) - Math.abs(a.pct))
85 + .slice(0, 12)
86 + .map((m) => {
87 + const inst = instruments.get(m.id);
88 + return inst ? { ...publicInstrument(inst), window_change_percent: round(m.pct, 3), quote: publicQuote(quoteStore.get(inst.id), inst) } : null;
89 + })
90 + .filter(Boolean);
91 + return envelope({
92 + window: p.window,
93 + since,
94 + instruments_tracked: moved.length,
95 + instruments_moved_over_1pct: big,
96 + events: events.rows[0]?.n ?? 0,
97 + filings: filings.rows[0]?.n ?? 0,
98 + events_by_type: byType.rows,
99 + top_movers: top,
100 + });
101 + });
102 +
103 + app.get("/v1/exchanges", async () => {
104 + return envelope(
105 + calendar.list().map((e) => {
106 + const list = quoted((i) => i.exchangeId === e.id);
107 + return {
108 + ...pubExchange(e),
109 + status: calendar.status(e.id),
110 + instruments_tracked: instruments.all().filter((i) => i.exchangeId === e.id && i.isActive).length,
111 + instruments_quoted: list.length,
112 + breadth: list.length ? breadth(list) : null,
113 + };
114 + }),
115 + );
116 + });
117 +
118 + app.get<{ Params: { id: string } }>("/v1/exchanges/:id", async (req) => {
119 + const e = calendar.exchange(req.params.id.toLowerCase()) ?? calendar.list().find((x) => x.mic?.toLowerCase() === req.params.id.toLowerCase());
120 + if (!e) throw notFound("exchange");
121 + const list = quoted((i) => i.exchangeId === e.id);
122 + const sorted = [...list].filter((x) => x.q!.changePercent != null).sort((a, b) => (b.q!.changePercent ?? 0) - (a.q!.changePercent ?? 0));
123 + const events = await pool.query(
124 + `select id, type, ts, instrument_ids, severity, confidence, source_count, sources, title, summary, data, confirmed_at from market_events where (data->>'exchangeId' = $1 or instrument_ids && $2::text[]) order by ts desc limit 30`,
125 + [e.id, list.map((x) => x.inst.id)],
126 + );
127 + return envelope({
128 + exchange: pubExchange(e),
129 + status: calendar.status(e.id),
130 + holidays: calendar.holidaysFor(e.id).filter((h) => h.date >= new Date().toISOString().slice(0, 10)).slice(0, 12),
131 + instruments_tracked: instruments.all().filter((i) => i.exchangeId === e.id && i.isActive).length,
132 + instruments_quoted: list.length,
133 + breadth: breadth(list),
134 + gainers: sorted.slice(0, 10).map(view),
135 + losers: sorted.slice(-10).reverse().map(view),
136 + most_active: [...list].sort((a, b) => (b.q!.volume ?? 0) - (a.q!.volume ?? 0)).slice(0, 10).map(view),
137 + events: events.rows.map(rowToEvent),
138 + });
139 + });
140 +
141 + app.get("/v1/countries", async () => {
142 + const rows = await pool.query<{ code: string; name: string; region: string; currency: string | null }>("select code, name, region, currency from countries order by name");
143 + return envelope(
144 + rows.rows.map((c) => {
145 + const ex = calendar.list().filter((e) => e.country === c.code);
146 + return { ...c, exchanges: ex.map((e) => ({ id: e.id, name: e.name, status: calendar.status(e.id).state })), instruments_tracked: instruments.all().filter((i) => i.country === c.code && i.isActive).length };
147 + }),
148 + );
149 + });
150 +
151 + app.get<{ Params: { code: string } }>("/v1/countries/:code", async (req) => {
152 + const code = req.params.code.toUpperCase();
153 + if (!/^[A-Z]{2}$/.test(code)) throw badRequest("country code must be ISO-3166 alpha-2");
154 + const c = (await pool.query("select code, name, region, currency from countries where code = $1", [code])).rows[0];
155 + if (!c) throw notFound("country");
156 + const ex = calendar.list().filter((e) => e.country === code);
157 + const byClass = (cls: AssetClass[]) => quoted((i) => i.country === code && cls.includes(i.assetClass)).map(view);
158 + const fx = quoted((i) => i.assetClass === "FOREX" && (i.base === c.currency || i.quote === c.currency)).map(view);
159 + const eq = quoted((i) => i.country === code && (i.assetClass === "EQUITY" || i.assetClass === "ETF"));
160 + const events = await pool.query(
161 + `select id, type, ts, instrument_ids, severity, confidence, source_count, sources, title, summary, data, confirmed_at from market_events where instrument_ids && $1::text[] order by ts desc limit 30`,
162 + [instruments.all().filter((i) => i.country === code).map((i) => i.id)],
163 + );
164 + return envelope({
165 + country: c,
166 + exchanges: ex.map((e) => ({ ...pubExchange(e), status: calendar.status(e.id) })),
167 + indices: byClass(["INDEX"]),
168 + rates: byClass(["TREASURY", "INTEREST_RATE", "BOND"]),
169 + fx,
170 + equities: [...eq].sort((a, b) => (b.q!.volume ?? 0) - (a.q!.volume ?? 0)).slice(0, 20).map(view),
171 + breadth: eq.length ? breadth(eq) : null,
172 + events: events.rows.map(rowToEvent),
173 + instruments_tracked: instruments.all().filter((i) => i.country === code && i.isActive).length,
174 + });
175 + });
176 +
177 + /** Cross-asset comparison: normalized daily closes for up to 8 instruments. */
178 + app.get("/v1/compare", async (req) => {
179 + const p = z.object({ ids: z.string(), resolution: z.enum(["1m", "1h", "1d"]).default("1d"), limit: z.coerce.number().int().min(10).max(2000).default(365) }).parse(req.query);
180 + const ids = p.ids.split(",").map((s) => s.trim()).filter(Boolean).slice(0, 8);
181 + const series = [];
182 + for (const raw of ids) {
183 + const inst = instruments.get(raw) ?? instruments.bySymbolCandidates(raw)[0];
184 + if (!inst) continue;
185 + const r = await pool.query("select ts, close from bars where instrument_id = $1 and resolution = $2 order by ts desc limit $3", [inst.id, p.resolution, p.limit]);
186 + const pts = r.rows.reverse().map((b) => ({ t: Date.parse(b.ts), c: b.close as number }));
187 + const base = pts[0]?.c ?? null;
188 + const rets = pts.slice(1).map((pt, i) => (pts[i]!.c ? pt.c / pts[i]!.c - 1 : 0));
189 + const vol = rets.length > 1 ? Math.sqrt(rets.reduce((a, r) => a + r * r, 0) / rets.length) * Math.sqrt(p.resolution === "1d" ? 252 : p.resolution === "1h" ? 252 * 6.5 : 252 * 390) : null;
190 + let peak = -Infinity;
191 + let mdd = 0;
192 + for (const pt of pts) {
193 + peak = Math.max(peak, pt.c);
194 + mdd = Math.min(mdd, pt.c / peak - 1);
195 + }
196 + series.push({
197 + instrument: publicInstrument(inst),
198 + quote: publicQuote(quoteStore.get(inst.id), inst),
199 + points: pts.map((pt) => ({ t: pt.t, c: pt.c, n: base ? (pt.c / base) * 100 : null })),
200 + stats: { return_percent: base && pts.length ? round((pts[pts.length - 1]!.c / base - 1) * 100, 2) : null, annualized_volatility_percent: vol == null ? null : round(vol * 100, 2), max_drawdown_percent: round(mdd * 100, 2), points: pts.length },
201 + });
202 + }
203 + // Pairwise correlation of returns on aligned timestamps.
204 + const correlations: Array<{ a: string; b: string; rho: number | null }> = [];
205 + for (let i = 0; i < series.length; i++)
206 + for (let j = i + 1; j < series.length; j++) {
207 + correlations.push({ a: series[i]!.instrument.id, b: series[j]!.instrument.id, rho: correlation(series[i]!.points, series[j]!.points) });
208 + }
209 + return envelope({ resolution: p.resolution, series, correlations });
210 + });
211 +}
212 +
213 +function correlation(a: Array<{ t: number; c: number }>, b: Array<{ t: number; c: number }>): number | null {
214 + const mb = new Map(b.map((p) => [p.t, p.c]));
215 + const pairs: Array<[number, number]> = [];
216 + let prevA: number | null = null;
217 + let prevB: number | null = null;
218 + for (const p of a) {
219 + const bc = mb.get(p.t);
220 + if (bc == null) continue;
221 + if (prevA != null && prevB != null && prevA && prevB) pairs.push([p.c / prevA - 1, bc / prevB - 1]);
222 + prevA = p.c;
223 + prevB = bc;
224 + }
225 + if (pairs.length < 5) return null;
226 + const ma = pairs.reduce((s, x) => s + x[0], 0) / pairs.length;
227 + const mbb = pairs.reduce((s, x) => s + x[1], 0) / pairs.length;
228 + let num = 0,
229 + da = 0,
230 + db = 0;
231 + for (const [x, y] of pairs) {
232 + num += (x - ma) * (y - mbb);
233 + da += (x - ma) ** 2;
234 + db += (y - mbb) ** 2;
235 + }
236 + return da && db ? round(num / Math.sqrt(da * db), 3) : null;
237 +}
238 +
239 +function breadth(list: Array<{ inst: Instrument; q: ReturnType<typeof quoteStore.get> }>) {
240 + const ch = list.map((x) => x.q!.changePercent).filter((x): x is number => x != null);
241 + const adv = ch.filter((x) => x > 0).length;
242 + const dec = ch.filter((x) => x < 0).length;
243 + const sorted = [...ch].sort((a, b) => a - b);
244 + return {
245 + instruments: list.length,
246 + advancers: adv,
247 + decliners: dec,
248 + unchanged: ch.length - adv - dec,
249 + median_change_percent: sorted.length ? round(sorted[Math.floor(sorted.length / 2)]!, 3) : null,
250 + mean_change_percent: ch.length ? round(ch.reduce((a, b) => a + b, 0) / ch.length, 3) : null,
251 + new_session_highs: list.filter((x) => x.q!.sessionHigh != null && x.q!.price === x.q!.sessionHigh && x.q!.realtimeStatus === "REALTIME").length,
252 + new_session_lows: list.filter((x) => x.q!.sessionLow != null && x.q!.price === x.q!.sessionLow && x.q!.realtimeStatus === "REALTIME").length,
253 + };
254 +}
255 +
256 +const pubExchange = (e: ReturnType<typeof calendar.list>[number]) => ({
257 + id: e.id,
258 + mic: e.mic,
259 + name: e.name,
260 + operator: e.operator,
261 + country: e.country,
262 + city: e.city,
263 + timezone: e.timezone,
264 + currency: e.currency,
265 + website: e.website,
266 + sessions: e.sessions,
267 + lat: e.lat,
268 + lon: e.lon,
269 + asset_classes: e.assetClasses,
270 +});
added apps/api/src/api/routes/public.ts +484 −0
@@ -0,0 +1,484 @@
1 +import type { FastifyInstance } from "fastify";
2 +import { z } from "zod";
3 +import { ASSET_CLASSES, BAR_RESOLUTIONS, PUBLICLY_REDISTRIBUTABLE, canonicalSymbol } from "@market-atlas/market-model";
4 +import { pool, tsToMs } from "../../db/pool.js";
5 +import { config } from "../../config.js";
6 +import { instruments } from "../../core/instruments.js";
7 +import { quoteStore } from "../../core/quotes.js";
8 +import { health } from "../../core/health.js";
9 +import { connectorManager } from "../../core/connector-manager.js";
10 +import { telemetry } from "../../core/telemetry.js";
11 +import { calendar } from "../../core/calendar.js";
12 +import { pipeline } from "../../core/pipeline.js";
13 +import { badRequest, envelope, notFound } from "../server.js";
14 +import { publicInstrument, publicQuote } from "../public-view.js";
15 +import { SOURCES } from "@market-atlas/connectors";
16 +
17 +const q = <T extends z.ZodTypeAny>(schema: T, input: unknown): z.infer<T> => {
18 + const r = schema.safeParse(input);
19 + if (!r.success) throw badRequest(r.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`).join("; "));
20 + return r.data;
21 +};
22 +
23 +const ListQuery = z.object({
24 + asset_class: z.enum(ASSET_CLASSES).optional(),
25 + exchange: z.string().optional(),
26 + country: z.string().length(2).optional(),
27 + q: z.string().max(80).optional(),
28 + quoted: z.enum(["1", "0"]).optional(),
29 + limit: z.coerce.number().int().min(1).max(500).default(100),
30 + offset: z.coerce.number().int().min(0).default(0),
31 + sort: z.enum(["symbol", "change", "-change", "volume", "name"]).default("symbol"),
32 +});
33 +
34 +/** Resolve an id or a symbol (optionally EXCHANGE:SYMBOL) to an instrument; quoted instruments win for bare symbols. */
35 +export function resolveInstrument(idOrSymbol: string) {
36 + const direct = instruments.get(idOrSymbol);
37 + if (direct) return direct;
38 + let sym = idOrSymbol;
39 + let venue: string | null = null;
40 + const m = idOrSymbol.match(/^([A-Za-z]+):(.+)$/);
41 + if (m) {
42 + venue = m[1]!.toLowerCase();
43 + sym = m[2]!;
44 + }
45 + const candidates = instruments.bySymbolCandidates(sym).filter((i) => !venue || i.exchangeId === venue || i.mic?.toLowerCase() === venue);
46 + if (!candidates.length) return undefined;
47 + const quoted = candidates.filter((c) => quoteStore.get(c.id)?.price != null);
48 + const pool_ = quoted.length ? quoted : candidates;
49 + const rank = (c: (typeof candidates)[number]) => ({ EQUITY: 0, ETF: 1, INDEX: 2, CRYPTO: 3, FOREX: 4 } as Record<string, number>)[c.assetClass] ?? 9;
50 + return pool_.sort((a, b) => rank(a) - rank(b))[0];
51 +}
52 +
53 +export async function registerPublicRoutes(app: FastifyInstance) {
54 + app.get("/v1/health", async () => {
55 + const db = await pool.query("select 1").then(() => "ok").catch(() => "down");
56 + const healthy = connectorManager.list().filter((r) => health.state(r.def.metadata.id) === "HEALTHY").length;
57 + const status = db === "ok" ? "ok" : "degraded";
58 + return { status, version: config.version, role: config.role, db, connectors: { total: connectorManager.list().length, healthy }, quotes: quoteStore.size(), uptime_s: Math.round(process.uptime()) };
59 + });
60 +
61 + app.get("/v1/status", async () => {
62 + const list = connectorManager.list();
63 + const states = list.map((r) => health.state(r.def.metadata.id));
64 + const count = (s: string) => states.filter((x) => x === s).length;
65 + const db = await pool.query("select 1").then(() => "operational").catch(() => "outage");
66 + const streaming = states.some((s) => s === "HEALTHY") ? "operational" : "degraded";
67 + return envelope({
68 + components: [
69 + { id: "website", name: "Website", status: "operational" },
70 + { id: "api", name: "REST API", status: db === "operational" ? "operational" : "degraded" },
71 + { id: "stream", name: "Live stream", status: streaming },
72 + { id: "market-data", name: "Market data ingestion", status: count("FAILED") > list.length / 2 ? "degraded" : "operational" },
73 + { id: "historical", name: "Historical database", status: db },
74 + { id: "crypto", name: "Crypto sources", status: groupStatus(list, "CRYPTO") },
75 + { id: "equities", name: "Equity & index sources", status: groupStatus(list, "EQUITY") },
76 + { id: "official", name: "Official & regulatory sources", status: groupStatus(list, "TREASURY", "FOREX") },
77 + ],
78 + connectors: { total: list.length, healthy: count("HEALTHY"), degraded: count("DEGRADED") + count("STALE") + count("RECONNECTING"), failed: count("FAILED"), paused: count("PAUSED") },
79 + version: config.version,
80 + });
81 + });
82 +
83 + app.get("/v1/metrics", async (_req, reply) => reply.type("text/plain; version=0.0.4").send(telemetry.prometheus()));
84 +
85 + /** Homepage telemetry — every number comes from the running system. */
86 + app.get("/v1/stats", async () => {
87 + const [daily, filings, evts] = await Promise.all([
88 + pool.query<{ observations: number; events: number; filings: number }>("select observations, events, filings from daily_stats where day = current_date"),
89 + pool.query<{ n: number }>("select count(*)::int as n from filings where filed_at > now() - interval '24 hours'"),
90 + pool.query<{ n: number }>("select count(*)::int as n from market_events where ts > now() - interval '24 hours'"),
91 + ]);
92 + const list = connectorManager.list();
93 + const states = list.map((r) => health.state(r.def.metadata.id));
94 + const quoted = quoteStore.all().filter((x) => x.price != null && PUBLICLY_REDISTRIBUTABLE.has(x.rightsStatus));
95 + const exchanges = calendar.list();
96 + const open = exchanges.filter((e) => calendar.status(e.id).state === "OPEN").length;
97 + return envelope({
98 + instruments_total: instruments.count(),
99 + instruments_quoted: quoted.length,
100 + instruments_live: quoted.filter((x) => x.realtimeStatus === "REALTIME" && Date.now() - x.updatedAt < 60_000).length,
101 + exchanges_total: exchanges.length,
102 + exchanges_open: open,
103 + sources_total: SOURCES.length,
104 + connectors_total: list.length,
105 + connectors_healthy: states.filter((s) => s === "HEALTHY").length,
106 + observations_today: daily.rows[0]?.observations ?? 0,
107 + observations_per_sec: round(telemetry.rate("observations_received_total"), 1),
108 + observations_total: telemetry.counter("observations_written_total"),
109 + events_today: daily.rows[0]?.events ?? 0,
110 + events_24h: evts.rows[0]?.n ?? 0,
111 + events_per_min: round(telemetry.rate("events_total") * 60, 2),
112 + filings_24h: filings.rows[0]?.n ?? 0,
113 + median_freshness_ms: median(quoted.filter((x) => x.realtimeStatus === "REALTIME").map((x) => (x.freshnessMs ?? 0) + (Date.now() - x.updatedAt))),
114 + stream_clients: telemetry.counter("stream_clients_connected") - telemetry.counter("stream_clients_disconnected"),
115 + uptime_s: Math.round(process.uptime()),
116 + });
117 + });
118 +
119 + app.get("/v1/instruments", async (req) => {
120 + const p = q(ListQuery, req.query);
121 + let list = instruments.all().filter((i) => i.isActive);
122 + if (p.asset_class) list = list.filter((i) => i.assetClass === p.asset_class);
123 + if (p.exchange) list = list.filter((i) => i.exchangeId === p.exchange!.toLowerCase() || i.mic?.toLowerCase() === p.exchange!.toLowerCase());
124 + if (p.country) list = list.filter((i) => i.country === p.country!.toUpperCase());
125 + if (p.q) {
126 + const needle = p.q.toLowerCase();
127 + list = list.filter((i) => i.symbol.toLowerCase().includes(needle) || i.name.toLowerCase().includes(needle));
128 + }
129 + if (p.quoted === "1") list = list.filter((i) => quoteStore.get(i.id)?.price != null);
130 + const quote = (id: string) => quoteStore.get(id);
131 + list.sort((a, b) => {
132 + switch (p.sort) {
133 + case "change":
134 + return (quote(a.id)?.changePercent ?? -Infinity) - (quote(b.id)?.changePercent ?? -Infinity);
135 + case "-change":
136 + return (quote(b.id)?.changePercent ?? -Infinity) - (quote(a.id)?.changePercent ?? -Infinity);
137 + case "volume":
138 + return (quote(b.id)?.volume ?? -1) - (quote(a.id)?.volume ?? -1);
139 + case "name":
140 + return a.name.localeCompare(b.name);
141 + default:
142 + return a.symbol.localeCompare(b.symbol);
143 + }
144 + });
145 + const total = list.length;
146 + const page = list.slice(p.offset, p.offset + p.limit);
147 + return envelope(
148 + page.map((i) => ({ ...publicInstrument(i), quote: publicQuote(quoteStore.get(i.id), i) })),
149 + { total, limit: p.limit, offset: p.offset },
150 + );
151 + });
152 +
153 + app.get<{ Params: { id: string } }>("/v1/instruments/:id", async (req) => {
154 + const inst = resolveInstrument(req.params.id);
155 + if (!inst) throw notFound("instrument");
156 + const [company, exchange] = await Promise.all([inst.companyId ? instruments.getCompany(inst.companyId) : null, Promise.resolve(instruments.exchange(inst.exchangeId))]);
157 + const quote = quoteStore.get(inst.id);
158 + const related = instruments
159 + .all()
160 + .filter((i) => i.id !== inst.id && ((inst.companyId && i.companyId === inst.companyId) || (inst.base && i.base === inst.base)))
161 + .slice(0, 12)
162 + .map(publicInstrument);
163 + const sources = pipeline.consensus.sourcesFor(inst.id);
164 + return envelope({
165 + instrument: publicInstrument(inst),
166 + company,
167 + exchange: exchange ? { ...exchange, status: calendar.status(exchange.id) } : null,
168 + quote: publicQuote(quote, inst),
169 + aliases: instruments.aliasesOf(inst.id).map((a) => ({ alias: a.alias, source_id: a.sourceId })),
170 + sources,
171 + related,
172 + });
173 + });
174 +
175 + app.get<{ Params: { id: string } }>("/v1/quotes/:id", async (req) => {
176 + const inst = resolveInstrument(req.params.id);
177 + if (!inst) throw notFound("instrument");
178 + const quote = quoteStore.get(inst.id);
179 + return envelope(publicQuote(quote, inst), { data_status: quote?.realtimeStatus ?? "UNKNOWN" });
180 + });
181 +
182 + app.get("/v1/quotes", async (req) => {
183 + const p = q(z.object({ ids: z.string().max(4000) }), req.query);
184 + const ids = p.ids.split(",").map((s) => s.trim()).filter(Boolean).slice(0, 200);
185 + const out = ids.map((id) => {
186 + const inst = resolveInstrument(id);
187 + return inst ? publicQuote(quoteStore.get(inst.id), inst) : null;
188 + });
189 + return envelope(out);
190 + });
191 +
192 + /** "Why this price?" — every contribution, included or not, with the reason. */
193 + app.get<{ Params: { id: string } }>("/v1/quotes/:id/provenance", async (req) => {
194 + const inst = resolveInstrument(req.params.id);
195 + if (!inst) throw notFound("instrument");
196 + const quote = quoteStore.get(inst.id);
197 + if (!quote) throw notFound("quote");
198 + const redistributable = PUBLICLY_REDISTRIBUTABLE.has(quote.rightsStatus);
199 + const sourceMeta = Object.fromEntries(SOURCES.map((s) => [s.id, { name: s.name, organization: s.organization, family: s.family, rights_status: s.rightsStatus, realtime_status: s.realtimeStatus, category: s.category }]));
200 + return envelope({
201 + instrument_id: inst.id,
202 + symbol: inst.symbol,
203 + canonical_price: redistributable ? quote.price : null,
204 + currency: quote.currency,
205 + confidence: quote.confidence,
206 + dispersion_bps: quote.dispersionBps,
207 + independent_sources: quote.sourceCount,
208 + freshness_ms: quote.freshnessMs,
209 + realtime_status: quote.realtimeStatus,
210 + rights_status: quote.rightsStatus,
211 + consensus_version: quote.consensusVersion,
212 + computed_at: new Date(quote.updatedAt).toISOString(),
213 + method: "weighted median over fresh observations; one vote per upstream source family; outliers > 2% from the median excluded; weights = source reliability × timestamp quality × real-time class × freshness decay",
214 + contributions: quote.contributions.map((c) => ({
215 + source_id: c.sourceId,
216 + source: sourceMeta[c.sourceId] ?? null,
217 + connector_id: c.connectorId,
218 + value: PUBLICLY_REDISTRIBUTABLE.has(c.rightsStatus) ? c.value : null,
219 + source_timestamp: c.sourceTimestamp == null ? null : new Date(c.sourceTimestamp).toISOString(),
220 + received_at: new Date(c.receivedAt).toISOString(),
221 + age_ms: c.ageMs,
222 + weight: c.weight,
223 + included: c.included,
224 + reason: c.reason ?? null,
225 + realtime_status: c.realtimeStatus,
226 + rights_status: c.rightsStatus,
227 + reliability_score: health.sourceReliability(c.sourceId),
228 + })),
229 + });
230 + });
231 +
232 + const HistoryQuery = z.object({
233 + resolution: z.enum(BAR_RESOLUTIONS).default("1d"),
234 + from: z.string().optional(),
235 + to: z.string().optional(),
236 + limit: z.coerce.number().int().min(1).max(5000).default(500),
237 + });
238 + app.get<{ Params: { id: string } }>("/v1/history/:id", async (req) => {
239 + const inst = resolveInstrument(req.params.id);
240 + if (!inst) throw notFound("instrument");
241 + const p = q(HistoryQuery, req.query);
242 + const from = p.from ? new Date(p.from) : null;
243 + const to = p.to ? new Date(p.to) : null;
244 + if ((from && Number.isNaN(from.getTime())) || (to && Number.isNaN(to.getTime()))) throw badRequest("invalid from/to");
245 + const r = await pool.query(
246 + `select ts, open, high, low, close, volume, source_count, producer from bars where instrument_id = $1 and resolution = $2
247 + and ($3::timestamptz is null or ts >= $3) and ($4::timestamptz is null or ts <= $4) order by ts desc limit $5`,
248 + [inst.id, p.resolution, from?.toISOString() ?? null, to?.toISOString() ?? null, p.limit],
249 + );
250 + const bars = r.rows.reverse().map((b) => ({ t: tsToMs(b.ts), o: b.open, h: b.high, l: b.low, c: b.close, v: b.volume, n: b.source_count, p: b.producer }));
251 + const producers = [...new Set(bars.map((b) => b.p))];
252 + return envelope(bars, { instrument_id: inst.id, resolution: p.resolution, count: bars.length, producers, data_status: producers.includes("consensus") ? "DERIVED" : "END_OF_DAY" });
253 + });
254 +
255 + const EventsQuery = z.object({
256 + type: z.string().optional(),
257 + instrument: z.string().optional(),
258 + country: z.string().optional(),
259 + asset_class: z.enum(ASSET_CLASSES).optional(),
260 + severity: z.string().optional(),
261 + since: z.string().optional(),
262 + before: z.string().optional(),
263 + limit: z.coerce.number().int().min(1).max(500).default(100),
264 + });
265 + app.get("/v1/events", async (req) => {
266 + const p = q(EventsQuery, req.query);
267 + const types = p.type ? p.type.split(",").map((s) => s.trim().toUpperCase()) : null;
268 + let instIds: string[] | null = null;
269 + if (p.instrument) {
270 + const inst = resolveInstrument(p.instrument);
271 + instIds = inst ? [inst.id] : ["__none__"];
272 + } else if (p.country || p.asset_class) {
273 + instIds = instruments
274 + .all()
275 + .filter((i) => (!p.country || i.country === p.country!.toUpperCase()) && (!p.asset_class || i.assetClass === p.asset_class))
276 + .map((i) => i.id);
277 + if (!instIds.length) instIds = ["__none__"];
278 + }
279 + const r = await pool.query(
280 + `select id, type, ts, instrument_ids, severity, confidence, source_count, sources, title, summary, data, confirmed_at from market_events
281 + where ($1::text[] is null or type = any($1)) and ($2::text[] is null or instrument_ids && $2) and ($3::text is null or severity = $3)
282 + and ($4::timestamptz is null or ts >= $4) and ($5::timestamptz is null or ts < $5)
283 + order by ts desc limit $6`,
284 + [types, instIds, p.severity ?? null, p.since ?? null, p.before ?? null, p.limit],
285 + );
286 + return envelope(r.rows.map(rowToEvent));
287 + });
288 +
289 + app.get<{ Params: { id: string } }>("/v1/events/:id", async (req) => {
290 + const r = await pool.query("select * from market_events where id = $1", [req.params.id]);
291 + if (!r.rows[0]) throw notFound("event");
292 + return envelope(rowToEvent(r.rows[0]));
293 + });
294 +
295 + app.get("/v1/filings", async (req) => {
296 + const p = q(z.object({ form: z.string().optional(), cik: z.string().optional(), q: z.string().optional(), limit: z.coerce.number().int().min(1).max(500).default(100), offset: z.coerce.number().int().min(0).default(0) }), req.query);
297 + const forms = p.form ? p.form.split(",").map((s) => s.trim().toUpperCase()) : null;
298 + const r = await pool.query(
299 + `select id, source_id, cik, company_name, form_type, filed_at, url, instrument_ids, metadata from filings
300 + where ($1::text[] is null or upper(form_type) = any($1)) and ($2::text is null or cik = $2) and ($3::text is null or company_name ilike '%' || $3 || '%')
301 + order by filed_at desc limit $4 offset $5`,
302 + [forms, p.cik ? p.cik.replace(/^0+/, "") : null, p.q ?? null, p.limit, p.offset],
303 + );
304 + const total = await pool.query<{ n: number }>("select count(*)::int as n from filings where filed_at > now() - interval '7 days'");
305 + return envelope(
306 + r.rows.map((f) => ({ id: f.id, source_id: f.source_id, cik: f.cik, company_name: f.company_name, form_type: f.form_type, filed_at: f.filed_at, url: f.url, instrument_ids: f.instrument_ids, metadata: f.metadata })),
307 + { last_7_days: total.rows[0]?.n ?? 0 },
308 + );
309 + });
310 +
311 + app.get("/v1/search", async (req) => {
312 + const p = q(z.object({ q: z.string().min(1).max(80), limit: z.coerce.number().int().min(1).max(50).default(20) }), req.query);
313 + const needle = p.q.trim();
314 + const lower = needle.toLowerCase();
315 + const cs = canonicalSymbol(needle);
316 + const scored: Array<{ score: number; item: unknown; group: string }> = [];
317 + for (const i of instruments.all()) {
318 + if (!i.isActive) continue;
319 + const sym = i.symbol.toLowerCase();
320 + const name = i.name.toLowerCase();
321 + let score = 0;
322 + if (canonicalSymbol(i.symbol) === cs) score = 100;
323 + else if (sym.startsWith(lower)) score = 80 - (sym.length - lower.length);
324 + else if (name === lower) score = 75;
325 + else if (name.startsWith(lower)) score = 60;
326 + else if (name.includes(lower) || sym.includes(lower)) score = 40;
327 + else if (i.base && i.base.toLowerCase() === lower) score = 70;
328 + else if (i.id.includes(cs)) score = 30;
329 + if (!score) continue;
330 + const quote = quoteStore.get(i.id);
331 + if (quote?.price != null) score += 10;
332 + if (i.assetClass === "EQUITY" && quote?.price == null) score -= 5;
333 + scored.push({ score, group: i.assetClass, item: { ...publicInstrument(i), quote: publicQuote(quote, i) } });
334 + }
335 + for (const e of calendar.list()) {
336 + const hay = `${e.name} ${e.mic ?? ""} ${e.id} ${e.city ?? ""}`.toLowerCase();
337 + if (hay.includes(lower)) scored.push({ score: e.name.toLowerCase().startsWith(lower) || (e.mic ?? "").toLowerCase() === lower ? 85 : 45, group: "EXCHANGE", item: { id: e.id, name: e.name, mic: e.mic, country: e.country, status: calendar.status(e.id).state } });
338 + }
339 + const countries = await pool.query<{ code: string; name: string; region: string }>("select code, name, region from countries where lower(name) like $1 or lower(code) = $2 limit 5", [`%${lower}%`, lower]);
340 + for (const c of countries.rows) scored.push({ score: c.name.toLowerCase().startsWith(lower) ? 70 : 40, group: "COUNTRY", item: c });
341 + scored.sort((a, b) => b.score - a.score);
342 + const top = scored.slice(0, p.limit);
343 + const groups: Record<string, unknown[]> = {};
344 + for (const s of top) (groups[s.group] ??= []).push(s.item);
345 + return envelope({ query: needle, results: top.map((s) => ({ group: s.group, score: s.score, item: s.item })), groups });
346 + });
347 +
348 + app.get("/v1/sources", async () => {
349 + const rows = await pool.query("select id, name, organization, source_type, homepage, jurisdiction, rights_status, realtime_status, family, category, enabled from sources order by category, name");
350 + const byConnector = connectorManager.list();
351 + return envelope(
352 + rows.rows.map((s) => {
353 + const conns = byConnector.filter((r) => r.def.metadata.sourceId === s.id);
354 + const snaps = conns.map((c) => health.snapshot(c.def.metadata.id));
355 + const last = snaps.map((x) => x.lastMessageAt ?? 0).reduce((a, b) => Math.max(a, b), 0);
356 + return {
357 + ...s,
358 + connectors: conns.length,
359 + status: conns.length ? worst(snaps.map((x) => x.state)) : "NONE",
360 + coverage: snaps.reduce((a, x) => a + x.instrumentsCovered, 0),
361 + last_observation: last ? new Date(last).toISOString() : null,
362 + reliability_score: snaps.map((x) => x.reliabilityScore).filter((x): x is number => x != null).sort((a, b) => b - a)[0] ?? null,
363 + asset_classes: [...new Set(conns.flatMap((c) => c.def.metadata.assetClasses))],
364 + description: conns[0]?.def.metadata.description ?? null,
365 + rights_notes: conns[0]?.def.metadata.rightsNotes ?? null,
366 + terms_url: conns[0]?.def.metadata.termsUrl ?? null,
367 + };
368 + }),
369 + );
370 + });
371 +
372 + /** Public connector directory: operational facts only, no endpoints. */
373 + app.get("/v1/connectors", async () => {
374 + return envelope(
375 + connectorManager.list().map((r) => {
376 + const m = r.def.metadata;
377 + const s = health.snapshot(m.id);
378 + return {
379 + id: m.id,
380 + name: m.name,
381 + source_id: m.sourceId,
382 + organization: m.organization,
383 + source_type: m.sourceType,
384 + rights_status: m.rightsStatus,
385 + realtime_status: m.realtimeStatus,
386 + asset_classes: m.assetClasses,
387 + exchanges: m.exchanges,
388 + jurisdiction: m.jurisdiction,
389 + supports_streaming: m.supportsStreaming,
390 + supports_historical: m.supportsHistorical,
391 + expected_latency_ms: m.expectedLatencyMs,
392 + description: m.description,
393 + version: m.version,
394 + status: s.state,
395 + last_message_at: s.lastMessageAt ? new Date(s.lastMessageAt).toISOString() : null,
396 + messages_1m: s.messages1m,
397 + messages_total: s.messagesTotal,
398 + median_latency_ms: s.medianLatencyMs,
399 + p95_latency_ms: s.p95LatencyMs,
400 + parse_success_rate: s.parseSuccessRate,
401 + instruments_covered: s.instrumentsCovered,
402 + reconnects: s.reconnects,
403 + reliability_score: s.reliabilityScore,
404 + next_poll_at: s.nextPollAt ? new Date(s.nextPollAt).toISOString() : null,
405 + };
406 + }),
407 + );
408 + });
409 +
410 + app.get("/v1/data-health", async () => {
411 + const list = connectorManager.list();
412 + const snaps = list.map((r) => health.snapshot(r.def.metadata.id));
413 + const states = snaps.map((s) => s.state);
414 + const count = (...s: string[]) => states.filter((x) => s.includes(x)).length;
415 + const quotes = quoteStore.all().filter((x) => x.price != null && PUBLICLY_REDISTRIBUTABLE.has(x.rightsStatus));
416 + const now = Date.now();
417 + const realtime = quotes.filter((x) => x.realtimeStatus === "REALTIME");
418 + const incidents = await pool.query(
419 + `select id, type, ts, severity, title, data from market_events where type in ('SOURCE_FAILURE','SOURCE_RECOVERY','SCHEMA_DRIFT','SOURCE_DIVERGENCE') and ts > now() - interval '7 days' order by ts desc limit 50`,
420 + );
421 + const history = await pool.query(
422 + `select date_trunc('hour', ts) as hour, avg(case when status = 'HEALTHY' then 1.0 else 0.0 end) as healthy_ratio, sum(messages_1m) as messages, avg(median_latency_ms) as latency
423 + from connector_health where ts > now() - interval '48 hours' group by 1 order by 1`,
424 + );
425 + return envelope({
426 + connectors: { total: list.length, healthy: count("HEALTHY"), degraded: count("DEGRADED", "STALE"), recovering: count("RECONNECTING", "STARTING"), failed: count("FAILED"), paused: count("PAUSED", "DISABLED") },
427 + healthy_ratio: list.length ? count("HEALTHY") / list.length : null,
428 + quotes_public: quotes.length,
429 + quotes_realtime: realtime.length,
430 + median_freshness_ms: median(realtime.map((x) => (x.freshnessMs ?? 0) + (now - x.updatedAt))),
431 + p95_freshness_ms: quantile(realtime.map((x) => (x.freshnessMs ?? 0) + (now - x.updatedAt)), 0.95),
432 + multi_source_quotes: quotes.filter((x) => x.sourceCount >= 2).length,
433 + mean_confidence: quotes.length ? round(quotes.reduce((a, x) => a + x.confidence, 0) / quotes.length, 3) : null,
434 + mean_dispersion_bps: round(mean(quotes.map((x) => x.dispersionBps).filter((x): x is number => x != null)), 2),
435 + observations_per_sec: round(telemetry.rate("observations_received_total"), 1),
436 + queue_depth: telemetry.counter("observation_queue_depth"),
437 + by_connector: snaps.map((s) => ({ id: s.connectorId, state: s.state, messages_1m: s.messages1m, median_latency_ms: s.medianLatencyMs, reliability_score: s.reliabilityScore, instruments: s.instrumentsCovered, last_message_at: s.lastMessageAt ? new Date(s.lastMessageAt).toISOString() : null, last_error: s.lastError })),
438 + incidents: incidents.rows.map((r) => ({ id: r.id, type: r.type, ts: r.ts, severity: r.severity, title: r.title, data: r.data })),
439 + history: history.rows.map((r) => ({ hour: r.hour, healthy_ratio: Number(r.healthy_ratio), messages: Number(r.messages), latency_ms: r.latency == null ? null : Math.round(Number(r.latency)) })),
440 + });
441 + });
442 +}
443 +
444 +export const rowToEvent = (r: any) => ({
445 + id: r.id,
446 + type: r.type,
447 + timestamp: r.ts,
448 + instrument_ids: r.instrument_ids,
449 + severity: r.severity,
450 + confidence: r.confidence,
451 + source_count: r.source_count,
452 + sources: r.sources,
453 + title: r.title,
454 + summary: r.summary,
455 + data: r.data,
456 + confirmed_at: r.confirmed_at,
457 + instruments: (r.instrument_ids as string[]).map((id: string) => {
458 + const i = instruments.get(id);
459 + return i ? { id, symbol: i.symbol, name: i.name, asset_class: i.assetClass } : { id };
460 + }),
461 +});
462 +
463 +function groupStatus(list: ReturnType<typeof connectorManager.list>, ...classes: string[]) {
464 + const rel = list.filter((r) => r.def.metadata.assetClasses.some((c) => classes.includes(c)));
465 + if (!rel.length) return "unknown";
466 + const st = rel.map((r) => health.state(r.def.metadata.id));
467 + if (st.every((s) => s === "HEALTHY")) return "operational";
468 + if (st.some((s) => s === "HEALTHY")) return "degraded";
469 + return "outage";
470 +}
471 +
472 +function worst(states: string[]): string {
473 + const order = ["FAILED", "PAUSED", "DISABLED", "RECONNECTING", "STALE", "DEGRADED", "STARTING", "HEALTHY"];
474 + return states.sort((a, b) => order.indexOf(a) - order.indexOf(b))[0] ?? "NONE";
475 +}
476 +
477 +export const median = (xs: number[]) => quantile(xs, 0.5);
478 +export function quantile(xs: number[], qq: number): number | null {
479 + if (!xs.length) return null;
480 + const s = [...xs].sort((a, b) => a - b);
481 + return s[Math.min(s.length - 1, Math.floor(qq * s.length))] ?? null;
482 +}
483 +export const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : NaN);
484 +export const round = (n: number, d: number) => (Number.isFinite(n) ? Math.round(n * 10 ** d) / 10 ** d : null);
added apps/api/src/api/server.ts +103 −0
@@ -0,0 +1,103 @@
1 +import Fastify, { type FastifyInstance, type FastifyReply, type FastifyRequest } from "fastify";
2 +import cors from "@fastify/cors";
3 +import websocket from "@fastify/websocket";
4 +import { randomUUID } from "node:crypto";
5 +import { config } from "../config.js";
6 +import { logger } from "../logger.js";
7 +import { telemetry } from "../core/telemetry.js";
8 +import { registerPublicRoutes } from "./routes/public.js";
9 +import { registerMarketRoutes } from "./routes/markets.js";
10 +import { registerAdminRoutes } from "./routes/admin.js";
11 +import { registerStream } from "./stream.js";
12 +
13 +export class ApiError extends Error {
14 + constructor(
15 + public status: number,
16 + public code: string,
17 + message: string,
18 + ) {
19 + super(message);
20 + }
21 +}
22 +
23 +export const notFound = (what: string) => new ApiError(404, `${what.toUpperCase()}_NOT_FOUND`, `${what} not found`);
24 +export const badRequest = (msg: string) => new ApiError(400, "BAD_REQUEST", msg);
25 +
26 +/** Standard envelope: { data, meta } — meta carries request id, timestamp and the data status of the payload. */
27 +export function envelope<T>(data: T, meta: Record<string, unknown> = {}) {
28 + return { data, meta: { request_id: randomUUID(), timestamp: new Date().toISOString(), ...meta } };
29 +}
30 +
31 +export async function buildServer(): Promise<FastifyInstance> {
32 + const app = Fastify({
33 + logger: false,
34 + trustProxy: true,
35 + bodyLimit: 256 * 1024,
36 + disableRequestLogging: true,
37 + });
38 + await app.register(cors, { origin: true, methods: ["GET", "POST", "OPTIONS"], allowedHeaders: ["content-type", "x-ma-admin-token"] });
39 + await app.register(websocket, { options: { maxPayload: 64 * 1024 } });
40 +
41 + // Simple per-IP token bucket (public REST). 240 req/min sustained, 60 burst.
42 + const buckets = new Map<string, { tokens: number; last: number }>();
43 + app.addHook("onRequest", async (req, reply) => {
44 + (req as FastifyRequest & { startedAt: number }).startedAt = Date.now();
45 + if (req.url.startsWith("/v1/stream") || req.url.startsWith("/v1/sse")) return;
46 + const ip = req.ip;
47 + const now = Date.now();
48 + let b = buckets.get(ip);
49 + if (!b) {
50 + b = { tokens: 60, last: now };
51 + buckets.set(ip, b);
52 + }
53 + b.tokens = Math.min(60, b.tokens + ((now - b.last) / 1000) * 4);
54 + b.last = now;
55 + if (b.tokens < 1) {
56 + telemetry.inc("api_rate_limited_total");
57 + reply.header("retry-after", "5");
58 + return reply.code(429).send({ error: { code: "RATE_LIMITED", message: "too many requests" } });
59 + }
60 + b.tokens -= 1;
61 + if (buckets.size > 50_000) for (const [k, v] of buckets) if (now - v.last > 600_000) buckets.delete(k);
62 + });
63 + app.addHook("onSend", async (req, reply, payload) => {
64 + reply.header("x-content-type-options", "nosniff");
65 + reply.header("referrer-policy", "strict-origin-when-cross-origin");
66 + reply.header("x-frame-options", "DENY");
67 + if (config.isProd) reply.header("strict-transport-security", "max-age=31536000; includeSubDomains");
68 + const ms = Date.now() - ((req as FastifyRequest & { startedAt?: number }).startedAt ?? Date.now());
69 + telemetry.inc("api_requests_total", 1, { status: String(reply.statusCode) });
70 + telemetry.observe("api_latency_ms", ms);
71 + return payload;
72 + });
73 + app.setErrorHandler((err: unknown, req: FastifyRequest, reply: FastifyReply) => {
74 + if (err instanceof ApiError) return reply.code(err.status).send({ error: { code: err.code, message: err.message } });
75 + const e = err as { statusCode?: number; validation?: unknown; message?: string };
76 + if (e.validation) return reply.code(400).send({ error: { code: "VALIDATION_ERROR", message: e.message } });
77 + if (e.statusCode && e.statusCode < 500) return reply.code(e.statusCode).send({ error: { code: "REQUEST_ERROR", message: e.message } });
78 + logger.error({ err, url: req.url }, "unhandled API error");
79 + telemetry.inc("api_errors_total");
80 + return reply.code(500).send({ error: { code: "INTERNAL_ERROR", message: "internal error" } });
81 + });
82 + app.setNotFoundHandler((_req, reply) => reply.code(404).send({ error: { code: "NOT_FOUND", message: "route not found" } }));
83 +
84 + await registerPublicRoutes(app);
85 + await registerMarketRoutes(app);
86 + await registerAdminRoutes(app);
87 + await registerStream(app);
88 +
89 + // Edge mode: one public port. /v1/* (REST, WebSocket, SSE) is served here; every other path goes to the web app.
90 + if (config.webUpstream) {
91 + const { default: proxy } = await import("@fastify/http-proxy");
92 + await app.register(proxy, {
93 + upstream: config.webUpstream,
94 + prefix: "/",
95 + rewritePrefix: "/",
96 + websocket: false,
97 + http2: false,
98 + replyOptions: { rewriteRequestHeaders: (req, headers) => ({ ...headers, "x-forwarded-host": req.headers.host ?? "", "x-forwarded-proto": (req.headers["x-forwarded-proto"] as string) ?? "https" }) },
99 + });
100 + logger.info({ upstream: config.webUpstream }, "edge mode: proxying non-API paths to the web app");
101 + }
102 + return app;
103 +}
added apps/api/src/api/stream.ts +210 −0
@@ -0,0 +1,210 @@
1 +import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
2 +import type { WebSocket } from "ws";
3 +import type { CanonicalQuote, MarketEvent } from "@market-atlas/market-model";
4 +import { StreamClientMessageSchema } from "@market-atlas/market-model";
5 +import { bus } from "../core/bus.js";
6 +import { instruments } from "../core/instruments.js";
7 +import { telemetry } from "../core/telemetry.js";
8 +import { streamQuote } from "./public-view.js";
9 +import { resolveInstrument } from "./routes/public.js";
10 +import { logger } from "../logger.js";
11 +
12 +/**
13 + * Market Atlas stream gateway (WebSocket + SSE). Channels:
14 + * quotes:<SYMBOL|id> canonical quotes for one instrument
15 + * quotes:* every canonical quote (throttled per instrument to 2/s)
16 + * events:* every canonical market event
17 + * events:<COUNTRY> events touching instruments of a country (ISO-2)
18 + * events:type:<TYPE> events of one type
19 + * market:<exchange> market state changes
20 + * tape compact tape of price changes (batched at 10 Hz)
21 + * Messages carry `seq` so clients can detect gaps; the gateway batches at 10–20 Hz per client.
22 + */
23 +interface Client {
24 + send(data: string): void;
25 + channels: Set<string>;
26 + pending: unknown[];
27 + timer: NodeJS.Timeout | null;
28 + seq: number;
29 + lastQuoteAt: Map<string, number>;
30 +}
31 +
32 +const clients = new Set<Client>();
33 +let globalSeq = 0;
34 +
35 +function channelsFor(msg: { kind: "quote"; q: CanonicalQuote } | { kind: "event"; e: MarketEvent } | { kind: "market"; exchangeId: string }): string[] {
36 + if (msg.kind === "quote") {
37 + const inst = instruments.get(msg.q.instrumentId);
38 + const out = [`quotes:${msg.q.instrumentId}`, `quotes:${msg.q.symbol.toUpperCase()}`, "quotes:*", "tape"];
39 + if (inst?.assetClass) out.push(`quotes:class:${inst.assetClass}`);
40 + return out;
41 + }
42 + if (msg.kind === "event") {
43 + const out = ["events:*", `events:type:${msg.e.type}`];
44 + for (const id of msg.e.instrumentIds) {
45 + out.push(`events:${id}`);
46 + const inst = instruments.get(id);
47 + if (inst?.country) out.push(`events:${inst.country}`);
48 + if (inst?.symbol) out.push(`events:${inst.symbol.toUpperCase()}`);
49 + }
50 + return out;
51 + }
52 + return [`market:${msg.exchangeId}`, "market:*"];
53 +}
54 +
55 +function enqueue(c: Client, payload: unknown) {
56 + c.pending.push(payload);
57 + if (c.pending.length > 2000) c.pending.splice(0, c.pending.length - 2000); // backpressure: drop oldest
58 + if (!c.timer) {
59 + c.timer = setTimeout(() => {
60 + c.timer = null;
61 + const batch = c.pending.splice(0);
62 + if (!batch.length) return;
63 + c.seq++;
64 + try {
65 + c.send(JSON.stringify({ type: "batch", seq: c.seq, ts: Date.now(), messages: batch }));
66 + } catch {
67 + /* closed */
68 + }
69 + }, 100);
70 + }
71 +}
72 +
73 +function broadcast(channels: string[], payload: unknown, perInstrumentKey?: string) {
74 + globalSeq++;
75 + for (const c of clients) {
76 + let hit = false;
77 + for (const ch of channels) if (c.channels.has(ch)) hit = true;
78 + if (!hit) continue;
79 + if (perInstrumentKey && c.channels.has("quotes:*") && !c.channels.has(`quotes:${perInstrumentKey}`)) {
80 + const last = c.lastQuoteAt.get(perInstrumentKey) ?? 0;
81 + const now = Date.now();
82 + if (now - last < 500) continue;
83 + c.lastQuoteAt.set(perInstrumentKey, now);
84 + }
85 + enqueue(c, payload);
86 + }
87 +}
88 +
89 +bus.subscribe("canonical.quote", (q) => {
90 + if (!clients.size) return;
91 + const w = streamQuote(q);
92 + if (!w) return;
93 + broadcast(channelsFor({ kind: "quote", q }), w, q.instrumentId);
94 +});
95 +bus.subscribe("market.event", (e) => {
96 + if (!clients.size) return;
97 + broadcast(channelsFor({ kind: "event", e }), { type: "event", event: publicEvent(e) });
98 +});
99 +bus.subscribe("market.state", (m) => {
100 + if (!clients.size) return;
101 + broadcast(channelsFor({ kind: "market", exchangeId: m.exchangeId }), { type: "market_state", ...m });
102 +});
103 +
104 +export function publicEvent(e: MarketEvent) {
105 + return {
106 + id: e.id,
107 + type: e.type,
108 + timestamp: e.timestamp,
109 + instrument_ids: e.instrumentIds,
110 + instruments: e.instrumentIds.map((id) => {
111 + const i = instruments.get(id);
112 + return i ? { id, symbol: i.symbol, name: i.name, asset_class: i.assetClass, country: i.country } : { id };
113 + }),
114 + severity: e.severity,
115 + confidence: e.confidence,
116 + source_count: e.sourceCount,
117 + sources: e.sources,
118 + title: e.title,
119 + summary: e.summary,
120 + data: e.data,
121 + };
122 +}
123 +
124 +function normalizeChannels(chs: string[]): string[] {
125 + return chs.map((ch) => {
126 + const m = ch.match(/^quotes:(.+)$/);
127 + if (m && m[1] !== "*" && !m[1]!.startsWith("class:")) {
128 + const inst = resolveInstrument(m[1]!);
129 + return inst ? `quotes:${inst.id}` : `quotes:${m[1]!.toUpperCase()}`;
130 + }
131 + const e = ch.match(/^events:(.+)$/);
132 + if (e && e[1] !== "*" && !e[1]!.startsWith("type:")) {
133 + const inst = resolveInstrument(e[1]!);
134 + if (inst) return `events:${inst.id}`;
135 + return `events:${e[1]!.toUpperCase()}`;
136 + }
137 + return ch;
138 + });
139 +}
140 +
141 +export async function registerStream(app: FastifyInstance) {
142 + app.get("/v1/stream", { websocket: true }, (socket: WebSocket, req: FastifyRequest) => {
143 + const client: Client = { send: (d) => socket.send(d), channels: new Set(), pending: [], timer: null, seq: 0, lastQuoteAt: new Map() };
144 + clients.add(client);
145 + telemetry.inc("stream_clients_connected");
146 + telemetry.gauge("stream_clients", clients.size);
147 + socket.send(JSON.stringify({ type: "hello", version: 1, server_seq: globalSeq, ts: Date.now(), channels_hint: ["quotes:AAPL", "quotes:*", "events:*", "events:US", "market:xnas", "tape"] }));
148 + const hb = setInterval(() => {
149 + try {
150 + socket.send(JSON.stringify({ type: "heartbeat", ts: Date.now(), seq: client.seq }));
151 + } catch {
152 + /* ignore */
153 + }
154 + }, 25_000);
155 + socket.on("message", (raw) => {
156 + let msg;
157 + try {
158 + msg = StreamClientMessageSchema.parse(JSON.parse(raw.toString()));
159 + } catch {
160 + socket.send(JSON.stringify({ type: "error", code: "BAD_MESSAGE", message: "expected {action: subscribe|unsubscribe|ping, channels: []}" }));
161 + return;
162 + }
163 + if (msg.action === "ping") return socket.send(JSON.stringify({ type: "pong", id: msg.id ?? null, ts: Date.now() }));
164 + const chs = normalizeChannels(msg.channels);
165 + if (msg.action === "subscribe") {
166 + for (const ch of chs) if (client.channels.size < 500) client.channels.add(ch);
167 + } else for (const ch of chs) client.channels.delete(ch);
168 + socket.send(JSON.stringify({ type: "subscriptions", channels: [...client.channels] }));
169 + });
170 + socket.on("close", () => {
171 + clients.delete(client);
172 + if (client.timer) clearTimeout(client.timer);
173 + clearInterval(hb);
174 + telemetry.inc("stream_clients_disconnected");
175 + telemetry.gauge("stream_clients", clients.size);
176 + });
177 + socket.on("error", (err) => logger.debug({ err: err.message, ip: req.ip }, "stream socket error"));
178 + });
179 +
180 + /** SSE: /v1/sse?channels=quotes:AAPL,events:* */
181 + app.get<{ Querystring: { channels?: string } }>("/v1/sse", async (req: FastifyRequest<{ Querystring: { channels?: string } }>, reply: FastifyReply) => {
182 + const chs = normalizeChannels((req.query.channels ?? "events:*").split(",").map((s) => s.trim()).filter(Boolean).slice(0, 100));
183 + reply.raw.writeHead(200, {
184 + "content-type": "text/event-stream",
185 + "cache-control": "no-cache, no-transform",
186 + connection: "keep-alive",
187 + "x-accel-buffering": "no",
188 + "access-control-allow-origin": "*",
189 + });
190 + reply.raw.write(`event: hello\ndata: ${JSON.stringify({ version: 1, channels: chs, ts: Date.now() })}\n\n`);
191 + const client: Client = {
192 + send: (d) => reply.raw.write(`event: batch\nid: ${++globalSeq}\ndata: ${d}\n\n`),
193 + channels: new Set(chs),
194 + pending: [],
195 + timer: null,
196 + seq: 0,
197 + lastQuoteAt: new Map(),
198 + };
199 + clients.add(client);
200 + telemetry.inc("stream_clients_connected");
201 + const hb = setInterval(() => reply.raw.write(`: keepalive ${Date.now()}\n\n`), 20_000);
202 + req.raw.on("close", () => {
203 + clients.delete(client);
204 + clearInterval(hb);
205 + if (client.timer) clearTimeout(client.timer);
206 + telemetry.inc("stream_clients_disconnected");
207 + });
208 + await new Promise(() => {}); // keep the handler alive until the socket closes
209 + });
210 +}
added apps/api/src/cli.ts +123 −0
@@ -0,0 +1,123 @@
1 +/**
2 + * Market Atlas CLI:
3 + * ma migrate apply SQL migrations
4 + * ma seed reference data (countries, exchanges, holidays)
5 + * ma connectors list connectors with rights/realtime status
6 + * ma run <connector> [--file p] run one poll (or normalize a fixture file offline) and print the batch
7 + * ma replay <day> [connector] re-run normalization on archived raw payloads of a day (no network)
8 + * ma status quick DB/quotes/events status
9 + * ma archive archive observation partitions older than the retention window
10 + * ma backup pg_dump metadata to data/backups
11 + */
12 +import { readFileSync, readdirSync, existsSync } from "node:fs";
13 +import { join } from "node:path";
14 +import { gunzipSync } from "node:zlib";
15 +import { CONNECTORS } from "@market-atlas/connectors";
16 +import { makeTestContext, normalizeFixture } from "@market-atlas/connector-sdk";
17 +import { config } from "./config.js";
18 +import { migrate } from "./db/migrate.js";
19 +import { pool } from "./db/pool.js";
20 +import { seed } from "./seed/index.js";
21 +import { archiveOldPartitions, backupMetadata } from "./core/jobs.js";
22 +
23 +const [cmd, ...args] = process.argv.slice(2);
24 +
25 +async function main() {
26 + switch (cmd) {
27 + case "migrate": {
28 + const applied = await migrate();
29 + console.log(applied.length ? `applied: ${applied.join(", ")}` : "up to date");
30 + break;
31 + }
32 + case "seed":
33 + await migrate();
34 + await seed();
35 + console.log("seed ok");
36 + break;
37 + case "connectors":
38 + for (const c of CONNECTORS) {
39 + const m = c.metadata;
40 + console.log(`${m.id.padEnd(30)} ${m.sourceType.padEnd(12)} ${m.rightsStatus.padEnd(34)} ${m.realtimeStatus.padEnd(12)} ${m.assetClasses.join(",")}`);
41 + }
42 + break;
43 + case "run": {
44 + const id = args[0];
45 + const def = CONNECTORS.find((c) => c.metadata.id === id);
46 + if (!def) throw new Error(`unknown connector ${id}`);
47 + const fileIdx = args.indexOf("--file");
48 + if (fileIdx >= 0) {
49 + const path = args[fileIdx + 1]!;
50 + const text = readFileSync(path, "utf8");
51 + const payload = path.endsWith(".json") ? JSON.parse(text) : text;
52 + const kind = args[args.indexOf("--kind") + 1] ?? "poll";
53 + const batch = normalizeFixture(def, args.includes("--kind") ? kind : "poll", payload);
54 + console.log(JSON.stringify(batch, null, 2));
55 + break;
56 + }
57 + if (!def.poll) throw new Error("streaming connector: use the running service or --file");
58 + const ctx = makeTestContext(def, { fetchImpl: fetch, secrets: process.env as Record<string, string> });
59 + const raws = await def.poll(ctx);
60 + let total = 0;
61 + for (const r of raws) {
62 + const b = def.normalize(r);
63 + total += b.observations.length;
64 + console.log(`kind=${r.kind} observations=${b.observations.length} events=${b.events?.length ?? 0} instruments=${b.instruments?.length ?? 0} filings=${b.filings?.length ?? 0}`);
65 + for (const o of b.observations.slice(0, 8)) console.log(` ${o.symbol.padEnd(12)} ${o.field.padEnd(15)} ${String(o.value).padEnd(14)} ${o.sourceTimestamp ? new Date(o.sourceTimestamp).toISOString() : "-"} ${o.realtimeStatus}`);
66 + }
67 + console.log(`total observations: ${total}`);
68 + break;
69 + }
70 + case "replay": {
71 + const day = args[0];
72 + if (!day) throw new Error("usage: ma replay YYYY-MM-DD [connector]");
73 + const dir = join(config.dataDir, "raw", day);
74 + if (!existsSync(dir)) throw new Error(`no raw archive for ${day}`);
75 + const connectors = args[1] ? [args[1]] : readdirSync(dir);
76 + for (const cid of connectors) {
77 + const def = CONNECTORS.find((c) => c.metadata.id === cid);
78 + if (!def) continue;
79 + let n = 0,
80 + obs = 0,
81 + failed = 0;
82 + for (const f of readdirSync(join(dir, cid))) {
83 + const rec = JSON.parse(gunzipSync(readFileSync(join(dir, cid, f))).toString("utf8"));
84 + try {
85 + const b = def.normalize({ connectorId: cid, sourceId: def.metadata.sourceId, kind: rec.kind, payload: rec.payload, receivedAt: Date.parse(rec.received_at) });
86 + obs += b.observations.length;
87 + } catch {
88 + failed++;
89 + }
90 + n++;
91 + }
92 + console.log(`${cid}: ${n} payloads → ${obs} observations, ${failed} normalization failures`);
93 + }
94 + break;
95 + }
96 + case "status": {
97 + const [i, q, e, o, c] = await Promise.all([
98 + pool.query<{ n: number }>("select count(*)::int as n from instruments"),
99 + pool.query<{ n: number }>("select count(*)::int as n from canonical_quotes where price is not null"),
100 + pool.query<{ n: number }>("select count(*)::int as n from market_events where ts > now() - interval '24 hours'"),
101 + pool.query<{ n: number }>("select coalesce(sum(observations),0)::bigint as n from daily_stats"),
102 + pool.query("select id, status, messages_total, reliability_score, last_message_at from connectors order by id"),
103 + ]);
104 + console.log(`instruments: ${i.rows[0]?.n} quotes: ${q.rows[0]?.n} events 24h: ${e.rows[0]?.n} observations total: ${o.rows[0]?.n}`);
105 + for (const r of c.rows) console.log(` ${String(r.id).padEnd(30)} ${String(r.status).padEnd(12)} msgs=${r.messages_total} score=${r.reliability_score ?? "-"} last=${r.last_message_at ?? "-"}`);
106 + break;
107 + }
108 + case "archive":
109 + console.log(await archiveOldPartitions());
110 + break;
111 + case "backup":
112 + console.log(await backupMetadata());
113 + break;
114 + default:
115 + console.log("usage: ma migrate|seed|connectors|run <id> [--file f --kind k]|replay <day> [connector]|status|archive|backup");
116 + }
117 + await pool.end();
118 +}
119 +
120 +main().catch((err) => {
121 + console.error(err instanceof Error ? err.message : err);
122 + process.exit(1);
123 +});
added apps/api/src/config.ts +51 −0
@@ -0,0 +1,51 @@
1 +import { mkdirSync } from "node:fs";
2 +import { resolve } from "node:path";
3 +
4 +function env(name: string, fallback?: string): string {
5 + const v = process.env[name];
6 + if (v === undefined || v === "") {
7 + if (fallback !== undefined) return fallback;
8 + throw new Error(`missing environment variable ${name}`);
9 + }
10 + return v;
11 +}
12 +
13 +export type Role = "all" | "api" | "worker";
14 +
15 +export const config = {
16 + appEnv: env("APP_ENV", "development"),
17 + isProd: env("APP_ENV", "development") === "production",
18 + siteUrl: env("MA_SITE_URL", "http://localhost:8390"),
19 + apiHost: env("MA_API_HOST", "127.0.0.1"),
20 + apiPort: Number(env("MA_API_PORT", "8391")),
21 + role: env("MA_ROLE", "all") as Role,
22 + databaseUrl: env("DATABASE_URL", "postgres://localhost:5432/market_atlas"),
23 + dataDir: resolve(env("MA_DATA_DIR", "./data")),
24 + adminToken: env("MA_ADMIN_TOKEN", ""),
25 + /** When set, the API process is the public edge: /v1/* served locally, everything else proxied to the Next.js web app. */
26 + webUpstream: env("MA_WEB_UPSTREAM", ""),
27 + logJson: env("MA_LOG_JSON", "0") === "1",
28 + contactEmail: env("MA_CONTACT_EMAIL", "contact@market-atlas.co"),
29 + userAgent: `MarketAtlas/0.1 (+https://www.market-atlas.co; ${env("MA_CONTACT_EMAIL", "contact@market-atlas.co")})`,
30 + disabledConnectors: new Set(
31 + env("MA_DISABLED_CONNECTORS", "")
32 + .split(",")
33 + .map((s) => s.trim())
34 + .filter(Boolean),
35 + ),
36 + rawSampleRate: Number(env("MA_RAW_SAMPLE_RATE", "0.02")),
37 + observationRetentionDays: Number(env("MA_OBSERVATION_RETENTION_DAYS", "45")),
38 + /** Observation writer batching. */
39 + writerFlushMs: 1000,
40 + writerMaxBatch: 2000, // 17 columns × 2000 rows stays under Postgres' 65,535 bind-parameter limit
41 + /** Consensus: observations older than this (ms) are excluded for REALTIME sources. */
42 + consensusFreshWindowMs: 15_000,
43 + version: "0.1.0",
44 +};
45 +
46 +export function ensureDataDirs(): void {
47 + for (const d of ["raw", "archive", "backups", "logs", "cache"]) mkdirSync(resolve(config.dataDir, d), { recursive: true });
48 +}
49 +
50 +export const runsWorker = () => config.role === "all" || config.role === "worker";
51 +export const runsApi = () => config.role === "all" || config.role === "api";
added apps/api/src/core/bars.ts +107 −0
@@ -0,0 +1,107 @@
1 +import type { Bar, BarResolution, CanonicalQuote } from "@market-atlas/market-model";
2 +import { CANONICAL_VERSIONS, RESOLUTION_MS, floorTo } from "@market-atlas/market-model";
3 +import { pool } from "../db/pool.js";
4 +import { logger } from "../logger.js";
5 +import { telemetry } from "./telemetry.js";
6 +
7 +/**
8 + * Live 1-minute bar aggregation from canonical prices; higher resolutions are rolled up in SQL
9 + * by the scheduler (reproducible from the 1m series).
10 + */
11 +export class BarAggregator {
12 + private open = new Map<string, Bar>(); // instrumentId -> current 1m bar
13 + private pending: Bar[] = [];
14 + private timer: NodeJS.Timeout | null = null;
15 +
16 + onQuote(q: CanonicalQuote): void {
17 + if (q.price == null || q.realtimeStatus === "STALE" || q.realtimeStatus === "END_OF_DAY") return;
18 + const ts = floorTo(q.sourceTimestamp ?? q.updatedAt, RESOLUTION_MS["1m"]);
19 + const cur = this.open.get(q.instrumentId);
20 + if (cur && cur.ts !== ts) {
21 + this.pending.push(cur);
22 + this.open.delete(q.instrumentId);
23 + }
24 + if (!this.open.has(q.instrumentId)) {
25 + this.open.set(q.instrumentId, {
26 + instrumentId: q.instrumentId,
27 + resolution: "1m",
28 + ts,
29 + open: q.price,
30 + high: q.price,
31 + low: q.price,
32 + close: q.price,
33 + volume: null,
34 + sourceCount: q.sourceCount,
35 + producer: "consensus",
36 + version: CANONICAL_VERSIONS.consensus,
37 + });
38 + } else {
39 + const b = this.open.get(q.instrumentId)!;
40 + b.high = Math.max(b.high, q.price);
41 + b.low = Math.min(b.low, q.price);
42 + b.close = q.price;
43 + b.sourceCount = Math.max(b.sourceCount, q.sourceCount);
44 + }
45 + if (!this.timer) this.timer = setTimeout(() => void this.flush(), 5000);
46 + }
47 +
48 + /** Flush completed bars plus a snapshot of open bars (upserted again when they close). */
49 + async flush(): Promise<void> {
50 + this.timer = null;
51 + const batch = [...this.pending, ...this.open.values()];
52 + this.pending = [];
53 + if (!batch.length) return;
54 + await upsertBars(batch);
55 + if (this.open.size) this.timer = setTimeout(() => void this.flush(), 5000);
56 + }
57 +}
58 +
59 +export async function upsertBars(bars: Bar[]): Promise<void> {
60 + if (!bars.length) return;
61 + const cols = 11;
62 + for (let i = 0; i < bars.length; i += 1000) {
63 + const chunk = bars.slice(i, i + 1000);
64 + const tuples: string[] = [];
65 + const values: unknown[] = [];
66 + chunk.forEach((b, j) => {
67 + const base = j * cols;
68 + tuples.push(`(${Array.from({ length: cols }, (_, k) => `$${base + k + 1}`).join(",")})`);
69 + values.push(b.instrumentId, b.resolution, new Date(b.ts).toISOString(), b.open, b.high, b.low, b.close, b.volume, b.sourceCount, b.producer, b.version);
70 + });
71 + try {
72 + await pool.query(
73 + `insert into bars (instrument_id, resolution, ts, open, high, low, close, volume, source_count, producer, version) values ${tuples.join(",")}
74 + on conflict (instrument_id, resolution, ts) do update set open = excluded.open,
75 + high = excluded.high, low = excluded.low, close = excluded.close, volume = coalesce(excluded.volume, bars.volume), source_count = excluded.source_count, producer = excluded.producer, version = excluded.version
76 + where bars.producer = 'consensus' or excluded.producer <> 'consensus'`,
77 + values,
78 + );
79 + telemetry.inc("bars_written_total", chunk.length);
80 + } catch (err) {
81 + logger.error({ err }, "bar upsert failed");
82 + }
83 + }
84 +}
85 +
86 +/** Roll 1m bars into 5m/15m/1h and 1m→1d (UTC days) for the last `hours` hours. */
87 +export async function rollupBars(hours = 3): Promise<void> {
88 + const targets: Array<[BarResolution, string]> = [
89 + ["5m", "5 minutes"],
90 + ["15m", "15 minutes"],
91 + ["1h", "1 hour"],
92 + ["1d", "1 day"],
93 + ];
94 + for (const [res, interval] of targets) {
95 + await pool.query(
96 + `insert into bars (instrument_id, resolution, ts, open, high, low, close, volume, source_count, producer, version)
97 + select instrument_id, $1, bucket, (array_agg(open order by ts))[1], max(high), min(low), (array_agg(close order by ts desc))[1], sum(volume), max(source_count), 'consensus', $3
98 + from (select *, date_bin($2::interval, ts, timestamptz '2000-01-01') as bucket from bars where resolution = '1m' and producer = 'consensus' and ts > now() - ($4 || ' hours')::interval) b
99 + group by instrument_id, bucket
100 + on conflict (instrument_id, resolution, ts) do update set open = excluded.open, high = excluded.high, low = excluded.low, close = excluded.close, volume = excluded.volume, source_count = excluded.source_count
101 + where bars.producer = 'consensus'`,
102 + [res, interval, CANONICAL_VERSIONS.consensus, String(hours)],
103 + );
104 + }
105 +}
106 +
107 +export const barAggregator = new BarAggregator();
added apps/api/src/core/bus.ts +38 −0
@@ -0,0 +1,38 @@
1 +import { EventEmitter } from "node:events";
2 +import type { CanonicalQuote, ConnectorHealth, MarketEvent, Observation, RawObservation } from "@market-atlas/market-model";
3 +
4 +/**
5 + * Internal market bus. In-process today (modular monolith); the topic names mirror the
6 + * spec so that a NATS/Redpanda transport can replace this class without touching producers.
7 + */
8 +export interface BusTopics {
9 + "raw.observation": RawObservation;
10 + "normalized.observation": Observation;
11 + "canonical.quote": CanonicalQuote;
12 + "market.event": MarketEvent;
13 + "connector.health": ConnectorHealth;
14 + "market.state": { exchangeId: string; state: string; at: number };
15 +}
16 +
17 +type Handler<T> = (msg: T) => void;
18 +
19 +export class MarketBus {
20 + private emitter = new EventEmitter({ captureRejections: false });
21 + readonly counters: Record<string, number> = {};
22 +
23 + constructor() {
24 + this.emitter.setMaxListeners(200);
25 + }
26 +
27 + publish<K extends keyof BusTopics>(topic: K, msg: BusTopics[K]): void {
28 + this.counters[topic] = (this.counters[topic] ?? 0) + 1;
29 + this.emitter.emit(topic, msg);
30 + }
31 +
32 + subscribe<K extends keyof BusTopics>(topic: K, handler: Handler<BusTopics[K]>): () => void {
33 + this.emitter.on(topic, handler);
34 + return () => this.emitter.off(topic, handler);
35 + }
36 +}
37 +
38 +export const bus = new MarketBus();
added apps/api/src/core/calendar.test.ts +54 −0
@@ -0,0 +1,54 @@
1 +import { describe, expect, it } from "vitest";
2 +import type { Exchange } from "@market-atlas/market-model";
3 +import { MarketCalendar } from "./calendar.js";
4 +
5 +const xnys: Exchange = { id: "xnys", mic: "XNYS", name: "NYSE", operator: null, country: "US", city: null, timezone: "America/New_York", currency: "USD", website: null, lat: null, lon: null, assetClasses: ["EQUITY"], sessions: { regular: [{ open: "09:30", close: "16:00" }], pre: { open: "04:00", close: "09:30" }, post: { open: "16:00", close: "20:00" } } };
6 +const xtks: Exchange = { ...xnys, id: "xtks", timezone: "Asia/Tokyo", sessions: { regular: [{ open: "09:00", close: "11:30" }, { open: "12:30", close: "15:30" }] } };
7 +const xsau: Exchange = { ...xnys, id: "xsau", timezone: "Asia/Riyadh", sessions: { regular: [{ open: "10:00", close: "15:00" }], weekdays: [0, 1, 2, 3, 4] } };
8 +const coinbase: Exchange = { ...xnys, id: "coinbase", timezone: "UTC", sessions: { regular: [], continuous: true } };
9 +
10 +const cal = new MarketCalendar();
11 +cal.load([xnys, xtks, xsau, coinbase], [
12 + { exchangeId: "xnys", date: "2026-09-07", name: "Labor Day", kind: "CLOSED", sourceId: "t" },
13 + { exchangeId: "xnys", date: "2026-11-27", name: "Early close", kind: "EARLY_CLOSE", closeTime: "13:00", sourceId: "t" },
14 +]);
15 +
16 +const at = (iso: string) => Date.parse(iso);
17 +
18 +describe("market calendar", () => {
19 + it("NYSE regular, pre, post, closed and weekend", () => {
20 + expect(cal.status("xnys", at("2026-09-11T14:00:00Z")).state).toBe("OPEN"); // 10:00 EDT Friday
21 + expect(cal.status("xnys", at("2026-09-11T12:00:00Z")).state).toBe("PRE"); // 08:00 EDT
22 + expect(cal.status("xnys", at("2026-09-11T21:00:00Z")).state).toBe("POST"); // 17:00 EDT
23 + expect(cal.status("xnys", at("2026-09-12T14:00:00Z")).state).toBe("CLOSED"); // Saturday
24 + expect(cal.status("xnys", at("2026-09-11T02:00:00Z")).state).toBe("CLOSED"); // 22:00 EDT Thursday
25 + });
26 + it("holidays and early closes", () => {
27 + const labor = cal.status("xnys", at("2026-09-07T14:00:00Z"));
28 + expect(labor.state).toBe("CLOSED");
29 + expect(labor.isHoliday).toBe(true);
30 + expect(labor.holidayName).toBe("Labor Day");
31 + expect(cal.status("xnys", at("2026-11-27T17:00:00Z")).state).toBe("OPEN"); // 12:00 EST
32 + expect(cal.status("xnys", at("2026-11-27T18:30:00Z")).state).toBe("CLOSED"); // 13:30 EST after early close
33 + });
34 + it("DST: winter session in UTC shifts by one hour", () => {
35 + expect(cal.status("xnys", at("2026-01-15T14:00:00Z")).state).toBe("PRE"); // 09:00 EST
36 + expect(cal.status("xnys", at("2026-01-15T14:30:00Z")).state).toBe("OPEN");
37 + });
38 + it("Tokyo lunch break and Riyadh Sunday trading", () => {
39 + expect(cal.status("xtks", at("2026-09-11T01:00:00Z")).state).toBe("OPEN"); // 10:00 JST
40 + expect(cal.status("xtks", at("2026-09-11T03:00:00Z")).state).toBe("CLOSED"); // 12:00 JST lunch
41 + expect(cal.status("xtks", at("2026-09-11T04:00:00Z")).state).toBe("OPEN"); // 13:00 JST
42 + expect(cal.status("xsau", at("2026-09-13T08:00:00Z")).state).toBe("OPEN"); // Sunday 11:00 AST
43 + expect(cal.status("xsau", at("2026-09-11T08:00:00Z")).state).toBe("CLOSED"); // Friday
44 + });
45 + it("continuous venues are always open; next transition computed for NYSE", () => {
46 + expect(cal.status("coinbase", at("2026-09-12T03:00:00Z")).state).toBe("OPEN");
47 + const s = cal.status("xnys", at("2026-09-11T14:00:00Z"));
48 + expect(s.nextTransition?.state).toBe("POST");
49 + expect(new Date(s.nextTransition!.at).toISOString()).toBe("2026-09-11T20:00:00.000Z");
50 + });
51 + it("session start for the trading day", () => {
52 + expect(new Date(cal.sessionStart("xnys", at("2026-09-11T14:00:00Z"))!).toISOString()).toBe("2026-09-11T08:00:00.000Z"); // pre-market 04:00 EDT
53 + });
54 +});
added apps/api/src/core/calendar.ts +135 −0
@@ -0,0 +1,135 @@
1 +import type { Exchange, ExchangeStatus, HolidaySpec, MarketState, TradingSessionSpec } from "@market-atlas/market-model";
2 +import { hmToMs, zonedParts, zonedTimeToUtc } from "@market-atlas/market-model";
3 +
4 +/**
5 + * Market hours engine. Time zones, DST, weekends, holidays and early closes are evaluated
6 + * with Intl — nothing is hard-coded to 09:30–16:00.
7 + */
8 +export class MarketCalendar {
9 + private exchanges = new Map<string, Exchange>();
10 + private holidays = new Map<string, Map<string, HolidaySpec>>(); // exchangeId -> date -> spec
11 +
12 + load(exchanges: Exchange[], holidays: HolidaySpec[]) {
13 + this.exchanges.clear();
14 + this.holidays.clear();
15 + for (const e of exchanges) this.exchanges.set(e.id, e);
16 + for (const h of holidays) this.addHoliday(h);
17 + }
18 +
19 + addHoliday(h: HolidaySpec) {
20 + let m = this.holidays.get(h.exchangeId);
21 + if (!m) {
22 + m = new Map();
23 + this.holidays.set(h.exchangeId, m);
24 + }
25 + m.set(h.date, h);
26 + }
27 +
28 + exchange(id: string): Exchange | undefined {
29 + return this.exchanges.get(id);
30 + }
31 +
32 + list(): Exchange[] {
33 + return [...this.exchanges.values()];
34 + }
35 +
36 + status(exchangeId: string, now = Date.now()): ExchangeStatus {
37 + const ex = this.exchanges.get(exchangeId);
38 + if (!ex) return { exchangeId, state: "UNKNOWN", localTime: "", nextTransition: null, isHoliday: false, holidayName: null };
39 + const p = zonedParts(now, ex.timezone);
40 + const sessions = ex.sessions;
41 + if (sessions.continuous) {
42 + return { exchangeId, state: "OPEN", localTime: p.time, nextTransition: null, isHoliday: false, holidayName: null };
43 + }
44 + const holiday = this.holidays.get(exchangeId)?.get(p.date) ?? null;
45 + const weekdays = sessions.weekdays ?? [1, 2, 3, 4, 5];
46 + const tradingDay = weekdays.includes(p.weekday) && holiday?.kind !== "CLOSED";
47 + const state = tradingDay ? this.stateWithin(sessions, p.time, holiday) : "CLOSED";
48 + return {
49 + exchangeId,
50 + state,
51 + localTime: p.time,
52 + nextTransition: this.nextTransition(ex, now),
53 + isHoliday: !!holiday,
54 + holidayName: holiday?.name ?? null,
55 + };
56 + }
57 +
58 + isOpen(exchangeId: string, now = Date.now()): boolean {
59 + return this.state(exchangeId, now) === "OPEN";
60 + }
61 +
62 + private stateCache = new Map<string, { at: number; state: MarketState }>();
63 +
64 + /** Cheap state lookup (no next-transition scan), memoized for 5 s per exchange. Hot path for list endpoints. */
65 + state(exchangeId: string, now = Date.now()): MarketState {
66 + const ex = this.exchanges.get(exchangeId);
67 + if (!ex) return "UNKNOWN";
68 + if (ex.sessions.continuous) return "OPEN";
69 + const c = this.stateCache.get(exchangeId);
70 + if (c && now - c.at < 5000) return c.state;
71 + const state = this.rawState(ex, now);
72 + this.stateCache.set(exchangeId, { at: now, state });
73 + return state;
74 + }
75 +
76 + /** True when any regular session of the exchange is trading, pre or post included. */
77 + isTradingDay(exchangeId: string, now = Date.now()): boolean {
78 + const s = this.status(exchangeId, now).state;
79 + return s === "OPEN" || s === "PRE" || s === "POST" || s === "AUCTION";
80 + }
81 +
82 + private stateWithin(sessions: TradingSessionSpec, time: string, holiday: HolidaySpec | null): MarketState {
83 + const t = hmToMs(time);
84 + const regular = sessions.regular.map((s) => ({ open: hmToMs(s.open), close: hmToMs(holiday?.kind === "EARLY_CLOSE" && holiday.closeTime ? holiday.closeTime : s.close) }));
85 + for (const s of regular) if (t >= s.open && t < s.close) return "OPEN";
86 + if (sessions.pre && t >= hmToMs(sessions.pre.open) && t < hmToMs(sessions.pre.close)) return "PRE";
87 + if (sessions.post && holiday?.kind !== "EARLY_CLOSE" && t >= hmToMs(sessions.post.open) && t < hmToMs(sessions.post.close)) return "POST";
88 + return "CLOSED";
89 + }
90 +
91 + /** Scan forward (minute granularity, up to 10 days) for the next state change. */
92 + private nextTransition(ex: Exchange, now: number): { state: MarketState; at: number } | null {
93 + const current = this.rawState(ex, now);
94 + const step = 60_000;
95 + const limit = now + 10 * 86_400_000;
96 + // Coarse scan by 15 minutes then refine by minute.
97 + for (let t = now + step; t <= limit; t += 15 * step) {
98 + if (this.rawState(ex, t) !== current) {
99 + for (let u = t - 15 * step; u <= t; u += step) {
100 + const s = this.rawState(ex, u);
101 + if (s !== current) return { state: s, at: u };
102 + }
103 + }
104 + }
105 + return null;
106 + }
107 +
108 + private rawState(ex: Exchange, at: number): MarketState {
109 + const p = zonedParts(at, ex.timezone);
110 + const holiday = this.holidays.get(ex.id)?.get(p.date) ?? null;
111 + const weekdays = ex.sessions.weekdays ?? [1, 2, 3, 4, 5];
112 + if (!weekdays.includes(p.weekday) || holiday?.kind === "CLOSED") return "CLOSED";
113 + return this.stateWithin(ex.sessions, p.time, holiday);
114 + }
115 +
116 + /** Session start (UTC ms) of the trading day containing `now`, used for session high/low resets. */
117 + sessionStart(exchangeId: string, now = Date.now()): number | null {
118 + const ex = this.exchanges.get(exchangeId);
119 + if (!ex) return null;
120 + if (ex.sessions.continuous) {
121 + const p = zonedParts(now, "UTC");
122 + return zonedTimeToUtc(p.date, "UTC");
123 + }
124 + const p = zonedParts(now, ex.timezone);
125 + const first = ex.sessions.pre?.open ?? ex.sessions.regular[0]?.open;
126 + if (!first) return null;
127 + return zonedTimeToUtc(`${p.date}T${first}:00`, ex.timezone);
128 + }
129 +
130 + holidaysFor(exchangeId: string): HolidaySpec[] {
131 + return [...(this.holidays.get(exchangeId)?.values() ?? [])].sort((a, b) => a.date.localeCompare(b.date));
132 + }
133 +}
134 +
135 +export const calendar = new MarketCalendar();
added apps/api/src/core/connector-manager.ts +371 −0
@@ -0,0 +1,371 @@
1 +import type { RawObservation } from "@market-atlas/market-model";
2 +import {
3 + HttpClient,
4 + ManagedWebSocket,
5 + RateLimiter,
6 + backoffMs,
7 + schemaFingerprint,
8 + type ConnectorContext,
9 + type ConnectorDefinition,
10 + type ManagedWebSocketOptions,
11 +} from "@market-atlas/connector-sdk";
12 +import { config } from "../config.js";
13 +import { pool } from "../db/pool.js";
14 +import { logger } from "../logger.js";
15 +import { calendar } from "./calendar.js";
16 +import { eventEngine } from "./events.js";
17 +import { health } from "./health.js";
18 +import { pipeline } from "./pipeline.js";
19 +import { rawArchive } from "./raw-archive.js";
20 +import { telemetry } from "./telemetry.js";
21 +
22 +interface Runtime {
23 + def: ConnectorDefinition;
24 + ctx: ConnectorContext;
25 + sockets: ManagedWebSocket[];
26 + pollTimer: NodeJS.Timeout | null;
27 + running: boolean;
28 + paused: boolean;
29 + consecutiveFailures: number;
30 + driftStrikes: number;
31 + symbols: string[];
32 + stateCache: Record<string, unknown>;
33 + stateDirty: boolean;
34 +}
35 +
36 +/**
37 + * Runs connectors: lifecycle, adaptive polling, per-host rate limits, circuit breaker with
38 + * jittered backoff, schema-drift detection, L0 archiving and hand-off to the pipeline.
39 + */
40 +export class ConnectorManager {
41 + private runtimes = new Map<string, Runtime>();
42 + readonly limiter = new RateLimiter({ ratePerSec: 2, burst: 4 });
43 + readonly http = new HttpClient({
44 + userAgent: config.userAgent,
45 + limiter: this.limiter,
46 + onRequest: (i) => {
47 + telemetry.inc("http_requests_total", 1, { host: i.host, status: String(i.status) });
48 + telemetry.observe("http_request_ms", i.durationMs, { host: i.host });
49 + },
50 + });
51 +
52 + list(): Runtime[] {
53 + return [...this.runtimes.values()];
54 + }
55 +
56 + get(id: string): Runtime | undefined {
57 + return this.runtimes.get(id);
58 + }
59 +
60 + async register(def: ConnectorDefinition, opts: { paused: boolean; state: Record<string, unknown>; fingerprints: Record<string, string[]>; symbols?: string[] }): Promise<void> {
61 + const id = def.metadata.id;
62 + for (const [host, rate] of Object.entries(def.rateLimits ?? {})) this.limiter.configure(host, rate);
63 + health.register(id, def.metadata.sourceId);
64 + health.loadFingerprints(id, opts.fingerprints);
65 + pipeline.registerSource(def.metadata.sourceId, {
66 + family: def.metadata.sourceFamily,
67 + isOfficial: def.metadata.rightsStatus === "OFFICIAL_OPEN_DATA",
68 + realtimeStatus: def.metadata.realtimeStatus,
69 + });
70 + const rt: Runtime = {
71 + def,
72 + ctx: null as unknown as ConnectorContext,
73 + sockets: [],
74 + pollTimer: null,
75 + running: false,
76 + paused: opts.paused,
77 + consecutiveFailures: 0,
78 + driftStrikes: 0,
79 + symbols: opts.symbols ?? def.defaultSymbols ?? [],
80 + stateCache: opts.state ?? {},
81 + stateDirty: false,
82 + };
83 + rt.ctx = this.makeContext(rt);
84 + this.runtimes.set(id, rt);
85 + }
86 +
87 + private makeContext(rt: Runtime): ConnectorContext {
88 + const id = rt.def.metadata.id;
89 + const log = logger.child({ connector: id });
90 + return {
91 + connectorId: id,
92 + logger: {
93 + debug: (m, d) => log.debug(d ?? {}, m),
94 + info: (m, d) => log.info(d ?? {}, m),
95 + warn: (m, d) => log.warn(d ?? {}, m),
96 + error: (m, d) => log.error(d ?? {}, m),
97 + },
98 + http: this.http,
99 + state: {
100 + get: async (k) => rt.stateCache[k] as never,
101 + set: async (k, v) => {
102 + rt.stateCache[k] = v;
103 + rt.stateDirty = true;
104 + },
105 + },
106 + emit: (raw) => {
107 + for (const r of Array.isArray(raw) ? raw : [raw]) void this.handleRaw(rt, r, false);
108 + },
109 + openWebSocket: (url: string, o: ManagedWebSocketOptions) => {
110 + const ws = new ManagedWebSocket(url, {
111 + ...o,
112 + onError: (err) => {
113 + health.error(id, err);
114 + o.onError?.(err);
115 + },
116 + });
117 + ws.on((ev, detail) => {
118 + if (ev === "connected") {
119 + rt.consecutiveFailures = 0;
120 + health.setState(id, "HEALTHY");
121 + } else if (ev === "reconnecting") {
122 + health.reconnect(id);
123 + health.setState(id, "RECONNECTING");
124 + if ((detail?.attempt as number) >= 5) this.markFailed(rt, `reconnect attempt ${detail?.attempt}`);
125 + } else if (ev === "stale") {
126 + health.setState(id, "STALE");
127 + }
128 + });
129 + rt.sockets.push(ws);
130 + return ws;
131 + },
132 + isMarketOpen: (exchangeId) => calendar.isOpen(exchangeId),
133 + watchedSymbols: () => rt.symbols,
134 + reportError: (err, context) => {
135 + health.error(id, err);
136 + log.warn({ err: err instanceof Error ? err.message : String(err), ...context }, "connector reported error");
137 + },
138 + secret: (name) => process.env[name],
139 + now: () => Date.now(),
140 + };
141 + }
142 +
143 + async startAll(): Promise<void> {
144 + for (const rt of this.runtimes.values()) {
145 + if (rt.paused) {
146 + health.setState(rt.def.metadata.id, "PAUSED");
147 + continue;
148 + }
149 + await this.start(rt.def.metadata.id);
150 + }
151 + }
152 +
153 + async start(id: string): Promise<void> {
154 + const rt = this.runtimes.get(id);
155 + if (!rt || rt.running) return;
156 + rt.running = true;
157 + rt.paused = false;
158 + health.setState(id, "STARTING");
159 + try {
160 + if (rt.def.start) {
161 + await rt.def.start(rt.ctx);
162 + if (!rt.sockets.length) health.setState(id, "HEALTHY");
163 + }
164 + if (rt.def.poll) this.schedulePoll(rt, 500 + Math.random() * 3000);
165 + } catch (err) {
166 + health.error(id, err);
167 + this.markFailed(rt, err instanceof Error ? err.message : String(err));
168 + this.scheduleRestart(rt);
169 + }
170 + }
171 +
172 + async stop(id: string, reason = "stop"): Promise<void> {
173 + const rt = this.runtimes.get(id);
174 + if (!rt) return;
175 + rt.running = false;
176 + if (rt.pollTimer) clearTimeout(rt.pollTimer);
177 + rt.pollTimer = null;
178 + for (const s of rt.sockets) s.close();
179 + rt.sockets = [];
180 + try {
181 + await rt.def.stop?.(rt.ctx);
182 + } catch (err) {
183 + logger.warn({ connector: id, err }, "connector stop failed");
184 + }
185 + await this.persistState(rt);
186 + logger.info({ connector: id, reason }, "connector stopped");
187 + }
188 +
189 + async pause(id: string): Promise<void> {
190 + const rt = this.runtimes.get(id);
191 + if (!rt) return;
192 + await this.stop(id, "pause");
193 + rt.paused = true;
194 + health.setState(id, "PAUSED");
195 + await pool.query("update connectors set paused = true, updated_at = now() where id = $1", [id]);
196 + }
197 +
198 + async resume(id: string): Promise<void> {
199 + const rt = this.runtimes.get(id);
200 + if (!rt) return;
201 + await pool.query("update connectors set paused = false, updated_at = now() where id = $1", [id]);
202 + rt.consecutiveFailures = 0;
203 + rt.driftStrikes = 0;
204 + await this.start(id);
205 + }
206 +
207 + async restart(id: string): Promise<void> {
208 + await this.stop(id, "restart");
209 + const rt = this.runtimes.get(id);
210 + if (rt) {
211 + rt.consecutiveFailures = 0;
212 + await this.start(id);
213 + }
214 + }
215 +
216 + /** Run one poll immediately (admin "run test"); returns accepted observation count. */
217 + async runOnce(id: string): Promise<{ accepted: number; raws: number; durationMs: number }> {
218 + const rt = this.runtimes.get(id);
219 + if (!rt) throw new Error("unknown connector");
220 + const started = Date.now();
221 + let accepted = 0;
222 + let raws = 0;
223 + if (rt.def.poll) {
224 + const out = await rt.def.poll(rt.ctx);
225 + raws = out.length;
226 + for (const r of out) accepted += await this.handleRaw(rt, r, true);
227 + } else if (rt.def.healthCheck) {
228 + const res = await rt.def.healthCheck(rt.ctx);
229 + if (!res.ok) throw new Error(res.detail ?? "health check failed");
230 + }
231 + return { accepted, raws, durationMs: Date.now() - started };
232 + }
233 +
234 + private schedulePoll(rt: Runtime, delayMs: number) {
235 + if (!rt.running) return;
236 + if (rt.pollTimer) clearTimeout(rt.pollTimer);
237 + const at = Date.now() + delayMs;
238 + health.poll(rt.def.metadata.id, at);
239 + rt.pollTimer = setTimeout(() => void this.poll(rt), delayMs);
240 + }
241 +
242 + private nextInterval(rt: Runtime): number {
243 + const s = rt.def.schedule!;
244 + if ("intervalMs" in s) return s.intervalMs;
245 + if (!s.exchangeId) return s.openMs;
246 + const st = calendar.status(s.exchangeId);
247 + if (st.state === "OPEN" || st.state === "PRE" || st.state === "POST") return s.openMs;
248 + const day = new Date().getUTCDay();
249 + if (day === 0 || day === 6 || st.isHoliday) return s.weekendMs;
250 + return s.closedMs;
251 + }
252 +
253 + private async poll(rt: Runtime) {
254 + const id = rt.def.metadata.id;
255 + if (!rt.running || !rt.def.poll) return;
256 + const started = Date.now();
257 + try {
258 + const raws = await rt.def.poll(rt.ctx);
259 + let accepted = 0;
260 + for (const r of raws) accepted += await this.handleRaw(rt, r, true);
261 + rt.consecutiveFailures = 0;
262 + if (health.state(id) !== "HEALTHY") health.setState(id, "HEALTHY");
263 + telemetry.observe("connector_poll_ms", Date.now() - started, { connector: id });
264 + logger.debug({ connector: id, raws: raws.length, accepted, ms: Date.now() - started }, "poll done");
265 + await this.persistState(rt);
266 + this.schedulePoll(rt, this.nextInterval(rt));
267 + } catch (err) {
268 + rt.consecutiveFailures++;
269 + health.error(id, err);
270 + const wait = Math.max(this.nextInterval(rt), backoffMs(rt.consecutiveFailures, 5000, 30 * 60_000));
271 + logger.warn({ connector: id, err: err instanceof Error ? err.message : String(err), failures: rt.consecutiveFailures, retryInMs: wait }, "poll failed");
272 + if (rt.consecutiveFailures >= 3) this.markFailed(rt, err instanceof Error ? err.message : String(err));
273 + else health.setState(id, "DEGRADED");
274 + this.schedulePoll(rt, wait);
275 + }
276 + }
277 +
278 + private markFailed(rt: Runtime, reason: string) {
279 + const id = rt.def.metadata.id;
280 + if (health.state(id) === "FAILED") return;
281 + health.setState(id, "FAILED");
282 + eventEngine.system("SOURCE_FAILURE", `${id}:${Math.floor(Date.now() / 3_600_000)}`, Date.now(), "WARNING", `Source failure: ${rt.def.metadata.name}`, { connectorId: id, reason: reason.slice(0, 200) }, [rt.def.metadata.sourceId]);
283 + }
284 +
285 + private scheduleRestart(rt: Runtime) {
286 + const wait = backoffMs(Math.min(8, rt.consecutiveFailures + 1), 10_000, 15 * 60_000);
287 + rt.consecutiveFailures++;
288 + setTimeout(() => {
289 + if (!rt.paused) {
290 + rt.running = false;
291 + void this.start(rt.def.metadata.id);
292 + }
293 + }, wait);
294 + }
295 +
296 + /** Every raw payload: fingerprint, archive, normalize, hand to the pipeline. Returns accepted count. */
297 + private async handleRaw(rt: Runtime, raw: RawObservation, forceArchive: boolean): Promise<number> {
298 + const id = rt.def.metadata.id;
299 + telemetry.inc("raw_total", 1, { connector: id });
300 + const fp = schemaFingerprint(raw.payload);
301 + const status = health.recordFingerprint(id, raw.kind, fp);
302 + let batch;
303 + try {
304 + batch = rt.def.normalize(raw);
305 + health.parse(id, true);
306 + } catch (err) {
307 + health.parse(id, false);
308 + health.error(id, err);
309 + rt.driftStrikes++;
310 + this.recordSchemaChange(rt, raw, fp, status).catch(() => {});
311 + if (rt.driftStrikes >= 20) this.driftPause(rt, raw.kind);
312 + return 0;
313 + }
314 + const produced = batch.observations.length + (batch.events?.length ?? 0) + (batch.instruments?.length ?? 0) + (batch.filings?.length ?? 0) + (batch.holidays?.length ?? 0);
315 + if (status === "new") {
316 + await this.recordSchemaChange(rt, raw, fp, status);
317 + if (produced === 0) {
318 + rt.driftStrikes++;
319 + if (rt.driftStrikes >= 5) this.driftPause(rt, raw.kind);
320 + }
321 + }
322 + if (produced > 0) rt.driftStrikes = Math.max(0, rt.driftStrikes - 1);
323 + const rawRef = rawArchive.store(raw, forceArchive || rt.def.metadata.sourceType !== "WEBSOCKET");
324 + if (health.state(id) === "STALE" || health.state(id) === "DEGRADED") health.setState(id, "HEALTHY");
325 + return pipeline.process(rt.def, raw, batch, rawRef);
326 + }
327 +
328 + private async recordSchemaChange(rt: Runtime, raw: RawObservation, fp: string, status: "known" | "new" | "first") {
329 + if (status === "known") return;
330 + const id = rt.def.metadata.id;
331 + const known = health.fingerprints(id)[raw.kind] ?? [];
332 + await pool.query(
333 + `insert into connector_schema_changes (connector_id, kind, old_fingerprint, new_fingerprint, sample) values ($1,$2,$3,$4,$5)`,
334 + [id, raw.kind, known.filter((k) => k !== fp).at(-1) ?? null, fp, JSON.stringify(truncate(raw.payload))],
335 + );
336 + if (status === "new") {
337 + telemetry.inc("schema_changes_total", 1, { connector: id });
338 + logger.warn({ connector: id, kind: raw.kind, fingerprint: fp }, "schema change detected");
339 + }
340 + }
341 +
342 + private driftPause(rt: Runtime, kind: string) {
343 + const id = rt.def.metadata.id;
344 + logger.error({ connector: id, kind }, "schema drift: pausing connector to avoid producing wrong values");
345 + eventEngine.system("SCHEMA_DRIFT", `${id}:${kind}:${Math.floor(Date.now() / 3_600_000)}`, Date.now(), "CRITICAL", `Schema drift: ${rt.def.metadata.name}`, { connectorId: id, kind }, [rt.def.metadata.sourceId]);
346 + void this.pause(id);
347 + }
348 +
349 + private async persistState(rt: Runtime) {
350 + if (!rt.stateDirty) return;
351 + rt.stateDirty = false;
352 + try {
353 + await pool.query("update connectors set state = $2, updated_at = now() where id = $1", [rt.def.metadata.id, JSON.stringify(rt.stateCache)]);
354 + } catch (err) {
355 + rt.stateDirty = true;
356 + logger.warn({ connector: rt.def.metadata.id, err }, "state persist failed");
357 + }
358 + }
359 +
360 + async stopAll(): Promise<void> {
361 + await Promise.all([...this.runtimes.keys()].map((id) => this.stop(id, "shutdown")));
362 + }
363 +}
364 +
365 +function truncate(v: unknown): unknown {
366 + const s = JSON.stringify(v);
367 + if (!s) return null;
368 + return s.length > 4000 ? { truncated: s.slice(0, 4000) } : v;
369 +}
370 +
371 +export const connectorManager = new ConnectorManager();
added apps/api/src/core/consensus.test.ts +132 −0
@@ -0,0 +1,132 @@
1 +import { describe, expect, it } from "vitest";
2 +import type { Observation } from "@market-atlas/market-model";
3 +import { ConsensusEngine, weightedMedian } from "./consensus.js";
4 +
5 +const NOW = 1_789_200_000_000;
6 +const profiles: Record<string, { family: string | null; reliability: number; isOfficial: boolean }> = {
7 + a: { family: "fa", reliability: 0.9, isOfficial: false },
8 + b: { family: "fb", reliability: 0.9, isOfficial: false },
9 + c: { family: "fc", reliability: 0.9, isOfficial: false },
10 + d: { family: "fd", reliability: 0.9, isOfficial: false },
11 + e: { family: "fa", reliability: 0.9, isOfficial: false }, // same upstream as a
12 + eod: { family: "official", reliability: 0.95, isOfficial: true },
13 +};
14 +const engine = () => new ConsensusEngine((id) => ({ ...(profiles[id] ?? { family: null, reliability: 0.6, isOfficial: false }), realtimeStatus: "REALTIME" }));
15 +
16 +function obs(source: string, value: number, ageMs = 500, field: Observation["field"] = "LAST_PRICE", rt: Observation["realtimeStatus"] = "REALTIME", trust: Observation["timestampTrust"] = "EXCHANGE"): Observation {
17 + return {
18 + observationId: `${source}-${field}-${value}-${ageMs}`,
19 + instrumentId: "crypto_btc_usd",
20 + symbol: "BTC-USD",
21 + field,
22 + value,
23 + currency: "USD",
24 + sourceTimestamp: NOW - ageMs,
25 + timestampTrust: trust,
26 + rightsStatus: "PUBLIC_ATTRIBUTED",
27 + realtimeStatus: rt,
28 + sourceId: source,
29 + connectorId: `${source}-ws`,
30 + receivedAt: NOW - ageMs + 50,
31 + latencyMs: 50,
32 + rawRef: null,
33 + normalizerVersion: "1.0",
34 + };
35 +}
36 +
37 +describe("consensus engine", () => {
38 + it("all sources agree → canonical price with high confidence", () => {
39 + const e = engine();
40 + for (const [s, v] of [["a", 100], ["b", 100], ["c", 100]] as const) e.ingest(obs(s, v));
41 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
42 + expect(q.price).toBe(100);
43 + expect(q.sourceCount).toBe(3);
44 + expect(q.dispersionBps).toBe(0);
45 + expect(q.confidence).toBeGreaterThan(0.85);
46 + expect(q.confidence).toBeLessThanOrEqual(0.995);
47 + });
48 + it("one stale source is excluded and labelled", () => {
49 + const e = engine();
50 + e.ingest(obs("a", 100));
51 + e.ingest(obs("b", 100.1));
52 + e.ingest(obs("c", 90, 60_000)); // 60 s old realtime → stale
53 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
54 + expect(q.price).toBeGreaterThanOrEqual(100);
55 + expect(q.contributions.find((c) => c.sourceId === "c")).toMatchObject({ included: false, reason: "stale" });
56 + expect(q.sourceCount).toBe(2);
57 + });
58 + it("one extreme outlier among ≥3 is rejected", () => {
59 + const e = engine();
60 + e.ingest(obs("a", 100));
61 + e.ingest(obs("b", 100.2));
62 + e.ingest(obs("c", 99.9));
63 + e.ingest(obs("d", 130));
64 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
65 + expect(q.price).toBeLessThan(101);
66 + expect(q.contributions.find((c) => c.sourceId === "d")).toMatchObject({ included: false, reason: "outlier" });
67 + });
68 + it("two source clusters disagree → low confidence, large dispersion", () => {
69 + const e = engine();
70 + e.ingest(obs("a", 100));
71 + e.ingest(obs("b", 101.5));
72 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
73 + expect(q.dispersionBps).toBeGreaterThan(100);
74 + expect(q.confidence).toBeLessThan(0.7);
75 + });
76 + it("only one source available → modest confidence, count 1", () => {
77 + const e = engine();
78 + e.ingest(obs("a", 100));
79 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
80 + expect(q.price).toBe(100);
81 + expect(q.sourceCount).toBe(1);
82 + expect(q.confidence).toBeLessThan(0.85);
83 + });
84 + it("sources sharing an upstream family count once", () => {
85 + const e = engine();
86 + e.ingest(obs("a", 100));
87 + e.ingest(obs("e", 100));
88 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
89 + expect(q.sourceCount).toBe(1);
90 + });
91 + it("out-of-order older observation does not overwrite the newer one", () => {
92 + const e = engine();
93 + e.ingest(obs("a", 100, 100));
94 + e.ingest(obs("a", 90, 5000));
95 + expect(e.compute("crypto_btc_usd", "BTC-USD", NOW)!.price).toBe(100);
96 + });
97 + it("end-of-day value is superseded by a live source, but used when alone", () => {
98 + const e = engine();
99 + e.ingest(obs("eod", 95, 3_600_000, "LAST_PRICE", "END_OF_DAY", "SOURCE"));
100 + expect(e.compute("crypto_btc_usd", "BTC-USD", NOW)!.price).toBe(95);
101 + expect(e.compute("crypto_btc_usd", "BTC-USD", NOW)!.realtimeStatus).toBe("END_OF_DAY");
102 + e.ingest(obs("a", 100));
103 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
104 + expect(q.price).toBe(100);
105 + expect(q.realtimeStatus).toBe("REALTIME");
106 + expect(q.contributions.find((c) => c.sourceId === "eod")).toMatchObject({ included: false, reason: "superseded_by_live" });
107 + });
108 + it("market closed / everything stale → STALE status with last known value and zero confidence", () => {
109 + const e = engine();
110 + e.ingest(obs("a", 100, 10 * 60_000));
111 + const q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
112 + expect(q.realtimeStatus).toBe("STALE");
113 + expect(q.price).toBe(100);
114 + expect(q.confidence).toBe(0);
115 + });
116 + it("derives change from previous close and tracks session high/low", () => {
117 + const e = engine();
118 + e.ingest(obs("a", 100));
119 + e.ingest(obs("a", 95, 500, "PREVIOUS_CLOSE"));
120 + let q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
121 + expect(q.change).toBeCloseTo(5);
122 + expect(q.changePercent).toBeCloseTo(5.263, 2);
123 + e.ingest(obs("a", 102, 100));
124 + q = e.compute("crypto_btc_usd", "BTC-USD", NOW)!;
125 + expect(q.sessionHigh).toBe(102);
126 + expect(q.sessionLow).toBe(100);
127 + });
128 + it("weighted median", () => {
129 + expect(weightedMedian([{ v: 1, w: 1 }, { v: 2, w: 1 }, { v: 100, w: 1 }])).toBe(2);
130 + expect(weightedMedian([{ v: 1, w: 10 }, { v: 2, w: 1 }, { v: 100, w: 1 }])).toBe(1);
131 + });
132 +});
added apps/api/src/core/consensus.ts +279 −0
@@ -0,0 +1,279 @@
1 +import type { CanonicalQuote, Observation, RealtimeStatus, RightsStatus, SourceContribution } from "@market-atlas/market-model";
2 +import { CANONICAL_VERSIONS, PUBLICLY_REDISTRIBUTABLE } from "@market-atlas/market-model";
3 +import { config } from "../config.js";
4 +
5 +export interface SourceProfile {
6 + family: string | null;
7 + reliability: number; // 0..1 operational reliability (from health engine)
8 + realtimeStatus: RealtimeStatus;
9 + isOfficial: boolean;
10 +}
11 +
12 +interface Latest {
13 + obs: Observation;
14 +}
15 +
16 +/** Fresh window per realtime class: realtime ticks age out fast, EOD values live a day+. */
17 +const FRESH_WINDOW_MS: Record<RealtimeStatus, number> = {
18 + REALTIME: config.consensusFreshWindowMs,
19 + DELAYED: 30 * 60_000,
20 + INDICATIVE: 60 * 60_000,
21 + END_OF_DAY: 3 * 86_400_000,
22 + STALE: 0,
23 + UNKNOWN: 60_000,
24 +};
25 +
26 +const REALTIME_RANK: Record<RealtimeStatus, number> = { REALTIME: 4, DELAYED: 3, INDICATIVE: 2, END_OF_DAY: 1, UNKNOWN: 0, STALE: -1 };
27 +
28 +/**
29 + * Per-instrument consensus. Keeps the latest observation of each (source, field), computes a
30 + * weighted median price over fresh observations, discounts sources sharing an upstream family,
31 + * flags outliers, and exposes the full contribution list ("Why this price?").
32 + */
33 +export class ConsensusEngine {
34 + private latest = new Map<string, Map<string, Latest>>(); // instrumentId -> `${sourceId}|${field}` -> latest
35 + private session = new Map<string, { high: number; low: number; startedAt: number }>();
36 +
37 + /**
38 + * @param profiles source reliability/family lookup
39 + * @param marketOpen true/false when the instrument's venue has a session calendar, null for continuous or unknown venues
40 + */
41 + constructor(
42 + private profiles: (sourceId: string) => SourceProfile,
43 + private marketOpen: (instrumentId: string) => boolean | null = () => null,
44 + ) {}
45 +
46 + ingest(o: Observation): void {
47 + let m = this.latest.get(o.instrumentId);
48 + if (!m) {
49 + m = new Map();
50 + this.latest.set(o.instrumentId, m);
51 + }
52 + const key = `${o.sourceId}|${o.field}`;
53 + const prev = m.get(key);
54 + // Ordering: prefer source timestamp, fall back to receive time. Ignore out-of-order values.
55 + if (prev) {
56 + const pt = prev.obs.sourceTimestamp ?? prev.obs.receivedAt;
57 + const nt = o.sourceTimestamp ?? o.receivedAt;
58 + if (nt < pt) return;
59 + }
60 + m.set(key, { obs: o });
61 + }
62 +
63 + resetSession(instrumentId: string, at: number) {
64 + this.session.set(instrumentId, { high: -Infinity, low: Infinity, startedAt: at });
65 + }
66 +
67 + sessionOf(instrumentId: string) {
68 + return this.session.get(instrumentId);
69 + }
70 +
71 + instrumentsTracked(): number {
72 + return this.latest.size;
73 + }
74 +
75 + sourcesFor(instrumentId: string): string[] {
76 + const m = this.latest.get(instrumentId);
77 + if (!m) return [];
78 + return [...new Set([...m.keys()].map((k) => k.split("|")[0]!))];
79 + }
80 +
81 + compute(instrumentId: string, symbol: string, now = Date.now()): CanonicalQuote | null {
82 + const m = this.latest.get(instrumentId);
83 + if (!m) return null;
84 + const closed = this.marketOpen(instrumentId) === false;
85 + const price = this.aggregate(m, "LAST_PRICE", now, closed);
86 + const pick = (field: string) => this.aggregate(m, field, now, closed);
87 + const open = pick("OPEN");
88 + const high = pick("HIGH");
89 + const low = pick("LOW");
90 + const prevClose = pick("PREVIOUS_CLOSE");
91 + const volume = pick("VOLUME");
92 + const bid = pick("BID");
93 + const ask = pick("ASK");
94 + const changeObs = pick("CHANGE");
95 + const changePctObs = pick("CHANGE_PERCENT");
96 + const p = price.value;
97 + let change = changeObs.value;
98 + let changePercent = changePctObs.value;
99 + if (p != null && prevClose.value != null && prevClose.value !== 0) {
100 + change = p - prevClose.value;
101 + changePercent = (change / prevClose.value) * 100;
102 + } else if (p != null && change == null && open.value != null && open.value !== 0) {
103 + // Continuous markets (crypto) publish a rolling 24 h open instead of a previous close.
104 + change = p - open.value;
105 + changePercent = (change / open.value) * 100;
106 + }
107 + // Session high/low derived by Market Atlas itself.
108 + let sess = this.session.get(instrumentId);
109 + if (p != null) {
110 + if (!sess) {
111 + sess = { high: p, low: p, startedAt: now };
112 + this.session.set(instrumentId, sess);
113 + } else {
114 + if (p > sess.high) sess.high = p;
115 + if (p < sess.low) sess.low = p;
116 + }
117 + }
118 + const all = [...m.values()].map((l) => l.obs);
119 + const currency = all.find((o) => o.currency)?.currency ?? null;
120 + const anyNonPublic = price.contributions.some((c) => c.included && !PUBLICLY_REDISTRIBUTABLE.has(c.rightsStatus));
121 + const rightsBasis = price.contributions.find((c) => c.included) ?? [...price.contributions].sort((a, b) => a.ageMs - b.ageMs)[0];
122 + const rights: RightsStatus = anyNonPublic ? "INTERNAL_ONLY" : (rightsBasis?.rightsStatus ?? "UNKNOWN");
123 + return {
124 + instrumentId,
125 + symbol,
126 + price: p,
127 + open: open.value,
128 + high: high.value ?? (sess && Number.isFinite(sess.high) ? sess.high : null),
129 + low: low.value ?? (sess && Number.isFinite(sess.low) ? sess.low : null),
130 + previousClose: prevClose.value,
131 + change: change ?? null,
132 + changePercent: changePercent ?? null,
133 + volume: volume.value,
134 + bid: bid.value,
135 + ask: ask.value,
136 + currency,
137 + sourceCount: price.independentFamilies,
138 + dispersionBps: price.dispersionBps,
139 + confidence: price.confidence,
140 + freshnessMs: price.freshnessMs,
141 + realtimeStatus: price.realtimeStatus,
142 + rightsStatus: rights,
143 + updatedAt: now,
144 + sourceTimestamp: price.sourceTimestamp,
145 + contributions: price.contributions,
146 + consensusVersion: CANONICAL_VERSIONS.consensus,
147 + sessionHigh: sess && Number.isFinite(sess.high) ? sess.high : null,
148 + sessionLow: sess && Number.isFinite(sess.low) ? sess.low : null,
149 + };
150 + }
151 +
152 + private aggregate(m: Map<string, Latest>, field: string, now: number, marketClosed = false) {
153 + const contributions: SourceContribution[] = [];
154 + const candidates: Array<{ value: number; weight: number; family: string; obs: Observation }> = [];
155 + for (const [key, l] of m) {
156 + if (!key.endsWith(`|${field}`)) continue;
157 + const o = l.obs;
158 + const prof = this.profiles(o.sourceId);
159 + const eventTs = o.sourceTimestamp ?? o.receivedAt;
160 + const ageMs = Math.max(0, now - eventTs);
161 + // While the venue is closed, the last session value is the truth for up to 4 days (long weekends).
162 + const window = marketClosed && (o.realtimeStatus === "REALTIME" || o.realtimeStatus === "DELAYED") ? 4 * 86_400_000 : (FRESH_WINDOW_MS[o.realtimeStatus] ?? 60_000);
163 + const fresh = ageMs <= window;
164 + const base = 0.35 + 0.65 * prof.reliability;
165 + const tsQuality = o.timestampTrust === "EXCHANGE" ? 1 : o.timestampTrust === "SOURCE" ? 0.9 : 0.75;
166 + const rtQuality = o.realtimeStatus === "REALTIME" ? 1 : o.realtimeStatus === "DELAYED" ? 0.6 : o.realtimeStatus === "INDICATIVE" ? 0.5 : 0.4;
167 + const official = prof.isOfficial ? 1.15 : 1;
168 + const decay = fresh ? Math.exp(-ageMs / Math.max(1000, window)) : 0;
169 + const weight = base * tsQuality * rtQuality * official * (0.4 + 0.6 * decay) * (o.confidence ?? 1);
170 + const c: SourceContribution = {
171 + sourceId: o.sourceId,
172 + connectorId: o.connectorId,
173 + value: o.value,
174 + sourceTimestamp: o.sourceTimestamp,
175 + receivedAt: o.receivedAt,
176 + ageMs,
177 + weight: fresh ? round(weight, 4) : 0,
178 + included: fresh,
179 + realtimeStatus: fresh ? o.realtimeStatus : "STALE",
180 + rightsStatus: o.rightsStatus,
181 + ...(fresh ? {} : { reason: "stale" }),
182 + };
183 + contributions.push(c);
184 + if (fresh) candidates.push({ value: o.value, weight, family: prof.family ?? o.sourceId, obs: o });
185 + }
186 + if (!candidates.length) {
187 + // Fall back to the most recent stale value so the UI can show "last known", clearly labelled STALE.
188 + const newest = contributions.sort((a, b) => a.ageMs - b.ageMs)[0];
189 + return {
190 + value: newest?.value ?? null,
191 + confidence: 0,
192 + dispersionBps: null,
193 + independentFamilies: 0,
194 + freshnessMs: newest?.ageMs ?? null,
195 + realtimeStatus: "STALE" as RealtimeStatus,
196 + sourceTimestamp: newest?.sourceTimestamp ?? null,
197 + contributions,
198 + };
199 + }
200 + // Tiering: when live (REALTIME/DELAYED) observations exist, end-of-day/indicative values are superseded.
201 + const bestRank = Math.max(...candidates.map((c) => REALTIME_RANK[c.obs.realtimeStatus]));
202 + if (bestRank >= 3) {
203 + for (const c of candidates) {
204 + if (REALTIME_RANK[c.obs.realtimeStatus] <= 2) {
205 + const contrib = contributions.find((x) => x.sourceId === c.obs.sourceId && x.included);
206 + if (contrib) {
207 + contrib.included = false;
208 + contrib.reason = "superseded_by_live";
209 + contrib.weight = 0;
210 + }
211 + }
212 + }
213 + const live = candidates.filter((c) => REALTIME_RANK[c.obs.realtimeStatus] >= 3);
214 + candidates.length = 0;
215 + candidates.push(...live);
216 + }
217 + // Family de-duplication: one vote per upstream family (the freshest), others get a small weight.
218 + const seenFamily = new Map<string, number>();
219 + for (const c of candidates) {
220 + const prior = seenFamily.get(c.family);
221 + if (prior == null) seenFamily.set(c.family, c.weight);
222 + else c.weight *= 0.25;
223 + }
224 + const median = weightedMedian(candidates.map((c) => ({ v: c.value, w: c.weight })));
225 + // Outlier rejection: > 2% away from median with ≥ 3 candidates → excluded.
226 + let usable = candidates;
227 + if (candidates.length >= 3 && median !== 0) {
228 + usable = candidates.filter((c) => Math.abs(c.value - median) / Math.abs(median) <= 0.02);
229 + for (const c of candidates) {
230 + if (!usable.includes(c)) {
231 + const contrib = contributions.find((x) => x.sourceId === c.obs.sourceId && x.included);
232 + if (contrib) {
233 + contrib.included = false;
234 + contrib.reason = "outlier";
235 + contrib.weight = 0;
236 + }
237 + }
238 + }
239 + if (!usable.length) usable = candidates;
240 + }
241 + const finalMedian = weightedMedian(usable.map((c) => ({ v: c.value, w: c.weight })));
242 + const values = usable.map((c) => c.value);
243 + const spread = values.length > 1 ? (Math.max(...values) - Math.min(...values)) / Math.abs(finalMedian || 1) : 0;
244 + const dispersionBps = round(spread * 10_000, 2);
245 + const families = new Set(usable.map((c) => c.family)).size;
246 + const newest = Math.min(...usable.map((c) => Math.max(0, now - (c.obs.sourceTimestamp ?? c.obs.receivedAt))));
247 + const bestRt = usable.map((c) => c.obs.realtimeStatus).sort((a, b) => REALTIME_RANK[b] - REALTIME_RANK[a])[0] ?? "UNKNOWN";
248 + // Confidence: agreement (dispersion), redundancy (families), freshness. Never claims > 0.995.
249 + const agreement = Math.max(0, 1 - Math.min(1, spread / 0.01)); // 1% spread → 0
250 + const redundancy = 1 - Math.exp(-families / 2); // 1 fam → 0.39, 3 → 0.78, 5 → 0.92
251 + const freshness = marketClosed ? 1 : Math.exp(-newest / 60_000);
252 + const avgReliability = usable.reduce((a, c) => a + this.profiles(c.obs.sourceId).reliability, 0) / usable.length;
253 + const confidence = round(Math.min(0.995, 0.15 + 0.35 * agreement + 0.25 * redundancy + 0.1 * freshness + 0.15 * avgReliability), 3);
254 + return {
255 + value: finalMedian,
256 + confidence,
257 + dispersionBps,
258 + independentFamilies: families,
259 + freshnessMs: Math.round(newest),
260 + realtimeStatus: bestRt,
261 + sourceTimestamp: usable.map((c) => c.obs.sourceTimestamp).filter((t): t is number => t != null).sort((a, b) => b - a)[0] ?? null,
262 + contributions: contributions.sort((a, b) => Number(b.included) - Number(a.included) || a.ageMs - b.ageMs),
263 + };
264 + }
265 +}
266 +
267 +export function weightedMedian(items: Array<{ v: number; w: number }>): number {
268 + if (!items.length) return NaN;
269 + const sorted = [...items].sort((a, b) => a.v - b.v);
270 + const total = sorted.reduce((a, b) => a + Math.max(b.w, 1e-9), 0);
271 + let acc = 0;
272 + for (const it of sorted) {
273 + acc += Math.max(it.w, 1e-9);
274 + if (acc >= total / 2) return it.v;
275 + }
276 + return sorted[sorted.length - 1]!.v;
277 +}
278 +
279 +export const round = (n: number, d: number) => Math.round(n * 10 ** d) / 10 ** d;
added apps/api/src/core/events.ts +247 −0
@@ -0,0 +1,247 @@
1 +import type { CanonicalQuote, EventType, MarketEvent, Severity } from "@market-atlas/market-model";
2 +import { makeEventId } from "@market-atlas/market-model";
3 +import { shortHash } from "@market-atlas/connector-sdk";
4 +import type { ProposedEvent } from "@market-atlas/connector-sdk";
5 +import { pool } from "../db/pool.js";
6 +import { logger } from "../logger.js";
7 +import { bus } from "./bus.js";
8 +import { telemetry } from "./telemetry.js";
9 +
10 +interface InstrumentStats {
11 + lastPrice: number | null;
12 + lastEventPrice: number | null;
13 + sessionHigh: number;
14 + sessionLow: number;
15 + /** EWMA of absolute 1-tick returns for volatility baselines. */
16 + ewmaAbsRet: number;
17 + ewmaVar: number;
18 + ticks: number;
19 + lastEventAt: Record<string, number>;
20 + lastSourceDivergenceAt: number;
21 + recentReturns: number[];
22 +}
23 +
24 +/**
25 + * Event engine: turns canonical quotes into meaningful, deduplicated events and canonicalizes
26 + * connector-proposed events (filings, halts…). Thresholds are relative to per-instrument baselines,
27 + * so a 0.5% move in a treasury yield is not treated like 0.5% on a crypto pair.
28 + */
29 +export class EventEngine {
30 + private stats = new Map<string, InstrumentStats>();
31 + private recentFingerprints = new Map<string, number>();
32 + private pending: MarketEvent[] = [];
33 + private timer: NodeJS_Timeout | null = null;
34 +
35 + private stat(id: string): InstrumentStats {
36 + let s = this.stats.get(id);
37 + if (!s) {
38 + s = { lastPrice: null, lastEventPrice: null, sessionHigh: -Infinity, sessionLow: Infinity, ewmaAbsRet: 0, ewmaVar: 0, ticks: 0, lastEventAt: {}, lastSourceDivergenceAt: 0, recentReturns: [] };
39 + this.stats.set(id, s);
40 + }
41 + return s;
42 + }
43 +
44 + resetSession(instrumentId: string) {
45 + const s = this.stat(instrumentId);
46 + s.sessionHigh = -Infinity;
47 + s.sessionLow = Infinity;
48 + }
49 +
50 + onQuote(q: CanonicalQuote, name: string): void {
51 + if (q.price == null || q.realtimeStatus === "STALE") return;
52 + const s = this.stat(q.instrumentId);
53 + const now = q.updatedAt;
54 + const p = q.price;
55 + if (s.lastPrice != null && s.lastPrice !== 0) {
56 + const ret = (p - s.lastPrice) / s.lastPrice;
57 + const absRet = Math.abs(ret);
58 + const alpha = 0.05;
59 + s.ewmaAbsRet = s.ticks < 20 ? (s.ewmaAbsRet * s.ticks + absRet) / (s.ticks + 1) : s.ewmaAbsRet * (1 - alpha) + absRet * alpha;
60 + s.ewmaVar = s.ticks < 20 ? (s.ewmaVar * s.ticks + ret * ret) / (s.ticks + 1) : s.ewmaVar * (1 - alpha) + ret * ret * alpha;
61 + s.recentReturns.push(ret);
62 + if (s.recentReturns.length > 60) s.recentReturns.shift();
63 + // Volatility spike: realized volatility over the last 30 ticks vs long-run baseline (needs history).
64 + if (s.ticks > 120 && s.recentReturns.length >= 30) {
65 + const recent = Math.sqrt(s.recentReturns.slice(-30).reduce((a, r) => a + r * r, 0) / 30);
66 + const baseline = Math.sqrt(s.ewmaVar);
67 + if (baseline > 0 && recent > 4 * baseline && this.cooldown(s, "VOLATILITY_SPIKE", now, 10 * 60_000)) {
68 + this.emitDerived("VOLATILITY_SPIKE", q, name, now, "WARNING", `${q.symbol} volatility ${(recent / baseline).toFixed(1)}× baseline`, {
69 + ratio: round(recent / baseline, 2),
70 + realized: recent,
71 + baseline,
72 + });
73 + }
74 + }
75 + }
76 + // Price change events on a per-instrument adaptive threshold (≥ 0.5% and ≥ 8× typical tick move).
77 + const ref = s.lastEventPrice ?? s.lastPrice;
78 + if (ref != null && ref !== 0 && s.ticks > 5) {
79 + const move = (p - ref) / ref;
80 + const threshold = Math.max(0.005, 8 * s.ewmaAbsRet);
81 + if (Math.abs(move) >= threshold && this.cooldown(s, "PRICE_CHANGE", now, 60_000)) {
82 + this.emitDerived("PRICE_CHANGE", q, name, now, Math.abs(move) >= 0.03 ? "WARNING" : "INFO", `${q.symbol} ${move >= 0 ? "+" : ""}${(move * 100).toFixed(2)}%`, {
83 + from: ref,
84 + to: p,
85 + changePercent: round(move * 100, 3),
86 + });
87 + s.lastEventPrice = p;
88 + }
89 + }
90 + if (s.lastEventPrice == null) s.lastEventPrice = p;
91 + // Session highs/lows: only after meaningful history, when the previous extreme is beaten by ≥ 0.1 %, with a 15-minute cooldown.
92 + if (s.ticks > 200) {
93 + if (p > s.sessionHigh * 1.001 && Number.isFinite(s.sessionHigh) && this.cooldown(s, "SESSION_HIGH", now, 15 * 60_000)) {
94 + this.emitDerived("SESSION_HIGH", q, name, now, "INFO", `${q.symbol} new session high ${fmt(p)}`, { price: p, previousHigh: s.sessionHigh });
95 + }
96 + if (p < s.sessionLow * 0.999 && Number.isFinite(s.sessionLow) && this.cooldown(s, "SESSION_LOW", now, 15 * 60_000)) {
97 + this.emitDerived("SESSION_LOW", q, name, now, "INFO", `${q.symbol} new session low ${fmt(p)}`, { price: p, previousLow: s.sessionLow });
98 + }
99 + }
100 + if (p > s.sessionHigh) s.sessionHigh = p;
101 + if (p < s.sessionLow) s.sessionLow = p;
102 + // Source divergence: included sources disagree by more than 50 bps (crypto venues legitimately differ; flag ≥ 50).
103 + if (q.dispersionBps != null && q.dispersionBps > 50 && q.sourceCount >= 2 && now - s.lastSourceDivergenceAt > 10 * 60_000) {
104 + s.lastSourceDivergenceAt = now;
105 + this.emitDerived("SOURCE_DIVERGENCE", q, name, now, "NOTICE", `${q.symbol}: sources diverge by ${q.dispersionBps.toFixed(0)} bps`, {
106 + dispersionBps: q.dispersionBps,
107 + contributions: q.contributions.filter((c) => c.included).map((c) => ({ source: c.sourceId, value: c.value, ageMs: c.ageMs })),
108 + });
109 + }
110 + s.lastPrice = p;
111 + s.ticks++;
112 + }
113 +
114 + private cooldown(s: InstrumentStats, type: string, now: number, ms: number): boolean {
115 + const last = s.lastEventAt[type] ?? 0;
116 + if (now - last < ms) return false;
117 + s.lastEventAt[type] = now;
118 + return true;
119 + }
120 +
121 + private emitDerived(type: EventType, q: CanonicalQuote, name: string, ts: number, severity: Severity, title: string, data: Record<string, unknown>) {
122 + const minute = Math.floor(ts / 60_000);
123 + const fingerprint = shortHash(`${type}|${q.instrumentId}|${minute}|${JSON.stringify(data)}`, 24);
124 + this.publish({
125 + version: 1,
126 + id: makeEventId(ts, fingerprint),
127 + type,
128 + instrumentIds: [q.instrumentId],
129 + entityIds: [],
130 + timestamp: ts,
131 + severity,
132 + confidence: q.confidence,
133 + sourceCount: q.sourceCount,
134 + sources: q.contributions.filter((c) => c.included).map((c) => c.sourceId),
135 + title,
136 + summary: `${name} · ${q.sourceCount} independent source${q.sourceCount === 1 ? "" : "s"} · confidence ${(q.confidence * 100).toFixed(1)}%`,
137 + data: { ...data, symbol: q.symbol, price: q.price, currency: q.currency },
138 + fingerprint,
139 + supportingObservations: [],
140 + });
141 + }
142 +
143 + /** Canonicalize an event proposed by a connector or the runtime (dedupe on dedupeKey within a 24 h window). */
144 + proposed(e: ProposedEvent, sourceId: string, instrumentIds: string[]): MarketEvent | null {
145 + const fingerprint = shortHash(`${e.type}|${e.dedupeKey}`, 24);
146 + const ev: MarketEvent = {
147 + version: 1,
148 + id: makeEventId(e.timestamp, fingerprint),
149 + type: e.type,
150 + instrumentIds,
151 + entityIds: e.entityIds ?? [],
152 + timestamp: e.timestamp,
153 + severity: e.severity ?? "INFO",
154 + confidence: e.confidence ?? 0.9,
155 + sourceCount: 1,
156 + sources: [sourceId],
157 + title: e.title,
158 + summary: e.summary ?? null,
159 + data: e.data ?? {},
160 + fingerprint,
161 + supportingObservations: [],
162 + };
163 + return this.publish(ev) ? ev : null;
164 + }
165 +
166 + /** System events (connector failure/recovery, schema drift, market open/close). */
167 + system(type: EventType, key: string, ts: number, severity: Severity, title: string, data: Record<string, unknown>, sources: string[] = [], instrumentIds: string[] = []) {
168 + const fingerprint = shortHash(`${type}|${key}`, 24);
169 + this.publish({
170 + version: 1,
171 + id: makeEventId(ts, fingerprint),
172 + type,
173 + instrumentIds,
174 + entityIds: [],
175 + timestamp: ts,
176 + severity,
177 + confidence: 1,
178 + sourceCount: sources.length,
179 + sources,
180 + title,
181 + summary: null,
182 + data,
183 + fingerprint,
184 + supportingObservations: [],
185 + });
186 + }
187 +
188 + private publish(ev: MarketEvent): boolean {
189 + const now = Date.now();
190 + const prev = this.recentFingerprints.get(ev.fingerprint);
191 + if (prev && now - prev < 86_400_000) {
192 + // Another source confirms an existing event → bump confirmation in DB, no new feed item.
193 + this.confirm(ev).catch(() => {});
194 + return false;
195 + }
196 + this.recentFingerprints.set(ev.fingerprint, now);
197 + if (this.recentFingerprints.size > 200_000) for (const [k, t] of this.recentFingerprints) if (now - t > 86_400_000) this.recentFingerprints.delete(k);
198 + this.pending.push(ev);
199 + bus.publish("market.event", ev);
200 + telemetry.inc("events_total", 1, { type: ev.type });
201 + if (!this.timer) this.timer = setTimeout(() => void this.flush(), 500);
202 + return true;
203 + }
204 +
205 + private async confirm(ev: MarketEvent) {
206 + await pool.query(
207 + `update market_events set source_count = source_count + 1, sources = array(select distinct unnest(sources || $2::text[])), confirmed_at = coalesce(confirmed_at, now()) where fingerprint = $1`,
208 + [ev.fingerprint, ev.sources],
209 + );
210 + }
211 +
212 + async flush(): Promise<void> {
213 + this.timer = null;
214 + const batch = this.pending.splice(0);
215 + if (!batch.length) return;
216 + const cols = 14;
217 + const tuples: string[] = [];
218 + const values: unknown[] = [];
219 + batch.forEach((e, i) => {
220 + const b = i * cols;
221 + tuples.push(`(${Array.from({ length: cols }, (_, j) => `$${b + j + 1}`).join(",")})`);
222 + values.push(e.id, e.type, new Date(e.timestamp).toISOString(), e.instrumentIds, e.entityIds, e.severity, e.confidence, e.sourceCount, e.sources, e.title, e.summary, JSON.stringify(e.data), e.fingerprint, e.supportingObservations);
223 + });
224 + try {
225 + await pool.query(
226 + `insert into market_events (id, type, ts, instrument_ids, entity_ids, severity, confidence, source_count, sources, title, summary, data, fingerprint, supporting_observations)
227 + values ${tuples.join(",")} on conflict (fingerprint) do update set source_count = market_events.source_count + 1, confirmed_at = coalesce(market_events.confirmed_at, now())`,
228 + values,
229 + );
230 + await pool.query(`insert into daily_stats (day, events) values (current_date, $1) on conflict (day) do update set events = daily_stats.events + excluded.events, updated_at = now()`, [batch.length]);
231 + } catch (err) {
232 + logger.error({ err, size: batch.length }, "event insert failed");
233 + }
234 + }
235 +
236 + /** Warm the dedupe cache from recent DB events so restarts do not re-emit filings/halts. */
237 + async warm(): Promise<void> {
238 + const r = await pool.query<{ fingerprint: string; ts: string }>("select fingerprint, ts from market_events where ts > now() - interval '30 hours'");
239 + for (const row of r.rows) this.recentFingerprints.set(row.fingerprint, Date.parse(row.ts));
240 + }
241 +}
242 +
243 +type NodeJS_Timeout = ReturnType<typeof setTimeout>;
244 +const round = (n: number, d: number) => Math.round(n * 10 ** d) / 10 ** d;
245 +const fmt = (n: number) => (Math.abs(n) >= 1000 ? n.toLocaleString("en-US", { maximumFractionDigits: 2 }) : n.toPrecision(6).replace(/\.?0+$/, ""));
246 +
247 +export const eventEngine = new EventEngine();
added apps/api/src/core/health.ts +233 −0
@@ -0,0 +1,233 @@
1 +import type { ConnectorHealth, ConnectorState } from "@market-atlas/market-model";
2 +import { pool } from "../db/pool.js";
3 +import { logger } from "../logger.js";
4 +import { bus } from "./bus.js";
5 +import { Reservoir, telemetry } from "./telemetry.js";
6 +
7 +interface Tracker {
8 + connectorId: string;
9 + sourceId: string;
10 + state: ConnectorState;
11 + startedAt: number | null;
12 + lastMessageAt: number | null;
13 + lastSuccessAt: number | null;
14 + lastErrorAt: number | null;
15 + lastError: string | null;
16 + messagesTotal: number;
17 + errorsTotal: number;
18 + parseOk: number;
19 + parseFail: number;
20 + reconnects: number;
21 + latency: Reservoir;
22 + instruments: Set<string>;
23 + schemaFingerprints: Map<string, Set<string>>;
24 + nextPollAt: number | null;
25 + /** Rolling availability samples (1 per minute, 1 = healthy). */
26 + availability: number[];
27 +}
28 +
29 +/**
30 + * Operational health and reliability scoring per connector/source. The score measures the
31 + * connector's operational behaviour (availability, latency, parse success, stability) — never
32 + * financial truth.
33 + */
34 +export class HealthEngine {
35 + private trackers = new Map<string, Tracker>();
36 + private sourceOfConnector = new Map<string, string>();
37 +
38 + register(connectorId: string, sourceId: string): void {
39 + this.sourceOfConnector.set(connectorId, sourceId);
40 + if (!this.trackers.has(connectorId)) {
41 + this.trackers.set(connectorId, {
42 + connectorId,
43 + sourceId,
44 + state: "STARTING",
45 + startedAt: null,
46 + lastMessageAt: null,
47 + lastSuccessAt: null,
48 + lastErrorAt: null,
49 + lastError: null,
50 + messagesTotal: 0,
51 + errorsTotal: 0,
52 + parseOk: 0,
53 + parseFail: 0,
54 + reconnects: 0,
55 + latency: new Reservoir(256),
56 + instruments: new Set(),
57 + schemaFingerprints: new Map(),
58 + nextPollAt: null,
59 + availability: [],
60 + });
61 + }
62 + }
63 +
64 + private t(id: string): Tracker {
65 + const t = this.trackers.get(id);
66 + if (!t) throw new Error(`unknown connector ${id}`);
67 + return t;
68 + }
69 +
70 + setState(id: string, state: ConnectorState) {
71 + const t = this.t(id);
72 + if (t.state === state) return;
73 + const prev = t.state;
74 + t.state = state;
75 + if (state === "HEALTHY" && !t.startedAt) t.startedAt = Date.now();
76 + telemetry.gauge("connector_state", stateNum(state), { connector: id });
77 + bus.publish("connector.health", this.snapshot(id));
78 + logger.info({ connector: id, from: prev, to: state }, "connector state");
79 + }
80 +
81 + state(id: string): ConnectorState {
82 + return this.trackers.get(id)?.state ?? "DISABLED";
83 + }
84 +
85 + message(id: string, latencyMs: number | null, instrumentIds: string[] = []) {
86 + const t = this.t(id);
87 + t.messagesTotal++;
88 + t.lastMessageAt = Date.now();
89 + t.lastSuccessAt = t.lastMessageAt;
90 + if (latencyMs != null && latencyMs >= 0 && latencyMs < 3_600_000) t.latency.add(latencyMs);
91 + for (const i of instrumentIds) t.instruments.add(i);
92 + telemetry.inc("connector_messages_total", 1, { connector: id });
93 + telemetry.inc("observations_received_total");
94 + }
95 +
96 + parse(id: string, ok: boolean) {
97 + const t = this.t(id);
98 + if (ok) t.parseOk++;
99 + else t.parseFail++;
100 + }
101 +
102 + error(id: string, err: unknown) {
103 + const t = this.t(id);
104 + t.errorsTotal++;
105 + t.lastErrorAt = Date.now();
106 + t.lastError = (err instanceof Error ? err.message : String(err)).slice(0, 500);
107 + telemetry.inc("connector_errors_total", 1, { connector: id });
108 + }
109 +
110 + reconnect(id: string) {
111 + this.t(id).reconnects++;
112 + }
113 +
114 + poll(id: string, nextAt: number | null) {
115 + this.t(id).nextPollAt = nextAt;
116 + }
117 +
118 + recordFingerprint(id: string, kind: string, fp: string): "known" | "new" | "first" {
119 + const t = this.t(id);
120 + let set = t.schemaFingerprints.get(kind);
121 + if (!set) {
122 + set = new Set();
123 + t.schemaFingerprints.set(kind, set);
124 + }
125 + if (set.has(fp)) return "known";
126 + const first = set.size === 0;
127 + set.add(fp);
128 + return first ? "first" : "new";
129 + }
130 +
131 + loadFingerprints(id: string, fps: Record<string, string[]>) {
132 + const t = this.t(id);
133 + for (const [kind, arr] of Object.entries(fps)) t.schemaFingerprints.set(kind, new Set(arr));
134 + }
135 +
136 + fingerprints(id: string): Record<string, string[]> {
137 + const t = this.t(id);
138 + return Object.fromEntries([...t.schemaFingerprints].map(([k, v]) => [k, [...v]]));
139 + }
140 +
141 + /** 0–100 operational reliability. */
142 + score(id: string): number | null {
143 + const t = this.trackers.get(id);
144 + if (!t || t.messagesTotal === 0) return null;
145 + const avail = t.availability.length ? t.availability.reduce((a, b) => a + b, 0) / t.availability.length : t.state === "HEALTHY" ? 1 : 0.5;
146 + const parseTotal = t.parseOk + t.parseFail;
147 + const parse = parseTotal ? t.parseOk / parseTotal : 1;
148 + const p95 = t.latency.quantile(0.95);
149 + const latencyScore = p95 == null ? 0.8 : p95 < 1000 ? 1 : p95 < 5000 ? 0.85 : p95 < 60_000 ? 0.7 : 0.5;
150 + const stability = Math.max(0.4, 1 - t.reconnects / Math.max(50, t.messagesTotal / 100));
151 + const errorRate = t.errorsTotal / Math.max(1, t.messagesTotal + t.errorsTotal);
152 + const errScore = Math.max(0, 1 - errorRate * 10);
153 + return Math.round(100 * (0.4 * avail + 0.2 * parse + 0.15 * latencyScore + 0.15 * stability + 0.1 * errScore));
154 + }
155 +
156 + /** Reliability 0..1 for the consensus engine, keyed by source id (best connector of the source). */
157 + sourceReliability(sourceId: string): number {
158 + const scores: number[] = [];
159 + for (const t of this.trackers.values()) {
160 + if (t.sourceId !== sourceId) continue;
161 + const s = this.score(t.connectorId);
162 + if (s != null) scores.push(s / 100);
163 + }
164 + return scores.length ? Math.max(...scores) : 0.6;
165 + }
166 +
167 + snapshot(id: string): ConnectorHealth {
168 + const t = this.t(id);
169 + const parseTotal = t.parseOk + t.parseFail;
170 + return {
171 + connectorId: id,
172 + state: t.state,
173 + lastMessageAt: t.lastMessageAt,
174 + lastSuccessAt: t.lastSuccessAt,
175 + lastErrorAt: t.lastErrorAt,
176 + lastError: t.lastError,
177 + messages1m: telemetry.lastMinute("connector_messages_total", { connector: id }),
178 + messagesTotal: t.messagesTotal,
179 + errorsTotal: t.errorsTotal,
180 + reconnects: t.reconnects,
181 + medianLatencyMs: t.latency.quantile(0.5),
182 + p95LatencyMs: t.latency.quantile(0.95),
183 + parseSuccessRate: parseTotal ? t.parseOk / parseTotal : null,
184 + instrumentsCovered: t.instruments.size,
185 + schemaFingerprints: [...t.schemaFingerprints.values()].flatMap((s) => [...s]),
186 + reliabilityScore: this.score(id),
187 + startedAt: t.startedAt,
188 + nextPollAt: t.nextPollAt,
189 + };
190 + }
191 +
192 + all(): ConnectorHealth[] {
193 + return [...this.trackers.keys()].map((id) => this.snapshot(id));
194 + }
195 +
196 + /** Called every minute: staleness detection, availability sample, DB snapshot. */
197 + async tick(staleAfterMs: (id: string) => number): Promise<void> {
198 + const now = Date.now();
199 + const rows: unknown[][] = [];
200 + for (const t of this.trackers.values()) {
201 + if (t.state === "HEALTHY" && t.lastMessageAt && now - t.lastMessageAt > staleAfterMs(t.connectorId)) this.setState(t.connectorId, "STALE");
202 + const healthy = t.state === "HEALTHY" ? 1 : t.state === "DEGRADED" || t.state === "STALE" ? 0.5 : t.state === "PAUSED" || t.state === "DISABLED" ? NaN : 0;
203 + if (!Number.isNaN(healthy)) {
204 + t.availability.push(healthy);
205 + if (t.availability.length > 24 * 60) t.availability.shift();
206 + }
207 + const s = this.snapshot(t.connectorId);
208 + rows.push([new Date(now).toISOString(), t.connectorId, t.state, s.messages1m, telemetry.lastMinute("connector_errors_total", { connector: t.connectorId }), s.medianLatencyMs, s.p95LatencyMs, s.parseSuccessRate, s.instrumentsCovered, t.reconnects, s.reliabilityScore]);
209 + }
210 + if (!rows.length) return;
211 + const cols = 11;
212 + const tuples = rows.map((_, i) => `(${Array.from({ length: cols }, (_, j) => `$${i * cols + j + 1}`).join(",")})`);
213 + try {
214 + await pool.query(
215 + `insert into connector_health (ts, connector_id, status, messages_1m, errors_1m, median_latency_ms, p95_latency_ms, parse_success_rate, instruments_covered, reconnects, reliability_score) values ${tuples.join(",")} on conflict do nothing`,
216 + rows.flat(),
217 + );
218 + for (const t of this.trackers.values()) {
219 + await pool.query(
220 + `update connectors set status = $2, last_message_at = $3, last_success_at = $4, last_error_at = $5, last_error = $6, messages_total = $7, errors_total = $8, reconnects = $9, reliability_score = $10, schema_fingerprints = $11, updated_at = now() where id = $1`,
221 + [t.connectorId, t.state, iso(t.lastMessageAt), iso(t.lastSuccessAt), iso(t.lastErrorAt), t.lastError, t.messagesTotal, t.errorsTotal, t.reconnects, this.score(t.connectorId), JSON.stringify(this.fingerprints(t.connectorId))],
222 + );
223 + }
224 + } catch (err) {
225 + logger.error({ err }, "health snapshot failed");
226 + }
227 + }
228 +}
229 +
230 +const iso = (ms: number | null) => (ms == null ? null : new Date(ms).toISOString());
231 +const stateNum = (s: ConnectorState) => ({ HEALTHY: 1, DEGRADED: 2, STALE: 3, RECONNECTING: 4, FAILED: 5, PAUSED: 6, DISABLED: 7, STARTING: 0 })[s];
232 +
233 +export const health = new HealthEngine();
added apps/api/src/core/instruments.ts +242 −0
@@ -0,0 +1,242 @@
1 +import type { Company, Exchange, Instrument, InstrumentHint, SymbolAlias } from "@market-atlas/market-model";
2 +import { canonicalSymbol, makeCompanyId, makeInstrumentId } from "@market-atlas/market-model";
3 +import { pool, query, type Queryable } from "../db/pool.js";
4 +import { logger } from "../logger.js";
5 +import { telemetry } from "./telemetry.js";
6 +
7 +interface Row {
8 + id: string;
9 + symbol: string;
10 + name: string;
11 + asset_class: Instrument["assetClass"];
12 + exchange_id: string | null;
13 + mic: string | null;
14 + currency: string | null;
15 + country: string | null;
16 + company_id: string | null;
17 + isin: string | null;
18 + security_type: string | null;
19 + base: string | null;
20 + quote: string | null;
21 + is_active: boolean;
22 + metadata: Record<string, unknown>;
23 +}
24 +
25 +export const rowToInstrument = (r: Row): Instrument => ({
26 + id: r.id,
27 + symbol: r.symbol,
28 + name: r.name,
29 + assetClass: r.asset_class,
30 + exchangeId: r.exchange_id,
31 + mic: r.mic,
32 + currency: r.currency,
33 + country: r.country,
34 + companyId: r.company_id,
35 + isin: r.isin,
36 + securityType: r.security_type,
37 + base: r.base,
38 + quote: r.quote,
39 + isActive: r.is_active,
40 + metadata: r.metadata ?? {},
41 +});
42 +
43 +/**
44 + * Instrument master + resolver. Keeps an in-memory index (id, alias per source, canonical symbol)
45 + * so that the hot path never queries Postgres per tick.
46 + */
47 +export class InstrumentStore {
48 + private byId = new Map<string, Instrument>();
49 + private byAlias = new Map<string, string>(); // `${sourceId}|${canonicalAlias}` or `*|alias` -> id
50 + private bySymbol = new Map<string, string[]>(); // canonical symbol -> ids
51 + private exchangesById = new Map<string, Exchange>();
52 + private creating = new Map<string, Promise<Instrument>>();
53 +
54 + async load(): Promise<void> {
55 + const [inst, aliases, ex] = await Promise.all([
56 + query<Row>("select * from instruments"),
57 + query<{ alias: string; source_id: string; instrument_id: string }>("select alias, source_id, instrument_id from symbol_aliases"),
58 + query<any>("select * from exchanges"),
59 + ]);
60 + this.byId.clear();
61 + this.byAlias.clear();
62 + this.bySymbol.clear();
63 + for (const r of inst.rows) this.index(rowToInstrument(r));
64 + for (const a of aliases.rows) this.byAlias.set(`${a.source_id}|${canonicalSymbol(a.alias)}`, a.instrument_id);
65 + this.exchangesById.clear();
66 + for (const e of ex.rows) this.exchangesById.set(e.id, rowToExchange(e));
67 + telemetry.gauge("instruments_total", this.byId.size);
68 + logger.info({ instruments: this.byId.size, aliases: this.byAlias.size, exchanges: this.exchangesById.size }, "instrument master loaded");
69 + }
70 +
71 + private index(i: Instrument) {
72 + this.byId.set(i.id, i);
73 + const cs = canonicalSymbol(i.symbol);
74 + const arr = this.bySymbol.get(cs) ?? [];
75 + if (!arr.includes(i.id)) arr.push(i.id);
76 + this.bySymbol.set(cs, arr);
77 + this.byAlias.set(`*|${cs}`, i.id);
78 + }
79 +
80 + get(id: string): Instrument | undefined {
81 + return this.byId.get(id);
82 + }
83 +
84 + all(): Instrument[] {
85 + return [...this.byId.values()];
86 + }
87 +
88 + count(): number {
89 + return this.byId.size;
90 + }
91 +
92 + exchange(id: string | null | undefined): Exchange | undefined {
93 + return id ? this.exchangesById.get(id) : undefined;
94 + }
95 +
96 + exchanges(): Exchange[] {
97 + return [...this.exchangesById.values()];
98 + }
99 +
100 + /** Resolve a source-native symbol. Source-specific aliases win over global ones. */
101 + resolve(symbol: string, sourceId: string): Instrument | undefined {
102 + const cs = canonicalSymbol(symbol);
103 + const id = this.byAlias.get(`${sourceId}|${cs}`) ?? this.byAlias.get(`*|${cs}`);
104 + return id ? this.byId.get(id) : undefined;
105 + }
106 +
107 + /** Ids sharing a canonical symbol (e.g. AAPL on several venues). */
108 + bySymbolCandidates(symbol: string): Instrument[] {
109 + return (this.bySymbol.get(canonicalSymbol(symbol)) ?? []).map((id) => this.byId.get(id)!).filter(Boolean);
110 + }
111 +
112 + /** Resolve, or create from a hint (deterministic id). Concurrent creations of the same id are coalesced. */
113 + async resolveOrCreate(symbol: string, sourceId: string, hint: InstrumentHint | undefined): Promise<Instrument | undefined> {
114 + const found = this.resolve(symbol, sourceId);
115 + if (found) return found;
116 + if (!hint) return undefined;
117 + const id = makeInstrumentId(hint.assetClass, { symbol, ...hint });
118 + const existing = this.byId.get(id);
119 + if (existing) {
120 + await this.addAlias(symbol, sourceId, id);
121 + return existing;
122 + }
123 + let p = this.creating.get(id);
124 + if (!p) {
125 + p = this.create(id, symbol, sourceId, hint).finally(() => this.creating.delete(id));
126 + this.creating.set(id, p);
127 + }
128 + return p;
129 + }
130 +
131 + private async create(id: string, symbol: string, sourceId: string, hint: InstrumentHint): Promise<Instrument> {
132 + let companyId: string | null = null;
133 + if (hint.companyName) companyId = await this.upsertCompany({ name: hint.companyName, cik: hint.cik ?? null, country: hint.country ?? null });
134 + const inst: Instrument = {
135 + id,
136 + symbol: symbol.replace(/^[_^]/, ""),
137 + name: hint.name ?? symbol,
138 + assetClass: hint.assetClass,
139 + exchangeId: hint.exchangeId ?? null,
140 + mic: hint.mic ?? null,
141 + currency: hint.currency ?? null,
142 + country: hint.country ?? null,
143 + companyId,
144 + isin: null,
145 + securityType: hint.securityType ?? null,
146 + base: hint.base ?? null,
147 + quote: hint.quote ?? null,
148 + isActive: true,
149 + metadata: hint.metadata ?? {},
150 + };
151 + await this.upsert(inst);
152 + await this.addAlias(symbol, sourceId, id);
153 + telemetry.inc("instruments_created_total");
154 + return inst;
155 + }
156 +
157 + async upsert(i: Instrument, client: Queryable = pool): Promise<void> {
158 + await query(
159 + `insert into instruments (id, symbol, name, asset_class, exchange_id, mic, currency, country, company_id, isin, security_type, base, quote, is_active, metadata)
160 + values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15)
161 + on conflict (id) do update set
162 + name = case when excluded.name <> excluded.symbol then excluded.name else instruments.name end,
163 + exchange_id = coalesce(excluded.exchange_id, instruments.exchange_id),
164 + mic = coalesce(excluded.mic, instruments.mic),
165 + currency = coalesce(excluded.currency, instruments.currency),
166 + country = coalesce(excluded.country, instruments.country),
167 + company_id = coalesce(excluded.company_id, instruments.company_id),
168 + isin = coalesce(excluded.isin, instruments.isin),
169 + security_type = coalesce(excluded.security_type, instruments.security_type),
170 + is_active = excluded.is_active,
171 + metadata = instruments.metadata || excluded.metadata,
172 + updated_at = now()`,
173 + [i.id, i.symbol, i.name, i.assetClass, i.exchangeId, i.mic, i.currency, i.country, i.companyId, i.isin, i.securityType, i.base, i.quote, i.isActive, JSON.stringify(i.metadata ?? {})],
174 + client,
175 + );
176 + const merged = { ...(this.byId.get(i.id) ?? i), ...stripNulls(i) } as Instrument;
177 + this.index(merged);
178 + telemetry.gauge("instruments_total", this.byId.size);
179 + }
180 +
181 + async addAlias(alias: string, sourceId: string, instrumentId: string): Promise<void> {
182 + const key = `${sourceId}|${canonicalSymbol(alias)}`;
183 + if (this.byAlias.get(key) === instrumentId) return;
184 + this.byAlias.set(key, instrumentId);
185 + await query(
186 + `insert into symbol_aliases (alias, source_id, instrument_id) values ($1,$2,$3)
187 + on conflict (alias, source_id) do update set instrument_id = excluded.instrument_id`,
188 + [alias, sourceId, instrumentId],
189 + );
190 + }
191 +
192 + aliasesOf(instrumentId: string): SymbolAlias[] {
193 + const out: SymbolAlias[] = [];
194 + for (const [k, v] of this.byAlias) {
195 + if (v !== instrumentId) continue;
196 + const [sourceId, alias] = k.split("|");
197 + out.push({ alias: alias!, sourceId: sourceId === "*" ? null : sourceId!, instrumentId });
198 + }
199 + return out;
200 + }
201 +
202 + async upsertCompany(c: { name: string; cik: string | null; country: string | null; sector?: string | null; website?: string | null }): Promise<string> {
203 + const id = makeCompanyId(c.name);
204 + await query(
205 + `insert into companies (id, name, cik, country, sector, website) values ($1,$2,$3,$4,$5,$6)
206 + on conflict (id) do update set cik = coalesce(excluded.cik, companies.cik), country = coalesce(excluded.country, companies.country),
207 + sector = coalesce(excluded.sector, companies.sector), website = coalesce(excluded.website, companies.website), updated_at = now()`,
208 + [id, c.name, c.cik, c.country, c.sector ?? null, c.website ?? null],
209 + );
210 + return id;
211 + }
212 +
213 + async getCompany(id: string): Promise<Company | null> {
214 + const r = await query<any>("select * from companies where id = $1", [id]);
215 + const c = r.rows[0];
216 + return c ? { id: c.id, name: c.name, cik: c.cik, country: c.country, sector: c.sector, industry: c.industry, website: c.website } : null;
217 + }
218 +}
219 +
220 +function stripNulls<T extends object>(o: T): Partial<T> {
221 + return Object.fromEntries(Object.entries(o).filter(([, v]) => v !== null && v !== undefined)) as Partial<T>;
222 +}
223 +
224 +export function rowToExchange(e: any): Exchange {
225 + return {
226 + id: e.id,
227 + mic: e.mic,
228 + name: e.name,
229 + operator: e.operator,
230 + country: e.country,
231 + city: e.city,
232 + timezone: e.timezone,
233 + currency: e.currency,
234 + website: e.website,
235 + sessions: e.sessions,
236 + lat: e.lat,
237 + lon: e.lon,
238 + assetClasses: e.asset_classes ?? [],
239 + };
240 +}
241 +
242 +export const instruments = new InstrumentStore();
added apps/api/src/core/jobs.ts +155 −0
@@ -0,0 +1,155 @@
1 +import { spawn } from "node:child_process";
2 +import { createWriteStream, mkdirSync } from "node:fs";
3 +import { stat } from "node:fs/promises";
4 +import { join } from "node:path";
5 +import { createGzip } from "node:zlib";
6 +import { pipeline as streamPipeline } from "node:stream/promises";
7 +import { Readable } from "node:stream";
8 +import { config } from "../config.js";
9 +import { pool } from "../db/pool.js";
10 +import { logger } from "../logger.js";
11 +import { rollupBars } from "./bars.js";
12 +import { bus } from "./bus.js";
13 +import { calendar } from "./calendar.js";
14 +import { eventEngine } from "./events.js";
15 +import { health } from "./health.js";
16 +import { instruments } from "./instruments.js";
17 +import { pipeline } from "./pipeline.js";
18 +import { quoteStore } from "./quotes.js";
19 +import { telemetry } from "./telemetry.js";
20 +import { connectorManager } from "./connector-manager.js";
21 +
22 +/** Periodic maintenance: health snapshots, market open/close events, rollups, retention/archive, backups. */
23 +export class Scheduler {
24 + private timers: NodeJS.Timeout[] = [];
25 + private lastState = new Map<string, string>();
26 + private lastDaily: string | null = null;
27 +
28 + start(): void {
29 + this.every(60_000, () => health.tick((id) => staleAfter(id)));
30 + this.every(5_000, () => this.marketStates());
31 + this.every(10_000, () => pipeline.sweep());
32 + this.every(5 * 60_000, () => rollupBars(3));
33 + this.every(60_000, () => this.gauges());
34 + this.every(15 * 60_000, () => this.daily());
35 + this.marketStates();
36 + this.gauges();
37 + }
38 +
39 + stop(): void {
40 + for (const t of this.timers) clearInterval(t);
41 + this.timers = [];
42 + }
43 +
44 + private every(ms: number, fn: () => unknown) {
45 + const t = setInterval(() => {
46 + Promise.resolve()
47 + .then(fn)
48 + .catch((err) => logger.error({ err }, "scheduled job failed"));
49 + }, ms);
50 + this.timers.push(t);
51 + }
52 +
53 + private gauges() {
54 + telemetry.gauge("quotes_cached", quoteStore.size());
55 + telemetry.gauge("instruments_total", instruments.count());
56 + telemetry.gauge("connectors_healthy", connectorManager.list().filter((r) => health.state(r.def.metadata.id) === "HEALTHY").length);
57 + telemetry.gauge("connectors_total", connectorManager.list().length);
58 + }
59 +
60 + /** Emit MARKET_OPEN / MARKET_CLOSE events and reset session stats at session start. */
61 + private marketStates() {
62 + const now = Date.now();
63 + for (const ex of calendar.list()) {
64 + if (ex.sessions.continuous) continue;
65 + const st = calendar.status(ex.id, now);
66 + const prev = this.lastState.get(ex.id);
67 + this.lastState.set(ex.id, st.state);
68 + if (prev === undefined || prev === st.state) continue;
69 + bus.publish("market.state", { exchangeId: ex.id, state: st.state, at: now });
70 + if (st.state === "OPEN") {
71 + eventEngine.system("MARKET_OPEN", `${ex.id}:${new Date(now).toISOString().slice(0, 10)}`, now, "NOTICE", `${ex.name} market open`, { exchangeId: ex.id, localTime: st.localTime }, ["market-atlas-calendar"]);
72 + for (const inst of instruments.all()) {
73 + if (inst.exchangeId === ex.id) {
74 + pipeline.consensus.resetSession(inst.id, now);
75 + eventEngine.resetSession(inst.id);
76 + }
77 + }
78 + } else if (st.state === "CLOSED" && (prev === "OPEN" || prev === "POST")) {
79 + eventEngine.system("MARKET_CLOSE", `${ex.id}:${new Date(now).toISOString().slice(0, 10)}`, now, "NOTICE", `${ex.name} market close`, { exchangeId: ex.id, localTime: st.localTime }, ["market-atlas-calendar"]);
80 + }
81 + }
82 + }
83 +
84 + /** Once per day (UTC 04:40-ish): archive old observation partitions, backup metadata. */
85 + private async daily() {
86 + const now = new Date();
87 + const key = now.toISOString().slice(0, 10);
88 + if (this.lastDaily === key) return;
89 + if (now.getUTCHours() < 8) return; // ~04:00 America/Toronto
90 + this.lastDaily = key;
91 + await archiveOldPartitions().catch((err) => logger.error({ err }, "archive failed"));
92 + await backupMetadata().catch((err) => logger.error({ err }, "backup failed"));
93 + await pool.query("delete from connector_health where ts < now() - interval '90 days'");
94 + }
95 +}
96 +
97 +function staleAfter(connectorId: string): number {
98 + const rt = connectorManager.get(connectorId);
99 + const m = rt?.def.metadata;
100 + if (!m) return 10 * 60_000;
101 + if (m.sourceType === "WEBSOCKET") return 5 * 60_000;
102 + const s = rt?.def.schedule;
103 + if (s && "intervalMs" in s) return s.intervalMs * 3 + 60_000;
104 + if (s) return s.weekendMs * 2 + 60_000;
105 + return 24 * 3_600_000;
106 +}
107 +
108 +/** Export observation partitions older than the retention window to gzip NDJSON, then drop them. */
109 +export async function archiveOldPartitions(): Promise<string[]> {
110 + const cutoff = new Date(Date.now() - config.observationRetentionDays * 86_400_000).toISOString().slice(0, 10).replace(/-/g, "");
111 + const parts = await pool.query<{ relname: string }>(
112 + `select c.relname from pg_inherits i join pg_class c on c.oid = i.inhrelid join pg_class p on p.oid = i.inhparent where p.relname = 'observations' order by 1`,
113 + );
114 + const archived: string[] = [];
115 + for (const { relname } of parts.rows) {
116 + const day = relname.replace("observations_", "");
117 + if (day >= cutoff) continue;
118 + const dir = join(config.dataDir, "archive", day.slice(0, 4));
119 + mkdirSync(dir, { recursive: true });
120 + const path = join(dir, `${relname}.ndjson.gz`);
121 + const rows = await pool.query(`select * from ${relname}`);
122 + await streamPipeline(Readable.from(rows.rows.map((r) => JSON.stringify(r) + "\n")), createGzip({ level: 9 }), createWriteStream(path));
123 + const { size } = await stat(path);
124 + await pool.query(`insert into observation_archives (partition_name, day, rows_archived, path, bytes) values ($1,$2,$3,$4,$5) on conflict (partition_name) do update set rows_archived = excluded.rows_archived, bytes = excluded.bytes, archived_at = now()`, [
125 + relname,
126 + `${day.slice(0, 4)}-${day.slice(4, 6)}-${day.slice(6, 8)}`,
127 + rows.rowCount ?? 0,
128 + path,
129 + size,
130 + ]);
131 + await pool.query(`drop table ${relname}`);
132 + archived.push(relname);
133 + logger.info({ partition: relname, rows: rows.rowCount, bytes: size }, "partition archived");
134 + }
135 + return archived;
136 +}
137 +
138 +/** pg_dump of metadata tables (schema + data, excluding observations/bars) into data/backups. */
139 +export function backupMetadata(): Promise<string> {
140 + return new Promise((resolve, reject) => {
141 + const dir = join(config.dataDir, "backups");
142 + mkdirSync(dir, { recursive: true });
143 + const file = join(dir, `market-atlas-meta-${new Date().toISOString().slice(0, 10)}.sql.gz`);
144 + const pgDump = process.env.PG_DUMP ?? "pg_dump";
145 + const child = spawn(pgDump, ["--no-owner", "--exclude-table-data=observations*", "--exclude-table-data=bars", "--exclude-table-data=connector_health", config.databaseUrl], { stdio: ["ignore", "pipe", "pipe"] });
146 + const out = createWriteStream(file);
147 + child.stdout.pipe(createGzip()).pipe(out);
148 + let stderr = "";
149 + child.stderr.on("data", (d) => (stderr += d));
150 + child.on("error", reject);
151 + child.on("close", (code) => (code === 0 ? resolve(file) : reject(new Error(`pg_dump exited ${code}: ${stderr.slice(0, 300)}`))));
152 + });
153 +}
154 +
155 +export const scheduler = new Scheduler();
added apps/api/src/core/observations.ts +117 −0
@@ -0,0 +1,117 @@
1 +import type { Observation } from "@market-atlas/market-model";
2 +import { config } from "../config.js";
3 +import { pool } from "../db/pool.js";
4 +import { ensureObservationPartition } from "../db/migrate.js";
5 +import { logger } from "../logger.js";
6 +import { telemetry } from "./telemetry.js";
7 +
8 +/**
9 + * Batched, idempotent observation writer. Bounded memory: when the queue exceeds the hard cap,
10 + * P2 tick observations are dropped (counted), never events or non-REALTIME values.
11 + */
12 +export class ObservationWriter {
13 + private queue: Observation[] = [];
14 + private timer: NodeJS.Timeout | null = null;
15 + private flushing = false;
16 + private partitionsReady = new Set<string>();
17 + private seen = new Map<string, number>(); // fingerprint -> ts (dedupe replays for 10 min)
18 + readonly hardCap = 200_000;
19 +
20 + enqueue(o: Observation): void {
21 + const fp = o.observationId;
22 + const now = Date.now();
23 + const prev = this.seen.get(fp);
24 + if (prev && now - prev < 600_000) {
25 + telemetry.inc("observations_duplicate_total");
26 + return;
27 + }
28 + this.seen.set(fp, now);
29 + if (this.seen.size > 500_000) this.pruneSeen(now);
30 + if (this.queue.length >= this.hardCap) {
31 + if (o.realtimeStatus === "REALTIME") {
32 + telemetry.inc("observations_dropped_total", 1, { reason: "backpressure" });
33 + return;
34 + }
35 + }
36 + this.queue.push(o);
37 + if (this.queue.length >= config.writerMaxBatch) void this.flush();
38 + else if (!this.timer) this.timer = setTimeout(() => void this.flush(), config.writerFlushMs);
39 + }
40 +
41 + private pruneSeen(now: number) {
42 + for (const [k, t] of this.seen) if (now - t > 600_000) this.seen.delete(k);
43 + }
44 +
45 + depth(): number {
46 + return this.queue.length;
47 + }
48 +
49 + async flush(): Promise<void> {
50 + if (this.timer) {
51 + clearTimeout(this.timer);
52 + this.timer = null;
53 + }
54 + if (this.flushing || !this.queue.length) return;
55 + this.flushing = true;
56 + const batch = this.queue.splice(0, config.writerMaxBatch);
57 + const started = Date.now();
58 + try {
59 + const days = new Set(batch.map((o) => new Date(o.receivedAt).toISOString().slice(0, 10)));
60 + for (const d of days) {
61 + if (!this.partitionsReady.has(d)) {
62 + await ensureObservationPartition(Date.parse(d));
63 + this.partitionsReady.add(d);
64 + }
65 + }
66 + const cols = 17;
67 + const values: unknown[] = [];
68 + const tuples: string[] = [];
69 + batch.forEach((o, i) => {
70 + const b = i * cols;
71 + tuples.push(`(${Array.from({ length: cols }, (_, j) => `$${b + j + 1}`).join(",")})`);
72 + values.push(
73 + new Date(o.receivedAt).toISOString(),
74 + o.sourceTimestamp == null ? null : new Date(o.sourceTimestamp).toISOString(),
75 + o.instrumentId,
76 + o.field,
77 + o.value,
78 + o.currency ?? null,
79 + o.sourceId,
80 + o.connectorId,
81 + o.rightsStatus,
82 + o.realtimeStatus,
83 + o.timestampTrust,
84 + o.confidence ?? null,
85 + o.latencyMs,
86 + o.sequence == null ? null : String(o.sequence),
87 + o.observationId,
88 + o.rawRef,
89 + o.normalizerVersion,
90 + );
91 + });
92 + await pool.query(
93 + `insert into observations (received_at, source_ts, instrument_id, field, value, currency, source_id, connector_id, rights_status, realtime_status, timestamp_trust, confidence, latency_ms, sequence, fingerprint, raw_ref, normalizer_version)
94 + values ${tuples.join(",")}`,
95 + values,
96 + );
97 + await pool.query(
98 + `insert into daily_stats (day, observations) values (current_date, $1)
99 + on conflict (day) do update set observations = daily_stats.observations + excluded.observations, updated_at = now()`,
100 + [batch.length],
101 + );
102 + telemetry.inc("observations_written_total", batch.length);
103 + telemetry.observe("db_insert_latency_ms", Date.now() - started);
104 + } catch (err) {
105 + telemetry.inc("observations_write_errors_total");
106 + logger.error({ err: err instanceof Error ? err.message : String(err), size: batch.length }, "observation batch insert failed");
107 + // Re-queue once at the front (bounded), so a transient DB hiccup does not lose canonical history.
108 + if (this.queue.length + batch.length <= this.hardCap) this.queue.unshift(...batch);
109 + } finally {
110 + this.flushing = false;
111 + telemetry.gauge("observation_queue_depth", this.queue.length);
112 + if (this.queue.length) this.timer = setTimeout(() => void this.flush(), config.writerFlushMs);
113 + }
114 + }
115 +}
116 +
117 +export const observationWriter = new ObservationWriter();
added apps/api/src/core/pipeline.ts +300 −0
@@ -0,0 +1,300 @@
1 +import type { Bar, Instrument, NormalizedObservation, Observation, RawObservation } from "@market-atlas/market-model";
2 +import { CANONICAL_VERSIONS, NormalizedObservationSchema, PUBLICLY_REDISTRIBUTABLE } from "@market-atlas/market-model";
3 +import { observationFingerprint, type ConnectorDefinition, type NormalizedBatch } from "@market-atlas/connector-sdk";
4 +import { pool } from "../db/pool.js";
5 +import { logger } from "../logger.js";
6 +import { barAggregator, upsertBars } from "./bars.js";
7 +import { bus } from "./bus.js";
8 +import { calendar } from "./calendar.js";
9 +import { ConsensusEngine } from "./consensus.js";
10 +import { eventEngine } from "./events.js";
11 +import { health } from "./health.js";
12 +import { instruments } from "./instruments.js";
13 +import { observationWriter } from "./observations.js";
14 +import { quoteStore } from "./quotes.js";
15 +import { telemetry } from "./telemetry.js";
16 +
17 +export interface SourceInfo {
18 + family: string | null;
19 + isOfficial: boolean;
20 + realtimeStatus: Observation["realtimeStatus"];
21 +}
22 +
23 +/**
24 + * L1→L4 pipeline for one normalized batch: validate, resolve instruments, persist observations,
25 + * update consensus + canonical quotes, feed the event engine and the bar aggregator, and handle
26 + * side outputs (proposed events, instrument master updates, filings, holidays).
27 + */
28 +export class Pipeline {
29 + readonly consensus: ConsensusEngine;
30 + private sourceInfo = new Map<string, SourceInfo>();
31 + private recomputeQueue = new Map<string, Instrument>();
32 + private recomputeTimer: NodeJS.Timeout | null = null;
33 +
34 + constructor() {
35 + this.consensus = new ConsensusEngine(
36 + (sourceId) => {
37 + const info = this.sourceInfo.get(sourceId);
38 + return {
39 + family: info?.family ?? null,
40 + reliability: health.sourceReliability(sourceId),
41 + realtimeStatus: info?.realtimeStatus ?? "UNKNOWN",
42 + isOfficial: info?.isOfficial ?? false,
43 + };
44 + },
45 + (instrumentId) => {
46 + const inst = instruments.get(instrumentId);
47 + const ex = inst?.exchangeId ? calendar.exchange(inst.exchangeId) : undefined;
48 + if (!ex || ex.sessions.continuous) return null;
49 + return calendar.state(ex.id) === "OPEN";
50 + },
51 + );
52 + bus.subscribe("canonical.quote", (q) => barAggregator.onQuote(q));
53 + }
54 +
55 + registerSource(sourceId: string, info: SourceInfo) {
56 + this.sourceInfo.set(sourceId, info);
57 + }
58 +
59 + async process(def: ConnectorDefinition, raw: RawObservation, batch: NormalizedBatch, rawRef: string | null): Promise<number> {
60 + const connectorId = def.metadata.id;
61 + const sourceId = def.metadata.sourceId;
62 + let accepted = 0;
63 + const touched = new Map<string, Instrument>();
64 + const ids: string[] = [];
65 +
66 + if (batch.instruments?.length) await this.applyInstruments(batch, sourceId);
67 +
68 + for (const n of batch.observations) {
69 + const parsed = NormalizedObservationSchema.safeParse(n);
70 + if (!parsed.success) {
71 + telemetry.inc("observations_invalid_total", 1, { connector: connectorId });
72 + continue;
73 + }
74 + const inst = n.instrumentId ? instruments.get(n.instrumentId) : await instruments.resolveOrCreate(n.symbol, sourceId, n.instrumentHint);
75 + if (!inst) {
76 + telemetry.inc("observations_unresolved_total", 1, { connector: connectorId });
77 + continue;
78 + }
79 + if (n.instrumentId && !instruments.resolve(n.symbol, sourceId)) await instruments.addAlias(n.symbol, sourceId, inst.id);
80 + if (!plausible(n, inst)) {
81 + telemetry.inc("observations_rejected_total", 1, { connector: connectorId, reason: "implausible" });
82 + continue;
83 + }
84 + const obs = this.toObservation(n, inst, raw, rawRef);
85 + observationWriter.enqueue(obs);
86 + this.consensus.ingest(obs);
87 + bus.publish("normalized.observation", obs);
88 + touched.set(inst.id, inst);
89 + ids.push(inst.id);
90 + accepted++;
91 + }
92 + const latency = batch.observations.find((o) => o.sourceTimestamp)?.sourceTimestamp;
93 + health.message(connectorId, latency ? raw.receivedAt - latency : null, ids);
94 + for (const [id, inst] of touched) this.recomputeQueue.set(id, inst);
95 + this.scheduleRecompute();
96 +
97 + if (batch.events?.length) await this.applyEvents(batch, sourceId);
98 + if (batch.filings?.length) await this.applyFilings(batch, sourceId);
99 + if (batch.holidays?.length) await this.applyHolidays(batch, sourceId);
100 + if (batch.bars?.length) await this.applyBars(batch, def);
101 + return accepted;
102 + }
103 +
104 + private async applyBars(batch: NormalizedBatch, def: ConnectorDefinition) {
105 + const out: Bar[] = [];
106 + for (const b of batch.bars ?? []) {
107 + const inst = await instruments.resolveOrCreate(b.symbol, def.metadata.sourceId, b.instrumentHint);
108 + if (!inst) continue;
109 + if (!(b.high >= b.low && b.open > 0 && b.close > 0)) continue;
110 + out.push({ instrumentId: inst.id, resolution: b.resolution, ts: b.ts, open: b.open, high: b.high, low: b.low, close: b.close, volume: b.volume, sourceCount: 1, producer: def.metadata.id, version: CANONICAL_VERSIONS.normalizer });
111 + }
112 + await upsertBars(out);
113 + if (out.length) telemetry.inc("bars_backfilled_total", out.length, { connector: def.metadata.id });
114 + }
115 +
116 + private toObservation(n: NormalizedObservation, inst: Instrument, raw: RawObservation, rawRef: string | null): Observation {
117 + const fp = observationFingerprint({
118 + sourceId: raw.sourceId,
119 + instrumentId: inst.id,
120 + field: n.field,
121 + value: n.value,
122 + sourceTimestamp: n.sourceTimestamp,
123 + sequence: n.sequence ?? null,
124 + });
125 + return {
126 + observationId: fp,
127 + instrumentId: inst.id,
128 + symbol: n.symbol,
129 + field: n.field,
130 + value: n.value,
131 + currency: n.currency ?? inst.currency ?? null,
132 + sourceTimestamp: n.sourceTimestamp,
133 + timestampTrust: n.timestampTrust,
134 + sequence: n.sequence ?? null,
135 + rightsStatus: n.rightsStatus,
136 + realtimeStatus: n.realtimeStatus,
137 + confidence: n.confidence,
138 + meta: n.meta,
139 + sourceId: raw.sourceId,
140 + connectorId: raw.connectorId,
141 + receivedAt: raw.receivedAt,
142 + latencyMs: n.sourceTimestamp == null ? null : Math.max(0, raw.receivedAt - n.sourceTimestamp),
143 + rawRef,
144 + normalizerVersion: CANONICAL_VERSIONS.normalizer,
145 + };
146 + }
147 +
148 + /** Recompute canonical quotes at most ~10×/s per instrument (UI batching happens downstream too). */
149 + private scheduleRecompute() {
150 + if (this.recomputeTimer) return;
151 + this.recomputeTimer = setTimeout(() => {
152 + this.recomputeTimer = null;
153 + const batch = [...this.recomputeQueue.values()];
154 + this.recomputeQueue.clear();
155 + const now = Date.now();
156 + for (const inst of batch) {
157 + const q = this.consensus.compute(inst.id, inst.symbol, now);
158 + if (!q) continue;
159 + // Public redistribution guard: values built only from restricted sources stay internal.
160 + quoteStore.update(q);
161 + if (PUBLICLY_REDISTRIBUTABLE.has(q.rightsStatus)) eventEngine.onQuote(q, inst.name);
162 + }
163 + telemetry.gauge("consensus_instruments", this.consensus.instrumentsTracked());
164 + }, 100);
165 + }
166 +
167 + /** Recompute every tracked instrument (used by the freshness sweeper so stale quotes decay). */
168 + sweep(): void {
169 + for (const q of quoteStore.all()) {
170 + const inst = instruments.get(q.instrumentId);
171 + if (inst) this.recomputeQueue.set(inst.id, inst);
172 + }
173 + this.scheduleRecompute();
174 + }
175 +
176 + private async applyInstruments(batch: NormalizedBatch, sourceId: string) {
177 + let created = 0;
178 + for (const p of batch.instruments ?? []) {
179 + const before = instruments.count();
180 + const existing = instruments.resolve(p.symbol, sourceId);
181 + if (!existing && p.createIfMissing === false) continue;
182 + const inst = existing ?? (await instruments.resolveOrCreate(p.symbol, sourceId, p.hint));
183 + if (!inst) continue;
184 + if (instruments.count() > before) created++;
185 + for (const a of p.aliases ?? []) await instruments.addAlias(a, "*", inst.id);
186 + // Enrichment of an existing instrument: company/CIK, name, exchange when previously unknown.
187 + if (existing) {
188 + let companyId = existing.companyId;
189 + if (!companyId && p.hint.companyName) companyId = await instruments.upsertCompany({ name: p.hint.companyName, cik: p.hint.cik ?? null, country: p.hint.country ?? null });
190 + else if (companyId && p.hint.cik) await instruments.upsertCompany({ name: p.hint.companyName ?? existing.name, cik: p.hint.cik, country: p.hint.country ?? null });
191 + const patch: Partial<Instrument> = {};
192 + if (companyId !== existing.companyId) patch.companyId = companyId;
193 + if (!existing.exchangeId && p.hint.exchangeId) patch.exchangeId = p.hint.exchangeId;
194 + if (!existing.mic && p.hint.mic) patch.mic = p.hint.mic;
195 + if (existing.name === existing.symbol && p.hint.name) patch.name = p.hint.name;
196 + if (p.isActive === false && existing.isActive) patch.isActive = false;
197 + if (Object.keys(patch).length) await instruments.upsert({ ...existing, ...patch });
198 + }
199 + }
200 + if (created) logger.info({ source: sourceId, created, proposed: batch.instruments?.length }, "instrument master updated");
201 + }
202 +
203 + private async applyEvents(batch: NormalizedBatch, sourceId: string) {
204 + for (const e of batch.events ?? []) {
205 + const ids = new Set<string>(e.instrumentIds ?? []);
206 + for (const s of e.symbols ?? []) {
207 + const inst = instruments.resolve(s, sourceId) ?? instruments.bySymbolCandidates(s).find((i) => i.assetClass === "EQUITY" || i.assetClass === "ETF");
208 + if (inst) ids.add(inst.id);
209 + }
210 + eventEngine.proposed(e, sourceId, [...ids]);
211 + }
212 + }
213 +
214 + private async applyFilings(batch: NormalizedBatch, sourceId: string) {
215 + let inserted = 0;
216 + for (const f of batch.filings ?? []) {
217 + const ids: string[] = [];
218 + if (f.cik) {
219 + const r = await pool.query<{ id: string }>("select i.id from instruments i join companies c on c.id = i.company_id where c.cik = $1 and i.is_active limit 5", [f.cik.replace(/^0+/, "")]);
220 + ids.push(...r.rows.map((x) => x.id));
221 + }
222 + for (const s of f.symbols ?? []) {
223 + const inst = instruments.bySymbolCandidates(s)[0];
224 + if (inst) ids.push(inst.id);
225 + }
226 + const res = await pool.query(
227 + `insert into filings (id, source_id, cik, company_name, form_type, filed_at, url, instrument_ids, metadata) values ($1,$2,$3,$4,$5,$6,$7,$8,$9) on conflict (id) do nothing`,
228 + [f.id, sourceId, f.cik, f.companyName, f.formType, new Date(f.filedAt).toISOString(), f.url, ids, JSON.stringify(f.metadata ?? {})],
229 + );
230 + if (res.rowCount) {
231 + inserted++;
232 + if (f.metadata?.material === false) continue;
233 + eventEngine.proposed(
234 + {
235 + type: "FILING_PUBLISHED",
236 + timestamp: f.filedAt,
237 + title: `${f.companyName}: ${f.formType} filing`,
238 + summary: null,
239 + severity: "INFO",
240 + confidence: 1,
241 + dedupeKey: `filing:${f.id}`,
242 + data: { formType: f.formType, cik: f.cik, url: f.url, companyName: f.companyName, filingId: f.id },
243 + },
244 + sourceId,
245 + ids,
246 + );
247 + }
248 + }
249 + if (inserted) {
250 + await pool.query(`insert into daily_stats (day, filings) values (current_date, $1) on conflict (day) do update set filings = daily_stats.filings + excluded.filings, updated_at = now()`, [inserted]);
251 + telemetry.inc("filings_total", inserted);
252 + }
253 + }
254 +
255 + private async applyHolidays(batch: NormalizedBatch, sourceId: string) {
256 + for (const h of batch.holidays ?? []) {
257 + await pool.query(
258 + `insert into exchange_holidays (exchange_id, date, name, kind, close_time, source_id) values ($1,$2,$3,$4,$5,$6)
259 + on conflict (exchange_id, date) do update set name = excluded.name, kind = excluded.kind, close_time = excluded.close_time, source_id = excluded.source_id`,
260 + [h.exchangeId, h.date, h.name, h.kind, h.closeTime ?? null, sourceId],
261 + );
262 + calendar.addHoliday({ ...h, sourceId });
263 + }
264 + }
265 +}
266 +
267 +/** Reject values that cannot be right for the field/instrument (guards against schema drift producing garbage). */
268 +function plausible(n: NormalizedObservation, inst: Instrument): boolean {
269 + if (!Number.isFinite(n.value)) return false;
270 + switch (n.field) {
271 + case "LAST_PRICE":
272 + case "OPEN":
273 + case "HIGH":
274 + case "LOW":
275 + case "PREVIOUS_CLOSE":
276 + case "CLOSE":
277 + case "BID":
278 + case "ASK":
279 + case "VWAP":
280 + case "NAV":
281 + if (n.value <= 0 && inst.assetClass !== "INTEREST_RATE" && inst.assetClass !== "TREASURY") return false;
282 + if (n.value > 1e9) return false;
283 + return true;
284 + case "VOLUME":
285 + case "BID_SIZE":
286 + case "ASK_SIZE":
287 + case "OPEN_INTEREST":
288 + case "MARKET_CAP":
289 + return n.value >= 0;
290 + case "CHANGE_PERCENT":
291 + return Math.abs(n.value) < 1000;
292 + case "YIELD":
293 + case "RATE":
294 + return n.value > -20 && n.value < 200;
295 + default:
296 + return true;
297 + }
298 +}
299 +
300 +export const pipeline = new Pipeline();
added apps/api/src/core/quotes.ts +110 −0
@@ -0,0 +1,110 @@
1 +import type { CanonicalQuote } from "@market-atlas/market-model";
2 +import { pool, tsToMs } from "../db/pool.js";
3 +import { logger } from "../logger.js";
4 +import { bus } from "./bus.js";
5 +import { telemetry } from "./telemetry.js";
6 +
7 +/**
8 + * Current-state cache of canonical quotes with debounced persistence (one upsert per instrument
9 + * per second at most). The bus receives every recomputation; the DB receives the settled value.
10 + */
11 +export class QuoteStore {
12 + private quotes = new Map<string, CanonicalQuote>();
13 + private dirty = new Map<string, CanonicalQuote>();
14 + private timer: NodeJS.Timeout | null = null;
15 +
16 + async load(): Promise<void> {
17 + const r = await pool.query("select * from canonical_quotes");
18 + for (const row of r.rows) this.quotes.set(row.instrument_id, rowToQuote(row));
19 + logger.info({ quotes: this.quotes.size }, "canonical quotes loaded");
20 + }
21 +
22 + get(id: string): CanonicalQuote | undefined {
23 + return this.quotes.get(id);
24 + }
25 +
26 + all(): CanonicalQuote[] {
27 + return [...this.quotes.values()];
28 + }
29 +
30 + size() {
31 + return this.quotes.size;
32 + }
33 +
34 + update(q: CanonicalQuote): void {
35 + this.quotes.set(q.instrumentId, q);
36 + this.dirty.set(q.instrumentId, q);
37 + bus.publish("canonical.quote", q);
38 + telemetry.inc("canonical_quotes_total");
39 + if (!this.timer) this.timer = setTimeout(() => void this.flush(), 1000);
40 + }
41 +
42 + async flush(): Promise<void> {
43 + this.timer = null;
44 + if (!this.dirty.size) return;
45 + const batch = [...this.dirty.values()];
46 + this.dirty.clear();
47 + const cols = 25;
48 + const tuples: string[] = [];
49 + const values: unknown[] = [];
50 + batch.forEach((q, i) => {
51 + const b = i * cols;
52 + tuples.push(`(${Array.from({ length: cols }, (_, j) => `$${b + j + 1}`).join(",")})`);
53 + values.push(
54 + q.instrumentId, q.symbol, q.price, q.open, q.high, q.low, q.previousClose, q.change, q.changePercent, q.volume, q.bid, q.ask, q.currency,
55 + q.sourceCount, q.dispersionBps, q.confidence, q.freshnessMs, q.realtimeStatus, q.rightsStatus, new Date(q.updatedAt).toISOString(),
56 + q.sourceTimestamp == null ? null : new Date(q.sourceTimestamp).toISOString(), q.sessionHigh, q.sessionLow,
57 + JSON.stringify(q.contributions.slice(0, 20)), q.consensusVersion,
58 + );
59 + });
60 + try {
61 + await pool.query(
62 + `insert into canonical_quotes (instrument_id, symbol, price, open, high, low, previous_close, change, change_percent, volume, bid, ask, currency,
63 + source_count, dispersion_bps, confidence, freshness_ms, realtime_status, rights_status, updated_at, source_ts, session_high, session_low, contributions, consensus_version)
64 + values ${tuples.join(",")}
65 + on conflict (instrument_id) do update set symbol = excluded.symbol, price = excluded.price, open = excluded.open, high = excluded.high, low = excluded.low,
66 + previous_close = excluded.previous_close, change = excluded.change, change_percent = excluded.change_percent, volume = excluded.volume, bid = excluded.bid, ask = excluded.ask,
67 + currency = excluded.currency, source_count = excluded.source_count, dispersion_bps = excluded.dispersion_bps, confidence = excluded.confidence, freshness_ms = excluded.freshness_ms,
68 + realtime_status = excluded.realtime_status, rights_status = excluded.rights_status, updated_at = excluded.updated_at, source_ts = excluded.source_ts,
69 + session_high = excluded.session_high, session_low = excluded.session_low, contributions = excluded.contributions, consensus_version = excluded.consensus_version`,
70 + values,
71 + );
72 + } catch (err) {
73 + logger.error({ err, size: batch.length }, "canonical quote upsert failed");
74 + for (const q of batch) if (!this.dirty.has(q.instrumentId)) this.dirty.set(q.instrumentId, q);
75 + this.timer = setTimeout(() => void this.flush(), 5000);
76 + }
77 + }
78 +}
79 +
80 +export function rowToQuote(row: any): CanonicalQuote {
81 + return {
82 + instrumentId: row.instrument_id,
83 + symbol: row.symbol,
84 + price: row.price,
85 + open: row.open,
86 + high: row.high,
87 + low: row.low,
88 + previousClose: row.previous_close,
89 + change: row.change,
90 + changePercent: row.change_percent,
91 + volume: row.volume,
92 + bid: row.bid,
93 + ask: row.ask,
94 + currency: row.currency,
95 + sourceCount: row.source_count,
96 + dispersionBps: row.dispersion_bps,
97 + confidence: row.confidence,
98 + freshnessMs: row.freshness_ms,
99 + realtimeStatus: row.realtime_status,
100 + rightsStatus: row.rights_status,
101 + updatedAt: tsToMs(row.updated_at) ?? 0,
102 + sourceTimestamp: tsToMs(row.source_ts),
103 + sessionHigh: row.session_high,
104 + sessionLow: row.session_low,
105 + contributions: row.contributions ?? [],
106 + consensusVersion: row.consensus_version,
107 + };
108 +}
109 +
110 +export const quoteStore = new QuoteStore();
added apps/api/src/core/raw-archive.ts +71 −0
@@ -0,0 +1,71 @@
1 +import { createWriteStream, mkdirSync } from "node:fs";
2 +import { readFile } from "node:fs/promises";
3 +import { join } from "node:path";
4 +import { gunzipSync, gzipSync } from "node:zlib";
5 +import type { RawObservation } from "@market-atlas/market-model";
6 +import { redactObject, sha256 } from "@market-atlas/connector-sdk";
7 +import { config } from "../config.js";
8 +import { telemetry } from "./telemetry.js";
9 +
10 +/**
11 + * L0 archive: raw payloads stored gzip-compressed, content-addressed, one directory per day
12 + * and connector. Streaming frames are sampled (MA_RAW_SAMPLE_RATE); polled/bulk payloads are
13 + * always kept. Secrets are redacted before writing.
14 + */
15 +export class RawArchive {
16 + private pending = new Map<string, Buffer>();
17 + private timer: NodeJS.Timeout | null = null;
18 +
19 + constructor(private root = join(config.dataDir, "raw")) {}
20 +
21 + /** Returns a raw_ref (relative path) or null when the frame was not sampled. */
22 + store(raw: RawObservation, force = false): string | null {
23 + if (!force && Math.random() > config.rawSampleRate) return null;
24 + const day = new Date(raw.receivedAt).toISOString().slice(0, 10);
25 + const body = JSON.stringify({
26 + connector_id: raw.connectorId,
27 + source_id: raw.sourceId,
28 + kind: raw.kind,
29 + received_at: new Date(raw.receivedAt).toISOString(),
30 + meta: raw.meta ? redactObject(raw.meta) : undefined,
31 + payload: redactObject(raw.payload),
32 + });
33 + const hash = sha256(body).slice(0, 32);
34 + const rel = join(day, raw.connectorId, `${hash}.json.gz`);
35 + if (!this.pending.has(rel)) {
36 + this.pending.set(rel, gzipSync(body, { level: 6 }));
37 + this.schedule();
38 + }
39 + return rel;
40 + }
41 +
42 + private schedule() {
43 + if (this.timer) return;
44 + this.timer = setTimeout(() => {
45 + this.timer = null;
46 + this.flush();
47 + }, 500);
48 + }
49 +
50 + flush(): void {
51 + const batch = [...this.pending.entries()];
52 + this.pending.clear();
53 + for (const [rel, buf] of batch) {
54 + const abs = join(this.root, rel);
55 + mkdirSync(join(abs, ".."), { recursive: true });
56 + const ws = createWriteStream(abs, { flags: "wx" });
57 + ws.on("error", () => {}); // already exists → content-addressed, fine
58 + ws.end(buf);
59 + telemetry.inc("raw_archived_total");
60 + telemetry.inc("raw_archived_bytes", buf.length);
61 + }
62 + }
63 +
64 + async read(ref: string): Promise<unknown> {
65 + if (ref.includes("..")) throw new Error("invalid ref");
66 + const buf = await readFile(join(this.root, ref));
67 + return JSON.parse(gunzipSync(buf).toString("utf8"));
68 + }
69 +}
70 +
71 +export const rawArchive = new RawArchive();
added apps/api/src/core/registry.ts +47 −0
@@ -0,0 +1,47 @@
1 +import type { ConnectorDefinition } from "@market-atlas/connector-sdk";
2 +import { CONNECTORS, SOURCES } from "@market-atlas/connectors";
3 +import { config } from "../config.js";
4 +import { pool } from "../db/pool.js";
5 +import { logger } from "../logger.js";
6 +import { connectorManager } from "./connector-manager.js";
7 +import { instruments } from "./instruments.js";
8 +
9 +/** Syncs code-defined sources/connectors into the DB (metadata is code; runtime flags are DB) and registers them. */
10 +export async function syncRegistry(): Promise<ConnectorDefinition[]> {
11 + for (const s of SOURCES) {
12 + await pool.query(
13 + `insert into sources (id, name, organization, source_type, homepage, jurisdiction, rights_status, realtime_status, family, category, enabled)
14 + values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
15 + on conflict (id) do update set name = excluded.name, organization = excluded.organization, source_type = excluded.source_type, homepage = excluded.homepage,
16 + jurisdiction = excluded.jurisdiction, rights_status = excluded.rights_status, realtime_status = excluded.realtime_status, family = excluded.family, category = excluded.category, updated_at = now()`,
17 + [s.id, s.name, s.organization, s.sourceType, s.homepage, s.jurisdiction, s.rightsStatus, s.realtimeStatus, s.family, s.category, s.enabled],
18 + );
19 + }
20 + const registered: ConnectorDefinition[] = [];
21 + for (const def of CONNECTORS) {
22 + const m = def.metadata;
23 + const enabled = m.enabled && !config.disabledConnectors.has(m.id);
24 + await pool.query(
25 + `insert into connectors (id, source_id, name, version, source_type, rights_status, realtime_status, enabled, metadata)
26 + values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
27 + on conflict (id) do update set source_id = excluded.source_id, name = excluded.name, version = excluded.version, source_type = excluded.source_type,
28 + rights_status = excluded.rights_status, realtime_status = excluded.realtime_status, enabled = excluded.enabled, metadata = excluded.metadata, updated_at = now()`,
29 + [m.id, m.sourceId, m.name, m.version, m.sourceType, m.rightsStatus, m.realtimeStatus, enabled, JSON.stringify(m)],
30 + );
31 + await pool.query(`insert into connector_versions (connector_id, version, metadata) values ($1,$2,$3) on conflict do nothing`, [m.id, m.version, JSON.stringify({ description: m.description })]);
32 + const row = (await pool.query<{ paused: boolean; state: Record<string, unknown>; schema_fingerprints: Record<string, string[]> }>("select paused, state, schema_fingerprints from connectors where id = $1", [m.id])).rows[0]!;
33 + if (!enabled) {
34 + logger.info({ connector: m.id }, "connector disabled");
35 + continue;
36 + }
37 + // Seed static instruments declared by the connector.
38 + for (const seed of def.seeds ?? []) {
39 + const inst = await instruments.resolveOrCreate(seed.symbol, m.sourceId, seed.hint);
40 + if (inst) for (const a of seed.aliases ?? []) await instruments.addAlias(a, "*", inst.id);
41 + }
42 + await connectorManager.register(def, { paused: row.paused, state: row.state ?? {}, fingerprints: row.schema_fingerprints ?? {} });
43 + registered.push(def);
44 + }
45 + logger.info({ connectors: registered.length, sources: SOURCES.length }, "registry synced");
46 + return registered;
47 +}
added apps/api/src/core/telemetry.ts +124 −0
@@ -0,0 +1,124 @@
1 +/** Lightweight in-process metrics: counters, gauges, sliding-window rates, latency reservoirs. Prometheus text export. */
2 +
3 +class Window {
4 + private buckets: number[];
5 + private idx = 0;
6 + private last: number;
7 + constructor(
8 + private seconds: number,
9 + now = Date.now(),
10 + ) {
11 + this.buckets = new Array(seconds).fill(0);
12 + this.last = Math.floor(now / 1000);
13 + }
14 + private roll(now: number) {
15 + const sec = Math.floor(now / 1000);
16 + let steps = sec - this.last;
17 + if (steps <= 0) return;
18 + if (steps > this.seconds) steps = this.seconds;
19 + for (let i = 0; i < steps; i++) {
20 + this.idx = (this.idx + 1) % this.seconds;
21 + this.buckets[this.idx] = 0;
22 + }
23 + this.last = sec;
24 + }
25 + add(n: number, now = Date.now()) {
26 + this.roll(now);
27 + this.buckets[this.idx]! += n;
28 + }
29 + sum(now = Date.now()): number {
30 + this.roll(now);
31 + return this.buckets.reduce((a, b) => a + b, 0);
32 + }
33 + perSecond(now = Date.now()): number {
34 + return this.sum(now) / this.seconds;
35 + }
36 +}
37 +
38 +export class Reservoir {
39 + private values: number[] = [];
40 + constructor(private size = 512) {}
41 + add(v: number) {
42 + if (this.values.length < this.size) this.values.push(v);
43 + else this.values[Math.floor(Math.random() * this.size)] = v;
44 + }
45 + quantile(q: number): number | null {
46 + if (!this.values.length) return null;
47 + const s = [...this.values].sort((a, b) => a - b);
48 + return s[Math.min(s.length - 1, Math.floor(q * s.length))] ?? null;
49 + }
50 + count() {
51 + return this.values.length;
52 + }
53 +}
54 +
55 +export class Telemetry {
56 + private counters = new Map<string, number>();
57 + private gauges = new Map<string, number>();
58 + private windows = new Map<string, Window>();
59 + private reservoirs = new Map<string, Reservoir>();
60 + readonly startedAt = Date.now();
61 +
62 + inc(name: string, n = 1, labels?: Record<string, string>) {
63 + const key = this.key(name, labels);
64 + this.counters.set(key, (this.counters.get(key) ?? 0) + n);
65 + let w = this.windows.get(key);
66 + if (!w) {
67 + w = new Window(60);
68 + this.windows.set(key, w);
69 + }
70 + w.add(n);
71 + }
72 + gauge(name: string, v: number, labels?: Record<string, string>) {
73 + this.gauges.set(this.key(name, labels), v);
74 + }
75 + observe(name: string, v: number, labels?: Record<string, string>) {
76 + const key = this.key(name, labels);
77 + let r = this.reservoirs.get(key);
78 + if (!r) {
79 + r = new Reservoir();
80 + this.reservoirs.set(key, r);
81 + }
82 + r.add(v);
83 + }
84 + counter(name: string, labels?: Record<string, string>): number {
85 + return this.counters.get(this.key(name, labels)) ?? 0;
86 + }
87 + rate(name: string, labels?: Record<string, string>): number {
88 + return this.windows.get(this.key(name, labels))?.perSecond() ?? 0;
89 + }
90 + lastMinute(name: string, labels?: Record<string, string>): number {
91 + return this.windows.get(this.key(name, labels))?.sum() ?? 0;
92 + }
93 + quantile(name: string, q: number, labels?: Record<string, string>): number | null {
94 + return this.reservoirs.get(this.key(name, labels))?.quantile(q) ?? null;
95 + }
96 + private key(name: string, labels?: Record<string, string>) {
97 + if (!labels) return name;
98 + const l = Object.entries(labels)
99 + .sort()
100 + .map(([k, v]) => `${k}="${v.replace(/"/g, '\\"')}"`)
101 + .join(",");
102 + return `${name}{${l}}`;
103 + }
104 +
105 + /** Prometheus exposition format. */
106 + prometheus(): string {
107 + const lines: string[] = [];
108 + for (const [k, v] of this.counters) lines.push(`ma_${k} ${v}`);
109 + for (const [k, v] of this.gauges) lines.push(`ma_${k} ${v}`);
110 + for (const [k, r] of this.reservoirs) {
111 + const base = k.includes("{") ? k.replace("{", "_seconds{") : `${k}_seconds`;
112 + for (const q of [0.5, 0.95, 0.99]) {
113 + const val = r.quantile(q);
114 + if (val != null) lines.push(`ma_${base.includes("{") ? base.replace("}", `,quantile="${q}"}`) : `${base}{quantile="${q}"}`} ${val / 1000}`);
115 + }
116 + }
117 + lines.push(`ma_uptime_seconds ${(Date.now() - this.startedAt) / 1000}`);
118 + const mem = process.memoryUsage();
119 + lines.push(`ma_process_rss_bytes ${mem.rss}`, `ma_process_heap_used_bytes ${mem.heapUsed}`);
120 + return lines.join("\n") + "\n";
121 + }
122 +}
123 +
124 +export const telemetry = new Telemetry();
added apps/api/src/db/migrate.ts +66 −0
@@ -0,0 +1,66 @@
1 +import { readdirSync, readFileSync, existsSync } from "node:fs";
2 +import { dirname, join, resolve } from "node:path";
3 +import { fileURLToPath } from "node:url";
4 +import { pool } from "./pool.js";
5 +import { logger } from "../logger.js";
6 +
7 +/** Migrations live in infra/migrations (forward-only SQL files, applied in lexical order). */
8 +export function migrationsDir(): string {
9 + const here = dirname(fileURLToPath(import.meta.url));
10 + const candidates = [
11 + resolve(here, "../../../../infra/migrations"), // apps/api/src/db → repo root
12 + resolve(here, "../../../infra/migrations"), // apps/api/dist → repo root
13 + resolve(process.cwd(), "infra/migrations"),
14 + resolve(process.cwd(), "../../infra/migrations"),
15 + ];
16 + const found = candidates.find((c) => existsSync(c));
17 + if (!found) throw new Error(`migrations directory not found (tried ${candidates.join(", ")})`);
18 + return found;
19 +}
20 +
21 +export async function migrate(): Promise<string[]> {
22 + const dir = migrationsDir();
23 + const files = readdirSync(dir)
24 + .filter((f) => f.endsWith(".sql"))
25 + .sort();
26 + const client = await pool.connect();
27 + const applied: string[] = [];
28 + try {
29 + await client.query("create table if not exists schema_migrations (version text primary key, applied_at timestamptz not null default now())");
30 + await client.query("select pg_advisory_lock(7231)");
31 + const done = new Set((await client.query<{ version: string }>("select version from schema_migrations")).rows.map((r) => r.version));
32 + for (const f of files) {
33 + if (done.has(f)) continue;
34 + const sql = readFileSync(join(dir, f), "utf8");
35 + logger.info({ migration: f }, "applying migration");
36 + await client.query("begin");
37 + try {
38 + await client.query(sql);
39 + await client.query("insert into schema_migrations(version) values ($1)", [f]);
40 + await client.query("commit");
41 + applied.push(f);
42 + } catch (err) {
43 + await client.query("rollback");
44 + throw new Error(`migration ${f} failed: ${err instanceof Error ? err.message : String(err)}`);
45 + }
46 + }
47 + await client.query("select pg_advisory_unlock(7231)");
48 + } finally {
49 + client.release();
50 + }
51 + return applied;
52 +}
53 +
54 +/** Ensure the daily partition for `day` (and the next one) exists. Idempotent. */
55 +export async function ensureObservationPartition(dayMs: number): Promise<void> {
56 + const d = new Date(dayMs);
57 + d.setUTCHours(0, 0, 0, 0);
58 + for (const offset of [0, 1]) {
59 + const start = new Date(d.getTime() + offset * 86_400_000);
60 + const end = new Date(start.getTime() + 86_400_000);
61 + const name = `observations_${start.toISOString().slice(0, 10).replace(/-/g, "")}`;
62 + await pool.query(
63 + `create table if not exists ${name} partition of observations for values from ('${start.toISOString()}') to ('${end.toISOString()}')`,
64 + );
65 + }
66 +}
added apps/api/src/db/pool.ts +49 −0
@@ -0,0 +1,49 @@
1 +import pg from "pg";
2 +import { config } from "../config.js";
3 +import { logger } from "../logger.js";
4 +
5 +const { Pool, types } = pg;
6 +// Keep timestamptz as ISO strings → we convert to ms explicitly; numeric/int8 as numbers where safe.
7 +types.setTypeParser(1184, (v) => v);
8 +types.setTypeParser(20, (v) => Number(v));
9 +types.setTypeParser(1700, (v) => Number(v));
10 +
11 +export const pool = new Pool({
12 + connectionString: config.databaseUrl,
13 + max: config.role === "api" ? 8 : 12,
14 + idleTimeoutMillis: 30_000,
15 + statement_timeout: 60_000,
16 + application_name: `market-atlas-${config.role}`,
17 +});
18 +
19 +pool.on("error", (err) => logger.error({ err }, "pg pool error"));
20 +
21 +export type Queryable = pg.Pool | pg.PoolClient;
22 +
23 +export async function query<T extends pg.QueryResultRow = pg.QueryResultRow>(text: string, params: unknown[] = [], client: Queryable = pool) {
24 + return client.query<T>(text, params as never[]);
25 +}
26 +
27 +export async function withTransaction<T>(fn: (client: pg.PoolClient) => Promise<T>): Promise<T> {
28 + const client = await pool.connect();
29 + try {
30 + await client.query("begin");
31 + const out = await fn(client);
32 + await client.query("commit");
33 + return out;
34 + } catch (err) {
35 + await client.query("rollback").catch(() => {});
36 + throw err;
37 + } finally {
38 + client.release();
39 + }
40 +}
41 +
42 +export const tsToMs = (v: unknown): number | null => {
43 + if (v == null) return null;
44 + if (v instanceof Date) return v.getTime();
45 + const ms = Date.parse(String(v));
46 + return Number.isFinite(ms) ? ms : null;
47 +};
48 +
49 +export const msToTs = (ms: number | null | undefined): string | null => (ms == null ? null : new Date(ms).toISOString());
added apps/api/src/discovery/probe.ts +128 −0
@@ -0,0 +1,128 @@
1 +import type { HttpClient } from "@market-atlas/connector-sdk";
2 +import { redactUrl, schemaFingerprint } from "@market-atlas/connector-sdk";
3 +
4 +/**
5 + * Discovery prototype (network-level, no browser yet): fetches a public page unauthenticated,
6 + * inventories the data endpoints it references (JSON/XHR URLs, WebSocket URLs, SSE, embedded
7 + * JSON state such as __NEXT_DATA__), and scores which ones look like market-data feeds.
8 + * Output is a *candidate* report for the rights-review workflow — never an activated connector.
9 + */
10 +export interface DiscoveryReport {
11 + url: string;
12 + status: number;
13 + content_type: string | null;
14 + bytes: number;
15 + websocket_urls: string[];
16 + sse_urls: string[];
17 + json_endpoints: Array<{ url: string; score: number; hints: string[] }>;
18 + embedded_state: Array<{ kind: string; fingerprint: string; price_like_fields: string[]; symbol_like_fields: string[]; sample_keys: string[] }>;
19 + instrument_hints: string[];
20 + notes: string[];
21 +}
22 +
23 +const PRICE_KEYS = /^(last|lastprice|last_price|price|lp|px|close|c|regularmarketprice|current_price|bid|ask|mid|value|rate)$/i;
24 +const SYMBOL_KEYS = /^(symbol|sym|ticker|s|code|instrument|isin|pair|product_id|instid)$/i;
25 +const TS_KEYS = /^(ts|t|time|timestamp|datetime|eventtime|updated|last_trade_time)$/i;
26 +
27 +export async function runDiscovery(url: string, http: HttpClient): Promise<DiscoveryReport> {
28 + const res = await http.getText(url, { timeoutMs: 20_000, conditional: false, headers: { accept: "text/html,application/json;q=0.9,*/*;q=0.8" } });
29 + const html = res.text;
30 + const notes: string[] = [];
31 + const wsUrls = uniq([...html.matchAll(/wss?:\/\/[^\s"'<>\\)]+/g)].map((m) => redactUrl(m[0])));
32 + const sse = uniq([...html.matchAll(/EventSource\((["'`])([^"'`]+)\1/g)].map((m) => m[2]!));
33 + const urls = uniq([...html.matchAll(/https?:\/\/[^\s"'<>\\)]+/g)].map((m) => m[0]))
34 + .filter((u) => /api|quote|price|market|ticker|chart|data|feed|graphql|json|v\d\//i.test(u) && !/\.(png|jpg|jpeg|gif|svg|css|woff2?|ico)(\?|$)/i.test(u))
35 + .map((u) => redactUrl(u));
36 + const jsonEndpoints = urls
37 + .map((u) => {
38 + const hints: string[] = [];
39 + let score = 0;
40 + if (/quote|price|ticker|last/i.test(u)) (score += 40), hints.push("quote/price keyword");
41 + if (/graphql/i.test(u)) (score += 20), hints.push("GraphQL");
42 + if (/\.json(\?|$)|format=json/i.test(u)) (score += 15), hints.push("JSON");
43 + if (/api\.|\/api\//i.test(u)) (score += 15), hints.push("API path");
44 + if (/chart|history|candles|bars/i.test(u)) (score += 10), hints.push("historical/chart");
45 + if (/symbol=|ticker=|sym=/i.test(u)) (score += 20), hints.push("symbol parameter");
46 + return { url: u, score, hints };
47 + })
48 + .filter((e) => e.score > 0)
49 + .sort((a, b) => b.score - a.score)
50 + .slice(0, 40);
51 +
52 + const embedded: DiscoveryReport["embedded_state"] = [];
53 + const stateBlocks: Array<[string, RegExp]> = [
54 + ["__NEXT_DATA__", /<script id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/i],
55 + ["__NUXT__", /window\.__NUXT__\s*=\s*([\s\S]*?);\s*<\/script>/i],
56 + ["__INITIAL_STATE__", /window\.__INITIAL_STATE__\s*=\s*([\s\S]*?);\s*<\/script>/i],
57 + ["__PRELOADED_STATE__", /window\.__PRELOADED_STATE__\s*=\s*([\s\S]*?);\s*<\/script>/i],
58 + ["ld+json", /<script type="application\/ld\+json"[^>]*>([\s\S]*?)<\/script>/i],
59 + ];
60 + for (const [kind, re] of stateBlocks) {
61 + const m = html.match(re);
62 + if (!m?.[1]) continue;
63 + try {
64 + const obj = JSON.parse(m[1]);
65 + const keys = new Set<string>();
66 + const priceLike = new Set<string>();
67 + const symbolLike = new Set<string>();
68 + walk(obj, "", 0, (path, key, val) => {
69 + keys.add(key);
70 + if (PRICE_KEYS.test(key) && (typeof val === "number" || (typeof val === "string" && /^-?\d+(\.\d+)?$/.test(val)))) priceLike.add(path);
71 + if (SYMBOL_KEYS.test(key) && typeof val === "string") symbolLike.add(path);
72 + });
73 + embedded.push({ kind, fingerprint: schemaFingerprint(obj), price_like_fields: [...priceLike].slice(0, 20), symbol_like_fields: [...symbolLike].slice(0, 20), sample_keys: [...keys].slice(0, 60) });
74 + } catch {
75 + notes.push(`${kind} block present but not parseable as JSON`);
76 + }
77 + }
78 + if (res.headers["content-type"]?.includes("application/json")) {
79 + try {
80 + const obj = JSON.parse(html);
81 + const priceLike = new Set<string>();
82 + const symbolLike = new Set<string>();
83 + const tsLike = new Set<string>();
84 + walk(obj, "", 0, (path, key, val) => {
85 + if (PRICE_KEYS.test(key) && typeof val !== "object") priceLike.add(path);
86 + if (SYMBOL_KEYS.test(key) && typeof val === "string") symbolLike.add(path);
87 + if (TS_KEYS.test(key)) tsLike.add(path);
88 + });
89 + embedded.push({ kind: "response-json", fingerprint: schemaFingerprint(obj), price_like_fields: [...priceLike], symbol_like_fields: [...symbolLike], sample_keys: [...tsLike].map((t) => `ts:${t}`) });
90 + notes.push("URL returns JSON directly — candidate XHR/FETCH connector");
91 + } catch {
92 + /* ignore */
93 + }
94 + }
95 + const instrumentHints = uniq([...html.matchAll(/\b(?:NASDAQ|NYSE|TSX|LSE|XETRA|BINANCE|COINBASE):([A-Z.\-]{1,12})\b/g)].map((m) => m[0])).slice(0, 20);
96 + if (wsUrls.length) notes.push("WebSocket endpoints referenced — verify subscription protocol and terms before use");
97 + if (!wsUrls.length && !jsonEndpoints.length && !embedded.length) notes.push("No data endpoints found statically; the page may load data via bundled JavaScript (browser-based discovery required)");
98 + notes.push("Rights status must be classified manually (RIGHTS_REVIEW) before any connector reaches staging.");
99 + return {
100 + url: redactUrl(url),
101 + status: res.status,
102 + content_type: res.headers["content-type"] ?? null,
103 + bytes: html.length,
104 + websocket_urls: wsUrls,
105 + sse_urls: sse,
106 + json_endpoints: jsonEndpoints,
107 + embedded_state: embedded,
108 + instrument_hints: instrumentHints,
109 + notes,
110 + };
111 +}
112 +
113 +function walk(v: unknown, path: string, depth: number, fn: (path: string, key: string, val: unknown) => void) {
114 + if (depth > 8 || v == null) return;
115 + if (Array.isArray(v)) {
116 + for (const item of v.slice(0, 5)) walk(item, `${path}[]`, depth + 1, fn);
117 + return;
118 + }
119 + if (typeof v === "object") {
120 + for (const [k, val] of Object.entries(v as Record<string, unknown>)) {
121 + const p = path ? `${path}.${k}` : k;
122 + fn(p, k, val);
123 + walk(val, p, depth + 1, fn);
124 + }
125 + }
126 +}
127 +
128 +const uniq = <T>(xs: T[]) => [...new Set(xs)];
added apps/api/src/logger.ts +16 −0
@@ -0,0 +1,16 @@
1 +import pino from "pino";
2 +import { config } from "./config.js";
3 +import { redactObject } from "@market-atlas/connector-sdk";
4 +
5 +export const logger = pino({
6 + level: process.env.MA_LOG_LEVEL ?? (config.isProd ? "info" : "debug"),
7 + base: { service: "market-atlas" },
8 + timestamp: pino.stdTimeFunctions.isoTime,
9 + formatters: {
10 + level: (label) => ({ level: label }),
11 + log: (obj) => redactObject(obj),
12 + },
13 + ...(config.logJson ? {} : { transport: { target: "pino/file", options: { destination: 1 } } }),
14 +});
15 +
16 +export type Logger = typeof logger;
added apps/api/src/main.ts +86 −0
@@ -0,0 +1,86 @@
1 +import { config, ensureDataDirs, runsApi, runsWorker } from "./config.js";
2 +import { logger } from "./logger.js";
3 +import { migrate } from "./db/migrate.js";
4 +import { pool } from "./db/pool.js";
5 +import { seed } from "./seed/index.js";
6 +import { instruments } from "./core/instruments.js";
7 +import { calendar } from "./core/calendar.js";
8 +import { quoteStore } from "./core/quotes.js";
9 +import { eventEngine } from "./core/events.js";
10 +import { syncRegistry } from "./core/registry.js";
11 +import { connectorManager } from "./core/connector-manager.js";
12 +import { scheduler } from "./core/jobs.js";
13 +import { observationWriter } from "./core/observations.js";
14 +import { barAggregator } from "./core/bars.js";
15 +import { rawArchive } from "./core/raw-archive.js";
16 +import { buildServer } from "./api/server.js";
17 +import { rowToExchange } from "./core/instruments.js";
18 +
19 +async function loadCalendar() {
20 + const [ex, hol] = await Promise.all([pool.query("select * from exchanges"), pool.query("select exchange_id, to_char(date, 'YYYY-MM-DD') as date, name, kind, close_time, source_id from exchange_holidays")]);
21 + calendar.load(
22 + ex.rows.map(rowToExchange),
23 + hol.rows.map((h) => ({ exchangeId: h.exchange_id, date: h.date, name: h.name, kind: h.kind, closeTime: h.close_time, sourceId: h.source_id })),
24 + );
25 +}
26 +
27 +async function main() {
28 + logger.info({ env: config.appEnv, role: config.role, version: config.version }, "Market Atlas starting");
29 + ensureDataDirs();
30 + if (config.isProd && !config.adminToken) throw new Error("MA_ADMIN_TOKEN is required in production");
31 + await migrate();
32 + await seed();
33 + await loadCalendar();
34 + await instruments.load();
35 + await quoteStore.load();
36 + await eventEngine.warm();
37 +
38 + if (runsWorker()) {
39 + await syncRegistry();
40 + scheduler.start();
41 + await connectorManager.startAll();
42 + }
43 +
44 + let app: Awaited<ReturnType<typeof buildServer>> | null = null;
45 + if (runsApi()) {
46 + app = await buildServer();
47 + await app.listen({ host: config.apiHost, port: config.apiPort });
48 + logger.info({ host: config.apiHost, port: config.apiPort }, "API listening");
49 + }
50 +
51 + let shuttingDown = false;
52 + const shutdown = async (signal: string) => {
53 + if (shuttingDown) return;
54 + shuttingDown = true;
55 + logger.info({ signal }, "shutting down");
56 + const force = setTimeout(() => process.exit(1), 20_000);
57 + try {
58 + scheduler.stop();
59 + await connectorManager.stopAll();
60 + await app?.close();
61 + await observationWriter.flush();
62 + await quoteStore.flush();
63 + await barAggregator.flush();
64 + await eventEngine.flush();
65 + rawArchive.flush();
66 + await pool.end();
67 + } catch (err) {
68 + logger.error({ err }, "shutdown error");
69 + } finally {
70 + clearTimeout(force);
71 + process.exit(0);
72 + }
73 + };
74 + process.on("SIGINT", () => void shutdown("SIGINT"));
75 + process.on("SIGTERM", () => void shutdown("SIGTERM"));
76 + process.on("unhandledRejection", (err) => logger.error({ err }, "unhandled rejection"));
77 + process.on("uncaughtException", (err) => {
78 + logger.fatal({ err }, "uncaught exception");
79 + void shutdown("uncaughtException");
80 + });
81 +}
82 +
83 +main().catch((err) => {
84 + logger.fatal({ err }, "startup failed");
85 + process.exit(1);
86 +});
added apps/api/src/seed/countries.json +54 −0
@@ -0,0 +1,54 @@
1 +[
2 + { "code": "US", "name": "United States", "region": "North America", "currency": "USD" },
3 + { "code": "CA", "name": "Canada", "region": "North America", "currency": "CAD" },
4 + { "code": "MX", "name": "Mexico", "region": "North America", "currency": "MXN" },
5 + { "code": "BR", "name": "Brazil", "region": "South America", "currency": "BRL" },
6 + { "code": "AR", "name": "Argentina", "region": "South America", "currency": "ARS" },
7 + { "code": "CL", "name": "Chile", "region": "South America", "currency": "CLP" },
8 + { "code": "CO", "name": "Colombia", "region": "South America", "currency": "COP" },
9 + { "code": "GB", "name": "United Kingdom", "region": "Europe", "currency": "GBP" },
10 + { "code": "IE", "name": "Ireland", "region": "Europe", "currency": "EUR" },
11 + { "code": "FR", "name": "France", "region": "Europe", "currency": "EUR" },
12 + { "code": "DE", "name": "Germany", "region": "Europe", "currency": "EUR" },
13 + { "code": "NL", "name": "Netherlands", "region": "Europe", "currency": "EUR" },
14 + { "code": "BE", "name": "Belgium", "region": "Europe", "currency": "EUR" },
15 + { "code": "PT", "name": "Portugal", "region": "Europe", "currency": "EUR" },
16 + { "code": "ES", "name": "Spain", "region": "Europe", "currency": "EUR" },
17 + { "code": "IT", "name": "Italy", "region": "Europe", "currency": "EUR" },
18 + { "code": "CH", "name": "Switzerland", "region": "Europe", "currency": "CHF" },
19 + { "code": "AT", "name": "Austria", "region": "Europe", "currency": "EUR" },
20 + { "code": "SE", "name": "Sweden", "region": "Europe", "currency": "SEK" },
21 + { "code": "NO", "name": "Norway", "region": "Europe", "currency": "NOK" },
22 + { "code": "DK", "name": "Denmark", "region": "Europe", "currency": "DKK" },
23 + { "code": "FI", "name": "Finland", "region": "Europe", "currency": "EUR" },
24 + { "code": "PL", "name": "Poland", "region": "Europe", "currency": "PLN" },
25 + { "code": "CZ", "name": "Czechia", "region": "Europe", "currency": "CZK" },
26 + { "code": "HU", "name": "Hungary", "region": "Europe", "currency": "HUF" },
27 + { "code": "GR", "name": "Greece", "region": "Europe", "currency": "EUR" },
28 + { "code": "TR", "name": "Türkiye", "region": "Europe", "currency": "TRY" },
29 + { "code": "RU", "name": "Russia", "region": "Europe", "currency": "RUB" },
30 + { "code": "IL", "name": "Israel", "region": "Middle East", "currency": "ILS" },
31 + { "code": "SA", "name": "Saudi Arabia", "region": "Middle East", "currency": "SAR" },
32 + { "code": "AE", "name": "United Arab Emirates", "region": "Middle East", "currency": "AED" },
33 + { "code": "QA", "name": "Qatar", "region": "Middle East", "currency": "QAR" },
34 + { "code": "EG", "name": "Egypt", "region": "Africa", "currency": "EGP" },
35 + { "code": "ZA", "name": "South Africa", "region": "Africa", "currency": "ZAR" },
36 + { "code": "NG", "name": "Nigeria", "region": "Africa", "currency": "NGN" },
37 + { "code": "KE", "name": "Kenya", "region": "Africa", "currency": "KES" },
38 + { "code": "JP", "name": "Japan", "region": "Asia-Pacific", "currency": "JPY" },
39 + { "code": "CN", "name": "China", "region": "Asia-Pacific", "currency": "CNY" },
40 + { "code": "HK", "name": "Hong Kong", "region": "Asia-Pacific", "currency": "HKD" },
41 + { "code": "TW", "name": "Taiwan", "region": "Asia-Pacific", "currency": "TWD" },
42 + { "code": "KR", "name": "South Korea", "region": "Asia-Pacific", "currency": "KRW" },
43 + { "code": "SG", "name": "Singapore", "region": "Asia-Pacific", "currency": "SGD" },
44 + { "code": "MY", "name": "Malaysia", "region": "Asia-Pacific", "currency": "MYR" },
45 + { "code": "TH", "name": "Thailand", "region": "Asia-Pacific", "currency": "THB" },
46 + { "code": "ID", "name": "Indonesia", "region": "Asia-Pacific", "currency": "IDR" },
47 + { "code": "PH", "name": "Philippines", "region": "Asia-Pacific", "currency": "PHP" },
48 + { "code": "VN", "name": "Vietnam", "region": "Asia-Pacific", "currency": "VND" },
49 + { "code": "IN", "name": "India", "region": "Asia-Pacific", "currency": "INR" },
50 + { "code": "AU", "name": "Australia", "region": "Asia-Pacific", "currency": "AUD" },
51 + { "code": "NZ", "name": "New Zealand", "region": "Asia-Pacific", "currency": "NZD" },
52 + { "code": "EU", "name": "Euro area", "region": "Europe", "currency": "EUR" },
53 + { "code": "XX", "name": "Global / cross-border", "region": "Global", "currency": null }
54 +]
added apps/api/src/seed/exchanges.json +63 −0
@@ -0,0 +1,63 @@
1 +[
2 + { "id": "xnys", "mic": "XNYS", "name": "New York Stock Exchange", "operator": "Intercontinental Exchange", "country": "US", "city": "New York", "timezone": "America/New_York", "currency": "USD", "website": "https://www.nyse.com", "lat": 40.7069, "lon": -74.0113, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:30", "close": "16:00" }], "pre": { "open": "04:00", "close": "09:30" }, "post": { "open": "16:00", "close": "20:00" } } },
3 + { "id": "xnas", "mic": "XNAS", "name": "Nasdaq Stock Market", "operator": "Nasdaq, Inc.", "country": "US", "city": "New York", "timezone": "America/New_York", "currency": "USD", "website": "https://www.nasdaq.com", "lat": 40.7565, "lon": -73.9860, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:30", "close": "16:00" }], "pre": { "open": "04:00", "close": "09:30" }, "post": { "open": "16:00", "close": "20:00" } } },
4 + { "id": "xase", "mic": "XASE", "name": "NYSE American", "operator": "Intercontinental Exchange", "country": "US", "city": "New York", "timezone": "America/New_York", "currency": "USD", "website": "https://www.nyse.com/markets/nyse-american", "lat": 40.7069, "lon": -74.0113, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:30", "close": "16:00" }], "pre": { "open": "04:00", "close": "09:30" }, "post": { "open": "16:00", "close": "20:00" } } },
5 + { "id": "arcx", "mic": "ARCX", "name": "NYSE Arca", "operator": "Intercontinental Exchange", "country": "US", "city": "Chicago", "timezone": "America/New_York", "currency": "USD", "website": "https://www.nyse.com/markets/nyse-arca", "lat": 41.8781, "lon": -87.6298, "asset_classes": ["ETF", "EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "16:00" }], "pre": { "open": "04:00", "close": "09:30" }, "post": { "open": "16:00", "close": "20:00" } } },
6 + { "id": "bats", "mic": "BATS", "name": "Cboe BZX Exchange", "operator": "Cboe Global Markets", "country": "US", "city": "Chicago", "timezone": "America/Chicago", "currency": "USD", "website": "https://www.cboe.com", "lat": 41.8827, "lon": -87.6324, "asset_classes": ["EQUITY", "ETF", "INDEX"], "sessions": { "regular": [{ "open": "08:30", "close": "15:00" }], "pre": { "open": "03:00", "close": "08:30" }, "post": { "open": "15:00", "close": "19:00" } } },
7 + { "id": "xcbo", "mic": "XCBO", "name": "Cboe Options Exchange", "operator": "Cboe Global Markets", "country": "US", "city": "Chicago", "timezone": "America/Chicago", "currency": "USD", "website": "https://www.cboe.com", "lat": 41.8827, "lon": -87.6324, "asset_classes": ["INDEX", "OPTION"], "sessions": { "regular": [{ "open": "08:30", "close": "15:15" }] } },
8 + { "id": "xcme", "mic": "XCME", "name": "CME Group (Globex)", "operator": "CME Group", "country": "US", "city": "Chicago", "timezone": "America/Chicago", "currency": "USD", "website": "https://www.cmegroup.com", "lat": 41.8785, "lon": -87.6355, "asset_classes": ["FUTURE", "COMMODITY"], "sessions": { "regular": [{ "open": "17:00", "close": "24:00" }, { "open": "00:00", "close": "16:00" }], "weekdays": [0, 1, 2, 3, 4, 5] } },
9 + { "id": "xtse", "mic": "XTSE", "name": "Toronto Stock Exchange", "operator": "TMX Group", "country": "CA", "city": "Toronto", "timezone": "America/Toronto", "currency": "CAD", "website": "https://www.tsx.com", "lat": 43.6487, "lon": -79.3817, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:30", "close": "16:00" }], "post": { "open": "16:15", "close": "17:00" } } },
10 + { "id": "xmex", "mic": "XMEX", "name": "Bolsa Mexicana de Valores", "operator": "Grupo BMV", "country": "MX", "city": "Mexico City", "timezone": "America/Mexico_City", "currency": "MXN", "website": "https://www.bmv.com.mx", "lat": 19.4260, "lon": -99.1730, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "08:30", "close": "15:00" }] } },
11 + { "id": "bvmf", "mic": "BVMF", "name": "B3 — Brasil Bolsa Balcão", "operator": "B3 S.A.", "country": "BR", "city": "São Paulo", "timezone": "America/Sao_Paulo", "currency": "BRL", "website": "https://www.b3.com.br", "lat": -23.5475, "lon": -46.6361, "asset_classes": ["EQUITY", "FUTURE"], "sessions": { "regular": [{ "open": "10:00", "close": "17:55" }] } },
12 + { "id": "xbue", "mic": "XBUE", "name": "Bolsas y Mercados Argentinos", "operator": "BYMA", "country": "AR", "city": "Buenos Aires", "timezone": "America/Argentina/Buenos_Aires", "currency": "ARS", "website": "https://www.byma.com.ar", "lat": -34.6037, "lon": -58.3737, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "11:00", "close": "17:00" }] } },
13 + { "id": "xsgo", "mic": "XSGO", "name": "Bolsa de Santiago", "operator": "Bolsa de Santiago", "country": "CL", "city": "Santiago", "timezone": "America/Santiago", "currency": "CLP", "website": "https://www.bolsadesantiago.com", "lat": -33.4390, "lon": -70.6510, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "16:00" }] } },
14 + { "id": "xlon", "mic": "XLON", "name": "London Stock Exchange", "operator": "LSEG", "country": "GB", "city": "London", "timezone": "Europe/London", "currency": "GBP", "website": "https://www.londonstockexchange.com", "lat": 51.5155, "lon": -0.0987, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "08:00", "close": "16:30" }] } },
15 + { "id": "xdub", "mic": "XDUB", "name": "Euronext Dublin", "operator": "Euronext", "country": "IE", "city": "Dublin", "timezone": "Europe/Dublin", "currency": "EUR", "website": "https://www.euronext.com/en/markets/dublin", "lat": 53.3441, "lon": -6.2675, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "08:00", "close": "16:28" }] } },
16 + { "id": "xpar", "mic": "XPAR", "name": "Euronext Paris", "operator": "Euronext", "country": "FR", "city": "Paris", "timezone": "Europe/Paris", "currency": "EUR", "website": "https://www.euronext.com/en/markets/paris", "lat": 48.8698, "lon": 2.3397, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
17 + { "id": "xams", "mic": "XAMS", "name": "Euronext Amsterdam", "operator": "Euronext", "country": "NL", "city": "Amsterdam", "timezone": "Europe/Amsterdam", "currency": "EUR", "website": "https://www.euronext.com/en/markets/amsterdam", "lat": 52.3702, "lon": 4.8952, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
18 + { "id": "xbru", "mic": "XBRU", "name": "Euronext Brussels", "operator": "Euronext", "country": "BE", "city": "Brussels", "timezone": "Europe/Brussels", "currency": "EUR", "website": "https://www.euronext.com/en/markets/brussels", "lat": 50.8467, "lon": 4.3499, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
19 + { "id": "xlis", "mic": "XLIS", "name": "Euronext Lisbon", "operator": "Euronext", "country": "PT", "city": "Lisbon", "timezone": "Europe/Lisbon", "currency": "EUR", "website": "https://www.euronext.com/en/markets/lisbon", "lat": 38.7078, "lon": -9.1366, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "08:00", "close": "16:30" }] } },
20 + { "id": "xmad", "mic": "XMAD", "name": "Bolsa de Madrid", "operator": "SIX / BME", "country": "ES", "city": "Madrid", "timezone": "Europe/Madrid", "currency": "EUR", "website": "https://www.bolsamadrid.es", "lat": 40.4168, "lon": -3.6936, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
21 + { "id": "xmil", "mic": "XMIL", "name": "Borsa Italiana", "operator": "Euronext", "country": "IT", "city": "Milan", "timezone": "Europe/Rome", "currency": "EUR", "website": "https://www.borsaitaliana.it", "lat": 45.4654, "lon": 9.1859, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
22 + { "id": "xetr", "mic": "XETR", "name": "Xetra (Frankfurt)", "operator": "Deutsche Börse", "country": "DE", "city": "Frankfurt", "timezone": "Europe/Berlin", "currency": "EUR", "website": "https://www.xetra.com", "lat": 50.1155, "lon": 8.6742, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
23 + { "id": "xswx", "mic": "XSWX", "name": "SIX Swiss Exchange", "operator": "SIX Group", "country": "CH", "city": "Zurich", "timezone": "Europe/Zurich", "currency": "CHF", "website": "https://www.six-group.com", "lat": 47.3667, "lon": 8.5378, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
24 + { "id": "xwbo", "mic": "XWBO", "name": "Wiener Börse", "operator": "Wiener Börse AG", "country": "AT", "city": "Vienna", "timezone": "Europe/Vienna", "currency": "EUR", "website": "https://www.wienerborse.at", "lat": 48.2116, "lon": 16.3702, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
25 + { "id": "xsto", "mic": "XSTO", "name": "Nasdaq Stockholm", "operator": "Nasdaq Nordic", "country": "SE", "city": "Stockholm", "timezone": "Europe/Stockholm", "currency": "SEK", "website": "https://www.nasdaqomxnordic.com", "lat": 59.3300, "lon": 18.0700, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:30" }] } },
26 + { "id": "xcse", "mic": "XCSE", "name": "Nasdaq Copenhagen", "operator": "Nasdaq Nordic", "country": "DK", "city": "Copenhagen", "timezone": "Europe/Copenhagen", "currency": "DKK", "website": "https://www.nasdaqomxnordic.com", "lat": 55.6761, "lon": 12.5683, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:00" }] } },
27 + { "id": "xhel", "mic": "XHEL", "name": "Nasdaq Helsinki", "operator": "Nasdaq Nordic", "country": "FI", "city": "Helsinki", "timezone": "Europe/Helsinki", "currency": "EUR", "website": "https://www.nasdaqomxnordic.com", "lat": 60.1699, "lon": 24.9384, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "18:30" }] } },
28 + { "id": "xosl", "mic": "XOSL", "name": "Oslo Børs", "operator": "Euronext", "country": "NO", "city": "Oslo", "timezone": "Europe/Oslo", "currency": "NOK", "website": "https://www.euronext.com/en/markets/oslo", "lat": 59.9139, "lon": 10.7522, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "16:20" }] } },
29 + { "id": "xwar", "mic": "XWAR", "name": "Warsaw Stock Exchange", "operator": "GPW", "country": "PL", "city": "Warsaw", "timezone": "Europe/Warsaw", "currency": "PLN", "website": "https://www.gpw.pl", "lat": 52.2297, "lon": 21.0122, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:00" }] } },
30 + { "id": "xpra", "mic": "XPRA", "name": "Prague Stock Exchange", "operator": "Wiener Börse AG", "country": "CZ", "city": "Prague", "timezone": "Europe/Prague", "currency": "CZK", "website": "https://www.pse.cz", "lat": 50.0755, "lon": 14.4378, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "16:20" }] } },
31 + { "id": "xbud", "mic": "XBUD", "name": "Budapest Stock Exchange", "operator": "BSE", "country": "HU", "city": "Budapest", "timezone": "Europe/Budapest", "currency": "HUF", "website": "https://www.bse.hu", "lat": 47.4979, "lon": 19.0402, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:00" }] } },
32 + { "id": "xath", "mic": "XATH", "name": "Athens Stock Exchange", "operator": "ATHEX", "country": "GR", "city": "Athens", "timezone": "Europe/Athens", "currency": "EUR", "website": "https://www.athexgroup.gr", "lat": 37.9838, "lon": 23.7275, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "17:20" }] } },
33 + { "id": "xist", "mic": "XIST", "name": "Borsa İstanbul", "operator": "Borsa İstanbul", "country": "TR", "city": "Istanbul", "timezone": "Europe/Istanbul", "currency": "TRY", "website": "https://www.borsaistanbul.com", "lat": 41.1085, "lon": 29.0288, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "18:00" }] } },
34 + { "id": "xmos", "mic": "MISX", "name": "Moscow Exchange", "operator": "MOEX", "country": "RU", "city": "Moscow", "timezone": "Europe/Moscow", "currency": "RUB", "website": "https://www.moex.com", "lat": 55.7558, "lon": 37.6173, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "18:50" }] } },
35 + { "id": "xtae", "mic": "XTAE", "name": "Tel Aviv Stock Exchange", "operator": "TASE", "country": "IL", "city": "Tel Aviv", "timezone": "Asia/Jerusalem", "currency": "ILS", "website": "https://www.tase.co.il", "lat": 32.0636, "lon": 34.7708, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "17:15" }], "weekdays": [0, 1, 2, 3, 4] } },
36 + { "id": "xsau", "mic": "XSAU", "name": "Saudi Exchange (Tadawul)", "operator": "Saudi Tadawul Group", "country": "SA", "city": "Riyadh", "timezone": "Asia/Riyadh", "currency": "SAR", "website": "https://www.saudiexchange.sa", "lat": 24.7136, "lon": 46.6753, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "15:00" }], "weekdays": [0, 1, 2, 3, 4] } },
37 + { "id": "xdfm", "mic": "XDFM", "name": "Dubai Financial Market", "operator": "DFM", "country": "AE", "city": "Dubai", "timezone": "Asia/Dubai", "currency": "AED", "website": "https://www.dfm.ae", "lat": 25.2172, "lon": 55.2790, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "14:45" }] } },
38 + { "id": "dsmd", "mic": "DSMD", "name": "Qatar Stock Exchange", "operator": "QSE", "country": "QA", "city": "Doha", "timezone": "Asia/Qatar", "currency": "QAR", "website": "https://www.qe.com.qa", "lat": 25.2854, "lon": 51.5310, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "13:15" }], "weekdays": [0, 1, 2, 3, 4] } },
39 + { "id": "xcai", "mic": "XCAI", "name": "Egyptian Exchange", "operator": "EGX", "country": "EG", "city": "Cairo", "timezone": "Africa/Cairo", "currency": "EGP", "website": "https://www.egx.com.eg", "lat": 30.0444, "lon": 31.2357, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "14:30" }], "weekdays": [0, 1, 2, 3, 4] } },
40 + { "id": "xjse", "mic": "XJSE", "name": "Johannesburg Stock Exchange", "operator": "JSE Limited", "country": "ZA", "city": "Johannesburg", "timezone": "Africa/Johannesburg", "currency": "ZAR", "website": "https://www.jse.co.za", "lat": -26.1076, "lon": 28.0567, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "17:00" }] } },
41 + { "id": "xnsa", "mic": "XNSA", "name": "Nigerian Exchange", "operator": "NGX Group", "country": "NG", "city": "Lagos", "timezone": "Africa/Lagos", "currency": "NGN", "website": "https://ngxgroup.com", "lat": 6.4541, "lon": 3.3947, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "14:30" }] } },
42 + { "id": "xnai", "mic": "XNAI", "name": "Nairobi Securities Exchange", "operator": "NSE", "country": "KE", "city": "Nairobi", "timezone": "Africa/Nairobi", "currency": "KES", "website": "https://www.nse.co.ke", "lat": -1.2921, "lon": 36.8219, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "15:00" }] } },
43 + { "id": "xnse", "mic": "XNSE", "name": "National Stock Exchange of India", "operator": "NSE India", "country": "IN", "city": "Mumbai", "timezone": "Asia/Kolkata", "currency": "INR", "website": "https://www.nseindia.com", "lat": 19.0607, "lon": 72.8611, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:15", "close": "15:30" }] } },
44 + { "id": "xbom", "mic": "XBOM", "name": "BSE (Bombay Stock Exchange)", "operator": "BSE Ltd", "country": "IN", "city": "Mumbai", "timezone": "Asia/Kolkata", "currency": "INR", "website": "https://www.bseindia.com", "lat": 18.9299, "lon": 72.8336, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:15", "close": "15:30" }] } },
45 + { "id": "xshg", "mic": "XSHG", "name": "Shanghai Stock Exchange", "operator": "SSE", "country": "CN", "city": "Shanghai", "timezone": "Asia/Shanghai", "currency": "CNY", "website": "https://www.sse.com.cn", "lat": 31.2304, "lon": 121.4737, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "11:30" }, { "open": "13:00", "close": "15:00" }] } },
46 + { "id": "xshe", "mic": "XSHE", "name": "Shenzhen Stock Exchange", "operator": "SZSE", "country": "CN", "city": "Shenzhen", "timezone": "Asia/Shanghai", "currency": "CNY", "website": "https://www.szse.cn", "lat": 22.5431, "lon": 114.0579, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "11:30" }, { "open": "13:00", "close": "15:00" }] } },
47 + { "id": "xhkg", "mic": "XHKG", "name": "Hong Kong Stock Exchange", "operator": "HKEX", "country": "HK", "city": "Hong Kong", "timezone": "Asia/Hong_Kong", "currency": "HKD", "website": "https://www.hkex.com.hk", "lat": 22.2830, "lon": 114.1588, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:30", "close": "12:00" }, { "open": "13:00", "close": "16:00" }] } },
48 + { "id": "xtai", "mic": "XTAI", "name": "Taiwan Stock Exchange", "operator": "TWSE", "country": "TW", "city": "Taipei", "timezone": "Asia/Taipei", "currency": "TWD", "website": "https://www.twse.com.tw", "lat": 25.0330, "lon": 121.5654, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "13:30" }] } },
49 + { "id": "xkrx", "mic": "XKRX", "name": "Korea Exchange", "operator": "KRX", "country": "KR", "city": "Seoul", "timezone": "Asia/Seoul", "currency": "KRW", "website": "https://global.krx.co.kr", "lat": 37.5236, "lon": 126.9260, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "15:30" }] } },
50 + { "id": "xtks", "mic": "XTKS", "name": "Tokyo Stock Exchange", "operator": "Japan Exchange Group", "country": "JP", "city": "Tokyo", "timezone": "Asia/Tokyo", "currency": "JPY", "website": "https://www.jpx.co.jp", "lat": 35.6817, "lon": 139.7785, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "09:00", "close": "11:30" }, { "open": "12:30", "close": "15:30" }] } },
51 + { "id": "xses", "mic": "XSES", "name": "Singapore Exchange", "operator": "SGX Group", "country": "SG", "city": "Singapore", "timezone": "Asia/Singapore", "currency": "SGD", "website": "https://www.sgx.com", "lat": 1.2792, "lon": 103.8507, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "12:00" }, { "open": "13:00", "close": "17:00" }] } },
52 + { "id": "xkls", "mic": "XKLS", "name": "Bursa Malaysia", "operator": "Bursa Malaysia", "country": "MY", "city": "Kuala Lumpur", "timezone": "Asia/Kuala_Lumpur", "currency": "MYR", "website": "https://www.bursamalaysia.com", "lat": 3.1490, "lon": 101.6959, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "12:30" }, { "open": "14:30", "close": "17:00" }] } },
53 + { "id": "xbkk", "mic": "XBKK", "name": "Stock Exchange of Thailand", "operator": "SET", "country": "TH", "city": "Bangkok", "timezone": "Asia/Bangkok", "currency": "THB", "website": "https://www.set.or.th", "lat": 13.7563, "lon": 100.5018, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "12:30" }, { "open": "14:30", "close": "16:30" }] } },
54 + { "id": "xidx", "mic": "XIDX", "name": "Indonesia Stock Exchange", "operator": "IDX", "country": "ID", "city": "Jakarta", "timezone": "Asia/Jakarta", "currency": "IDR", "website": "https://www.idx.co.id", "lat": -6.2245, "lon": 106.8090, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "11:30" }, { "open": "13:30", "close": "15:50" }] } },
55 + { "id": "xphs", "mic": "XPHS", "name": "Philippine Stock Exchange", "operator": "PSE", "country": "PH", "city": "Manila", "timezone": "Asia/Manila", "currency": "PHP", "website": "https://www.pse.com.ph", "lat": 14.5547, "lon": 121.0244, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:30", "close": "12:00" }, { "open": "13:00", "close": "15:00" }] } },
56 + { "id": "xstc", "mic": "XSTC", "name": "Ho Chi Minh Stock Exchange", "operator": "HOSE", "country": "VN", "city": "Ho Chi Minh City", "timezone": "Asia/Ho_Chi_Minh", "currency": "VND", "website": "https://www.hsx.vn", "lat": 10.7769, "lon": 106.7009, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "09:00", "close": "11:30" }, { "open": "13:00", "close": "15:00" }] } },
57 + { "id": "xasx", "mic": "XASX", "name": "Australian Securities Exchange", "operator": "ASX Ltd", "country": "AU", "city": "Sydney", "timezone": "Australia/Sydney", "currency": "AUD", "website": "https://www.asx.com.au", "lat": -33.8651, "lon": 151.2099, "asset_classes": ["EQUITY", "ETF"], "sessions": { "regular": [{ "open": "10:00", "close": "16:00" }] } },
58 + { "id": "xnze", "mic": "XNZE", "name": "NZX", "operator": "NZX Limited", "country": "NZ", "city": "Wellington", "timezone": "Pacific/Auckland", "currency": "NZD", "website": "https://www.nzx.com", "lat": -41.2865, "lon": 174.7762, "asset_classes": ["EQUITY"], "sessions": { "regular": [{ "open": "10:00", "close": "16:45" }] } },
59 + { "id": "coinbase", "mic": null, "name": "Coinbase Exchange", "operator": "Coinbase Global", "country": "US", "city": "San Francisco", "timezone": "UTC", "currency": "USD", "website": "https://exchange.coinbase.com", "lat": 37.7749, "lon": -122.4194, "asset_classes": ["CRYPTO"], "sessions": { "regular": [], "continuous": true } },
60 + { "id": "kraken", "mic": null, "name": "Kraken", "operator": "Payward, Inc.", "country": "US", "city": "San Francisco", "timezone": "UTC", "currency": "USD", "website": "https://www.kraken.com", "lat": 37.79, "lon": -122.40, "asset_classes": ["CRYPTO"], "sessions": { "regular": [], "continuous": true } },
61 + { "id": "binance", "mic": null, "name": "Binance", "operator": "Binance", "country": "XX", "city": null, "timezone": "UTC", "currency": "USDT", "website": "https://www.binance.com", "lat": 43.7384, "lon": 7.4246, "asset_classes": ["CRYPTO"], "sessions": { "regular": [], "continuous": true } },
62 + { "id": "okx", "mic": null, "name": "OKX", "operator": "OKX", "country": "XX", "city": null, "timezone": "UTC", "currency": "USDT", "website": "https://www.okx.com", "lat": 1.3521, "lon": 103.8198, "asset_classes": ["CRYPTO"], "sessions": { "regular": [], "continuous": true } }
63 +]
added apps/api/src/seed/holidays.json +136 −0
@@ -0,0 +1,136 @@
1 +[
2 + { "exchanges": ["xnys", "xnas", "xase", "arcx", "bats", "xcbo"], "holidays": [
3 + { "date": "2026-01-01", "name": "New Year's Day" },
4 + { "date": "2026-01-19", "name": "Martin Luther King Jr. Day" },
5 + { "date": "2026-02-16", "name": "Presidents' Day" },
6 + { "date": "2026-04-03", "name": "Good Friday" },
7 + { "date": "2026-05-25", "name": "Memorial Day" },
8 + { "date": "2026-06-19", "name": "Juneteenth" },
9 + { "date": "2026-07-03", "name": "Independence Day (observed)" },
10 + { "date": "2026-09-07", "name": "Labor Day" },
11 + { "date": "2026-11-26", "name": "Thanksgiving Day" },
12 + { "date": "2026-11-27", "name": "Day after Thanksgiving (early close)", "kind": "EARLY_CLOSE", "closeTime": "13:00" },
13 + { "date": "2026-12-24", "name": "Christmas Eve (early close)", "kind": "EARLY_CLOSE", "closeTime": "13:00" },
14 + { "date": "2026-12-25", "name": "Christmas Day" },
15 + { "date": "2027-01-01", "name": "New Year's Day" },
16 + { "date": "2027-01-18", "name": "Martin Luther King Jr. Day" },
17 + { "date": "2027-02-15", "name": "Presidents' Day" },
18 + { "date": "2027-03-26", "name": "Good Friday" },
19 + { "date": "2027-05-31", "name": "Memorial Day" },
20 + { "date": "2027-06-18", "name": "Juneteenth (observed)" },
21 + { "date": "2027-07-05", "name": "Independence Day (observed)" },
22 + { "date": "2027-09-06", "name": "Labor Day" },
23 + { "date": "2027-11-25", "name": "Thanksgiving Day" },
24 + { "date": "2027-12-24", "name": "Christmas Day (observed)" }
25 + ] },
26 + { "exchanges": ["xtse"], "holidays": [
27 + { "date": "2026-01-01", "name": "New Year's Day" },
28 + { "date": "2026-02-16", "name": "Family Day" },
29 + { "date": "2026-04-03", "name": "Good Friday" },
30 + { "date": "2026-05-18", "name": "Victoria Day" },
31 + { "date": "2026-07-01", "name": "Canada Day" },
32 + { "date": "2026-08-03", "name": "Civic Holiday" },
33 + { "date": "2026-09-07", "name": "Labour Day" },
34 + { "date": "2026-10-12", "name": "Thanksgiving Day" },
35 + { "date": "2026-12-25", "name": "Christmas Day" },
36 + { "date": "2026-12-28", "name": "Boxing Day (observed)" },
37 + { "date": "2027-01-01", "name": "New Year's Day" }
38 + ] },
39 + { "exchanges": ["xlon"], "holidays": [
40 + { "date": "2026-01-01", "name": "New Year's Day" },
41 + { "date": "2026-04-03", "name": "Good Friday" },
42 + { "date": "2026-04-06", "name": "Easter Monday" },
43 + { "date": "2026-05-04", "name": "Early May Bank Holiday" },
44 + { "date": "2026-05-25", "name": "Spring Bank Holiday" },
45 + { "date": "2026-08-31", "name": "Summer Bank Holiday" },
46 + { "date": "2026-12-24", "name": "Christmas Eve (early close)", "kind": "EARLY_CLOSE", "closeTime": "12:30" },
47 + { "date": "2026-12-25", "name": "Christmas Day" },
48 + { "date": "2026-12-28", "name": "Boxing Day (substitute)" },
49 + { "date": "2026-12-31", "name": "New Year's Eve (early close)", "kind": "EARLY_CLOSE", "closeTime": "12:30" },
50 + { "date": "2027-01-01", "name": "New Year's Day" }
51 + ] },
52 + { "exchanges": ["xetr"], "holidays": [
53 + { "date": "2026-01-01", "name": "New Year's Day" },
54 + { "date": "2026-04-03", "name": "Good Friday" },
55 + { "date": "2026-04-06", "name": "Easter Monday" },
56 + { "date": "2026-05-01", "name": "Labour Day" },
57 + { "date": "2026-05-25", "name": "Whit Monday" },
58 + { "date": "2026-12-24", "name": "Christmas Eve" },
59 + { "date": "2026-12-25", "name": "Christmas Day" },
60 + { "date": "2026-12-31", "name": "New Year's Eve" },
61 + { "date": "2027-01-01", "name": "New Year's Day" }
62 + ] },
63 + { "exchanges": ["xpar", "xams", "xbru", "xlis", "xdub", "xmil", "xosl"], "holidays": [
64 + { "date": "2026-01-01", "name": "New Year's Day" },
65 + { "date": "2026-04-03", "name": "Good Friday" },
66 + { "date": "2026-04-06", "name": "Easter Monday" },
67 + { "date": "2026-05-01", "name": "Labour Day" },
68 + { "date": "2026-12-24", "name": "Christmas Eve (early close)", "kind": "EARLY_CLOSE", "closeTime": "14:05" },
69 + { "date": "2026-12-25", "name": "Christmas Day" },
70 + { "date": "2026-12-31", "name": "New Year's Eve (early close)", "kind": "EARLY_CLOSE", "closeTime": "14:05" },
71 + { "date": "2027-01-01", "name": "New Year's Day" }
72 + ] },
73 + { "exchanges": ["xswx"], "holidays": [
74 + { "date": "2026-01-01", "name": "New Year's Day" },
75 + { "date": "2026-01-02", "name": "Berchtoldstag" },
76 + { "date": "2026-04-03", "name": "Good Friday" },
77 + { "date": "2026-04-06", "name": "Easter Monday" },
78 + { "date": "2026-05-01", "name": "Labour Day" },
79 + { "date": "2026-05-14", "name": "Ascension Day" },
80 + { "date": "2026-05-25", "name": "Whit Monday" },
81 + { "date": "2026-08-01", "name": "Swiss National Day" },
82 + { "date": "2026-12-24", "name": "Christmas Eve" },
83 + { "date": "2026-12-25", "name": "Christmas Day" },
84 + { "date": "2026-12-31", "name": "New Year's Eve" }
85 + ] },
86 + { "exchanges": ["xtks"], "holidays": [
87 + { "date": "2026-01-01", "name": "New Year's Day" },
88 + { "date": "2026-01-02", "name": "Market holiday" },
89 + { "date": "2026-01-12", "name": "Coming of Age Day" },
90 + { "date": "2026-02-11", "name": "National Foundation Day" },
91 + { "date": "2026-02-23", "name": "Emperor's Birthday" },
92 + { "date": "2026-03-20", "name": "Vernal Equinox Day" },
93 + { "date": "2026-04-29", "name": "Shōwa Day" },
94 + { "date": "2026-05-04", "name": "Greenery Day" },
95 + { "date": "2026-05-05", "name": "Children's Day" },
96 + { "date": "2026-05-06", "name": "Constitution Memorial Day (observed)" },
97 + { "date": "2026-07-20", "name": "Marine Day" },
98 + { "date": "2026-08-11", "name": "Mountain Day" },
99 + { "date": "2026-09-21", "name": "Respect for the Aged Day" },
100 + { "date": "2026-09-22", "name": "National holiday" },
101 + { "date": "2026-09-23", "name": "Autumnal Equinox Day" },
102 + { "date": "2026-10-12", "name": "Sports Day" },
103 + { "date": "2026-11-03", "name": "Culture Day" },
104 + { "date": "2026-11-23", "name": "Labour Thanksgiving Day" },
105 + { "date": "2026-12-31", "name": "Market holiday" },
106 + { "date": "2027-01-01", "name": "New Year's Day" }
107 + ] },
108 + { "exchanges": ["xhkg"], "holidays": [
109 + { "date": "2026-01-01", "name": "New Year's Day" },
110 + { "date": "2026-02-17", "name": "Lunar New Year" },
111 + { "date": "2026-02-18", "name": "Lunar New Year (2nd day)" },
112 + { "date": "2026-02-19", "name": "Lunar New Year (3rd day)" },
113 + { "date": "2026-04-03", "name": "Good Friday" },
114 + { "date": "2026-04-06", "name": "Easter Monday" },
115 + { "date": "2026-04-07", "name": "Ching Ming Festival (observed)" },
116 + { "date": "2026-05-01", "name": "Labour Day" },
117 + { "date": "2026-05-25", "name": "Buddha's Birthday (observed)" },
118 + { "date": "2026-06-19", "name": "Tuen Ng Festival" },
119 + { "date": "2026-07-01", "name": "HKSAR Establishment Day" },
120 + { "date": "2026-09-26", "name": "Day after Mid-Autumn Festival" },
121 + { "date": "2026-10-01", "name": "National Day" },
122 + { "date": "2026-10-19", "name": "Chung Yeung Festival (observed)" },
123 + { "date": "2026-12-25", "name": "Christmas Day" },
124 + { "date": "2026-12-26", "name": "First weekday after Christmas" }
125 + ] },
126 + { "exchanges": ["xasx"], "holidays": [
127 + { "date": "2026-01-01", "name": "New Year's Day" },
128 + { "date": "2026-01-26", "name": "Australia Day" },
129 + { "date": "2026-04-03", "name": "Good Friday" },
130 + { "date": "2026-04-06", "name": "Easter Monday" },
131 + { "date": "2026-04-27", "name": "Anzac Day (observed)" },
132 + { "date": "2026-06-08", "name": "King's Birthday" },
133 + { "date": "2026-12-25", "name": "Christmas Day" },
134 + { "date": "2026-12-28", "name": "Boxing Day (observed)" }
135 + ] }
136 +]
added apps/api/src/seed/index.ts +61 −0
@@ -0,0 +1,61 @@
1 +import { readFileSync } from "node:fs";
2 +import { dirname, join } from "node:path";
3 +import { fileURLToPath } from "node:url";
4 +import { pool } from "../db/pool.js";
5 +import { logger } from "../logger.js";
6 +
7 +function seedDir(): string {
8 + // Works from src (tsx) and from dist (bundled): seeds are copied next to dist by the build.
9 + const here = dirname(fileURLToPath(import.meta.url));
10 + for (const c of [here, join(here, "seed"), join(here, "../seed"), join(here, "../src/seed")]) {
11 + try {
12 + readFileSync(join(c, "countries.json"));
13 + return c;
14 + } catch {
15 + /* try next */
16 + }
17 + }
18 + throw new Error("seed directory not found");
19 +}
20 +
21 +/** Idempotent reference data: countries, exchanges, holidays. Instruments are seeded by connectors. */
22 +export async function seed(): Promise<void> {
23 + const dir = seedDir();
24 + const countries = JSON.parse(readFileSync(join(dir, "countries.json"), "utf8")) as Array<Record<string, unknown>>;
25 + const exchanges = JSON.parse(readFileSync(join(dir, "exchanges.json"), "utf8")) as Array<Record<string, any>>;
26 + const holidayGroups = JSON.parse(readFileSync(join(dir, "holidays.json"), "utf8")) as Array<{ exchanges: string[]; holidays: Array<Record<string, any>> }>;
27 +
28 + for (const c of countries) {
29 + await pool.query(
30 + `insert into countries (code, name, region, currency) values ($1,$2,$3,$4) on conflict (code) do update set name = excluded.name, region = excluded.region, currency = excluded.currency`,
31 + [c.code, c.name, c.region, c.currency],
32 + );
33 + }
34 + for (const e of exchanges) {
35 + await pool.query(
36 + `insert into exchanges (id, mic, name, operator, country, city, timezone, currency, website, sessions, lat, lon, asset_classes)
37 + values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13)
38 + on conflict (id) do update set mic = excluded.mic, name = excluded.name, operator = excluded.operator, country = excluded.country, city = excluded.city, timezone = excluded.timezone,
39 + currency = excluded.currency, website = excluded.website, sessions = excluded.sessions, lat = excluded.lat, lon = excluded.lon, asset_classes = excluded.asset_classes, updated_at = now()`,
40 + [e.id, e.mic, e.name, e.operator, e.country, e.city, e.timezone, e.currency, e.website, JSON.stringify(e.sessions), e.lat, e.lon, e.asset_classes],
41 + );
42 + }
43 + let holidays = 0;
44 + for (const g of holidayGroups) {
45 + for (const ex of g.exchanges) {
46 + for (const h of g.holidays) {
47 + await pool.query(
48 + `insert into exchange_holidays (exchange_id, date, name, kind, close_time, source_id) values ($1,$2,$3,$4,$5,'market-atlas-seed')
49 + on conflict (exchange_id, date) do update set name = excluded.name, kind = excluded.kind, close_time = excluded.close_time where exchange_holidays.source_id = 'market-atlas-seed'`,
50 + [ex, h.date, h.name, h.kind ?? "CLOSED", h.closeTime ?? null],
51 + );
52 + holidays++;
53 + }
54 + }
55 + }
56 + await pool.query(
57 + `insert into sources (id, name, organization, source_type, homepage, jurisdiction, rights_status, realtime_status, family, category, enabled)
58 + values ('market-atlas-calendar','Market Atlas market-hours engine','Market Atlas','OFFICIAL_API',null,null,'INTERNAL_ONLY','REALTIME',null,'INTERNAL',true) on conflict (id) do nothing`,
59 + );
60 + logger.info({ countries: countries.length, exchanges: exchanges.length, holidays }, "seed applied");
61 +}
added apps/web/next.config.ts +56 −0
@@ -0,0 +1,56 @@
1 +import type { NextConfig } from "next";
2 +import { existsSync } from "node:fs";
3 +import path from "node:path";
4 +
5 +// Monorepo: a single `.env` lives at the repository root; Next only reads the app directory.
6 +for (const candidate of [path.resolve(process.cwd(), "../../.env"), path.resolve(process.cwd(), ".env")]) {
7 + if (existsSync(candidate)) {
8 + try {
9 + process.loadEnvFile(candidate);
10 + } catch {
11 + /* ignore malformed env */
12 + }
13 + }
14 +}
15 +
16 +// Dev: API on :8391 (repo .env). Production: the API process is the public edge (:8380) and proxies non-/v1 paths here (:8382).
17 +const API_URL = (process.env.API_URL ?? "http://127.0.0.1:8391").replace(/\/$/, "");
18 +const isDevApi = /:8391$/.test(API_URL) || /localhost|127\.0\.0\.1/.test(API_URL) && process.env.NODE_ENV !== "production";
19 +
20 +const nextConfig: NextConfig = {
21 + distDir: process.env.NEXT_DIST_DIR || ".next",
22 + reactStrictMode: true,
23 + poweredByHeader: false,
24 + allowedDevOrigins: ["127.0.0.1", "localhost"],
25 + outputFileTracingRoot: path.resolve(__dirname, "../.."),
26 + experimental: { optimizePackageImports: ["lucide-react"] },
27 + env: {
28 + // In development the browser talks to the API directly for WebSocket (rewrites do not proxy upgrades).
29 + NEXT_PUBLIC_WS_URL: process.env.NEXT_PUBLIC_WS_URL ?? (isDevApi ? API_URL.replace(/^http/, "ws") + "/v1/stream" : ""),
30 + NEXT_PUBLIC_SITE_URL: process.env.NEXT_PUBLIC_SITE_URL ?? process.env.MA_SITE_URL ?? "https://www.market-atlas.co",
31 + },
32 + async rewrites() {
33 + return [{ source: "/v1/:path*", destination: `${API_URL}/v1/:path*` }];
34 + },
35 + async headers() {
36 + const PUBLIC_CACHE = { key: "Cache-Control", value: "public, s-maxage=60, stale-while-revalidate=600" };
37 + const NO_STORE = { key: "Cache-Control", value: "private, no-store" };
38 + return [
39 + {
40 + source: "/(.*)",
41 + headers: [
42 + { key: "X-Content-Type-Options", value: "nosniff" },
43 + { key: "Referrer-Policy", value: "strict-origin-when-cross-origin" },
44 + { key: "X-Frame-Options", value: "SAMEORIGIN" },
45 + { key: "Permissions-Policy", value: "camera=(), microphone=(), geolocation=()" },
46 + ],
47 + },
48 + { source: "/methodology", headers: [PUBLIC_CACHE] },
49 + { source: "/developers", headers: [PUBLIC_CACHE] },
50 + { source: "/licensing", headers: [PUBLIC_CACHE] },
51 + { source: "/admin/:path*", headers: [NO_STORE] },
52 + ];
53 + },
54 +};
55 +
56 +export default nextConfig;
added apps/web/package.json +34 −0
@@ -0,0 +1,34 @@
1 +{
2 + "name": "@market-atlas/web",
3 + "version": "0.1.0",
4 + "private": true,
5 + "scripts": {
6 + "dev": "next dev -p 8390",
7 + "build": "next build",
8 + "start": "next start -p 8382 -H 127.0.0.1",
9 + "typecheck": "tsc -p tsconfig.json --noEmit",
10 + "qa": "node qa/screens.mjs"
11 + },
12 + "dependencies": {
13 + "d3-geo": "^3.1.1",
14 + "geist": "^1.5.1",
15 + "lightweight-charts": "^5.0.8",
16 + "lucide-react": "^1.0.0",
17 + "next": "16.3.4",
18 + "react": "19.2.8",
19 + "react-dom": "19.2.8",
20 + "server-only": "^0.0.1",
21 + "topojson-client": "^3.1.0",
22 + "world-atlas": "^2.0.2"
23 + },
24 + "devDependencies": {
25 + "@tailwindcss/postcss": "^4",
26 + "@types/d3-geo": "^3.1.0",
27 + "@types/node": "^24.0.0",
28 + "@types/react": "^19",
29 + "@types/react-dom": "^19",
30 + "@types/topojson-client": "^3.1.5",
31 + "tailwindcss": "^4",
32 + "typescript": "^5.9.3"
33 + }
34 +}
added apps/web/postcss.config.mjs +7 −0
@@ -0,0 +1,7 @@
1 +const config = {
2 + plugins: {
3 + "@tailwindcss/postcss": {},
4 + },
5 +};
6 +
7 +export default config;
added apps/web/src/app/globals.css +289 −0
@@ -0,0 +1,289 @@
1 +@import "tailwindcss";
2 +
3 +/*
4 + Market Atlas design tokens. Institutional, dense but breathable. Two themes on <html data-theme>.
5 + Light: cool paper canvas, near-black ink, atlas blue accent. Dark: graphite (never pure black), off-white ink.
6 + Semantic colours: positive / negative / neutral / stale / warning — always paired with text or an icon.
7 +*/
8 +:root,
9 +[data-theme="light"] {
10 + color-scheme: light;
11 + --canvas: #f5f6f4;
12 + --surface: #ffffff;
13 + --surface-2: #edeeeb;
14 + --surface-3: #e1e3de;
15 + --ink: #0f1217;
16 + --ink-2: #4a5062;
17 + --ink-3: #7b8294;
18 + --rule: rgba(15, 18, 23, 0.1);
19 + --rule-strong: rgba(15, 18, 23, 0.22);
20 + --accent: #1d4fd8;
21 + --accent-ink: #ffffff;
22 + --accent-soft: rgba(29, 79, 216, 0.1);
23 + --positive: #167a48;
24 + --positive-soft: rgba(22, 122, 72, 0.12);
25 + --negative: #c0323c;
26 + --negative-soft: rgba(192, 50, 60, 0.12);
27 + --warning: #b16d0b;
28 + --warning-soft: rgba(212, 130, 22, 0.14);
29 + --stale: #7b8294;
30 + --stale-soft: rgba(123, 130, 148, 0.14);
31 + --map-land: #e1e3de;
32 + --map-stroke: #ffffff;
33 + --brand-plate: #0f1217;
34 + --brand-ink: #f5f6f4;
35 + --series-1: #1d4fd8;
36 + --series-2: #c2560f;
37 + --series-3: #167a48;
38 + --series-4: #6d3fc7;
39 + --series-5: #0d7c86;
40 + --series-6: #b6337e;
41 + --series-7: #7b8294;
42 + --series-8: #c0323c;
43 + --header-h: 56px;
44 + --tabbar-h: 58px;
45 + --radius: 4px;
46 + --radius-lg: 8px;
47 +}
48 +[data-theme="dark"] {
49 + color-scheme: dark;
50 + --canvas: #0c0f14;
51 + --surface: #12161d;
52 + --surface-2: #191e27;
53 + --surface-3: #222834;
54 + --ink: #e8eaef;
55 + --ink-2: #a2a8b7;
56 + --ink-3: #6e7587;
57 + --rule: rgba(190, 200, 225, 0.11);
58 + --rule-strong: rgba(190, 200, 225, 0.26);
59 + --accent: #6b93ff;
60 + --accent-ink: #061024;
61 + --accent-soft: rgba(107, 147, 255, 0.14);
62 + --positive: #43c37d;
63 + --positive-soft: rgba(67, 195, 125, 0.14);
64 + --negative: #ff6b74;
65 + --negative-soft: rgba(255, 107, 116, 0.14);
66 + --warning: #f2b04a;
67 + --warning-soft: rgba(242, 176, 74, 0.14);
68 + --stale: #8b92a5;
69 + --stale-soft: rgba(139, 146, 165, 0.16);
70 + --map-land: #222834;
71 + --map-stroke: #0c0f14;
72 + --brand-plate: #e8eaef;
73 + --brand-ink: #0c0f14;
74 + --series-1: #6b93ff;
75 + --series-2: #ff9a4d;
76 + --series-3: #43c37d;
77 + --series-4: #b08cff;
78 + --series-5: #3fc3cf;
79 + --series-6: #ff7ab6;
80 + --series-7: #9aa3b8;
81 + --series-8: #ff6b74;
82 +}
83 +
84 +@theme inline {
85 + --color-canvas: var(--canvas);
86 + --color-surface: var(--surface);
87 + --color-surface-2: var(--surface-2);
88 + --color-surface-3: var(--surface-3);
89 + --color-ink: var(--ink);
90 + --color-ink-2: var(--ink-2);
91 + --color-ink-3: var(--ink-3);
92 + --color-rule: var(--rule);
93 + --color-rule-strong: var(--rule-strong);
94 + --color-accent: var(--accent);
95 + --color-accent-ink: var(--accent-ink);
96 + --color-accent-soft: var(--accent-soft);
97 + --color-positive: var(--positive);
98 + --color-positive-soft: var(--positive-soft);
99 + --color-negative: var(--negative);
100 + --color-negative-soft: var(--negative-soft);
101 + --color-warning: var(--warning);
102 + --color-warning-soft: var(--warning-soft);
103 + --color-stale: var(--stale);
104 + --color-stale-soft: var(--stale-soft);
105 + --font-sans: var(--font-geist-sans), ui-sans-serif, system-ui, sans-serif;
106 + --font-mono: var(--font-geist-mono), ui-monospace, SFMono-Regular, Menlo, monospace;
107 + --radius-sm: var(--radius);
108 + --radius-md: var(--radius-lg);
109 +}
110 +
111 +html {
112 + background: var(--canvas);
113 + color: var(--ink);
114 + font-size: 15px;
115 + -webkit-text-size-adjust: 100%;
116 + scroll-padding-top: calc(var(--header-h) + 12px);
117 +}
118 +body {
119 + font-family: var(--font-sans);
120 + font-feature-settings: "cv11", "ss01";
121 + overflow-x: hidden;
122 +}
123 +.tnum {
124 + font-variant-numeric: tabular-nums;
125 +}
126 +.mono {
127 + font-family: var(--font-mono);
128 + font-variant-numeric: tabular-nums;
129 +}
130 +
131 +/* Surfaces & rules */
132 +.panel {
133 + background: var(--surface);
134 + border: 1px solid var(--rule);
135 + border-radius: var(--radius-lg);
136 +}
137 +.rule-b {
138 + border-bottom: 1px solid var(--rule);
139 +}
140 +.rule-t {
141 + border-top: 1px solid var(--rule);
142 +}
143 +
144 +/* Price flash — subtle, directional, 600 ms */
145 +@keyframes flash-up {
146 + 0% {
147 + background-color: var(--positive-soft);
148 + }
149 + 100% {
150 + background-color: transparent;
151 + }
152 +}
153 +@keyframes flash-down {
154 + 0% {
155 + background-color: var(--negative-soft);
156 + }
157 + 100% {
158 + background-color: transparent;
159 + }
160 +}
161 +.flash-up {
162 + animation: flash-up 0.6s ease-out;
163 +}
164 +.flash-down {
165 + animation: flash-down 0.6s ease-out;
166 +}
167 +
168 +/* Live dot */
169 +@keyframes pulse-dot {
170 + 0%,
171 + 100% {
172 + opacity: 1;
173 + }
174 + 50% {
175 + opacity: 0.35;
176 + }
177 +}
178 +.live-dot {
179 + animation: pulse-dot 1.6s ease-in-out infinite;
180 +}
181 +
182 +/* Dense tables */
183 +.table-dense {
184 + width: 100%;
185 + border-collapse: separate;
186 + border-spacing: 0;
187 + font-size: 0.8125rem;
188 +}
189 +.table-dense th {
190 + position: sticky;
191 + top: 0;
192 + z-index: 1;
193 + background: var(--surface);
194 + color: var(--ink-3);
195 + font-weight: 500;
196 + font-size: 0.6875rem;
197 + letter-spacing: 0.04em;
198 + text-transform: uppercase;
199 + text-align: left;
200 + padding: 0.5rem 0.625rem;
201 + border-bottom: 1px solid var(--rule-strong);
202 + white-space: nowrap;
203 +}
204 +.table-dense td {
205 + padding: 0.45rem 0.625rem;
206 + border-bottom: 1px solid var(--rule);
207 + white-space: nowrap;
208 + vertical-align: middle;
209 +}
210 +.table-dense tbody tr:hover {
211 + background: var(--surface-2);
212 +}
213 +.table-dense .num {
214 + text-align: right;
215 + font-family: var(--font-mono);
216 + font-variant-numeric: tabular-nums;
217 +}
218 +
219 +/* Scrollbars */
220 +* {
221 + scrollbar-width: thin;
222 + scrollbar-color: var(--rule-strong) transparent;
223 +}
224 +
225 +/* Prose (methodology, developers…) */
226 +.prose-ma h2 {
227 + font-size: 1.125rem;
228 + font-weight: 600;
229 + margin-top: 2rem;
230 + margin-bottom: 0.5rem;
231 +}
232 +.prose-ma h3 {
233 + font-size: 0.95rem;
234 + font-weight: 600;
235 + margin-top: 1.25rem;
236 + margin-bottom: 0.35rem;
237 +}
238 +.prose-ma p,
239 +.prose-ma li {
240 + color: var(--ink-2);
241 + line-height: 1.6;
242 + margin-bottom: 0.6rem;
243 +}
244 +.prose-ma ul {
245 + list-style: disc;
246 + padding-left: 1.25rem;
247 +}
248 +.prose-ma code {
249 + font-family: var(--font-mono);
250 + font-size: 0.8em;
251 + background: var(--surface-2);
252 + padding: 0.1em 0.35em;
253 + border-radius: 3px;
254 +}
255 +.prose-ma pre {
256 + font-family: var(--font-mono);
257 + font-size: 0.78rem;
258 + background: var(--surface-2);
259 + border: 1px solid var(--rule);
260 + border-radius: var(--radius-lg);
261 + padding: 0.85rem 1rem;
262 + overflow-x: auto;
263 + margin: 0.75rem 0 1rem;
264 + line-height: 1.5;
265 +}
266 +.prose-ma pre code {
267 + background: none;
268 + padding: 0;
269 +}
270 +.prose-ma table {
271 + width: 100%;
272 + font-size: 0.85rem;
273 + margin: 0.5rem 0 1rem;
274 +}
275 +.prose-ma th,
276 +.prose-ma td {
277 + text-align: left;
278 + padding: 0.4rem 0.6rem;
279 + border-bottom: 1px solid var(--rule);
280 + vertical-align: top;
281 +}
282 +
283 +@media (prefers-reduced-motion: reduce) {
284 + .flash-up,
285 + .flash-down,
286 + .live-dot {
287 + animation: none;
288 + }
289 +}
added apps/web/src/app/icon.svg +1 −0
@@ -0,0 +1 @@
1 +<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32"><rect x="1" y="1" width="30" height="30" rx="7" fill="#0f1217"/><g stroke="#f5f6f4" stroke-width="1.2" fill="none" opacity="0.85"><circle cx="16" cy="16" r="9.5"/><ellipse cx="16" cy="16" rx="4.2" ry="9.5"/><line x1="6.5" y1="16" x2="25.5" y2="16"/><path d="M8.2 11.2 Q16 13.6 23.8 11.2"/><path d="M8.2 20.8 Q16 18.4 23.8 20.8"/></g><circle cx="20.5" cy="12.6" r="2.2" fill="#6b93ff"/></svg>
added apps/web/src/app/layout.tsx +55 −0
@@ -0,0 +1,55 @@
1 +import type { Metadata, Viewport } from "next";
2 +import Script from "next/script";
3 +import "./globals.css";
4 +import { MobileTabBar } from "@/components/layout/mobile-tab-bar";
5 +import { SearchProvider } from "@/components/layout/search-dialog";
6 +import { SiteFooter } from "@/components/layout/site-footer";
7 +import { SiteHeader } from "@/components/layout/site-header";
8 +import { fontMono, fontUi } from "@/lib/fonts";
9 +import { THEME_SCRIPT } from "@/lib/prepaint";
10 +import { DESCRIPTION, SITE_NAME, SITE_URL, TAGLINE, THEME_DARK, THEME_LIGHT } from "@/lib/site";
11 +
12 +export const metadata: Metadata = {
13 + metadataBase: new URL(SITE_URL),
14 + title: { default: `${SITE_NAME} — ${TAGLINE}`, template: `%s | ${SITE_NAME}` },
15 + description: DESCRIPTION,
16 + applicationName: SITE_NAME,
17 + robots: { index: true, follow: true },
18 + alternates: { canonical: "/" },
19 + openGraph: { type: "website", siteName: SITE_NAME, url: SITE_URL, title: `${SITE_NAME} — ${TAGLINE}`, description: DESCRIPTION },
20 + twitter: { card: "summary_large_image", title: `${SITE_NAME} — ${TAGLINE}`, description: DESCRIPTION },
21 + icons: { icon: [{ url: "/icon.svg", type: "image/svg+xml" }] },
22 +};
23 +
24 +export const viewport: Viewport = {
25 + width: "device-width",
26 + initialScale: 1,
27 + viewportFit: "cover",
28 + themeColor: [
29 + { media: "(prefers-color-scheme: light)", color: THEME_LIGHT },
30 + { media: "(prefers-color-scheme: dark)", color: THEME_DARK },
31 + ],
32 +};
33 +
34 +export default function RootLayout({ children }: { children: React.ReactNode }) {
35 + return (
36 + <html lang="en" className={`${fontUi.variable} ${fontMono.variable} h-full antialiased`} suppressHydrationWarning>
37 + <head>
38 + <Script id="ma-theme" strategy="beforeInteractive" dangerouslySetInnerHTML={{ __html: THEME_SCRIPT }} />
39 + </head>
40 + <body className="flex min-h-full flex-col pb-[calc(var(--tabbar-h)+env(safe-area-inset-bottom,0px))] lg:pb-0">
41 + <a href="#main" className="sr-only focus:not-sr-only focus:fixed focus:left-3 focus:top-3 focus:z-[200] focus:bg-accent focus:px-3 focus:py-2 focus:text-sm focus:text-accent-ink">
42 + Skip to content
43 + </a>
44 + <SearchProvider>
45 + <SiteHeader />
46 + <main id="main" className="flex-1">
47 + {children}
48 + </main>
49 + <SiteFooter />
50 + <MobileTabBar />
51 + </SearchProvider>
52 + </body>
53 + </html>
54 + );
55 +}
added apps/web/src/components/layout/logo.tsx +29 −0
@@ -0,0 +1,29 @@
1 +import { cx } from "@/lib/format";
2 +
3 +/** Atlas mark: a globe grid on a solid plate with a bright "now" node. */
4 +export function LogoMark({ size = 24, className }: { size?: number; className?: string }) {
5 + return (
6 + <svg width={size} height={size} viewBox="0 0 32 32" aria-hidden="true" className={cx("shrink-0", className)}>
7 + <rect x="1" y="1" width="30" height="30" rx="7" fill="var(--brand-plate)" />
8 + <g stroke="var(--brand-ink)" strokeWidth="1.2" fill="none" opacity="0.85">
9 + <circle cx="16" cy="16" r="9.5" />
10 + <ellipse cx="16" cy="16" rx="4.2" ry="9.5" />
11 + <line x1="6.5" y1="16" x2="25.5" y2="16" />
12 + <path d="M8.2 11.2 Q16 13.6 23.8 11.2" />
13 + <path d="M8.2 20.8 Q16 18.4 23.8 20.8" />
14 + </g>
15 + <circle cx="20.5" cy="12.6" r="2.2" fill="var(--accent)" />
16 + </svg>
17 + );
18 +}
19 +
20 +export function Wordmark({ className }: { className?: string }) {
21 + return (
22 + <span className={cx("inline-flex items-center gap-2 font-semibold tracking-tight", className)}>
23 + <LogoMark />
24 + <span>
25 + Market <span className="text-ink-2 font-medium">Atlas</span>
26 + </span>
27 + </span>
28 + );
29 +}
added apps/web/src/components/layout/market-clock.tsx +69 −0
@@ -0,0 +1,69 @@
1 +"use client";
2 +
3 +import { useEffect, useState } from "react";
4 +import { clientApi } from "@/lib/client-api";
5 +import { useNow } from "@/lib/stream";
6 +import { cx } from "@/lib/format";
7 +import type { Exchange } from "@/lib/types";
8 +
9 +const CLOCKS: Array<{ id: string; label: string; tz: string }> = [
10 + { id: "xnys", label: "New York", tz: "America/New_York" },
11 + { id: "xlon", label: "London", tz: "Europe/London" },
12 + { id: "xetr", label: "Frankfurt", tz: "Europe/Berlin" },
13 + { id: "xtks", label: "Tokyo", tz: "Asia/Tokyo" },
14 + { id: "xhkg", label: "Hong Kong", tz: "Asia/Hong_Kong" },
15 + { id: "xasx", label: "Sydney", tz: "Australia/Sydney" },
16 +];
17 +
18 +const fmts = new Map<string, Intl.DateTimeFormat>();
19 +const timeIn = (tz: string, now: number) => {
20 + let f = fmts.get(tz);
21 + if (!f) {
22 + f = new Intl.DateTimeFormat("en-GB", { timeZone: tz, hour: "2-digit", minute: "2-digit", hourCycle: "h23" });
23 + fmts.set(tz, f);
24 + }
25 + return f.format(new Date(now));
26 +};
27 +
28 +export const STATE_DOT: Record<string, string> = {
29 + OPEN: "bg-positive",
30 + PRE: "bg-warning",
31 + POST: "bg-warning",
32 + AUCTION: "bg-warning",
33 + HALTED: "bg-negative",
34 + CLOSED: "bg-stale",
35 + UNKNOWN: "bg-stale",
36 +};
37 +
38 +/** Global clock strip: local time and session state of the main market centres (refreshed every 60 s). */
39 +export function MarketClock({ className }: { className?: string }) {
40 + const now = useNow(1000);
41 + const [states, setStates] = useState<Record<string, string>>({});
42 + useEffect(() => {
43 + let alive = true;
44 + const load = () =>
45 + clientApi<Exchange[]>("/v1/exchanges")
46 + .then((xs) => alive && setStates(Object.fromEntries(xs.map((x) => [x.id, x.status?.state ?? "UNKNOWN"]))))
47 + .catch(() => {});
48 + load();
49 + const t = setInterval(load, 60_000);
50 + return () => {
51 + alive = false;
52 + clearInterval(t);
53 + };
54 + }, []);
55 + return (
56 + <div className={cx("flex items-center gap-4 overflow-x-auto whitespace-nowrap text-[11px] text-ink-3 [-ms-overflow-style:none] [scrollbar-width:none] [&::-webkit-scrollbar]:hidden", className)}>
57 + {CLOCKS.map((c) => {
58 + const st = states[c.id] ?? "UNKNOWN";
59 + return (
60 + <span key={c.id} className="inline-flex items-center gap-1.5" title={`${c.label}: ${st.toLowerCase()}`}>
61 + <span className={cx("inline-block h-1.5 w-1.5 rounded-full", STATE_DOT[st] ?? "bg-stale", st === "OPEN" && "live-dot")} />
62 + <span>{c.label}</span>
63 + <span className="mono text-ink-2">{now ? timeIn(c.tz, now) : "--:--"}</span>
64 + </span>
65 + );
66 + })}
67 + </div>
68 + );
69 +}
added apps/web/src/components/layout/mobile-tab-bar.tsx +46 −0
@@ -0,0 +1,46 @@
1 +"use client";
2 +
3 +import { Activity, Globe2, LineChart, Radio, Search } from "lucide-react";
4 +import Link from "next/link";
5 +import { usePathname } from "next/navigation";
6 +import { cx } from "@/lib/format";
7 +import { useSearch } from "./search-dialog";
8 +
9 +const TABS = [
10 + { href: "/markets", label: "Markets", icon: LineChart, match: /^\/(markets|stocks|etfs|indices|crypto|forex|rates|commodities|instruments|compare)/ },
11 + { href: "/live", label: "Live", icon: Radio, match: /^\/live/ },
12 + { href: "#search", label: "Search", icon: Search, match: /^\/search/ },
13 + { href: "/events", label: "Events", icon: Activity, match: /^\/(events|halts|filings)/ },
14 + { href: "/exchanges", label: "World", icon: Globe2, match: /^\/(exchanges|countries)/ },
15 +];
16 +
17 +export function MobileTabBar() {
18 + const path = usePathname();
19 + const { open } = useSearch();
20 + return (
21 + <nav aria-label="Mobile" className="fixed inset-x-0 bottom-0 z-[100] border-t border-rule bg-surface/95 pb-[env(safe-area-inset-bottom,0px)] backdrop-blur lg:hidden">
22 + <ul className="grid h-[var(--tabbar-h)] grid-cols-5">
23 + {TABS.map((t) => {
24 + const active = t.match.test(path);
25 + const cls = cx("flex h-full flex-col items-center justify-center gap-0.5 text-[10.5px]", active ? "text-accent" : "text-ink-3");
26 + const Icon = t.icon;
27 + return (
28 + <li key={t.href}>
29 + {t.href === "#search" ? (
30 + <button type="button" onClick={open} className={cx(cls, "w-full")}>
31 + <Icon size={19} />
32 + {t.label}
33 + </button>
34 + ) : (
35 + <Link href={t.href} className={cls}>
36 + <Icon size={19} />
37 + {t.label}
38 + </Link>
39 + )}
40 + </li>
41 + );
42 + })}
43 + </ul>
44 + </nav>
45 + );
46 +}
added apps/web/src/components/layout/search-dialog.tsx +172 −0
@@ -0,0 +1,172 @@
1 +"use client";
2 +
3 +import { Search, X } from "lucide-react";
4 +import { useRouter } from "next/navigation";
5 +import { createContext, useCallback, useContext, useEffect, useMemo, useRef, useState, type ReactNode } from "react";
6 +import { clientApi } from "@/lib/client-api";
7 +import { ASSET_CLASS_LABEL, cx, formatPercent, formatQuoteValue, instrumentHref } from "@/lib/format";
8 +import type { SearchResult } from "@/lib/types";
9 +
10 +const Ctx = createContext<{ open: () => void }>({ open: () => {} });
11 +export const useSearch = () => useContext(Ctx);
12 +
13 +interface Hit {
14 + group: string;
15 + href: string;
16 + title: string;
17 + subtitle: string;
18 + right?: string;
19 + rightTone?: "pos" | "neg" | "neutral";
20 +}
21 +
22 +function toHits(r: SearchResult): Hit[] {
23 + return r.results.map(({ group, item }) => {
24 + const it = item as Record<string, any>;
25 + if (group === "EXCHANGE") return { group, href: `/exchanges/${it.id}`, title: it.name, subtitle: `${it.mic ?? ""} · ${it.country} · ${String(it.status).toLowerCase()}`, right: "" };
26 + if (group === "COUNTRY") return { group, href: `/countries/${it.code}`, title: it.name, subtitle: it.region, right: "" };
27 + const q = it.quote as Record<string, any> | null;
28 + const pct = q?.change_percent as number | null | undefined;
29 + return {
30 + group,
31 + href: instrumentHref(it.id),
32 + title: it.symbol,
33 + subtitle: `${it.name} · ${ASSET_CLASS_LABEL[it.asset_class] ?? it.asset_class}${it.exchange_id ? ` · ${String(it.exchange_id).toUpperCase()}` : ""}`,
34 + right: q?.price != null ? `${formatQuoteValue(q.price, it.asset_class, q.currency)} ${formatPercent(pct)}` : "",
35 + rightTone: pct == null ? "neutral" : pct > 0 ? "pos" : pct < 0 ? "neg" : "neutral",
36 + };
37 + });
38 +}
39 +
40 +export function SearchProvider({ children }: { children: ReactNode }) {
41 + const [open, setOpen] = useState(false);
42 + const [q, setQ] = useState("");
43 + const [hits, setHits] = useState<Hit[]>([]);
44 + const [idx, setIdx] = useState(0);
45 + const [loading, setLoading] = useState(false);
46 + const router = useRouter();
47 + const inputRef = useRef<HTMLInputElement>(null);
48 + const seq = useRef(0);
49 +
50 + useEffect(() => {
51 + const onKey = (e: KeyboardEvent) => {
52 + if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === "k") {
53 + e.preventDefault();
54 + setOpen((o) => !o);
55 + } else if (e.key === "Escape") setOpen(false);
56 + };
57 + window.addEventListener("keydown", onKey);
58 + return () => window.removeEventListener("keydown", onKey);
59 + }, []);
60 +
61 + useEffect(() => {
62 + if (!open) return;
63 + setTimeout(() => inputRef.current?.focus(), 20);
64 + }, [open]);
65 +
66 + useEffect(() => {
67 + if (!open) return;
68 + const term = q.trim();
69 + if (!term) {
70 + setHits([]);
71 + return;
72 + }
73 + const my = ++seq.current;
74 + setLoading(true);
75 + const t = setTimeout(() => {
76 + clientApi<SearchResult>(`/v1/search?q=${encodeURIComponent(term)}&limit=14`)
77 + .then((r) => {
78 + if (my !== seq.current) return;
79 + setHits(toHits(r));
80 + setIdx(0);
81 + })
82 + .catch(() => {})
83 + .finally(() => my === seq.current && setLoading(false));
84 + }, 140);
85 + return () => clearTimeout(t);
86 + }, [q, open]);
87 +
88 + const go = useCallback(
89 + (h: Hit | undefined) => {
90 + if (!h) {
91 + if (q.trim()) router.push(`/search?q=${encodeURIComponent(q.trim())}`);
92 + } else router.push(h.href);
93 + setOpen(false);
94 + setQ("");
95 + },
96 + [router, q],
97 + );
98 +
99 + const ctx = useMemo(() => ({ open: () => setOpen(true) }), []);
100 + return (
101 + <Ctx.Provider value={ctx}>
102 + {children}
103 + {open && (
104 + <div role="dialog" aria-modal="true" aria-label="Search" data-palette className="fixed inset-0 z-[150] flex items-start justify-center bg-black/40 p-3 pt-[10vh] backdrop-blur-[2px]" onClick={() => setOpen(false)}>
105 + <div className="panel w-full max-w-xl overflow-hidden shadow-2xl" onClick={(e) => e.stopPropagation()}>
106 + <div className="flex items-center gap-2 border-b border-rule px-3">
107 + <Search size={16} className="text-ink-3" />
108 + <input
109 + ref={inputRef}
110 + value={q}
111 + onChange={(e) => setQ(e.target.value)}
112 + onKeyDown={(e) => {
113 + if (e.key === "ArrowDown") {
114 + e.preventDefault();
115 + setIdx((i) => Math.min(hits.length - 1, i + 1));
116 + } else if (e.key === "ArrowUp") {
117 + e.preventDefault();
118 + setIdx((i) => Math.max(0, i - 1));
119 + } else if (e.key === "Enter") go(hits[idx]);
120 + }}
121 + placeholder="Search AAPL, Bitcoin, S&P 500, Toronto Stock Exchange, US 10Y…"
122 + className="h-12 flex-1 bg-transparent text-[15px] outline-none placeholder:text-ink-3"
123 + autoComplete="off"
124 + />
125 + <button type="button" onClick={() => setOpen(false)} aria-label="Close" className="flex h-11 w-11 items-center justify-center text-ink-3 hover:text-ink">
126 + <X size={16} />
127 + </button>
128 + </div>
129 + <div className="max-h-[60vh] overflow-y-auto">
130 + {!q.trim() && (
131 + <div className="px-4 py-3 text-xs text-ink-3">
132 + Type a symbol, company, index, currency pair, exchange or country. <kbd className="mono rounded border border-rule px-1">↑↓</kbd> to move, <kbd className="mono rounded border border-rule px-1">↵</kbd> to open.
133 + </div>
134 + )}
135 + {q.trim() && !hits.length && !loading && <div className="px-4 py-6 text-center text-sm text-ink-3">No results for “{q}”.</div>}
136 + {hits.map((h, i) => (
137 + <button key={h.href + i} type="button" onMouseEnter={() => setIdx(i)} onClick={() => go(h)} className={cx("flex w-full items-center gap-3 px-4 py-2.5 text-left", i === idx ? "bg-accent-soft" : "hover:bg-surface-2")}>
138 + <span className="mono w-16 shrink-0 text-[10px] uppercase tracking-wide text-ink-3">{ASSET_CLASS_LABEL[h.group] ?? h.group.toLowerCase()}</span>
139 + <span className="min-w-0 flex-1">
140 + <span className="block truncate text-sm font-medium">{h.title}</span>
141 + <span className="block truncate text-xs text-ink-3">{h.subtitle}</span>
142 + </span>
143 + {h.right && <span className={cx("mono whitespace-pre text-xs", h.rightTone === "pos" ? "text-positive" : h.rightTone === "neg" ? "text-negative" : "text-ink-2")}>{h.right}</span>}
144 + </button>
145 + ))}
146 + {q.trim() && (
147 + <button type="button" onClick={() => go(undefined)} className="w-full border-t border-rule px-4 py-2.5 text-left text-xs text-accent hover:bg-surface-2">
148 + See all results for “{q}” →
149 + </button>
150 + )}
151 + </div>
152 + </div>
153 + </div>
154 + )}
155 + </Ctx.Provider>
156 + );
157 +}
158 +
159 +export function SearchButton({ className, compact }: { className?: string; compact?: boolean }) {
160 + const { open } = useSearch();
161 + return (
162 + <button type="button" onClick={open} className={cx("inline-flex h-11 items-center gap-2 rounded-md border border-rule bg-surface px-3 text-sm text-ink-3 hover:border-rule-strong hover:text-ink", className)} aria-label="Search">
163 + <Search size={15} />
164 + {!compact && (
165 + <>
166 + <span className="hidden sm:inline">Search markets</span>
167 + <kbd className="mono ml-2 hidden rounded border border-rule px-1 text-[10px] sm:inline">⌘K</kbd>
168 + </>
169 + )}
170 + </button>
171 + );
172 +}
added apps/web/src/components/layout/site-footer.tsx +45 −0
@@ -0,0 +1,45 @@
1 +import Link from "next/link";
2 +import { DISCLAIMER } from "@/lib/site";
3 +import { Wordmark } from "./logo";
4 +
5 +const COLS: Array<{ title: string; links: Array<[string, string]> }> = [
6 + { title: "Markets", links: [["/stocks", "Stocks"], ["/etfs", "ETFs"], ["/indices", "Indices"], ["/crypto", "Crypto"], ["/forex", "Forex"], ["/rates", "Rates"], ["/commodities", "Commodities"]] },
7 + { title: "Atlas", links: [["/live", "Live feed"], ["/exchanges", "Exchanges"], ["/countries", "Countries"], ["/events", "Events"], ["/halts", "Halts"], ["/filings", "Filings"], ["/compare", "Compare"]] },
8 + { title: "Data", links: [["/sources", "Sources"], ["/connectors", "Connectors"], ["/data-health", "Data health"], ["/status", "Status"], ["/methodology", "Methodology"], ["/licensing", "Data rights"], ["/developers", "Developers"]] },
9 +];
10 +
11 +export function SiteFooter() {
12 + return (
13 + <footer className="rule-t mt-12 bg-surface">
14 + <div className="mx-auto grid max-w-[1440px] gap-8 px-4 py-10 sm:px-5 md:grid-cols-[1.4fr_repeat(3,1fr)]">
15 + <div>
16 + <Wordmark />
17 + <p className="mt-3 max-w-sm text-xs leading-relaxed text-ink-3">{DISCLAIMER}</p>
18 + <p className="mt-3 text-xs text-ink-3">
19 + Prices may be delayed or end-of-day depending on the source; every quote carries its data status and provenance. See <Link href="/methodology" className="text-accent hover:underline">methodology</Link>.
20 + </p>
21 + </div>
22 + {COLS.map((c) => (
23 + <div key={c.title}>
24 + <h3 className="mb-2 text-[11px] font-medium uppercase tracking-wide text-ink-3">{c.title}</h3>
25 + <ul className="space-y-1.5 text-sm">
26 + {c.links.map(([href, label]) => (
27 + <li key={href}>
28 + <Link href={href} className="text-ink-2 hover:text-ink">
29 + {label}
30 + </Link>
31 + </li>
32 + ))}
33 + </ul>
34 + </div>
35 + ))}
36 + </div>
37 + <div className="rule-t">
38 + <div className="mx-auto flex max-w-[1440px] flex-wrap items-center justify-between gap-2 px-4 py-3 text-[11px] text-ink-3 sm:px-5">
39 + <span>© {new Date().getUTCFullYear()} Market Atlas · The live map of global markets.</span>
40 + <span>Data attributions: Coinbase, Kraken, Binance, OKX, Cboe (15-min delayed), ECB, Bank of Canada, U.S. Treasury, SEC EDGAR, Nasdaq Trader, HF Market Data.</span>
41 + </div>
42 + </div>
43 + </footer>
44 + );
45 +}
added apps/web/src/components/layout/site-header.tsx +53 −0
@@ -0,0 +1,53 @@
1 +"use client";
2 +
3 +import Link from "next/link";
4 +import { usePathname } from "next/navigation";
5 +import { cx } from "@/lib/format";
6 +import { Wordmark } from "./logo";
7 +import { MarketClock } from "./market-clock";
8 +import { SearchButton } from "./search-dialog";
9 +import { ThemeToggle } from "./theme-toggle";
10 +import { StreamIndicator } from "./stream-indicator";
11 +
12 +const NAV: Array<{ href: string; label: string; match?: RegExp }> = [
13 + { href: "/markets", label: "Markets", match: /^\/(markets|stocks|etfs|indices|crypto|forex|rates|commodities|instruments)/ },
14 + { href: "/live", label: "Live" },
15 + { href: "/events", label: "Events", match: /^\/(events|halts|filings)/ },
16 + { href: "/exchanges", label: "World", match: /^\/(exchanges|countries)/ },
17 + { href: "/sources", label: "Sources", match: /^\/(sources|connectors|data-health|status)/ },
18 + { href: "/compare", label: "Compare" },
19 +];
20 +
21 +export function SiteHeader() {
22 + const path = usePathname();
23 + return (
24 + <header className="sticky top-0 z-[100] border-b border-rule bg-canvas/85 backdrop-blur supports-[backdrop-filter]:bg-canvas/70">
25 + <div className="mx-auto flex h-[var(--header-h)] max-w-[1440px] items-center gap-3 px-3 sm:px-5">
26 + <Link href="/" className="flex h-11 items-center pr-1" aria-label="Market Atlas home">
27 + <Wordmark className="text-[15px]" />
28 + </Link>
29 + <nav className="hidden items-center gap-0.5 lg:flex" aria-label="Primary">
30 + {NAV.map((n) => {
31 + const active = (n.match ?? new RegExp(`^${n.href}`)).test(path);
32 + return (
33 + <Link key={n.href} href={n.href} className={cx("flex h-11 items-center rounded-md px-3 text-[13.5px]", active ? "text-ink font-medium" : "text-ink-2 hover:bg-surface-2 hover:text-ink")}>
34 + {n.label}
35 + </Link>
36 + );
37 + })}
38 + </nav>
39 + <div className="ml-auto flex items-center gap-1.5">
40 + <StreamIndicator className="hidden md:flex" />
41 + <SearchButton className="hidden sm:inline-flex" />
42 + <SearchButton className="sm:hidden" compact />
43 + <ThemeToggle />
44 + </div>
45 + </div>
46 + <div className="hidden border-t border-rule/60 md:block">
47 + <div className="mx-auto max-w-[1440px] px-5 py-1">
48 + <MarketClock />
49 + </div>
50 + </div>
51 + </header>
52 + );
53 +}
added apps/web/src/components/layout/stream-indicator.tsx +20 −0
@@ -0,0 +1,20 @@
1 +"use client";
2 +
3 +import { useSyncExternalStore } from "react";
4 +import { marketStream } from "@/lib/stream";
5 +import { cx } from "@/lib/format";
6 +
7 +export function StreamIndicator({ className }: { className?: string }) {
8 + const state = useSyncExternalStore(
9 + (l) => marketStream().listenState(l),
10 + () => marketStream().state,
11 + () => "idle" as const,
12 + );
13 + const label = state === "open" ? "Stream live" : state === "connecting" || state === "reconnecting" ? "Connecting…" : "Stream idle";
14 + return (
15 + <span className={cx("items-center gap-1.5 px-2 text-[11px] text-ink-3", className)} title="Market Atlas WebSocket stream">
16 + <span className={cx("inline-block h-1.5 w-1.5 rounded-full", state === "open" ? "bg-positive live-dot" : state === "idle" || state === "closed" ? "bg-stale" : "bg-warning")} />
17 + {label}
18 + </span>
19 + );
20 +}
added apps/web/src/components/layout/theme-toggle.tsx +28 −0
@@ -0,0 +1,28 @@
1 +"use client";
2 +
3 +import { Moon, Sun } from "lucide-react";
4 +import { useEffect, useState } from "react";
5 +import { THEME_KEY } from "@/lib/prepaint";
6 +
7 +export function ThemeToggle({ className }: { className?: string }) {
8 + const [theme, setTheme] = useState<"light" | "dark">("light");
9 + useEffect(() => {
10 + setTheme((document.documentElement.getAttribute("data-theme") as "light" | "dark") ?? "light");
11 + }, []);
12 + const toggle = () => {
13 + const next = theme === "dark" ? "light" : "dark";
14 + setTheme(next);
15 + document.documentElement.setAttribute("data-theme", next);
16 + document.documentElement.style.colorScheme = next;
17 + try {
18 + localStorage.setItem(THEME_KEY, next);
19 + } catch {
20 + /* ignore */
21 + }
22 + };
23 + return (
24 + <button type="button" onClick={toggle} aria-label={theme === "dark" ? "Switch to light mode" : "Switch to dark mode"} className={`inline-flex h-11 w-11 items-center justify-center rounded-md text-ink-2 hover:bg-surface-2 hover:text-ink ${className ?? ""}`}>
25 + {theme === "dark" ? <Sun size={17} /> : <Moon size={17} />}
26 + </button>
27 + );
28 +}
added apps/web/src/components/ui/confidence.tsx +20 −0
@@ -0,0 +1,20 @@
1 +import { cx } from "@/lib/format";
2 +
3 +/** Sources + confidence meter. Confidence is Market Atlas' consensus confidence (0–1), never a market prediction. */
4 +export function ConfidenceMeter({ confidence, sources, className, compact }: { confidence: number | null | undefined; sources: number | null | undefined; className?: string; compact?: boolean }) {
5 + const c = confidence ?? 0;
6 + const pct = Math.round(c * 100);
7 + const tone = c >= 0.85 ? "bg-positive" : c >= 0.6 ? "bg-accent" : c > 0 ? "bg-warning" : "bg-stale";
8 + return (
9 + <span className={cx("inline-flex items-center gap-1.5 whitespace-nowrap text-xs text-ink-2", className)} title={`Consensus confidence ${pct}% · ${sources ?? 0} independent source${sources === 1 ? "" : "s"}`}>
10 + <span className="inline-block h-1.5 w-10 overflow-hidden rounded-full bg-surface-3">
11 + <span className={cx("block h-full rounded-full", tone)} style={{ width: `${pct}%` }} />
12 + </span>
13 + {!compact && (
14 + <span className="mono tnum">
15 + {pct}% · {sources ?? 0} src
16 + </span>
17 + )}
18 + </span>
19 + );
20 +}
added apps/web/src/components/ui/freshness.tsx +60 −0
@@ -0,0 +1,60 @@
1 +"use client";
2 +
3 +import { cx, formatDateTime, formatDuration, relativeTime, STATUS_LABEL, toMs } from "@/lib/format";
4 +import { useLiveQuote, useNow } from "@/lib/stream";
5 +import type { Quote } from "@/lib/types";
6 +
7 +/**
8 + * Human freshness label, ticking every second:
9 + * REALTIME → "Updated 320 ms ago" · DELAYED → "Delayed 15 min · 12s ago" · AT_CLOSE → "At close · Fri 16:00 ET"
10 + * END_OF_DAY → "End of day · Sep 11" · STALE → "Stale · last 4h 12m ago" · WITHHELD → "Withheld (rights)".
11 + */
12 +export function FreshnessLabel({ quote, className, exchangeTz }: { quote: Quote | null | undefined; className?: string; exchangeTz?: string | null }) {
13 + const now = useNow(1000);
14 + const live = useLiveQuote(quote?.instrument_id);
15 + if (!quote) return <span className={cx("text-ink-3", className)}>No observation yet</span>;
16 + const status = quote.data_status;
17 + const liveTs = live && live.received >= Date.parse(quote.updated_at) ? live.timestamp : null;
18 + const srcTs = liveTs ?? toMs(quote.source_timestamp) ?? toMs(quote.updated_at);
19 + const age = srcTs != null && now ? Math.max(0, now - srcTs) : quote.freshness_ms;
20 + let text: string;
21 + switch (status) {
22 + case "REALTIME":
23 + text = age == null ? "Live" : age < 60_000 ? `Updated ${formatDuration(age)} ago` : `Live · last ${relativeTime(srcTs, now || Date.now())}`;
24 + break;
25 + case "DELAYED":
26 + text = `Delayed 15 min · ${relativeTime(srcTs, now || Date.now())}`;
27 + break;
28 + case "AT_CLOSE":
29 + text = `At close · ${formatDateTime(srcTs, { tz: exchangeTz ?? "America/New_York" })}`;
30 + break;
31 + case "END_OF_DAY":
32 + text = `End of day · ${formatDateTime(srcTs, { tz: "UTC", dateOnly: true })}`;
33 + break;
34 + case "STALE":
35 + text = `Stale · last update ${relativeTime(srcTs, now || Date.now())}`;
36 + break;
37 + case "WITHHELD":
38 + text = "Value withheld (data rights)";
39 + break;
40 + default:
41 + text = STATUS_LABEL[status] ?? status;
42 + }
43 + const tone = status === "REALTIME" ? "text-positive" : status === "STALE" || status === "WITHHELD" ? "text-stale" : status === "DELAYED" ? "text-warning" : "text-ink-3";
44 + return (
45 + <span className={cx("inline-flex items-center gap-1.5 whitespace-nowrap text-xs", tone, className)} suppressHydrationWarning>
46 + {status === "REALTIME" && <span className="live-dot inline-block h-1.5 w-1.5 rounded-full bg-positive" />}
47 + {text}
48 + </span>
49 + );
50 +}
51 +
52 +export function RelativeTime({ value, className }: { value: string | number | null | undefined; className?: string }) {
53 + const now = useNow(1000);
54 + const ms = typeof value === "number" ? value : toMs(value);
55 + return (
56 + <time className={cx("whitespace-nowrap", className)} dateTime={ms ? new Date(ms).toISOString() : undefined} title={ms ? formatDateTime(ms, { seconds: true }) : undefined} suppressHydrationWarning>
57 + {ms == null ? "—" : relativeTime(ms, now || Date.now())}
58 + </time>
59 + );
60 +}
added apps/web/src/components/ui/price.tsx +52 −0
@@ -0,0 +1,52 @@
1 +"use client";
2 +
3 +import { useEffect, useRef, useState } from "react";
4 +import { cx, formatChange, formatPercent, formatQuoteValue } from "@/lib/format";
5 +import { useLiveQuote } from "@/lib/stream";
6 +import type { Quote } from "@/lib/types";
7 +
8 +/** Change colour helpers — colour is supplementary; the sign is always in the text. */
9 +export const toneOf = (v: number | null | undefined) => (v == null || !Number.isFinite(v) || v === 0 ? "text-ink-2" : v > 0 ? "text-positive" : "text-negative");
10 +
11 +/**
12 + * Price cell that merges the server-rendered quote with the live stream (when the instrument is
13 + * subscribed by an ancestor). Flashes subtly on change. Never shows a live value when the
14 + * stream quote is older than the server snapshot.
15 + */
16 +export function LivePrice({ instrumentId, quote, assetClass, className, showChange = true, big = false }: { instrumentId: string; quote: Quote | null | undefined; assetClass?: string | null; className?: string; showChange?: boolean; big?: boolean }) {
17 + const live = useLiveQuote(instrumentId);
18 + const serverTs = quote ? Date.parse(quote.updated_at) : 0;
19 + const useLive = !!live && live.received >= serverTs && live.price != null;
20 + const price = useLive ? live.price : quote?.price ?? null;
21 + const change = useLive ? live.change : quote?.change ?? null;
22 + const pct = useLive ? live.change_pct : quote?.change_percent ?? null;
23 + const currency = useLive ? live.currency : quote?.currency;
24 + const prev = useRef<number | null>(null);
25 + const [flash, setFlash] = useState<"" | "flash-up" | "flash-down">("");
26 + useEffect(() => {
27 + if (price == null) return;
28 + if (prev.current != null && price !== prev.current) {
29 + setFlash(price > prev.current ? "flash-up" : "flash-down");
30 + const t = setTimeout(() => setFlash(""), 650);
31 + prev.current = price;
32 + return () => clearTimeout(t);
33 + }
34 + prev.current = price;
35 + }, [price]);
36 + if (price == null) return <span className={cx("mono text-ink-3", className)}>—</span>;
37 + return (
38 + <span className={cx("mono inline-flex items-baseline gap-2 rounded-[3px] px-0.5 tnum", flash, className)}>
39 + <span className={cx(big ? "text-3xl font-semibold tracking-tight sm:text-4xl" : "font-medium")}>{formatQuoteValue(price, assetClass, currency)}</span>
40 + {showChange && (
41 + <span className={cx(toneOf(pct), big ? "text-sm" : "text-xs")}>
42 + {big && change != null ? `${formatChange(change, assetClass)} ` : ""}
43 + {formatPercent(pct)}
44 + </span>
45 + )}
46 + </span>
47 + );
48 +}
49 +
50 +export function ChangeCell({ value, className }: { value: number | null | undefined; className?: string }) {
51 + return <span className={cx("mono tnum", toneOf(value), className)}>{formatPercent(value)}</span>;
52 +}
added apps/web/src/components/ui/status-badge.tsx +68 −0
@@ -0,0 +1,68 @@
1 +import { cx, STATUS_LABEL } from "@/lib/format";
2 +
3 +const TONE: Record<string, string> = {
4 + REALTIME: "bg-positive-soft text-positive",
5 + DELAYED: "bg-warning-soft text-warning",
6 + END_OF_DAY: "bg-surface-3 text-ink-2",
7 + AT_CLOSE: "bg-surface-3 text-ink-2",
8 + INDICATIVE: "bg-surface-3 text-ink-2",
9 + STALE: "bg-stale-soft text-stale",
10 + WITHHELD: "bg-stale-soft text-stale",
11 + UNKNOWN: "bg-stale-soft text-stale",
12 + // connector / market states
13 + HEALTHY: "bg-positive-soft text-positive",
14 + DEGRADED: "bg-warning-soft text-warning",
15 + RECONNECTING: "bg-warning-soft text-warning",
16 + STARTING: "bg-warning-soft text-warning",
17 + FAILED: "bg-negative-soft text-negative",
18 + PAUSED: "bg-stale-soft text-stale",
19 + DISABLED: "bg-stale-soft text-stale",
20 + NONE: "bg-stale-soft text-stale",
21 + OPEN: "bg-positive-soft text-positive",
22 + PRE: "bg-warning-soft text-warning",
23 + POST: "bg-warning-soft text-warning",
24 + AUCTION: "bg-warning-soft text-warning",
25 + HALTED: "bg-negative-soft text-negative",
26 + CLOSED: "bg-surface-3 text-ink-2",
27 + operational: "bg-positive-soft text-positive",
28 + degraded: "bg-warning-soft text-warning",
29 + outage: "bg-negative-soft text-negative",
30 + unknown: "bg-stale-soft text-stale",
31 + // severities
32 + INFO: "bg-surface-3 text-ink-2",
33 + NOTICE: "bg-accent-soft text-accent",
34 + WARNING: "bg-warning-soft text-warning",
35 + CRITICAL: "bg-negative-soft text-negative",
36 + DEBUG: "bg-stale-soft text-stale",
37 +};
38 +
39 +export function StatusBadge({ status, label, className, dot }: { status: string | null | undefined; label?: string; className?: string; dot?: boolean }) {
40 + const s = status ?? "UNKNOWN";
41 + const text = label ?? STATUS_LABEL[s] ?? s.replace(/_/g, " ").toLowerCase();
42 + return (
43 + <span className={cx("inline-flex h-5 items-center gap-1 whitespace-nowrap rounded-[3px] px-1.5 text-[10.5px] font-medium uppercase tracking-wide", TONE[s] ?? "bg-surface-3 text-ink-2", className)}>
44 + {dot && <span className={cx("inline-block h-1.5 w-1.5 rounded-full bg-current", s === "REALTIME" && "live-dot")} />}
45 + {text}
46 + </span>
47 + );
48 +}
49 +
50 +/** Rights classification chip with a plain-language label. */
51 +export const RIGHTS_LABEL: Record<string, string> = {
52 + PUBLIC_OPEN: "Public",
53 + PUBLIC_ATTRIBUTED: "Public · attribution",
54 + PUBLIC_RESTRICTED_REDISTRIBUTION: "Restricted redistribution",
55 + OFFICIAL_OPEN_DATA: "Official open data",
56 + LICENSED: "Licensed",
57 + DELAYED: "Delayed · attribution",
58 + INDICATIVE: "Indicative",
59 + RESEARCH_ONLY: "Research only",
60 + INTERNAL_ONLY: "Internal only",
61 + UNKNOWN: "Unknown",
62 + DISABLED: "Disabled",
63 +};
64 +
65 +export function RightsBadge({ status, className }: { status: string; className?: string }) {
66 + const restricted = ["PUBLIC_RESTRICTED_REDISTRIBUTION", "RESEARCH_ONLY", "INTERNAL_ONLY", "UNKNOWN", "DISABLED"].includes(status);
67 + return <span className={cx("inline-flex h-5 items-center whitespace-nowrap rounded-[3px] px-1.5 text-[10.5px] font-medium", restricted ? "bg-stale-soft text-stale" : status === "OFFICIAL_OPEN_DATA" ? "bg-accent-soft text-accent" : "bg-surface-3 text-ink-2", className)}>{RIGHTS_LABEL[status] ?? status}</span>;
68 +}
added apps/web/src/lib/api.ts +58 −0
@@ -0,0 +1,58 @@
1 +import "server-only";
2 +import { notFound } from "next/navigation";
3 +import type { Envelope } from "./types";
4 +
5 +export const API_URL = (process.env.API_URL ?? "http://127.0.0.1:8391").replace(/\/$/, "");
6 +
7 +export class ApiRequestError extends Error {
8 + constructor(
9 + public status: number,
10 + public code: string,
11 + message: string,
12 + ) {
13 + super(message);
14 + }
15 +}
16 +
17 +/** Server-side fetch of an API envelope. 404 → Next `notFound()`; other failures throw (caught by error.tsx). */
18 +export async function apiEnvelope<T>(path: string, init: { revalidate?: number; timeoutMs?: number } = {}): Promise<Envelope<T>> {
19 + const ctrl = new AbortController();
20 + const timer = setTimeout(() => ctrl.abort(), init.timeoutMs ?? 12_000);
21 + try {
22 + const res = await fetch(`${API_URL}${path}`, {
23 + signal: ctrl.signal,
24 + headers: { accept: "application/json" },
25 + ...(init.revalidate === undefined ? { cache: "no-store" } : { next: { revalidate: init.revalidate } }),
26 + });
27 + if (res.status === 404) notFound();
28 + if (!res.ok) {
29 + let code = "API_ERROR";
30 + let message = `API ${res.status}`;
31 + try {
32 + const body = (await res.json()) as { error?: { code?: string; message?: string } };
33 + code = body.error?.code ?? code;
34 + message = body.error?.message ?? message;
35 + } catch {
36 + /* ignore */
37 + }
38 + throw new ApiRequestError(res.status, code, message);
39 + }
40 + return (await res.json()) as Envelope<T>;
41 + } finally {
42 + clearTimeout(timer);
43 + }
44 +}
45 +
46 +export async function api<T>(path: string, init?: { revalidate?: number; timeoutMs?: number }): Promise<T> {
47 + return (await apiEnvelope<T>(path, init)).data;
48 +}
49 +
50 +/** Best-effort variant for non-critical widgets: returns null instead of throwing. */
51 +export async function apiOptional<T>(path: string, init?: { revalidate?: number; timeoutMs?: number }): Promise<T | null> {
52 + try {
53 + return await api<T>(path, init);
54 + } catch (err) {
55 + if (err && typeof err === "object" && "digest" in err && String((err as { digest: unknown }).digest).startsWith("NEXT_HTTP_ERROR_FALLBACK")) return null;
56 + return null;
57 + }
58 +}
added apps/web/src/lib/client-api.ts +14 −0
@@ -0,0 +1,14 @@
1 +"use client";
2 +
3 +import type { Envelope } from "./types";
4 +
5 +/** Browser-side API calls — always same-origin `/v1` (rewritten in dev, served by the edge in production). */
6 +export async function clientApi<T>(path: string, init?: RequestInit & { adminToken?: string }): Promise<T> {
7 + const headers: Record<string, string> = { accept: "application/json", ...((init?.headers as Record<string, string>) ?? {}) };
8 + if (init?.adminToken) headers["x-ma-admin-token"] = init.adminToken;
9 + if (init?.body && !headers["content-type"]) headers["content-type"] = "application/json";
10 + const res = await fetch(path, { ...init, headers });
11 + const body = (await res.json().catch(() => null)) as (Envelope<T> & { error?: { code: string; message: string } }) | null;
12 + if (!res.ok) throw new Error(body?.error?.message ?? `HTTP ${res.status}`);
13 + return (body as Envelope<T>).data;
14 +}
added apps/web/src/lib/fonts.ts +6 −0
@@ -0,0 +1,6 @@
1 +/** Geist Sans (UI) + Geist Mono (prices, ids, telemetry) self-hosted through the `geist` package — no network at build time. */
2 +import { GeistMono } from "geist/font/mono";
3 +import { GeistSans } from "geist/font/sans";
4 +
5 +export const fontUi = GeistSans;
6 +export const fontMono = GeistMono;
added apps/web/src/lib/format.ts +190 −0
@@ -0,0 +1,190 @@
1 +// Formatting helpers shared by server and client components. Deterministic (en-US) to avoid hydration drift.
2 +
3 +const nf = (opts: Intl.NumberFormatOptions) => new Intl.NumberFormat("en-US", opts);
4 +const cache = new Map<string, Intl.NumberFormat>();
5 +function fmt(min: number, max: number) {
6 + const k = `${min}:${max}`;
7 + let f = cache.get(k);
8 + if (!f) {
9 + f = nf({ minimumFractionDigits: min, maximumFractionDigits: max });
10 + cache.set(k, f);
11 + }
12 + return f;
13 +}
14 +
15 +/** Price precision by magnitude and asset class: crypto sub-cent, FX 4–5 decimals, yields 2–3. */
16 +export function priceDecimals(v: number, assetClass?: string | null): number {
17 + const a = Math.abs(v);
18 + if (assetClass === "FOREX") return a >= 20 ? 3 : a >= 1 ? 4 : 5;
19 + if (assetClass === "TREASURY" || assetClass === "INTEREST_RATE" || assetClass === "BOND") return 2;
20 + if (a === 0) return 2;
21 + if (a >= 10000) return 0;
22 + if (a >= 1000) return 1;
23 + if (a >= 1) return 2;
24 + if (a >= 0.1) return 4;
25 + if (a >= 0.01) return 5;
26 + return 6;
27 +}
28 +
29 +export function formatPrice(v: number | null | undefined, assetClass?: string | null): string {
30 + if (v == null || !Number.isFinite(v)) return "—";
31 + const d = priceDecimals(v, assetClass);
32 + return fmt(d, d).format(v);
33 +}
34 +
35 +export function isRateClass(assetClass?: string | null): boolean {
36 + return assetClass === "TREASURY" || assetClass === "INTEREST_RATE" || assetClass === "BOND";
37 +}
38 +
39 +/** Price with unit: yields get a % suffix, everything else the currency code when provided. */
40 +export function formatQuoteValue(v: number | null | undefined, assetClass?: string | null, currency?: string | null): string {
41 + if (v == null) return "—";
42 + if (isRateClass(assetClass)) return `${formatPrice(v, assetClass)}%`;
43 + const p = formatPrice(v, assetClass);
44 + return currency && currency !== "USD" && currency !== "USDT" ? `${p} ${currency}` : p;
45 +}
46 +
47 +export function formatPercent(v: number | null | undefined, digits = 2, sign = true): string {
48 + if (v == null || !Number.isFinite(v)) return "—";
49 + const s = fmt(digits, digits).format(Math.abs(v));
50 + return `${sign ? (v > 0 ? "+" : v < 0 ? "−" : "") : v < 0 ? "−" : ""}${s}%`;
51 +}
52 +
53 +export function formatChange(v: number | null | undefined, assetClass?: string | null): string {
54 + if (v == null || !Number.isFinite(v)) return "—";
55 + const d = priceDecimals(v, assetClass);
56 + const s = fmt(Math.min(d, 4), Math.min(d, 4)).format(Math.abs(v));
57 + return `${v > 0 ? "+" : v < 0 ? "−" : ""}${s}`;
58 +}
59 +
60 +export function formatCompact(v: number | null | undefined, digits = 1): string {
61 + if (v == null || !Number.isFinite(v)) return "—";
62 + const a = Math.abs(v);
63 + if (a >= 1e12) return `${fmt(0, digits).format(v / 1e12)}T`;
64 + if (a >= 1e9) return `${fmt(0, digits).format(v / 1e9)}B`;
65 + if (a >= 1e6) return `${fmt(0, digits).format(v / 1e6)}M`;
66 + if (a >= 1e4) return `${fmt(0, digits).format(v / 1e3)}K`;
67 + return fmt(0, a < 10 ? 2 : 0).format(v);
68 +}
69 +
70 +export function formatInt(v: number | null | undefined): string {
71 + if (v == null || !Number.isFinite(v)) return "—";
72 + return fmt(0, 0).format(v);
73 +}
74 +
75 +export function formatBps(v: number | null | undefined): string {
76 + if (v == null) return "—";
77 + return `${fmt(0, v < 10 ? 2 : 0).format(v)} bps`;
78 +}
79 +
80 +/** Parse API timestamps: ISO, epoch ms, or Postgres text "2026-09-11 19:50:00-04". */
81 +export function toMs(v: string | number | null | undefined): number | null {
82 + if (v == null) return null;
83 + if (typeof v === "number") return v;
84 + let s = v.trim();
85 + if (/^\d+$/.test(s)) return Number(s);
86 + s = s.replace(" ", "T");
87 + if (/[+-]\d{2}$/.test(s)) s += ":00";
88 + const ms = Date.parse(s);
89 + return Number.isFinite(ms) ? ms : null;
90 +}
91 +
92 +export function formatDuration(ms: number | null | undefined): string {
93 + if (ms == null || !Number.isFinite(ms)) return "—";
94 + if (ms < 1000) return `${Math.round(ms)} ms`;
95 + if (ms < 60_000) return `${(ms / 1000).toFixed(ms < 10_000 ? 1 : 0)} s`;
96 + if (ms < 3_600_000) return `${Math.floor(ms / 60_000)}m ${Math.floor((ms % 60_000) / 1000)}s`;
97 + if (ms < 86_400_000) return `${Math.floor(ms / 3_600_000)}h ${Math.floor((ms % 3_600_000) / 60_000)}m`;
98 + return `${Math.floor(ms / 86_400_000)}d ${Math.floor((ms % 86_400_000) / 3_600_000)}h`;
99 +}
100 +
101 +export function relativeTime(ms: number | null | undefined, now = Date.now()): string {
102 + if (ms == null) return "—";
103 + const d = now - ms;
104 + if (d < 0) return "just now";
105 + if (d < 1000) return `${d} ms ago`;
106 + if (d < 60_000) return `${Math.floor(d / 1000)}s ago`;
107 + if (d < 3_600_000) return `${Math.floor(d / 60_000)}m ago`;
108 + if (d < 86_400_000) return `${Math.floor(d / 3_600_000)}h ago`;
109 + return `${Math.floor(d / 86_400_000)}d ago`;
110 +}
111 +
112 +const dtfCache = new Map<string, Intl.DateTimeFormat>();
113 +export function formatDateTime(ms: number | string | null | undefined, opts: { tz?: string; seconds?: boolean; dateOnly?: boolean } = {}): string {
114 + const t = typeof ms === "string" ? toMs(ms) : ms;
115 + if (t == null) return "—";
116 + const key = `${opts.tz ?? "UTC"}:${opts.seconds ? 1 : 0}:${opts.dateOnly ? 1 : 0}`;
117 + let f = dtfCache.get(key);
118 + if (!f) {
119 + f = new Intl.DateTimeFormat("en-US", {
120 + timeZone: opts.tz ?? "UTC",
121 + year: "numeric",
122 + month: "short",
123 + day: "2-digit",
124 + ...(opts.dateOnly ? {} : { hour: "2-digit", minute: "2-digit", ...(opts.seconds ? { second: "2-digit" } : {}), hourCycle: "h23", timeZoneName: "short" }),
125 + });
126 + dtfCache.set(key, f);
127 + }
128 + return f.format(new Date(t));
129 +}
130 +
131 +export const ASSET_CLASS_LABEL: Record<string, string> = {
132 + EQUITY: "Stock",
133 + ETF: "ETF",
134 + ETN: "ETN",
135 + INDEX: "Index",
136 + CRYPTO: "Crypto",
137 + FOREX: "FX",
138 + COMMODITY: "Commodity",
139 + FUTURE: "Future",
140 + OPTION: "Option",
141 + BOND: "Bond yield",
142 + TREASURY: "Treasury",
143 + INTEREST_RATE: "Rate",
144 + MUTUAL_FUND: "Fund",
145 + ADR: "ADR",
146 + REIT: "REIT",
147 + WARRANT: "Warrant",
148 + OTHER: "Other",
149 +};
150 +
151 +export const EVENT_TYPE_LABEL: Record<string, string> = {
152 + PRICE_CHANGE: "Price move",
153 + SESSION_HIGH: "Session high",
154 + SESSION_LOW: "Session low",
155 + VOLATILITY_SPIKE: "Volatility spike",
156 + VOLUME_SPIKE: "Volume spike",
157 + SOURCE_DIVERGENCE: "Source divergence",
158 + SOURCE_FAILURE: "Source failure",
159 + SOURCE_RECOVERY: "Source recovery",
160 + SCHEMA_DRIFT: "Schema drift",
161 + MARKET_OPEN: "Market open",
162 + MARKET_CLOSE: "Market close",
163 + TRADING_HALT: "Trading halt",
164 + TRADING_RESUME: "Trading resumed",
165 + FILING_PUBLISHED: "Filing",
166 + DOCUMENT_CHANGED: "Document changed",
167 + INSTRUMENT_LISTED: "Listed",
168 + INSTRUMENT_DELISTED: "Delisted",
169 + REFERENCE_RATE_PUBLISHED: "Reference rate",
170 + RATE_DECISION: "Rate decision",
171 +};
172 +
173 +export const STATUS_LABEL: Record<string, string> = {
174 + REALTIME: "Live",
175 + DELAYED: "Delayed",
176 + END_OF_DAY: "End of day",
177 + AT_CLOSE: "At close",
178 + STALE: "Stale",
179 + WITHHELD: "Withheld",
180 + INDICATIVE: "Indicative",
181 + UNKNOWN: "Unknown",
182 +};
183 +
184 +export function instrumentHref(idOrSymbol: string): string {
185 + return `/instruments/${encodeURIComponent(idOrSymbol)}`;
186 +}
187 +
188 +export function cx(...xs: Array<string | false | null | undefined>): string {
189 + return xs.filter(Boolean).join(" ");
190 +}
added apps/web/src/lib/prepaint.ts +3 −0
@@ -0,0 +1,3 @@
1 +/** Inline script run in <head> before paint: applies the persisted theme so the first frame is right. */
2 +export const THEME_KEY = "ma-theme";
3 +export const THEME_SCRIPT = `(function(){try{var k='${THEME_KEY}',p=localStorage.getItem(k),m=window.matchMedia('(prefers-color-scheme: dark)'),t=(p==='light'||p==='dark')?p:(m.matches?'dark':'light');document.documentElement.setAttribute('data-theme',t);document.documentElement.style.colorScheme=t;}catch(e){}})();`;
added apps/web/src/lib/site.ts +9 −0
@@ -0,0 +1,9 @@
1 +export const SITE_NAME = "Market Atlas";
2 +export const TAGLINE = "The live map of global markets";
3 +export const DESCRIPTION =
4 + "Market Atlas observes thousands of market sources, normalizes their outputs, compares independent observations and exposes one continuously updating, provenance-backed picture of global markets.";
5 +export const SITE_URL = (process.env.NEXT_PUBLIC_SITE_URL ?? "https://www.market-atlas.co").replace(/\/$/, "");
6 +export const THEME_LIGHT = "#f5f6f4";
7 +export const THEME_DARK = "#0c0f14";
8 +export const DISCLAIMER =
9 + "Market Atlas is an informational market-data and analytics platform. Information may be delayed, indicative, incomplete, or sourced from third parties. Market Atlas is not an exchange, broker, investment adviser, or execution venue. Nothing here is investment advice.";
added apps/web/src/lib/stream.ts +283 −0
@@ -0,0 +1,283 @@
1 +"use client";
2 +
3 +import { useEffect, useMemo, useRef, useSyncExternalStore } from "react";
4 +import type { MarketEvent, StreamMessage, StreamQuote } from "./types";
5 +
6 +/**
7 + * Market Atlas stream client (singleton per tab). One WebSocket, reference-counted channel
8 + * subscriptions, a quote store keyed by instrument id, and per-key listeners notified at most
9 + * every ~100 ms so tables never re-render per tick.
10 + */
11 +type Listener = () => void;
12 +
13 +export type ConnectionState = "idle" | "connecting" | "open" | "reconnecting" | "closed";
14 +
15 +class MarketStreamClient {
16 + private ws: WebSocket | null = null;
17 + private refs = new Map<string, number>();
18 + private wanted = new Set<string>();
19 + private sent = new Set<string>();
20 + private quotes = new Map<string, StreamQuote>();
21 + private dirty = new Set<string>();
22 + private listeners = new Map<string, Set<Listener>>();
23 + private eventListeners = new Set<Listener>();
24 + private events: MarketEvent[] = [];
25 + private eventsVersion = 0;
26 + private tape: StreamQuote[] = [];
27 + private tapeVersion = 0;
28 + private flushTimer: ReturnType<typeof setTimeout> | null = null;
29 + private attempt = 0;
30 + private timer: ReturnType<typeof setTimeout> | null = null;
31 + state: ConnectionState = "idle";
32 + private stateListeners = new Set<Listener>();
33 + lastSeq = 0;
34 + frames = 0;
35 +
36 + private url(): string {
37 + const env = process.env.NEXT_PUBLIC_WS_URL;
38 + if (env) return env;
39 + const proto = window.location.protocol === "https:" ? "wss" : "ws";
40 + return `${proto}://${window.location.host}/v1/stream`;
41 + }
42 +
43 + private setState(s: ConnectionState) {
44 + if (this.state === s) return;
45 + this.state = s;
46 + for (const l of this.stateListeners) l();
47 + }
48 +
49 + private ensure() {
50 + if (typeof window === "undefined") return;
51 + if (this.ws || this.timer) return;
52 + if (!this.wanted.size) return;
53 + this.setState(this.attempt ? "reconnecting" : "connecting");
54 + let ws: WebSocket;
55 + try {
56 + ws = new WebSocket(this.url());
57 + } catch {
58 + this.scheduleReconnect();
59 + return;
60 + }
61 + this.ws = ws;
62 + ws.onopen = () => {
63 + this.attempt = 0;
64 + this.sent.clear();
65 + this.setState("open");
66 + this.sync();
67 + };
68 + ws.onmessage = (ev) => this.onFrame(String(ev.data));
69 + ws.onclose = () => {
70 + this.ws = null;
71 + this.sent.clear();
72 + if (this.wanted.size) this.scheduleReconnect();
73 + else this.setState("closed");
74 + };
75 + ws.onerror = () => {
76 + /* onclose follows */
77 + };
78 + }
79 +
80 + private scheduleReconnect() {
81 + if (this.timer) return;
82 + this.setState("reconnecting");
83 + const wait = Math.min(30_000, 1000 * 2 ** Math.min(6, this.attempt)) * (0.5 + Math.random());
84 + this.attempt++;
85 + this.timer = setTimeout(() => {
86 + this.timer = null;
87 + this.ensure();
88 + }, wait);
89 + }
90 +
91 + private sync() {
92 + if (!this.ws || this.ws.readyState !== WebSocket.OPEN) return;
93 + const add = [...this.wanted].filter((c) => !this.sent.has(c));
94 + const del = [...this.sent].filter((c) => !this.wanted.has(c));
95 + if (add.length) this.ws.send(JSON.stringify({ action: "subscribe", channels: add }));
96 + if (del.length) this.ws.send(JSON.stringify({ action: "unsubscribe", channels: del }));
97 + for (const c of add) this.sent.add(c);
98 + for (const c of del) this.sent.delete(c);
99 + if (!this.wanted.size) {
100 + this.ws.close();
101 + }
102 + }
103 +
104 + subscribe(channels: string[]): () => void {
105 + for (const c of channels) {
106 + this.refs.set(c, (this.refs.get(c) ?? 0) + 1);
107 + this.wanted.add(c);
108 + }
109 + this.ensure();
110 + this.sync();
111 + return () => {
112 + for (const c of channels) {
113 + const n = (this.refs.get(c) ?? 1) - 1;
114 + if (n <= 0) {
115 + this.refs.delete(c);
116 + this.wanted.delete(c);
117 + } else this.refs.set(c, n);
118 + }
119 + // Debounce unsubscribes so quick navigations reuse the socket.
120 + setTimeout(() => this.sync(), 500);
121 + };
122 + }
123 +
124 + private onFrame(raw: string) {
125 + let msg: { type?: string; seq?: number; messages?: StreamMessage[] };
126 + try {
127 + msg = JSON.parse(raw);
128 + } catch {
129 + return;
130 + }
131 + if (msg.type !== "batch" || !Array.isArray(msg.messages)) return;
132 + this.frames++;
133 + if (typeof msg.seq === "number") this.lastSeq = msg.seq;
134 + for (const m of msg.messages) {
135 + if (m.type === "quote") {
136 + const prev = this.quotes.get(m.instrument_id);
137 + if (prev && prev.received > m.received) continue;
138 + this.quotes.set(m.instrument_id, m);
139 + this.dirty.add(m.instrument_id);
140 + this.dirty.add(`symbol:${m.symbol.toUpperCase()}`);
141 + if (!prev || prev.price !== m.price) {
142 + this.tape.push(m);
143 + if (this.tape.length > 200) this.tape.splice(0, this.tape.length - 200);
144 + this.tapeVersion++;
145 + this.dirty.add("__tape__");
146 + }
147 + } else if (m.type === "event") {
148 + this.events.unshift(m.event);
149 + if (this.events.length > 500) this.events.length = 500;
150 + this.eventsVersion++;
151 + this.dirty.add("__events__");
152 + } else if (m.type === "market_state") {
153 + this.dirty.add("__market__");
154 + }
155 + }
156 + this.scheduleFlush();
157 + }
158 +
159 + private scheduleFlush() {
160 + if (this.flushTimer) return;
161 + this.flushTimer = setTimeout(() => {
162 + this.flushTimer = null;
163 + const keys = [...this.dirty];
164 + this.dirty.clear();
165 + for (const k of keys) {
166 + const ls = this.listeners.get(k);
167 + if (ls) for (const l of ls) l();
168 + }
169 + this.dirty.clear();
170 + }, 100);
171 + }
172 +
173 + listen(key: string, l: Listener): () => void {
174 + let s = this.listeners.get(key);
175 + if (!s) {
176 + s = new Set();
177 + this.listeners.set(key, s);
178 + }
179 + s.add(l);
180 + return () => {
181 + s!.delete(l);
182 + if (!s!.size) this.listeners.delete(key);
183 + };
184 + }
185 +
186 + listenState(l: Listener): () => void {
187 + this.stateListeners.add(l);
188 + return () => this.stateListeners.delete(l);
189 + }
190 +
191 + quote(id: string): StreamQuote | undefined {
192 + return this.quotes.get(id);
193 + }
194 + quoteBySymbol(symbol: string): StreamQuote | undefined {
195 + const u = symbol.toUpperCase();
196 + for (const q of this.quotes.values()) if (q.symbol.toUpperCase() === u) return q;
197 + return undefined;
198 + }
199 + recentEvents(): MarketEvent[] {
200 + return this.events;
201 + }
202 + eventsSnapshot(): number {
203 + return this.eventsVersion;
204 + }
205 + tapeSnapshot(): number {
206 + return this.tapeVersion;
207 + }
208 + recentTape(): StreamQuote[] {
209 + return this.tape;
210 + }
211 +}
212 +
213 +let client: MarketStreamClient | null = null;
214 +export function marketStream(): MarketStreamClient {
215 + if (!client) client = new MarketStreamClient();
216 + return client;
217 +}
218 +
219 +/** Keep the given channels subscribed while the component is mounted. */
220 +export function useMarketStream(channels: string[]): ConnectionState {
221 + const key = channels.join("|");
222 + useEffect(() => {
223 + if (!channels.length) return;
224 + return marketStream().subscribe(channels);
225 + // eslint-disable-next-line react-hooks/exhaustive-deps
226 + }, [key]);
227 + return useSyncExternalStore(
228 + (l) => marketStream().listenState(l),
229 + () => marketStream().state,
230 + () => "idle" as ConnectionState,
231 + );
232 +}
233 +
234 +/** Latest streamed quote for an instrument id (undefined until the first frame). Re-renders only that consumer. */
235 +export function useLiveQuote(instrumentId: string | null | undefined): StreamQuote | undefined {
236 + const id = instrumentId ?? "";
237 + return useSyncExternalStore(
238 + (l) => (id ? marketStream().listen(id, l) : () => {}),
239 + () => (id ? marketStream().quote(id) : undefined),
240 + () => undefined,
241 + );
242 +}
243 +
244 +/** Events received on the stream since mount (newest first), capped. */
245 +export function useLiveEvents(max = 100): MarketEvent[] {
246 + const version = useSyncExternalStore(
247 + (l) => marketStream().listen("__events__", l),
248 + () => marketStream().eventsSnapshot(),
249 + () => 0,
250 + );
251 + const ref = useRef<MarketEvent[]>([]);
252 + return useMemo(() => {
253 + void version;
254 + ref.current = marketStream().recentEvents().slice(0, max);
255 + return ref.current;
256 + }, [version, max]);
257 +}
258 +
259 +/** Recent price changes (tape). */
260 +export function useTape(max = 60): StreamQuote[] {
261 + const version = useSyncExternalStore(
262 + (l) => marketStream().listen("__tape__", l),
263 + () => marketStream().tapeSnapshot(),
264 + () => 0,
265 + );
266 + return useMemo(() => {
267 + void version;
268 + const t = marketStream().recentTape();
269 + return t.slice(Math.max(0, t.length - max)).reverse();
270 + }, [version, max]);
271 +}
272 +
273 +/** Ticks once per second — for relative "x s ago" labels. */
274 +export function useNow(intervalMs = 1000): number {
275 + return useSyncExternalStore(
276 + (l) => {
277 + const t = setInterval(l, intervalMs);
278 + return () => clearInterval(t);
279 + },
280 + () => Math.floor(Date.now() / intervalMs) * intervalMs,
281 + () => 0,
282 + );
283 +}
added apps/web/src/lib/types.ts +331 −0
@@ -0,0 +1,331 @@
1 +// Wire types of the Market Atlas API v1 (mirrors apps/api/src/api/*). Keep in sync with docs/API.md.
2 +
3 +export type DataStatus = "REALTIME" | "DELAYED" | "END_OF_DAY" | "AT_CLOSE" | "STALE" | "WITHHELD" | "INDICATIVE" | "UNKNOWN";
4 +export type MarketState = "PRE" | "OPEN" | "HALTED" | "CLOSED" | "POST" | "AUCTION" | "UNKNOWN";
5 +
6 +export interface Envelope<T> {
7 + data: T;
8 + meta: Record<string, unknown> & { request_id?: string; timestamp?: string; total?: number };
9 +}
10 +
11 +export interface Quote {
12 + instrument_id: string;
13 + symbol: string;
14 + name: string | null;
15 + asset_class: string | null;
16 + exchange_id: string | null;
17 + country: string | null;
18 + price: number | null;
19 + open: number | null;
20 + high: number | null;
21 + low: number | null;
22 + previous_close: number | null;
23 + change: number | null;
24 + change_percent: number | null;
25 + volume: number | null;
26 + bid: number | null;
27 + ask: number | null;
28 + currency: string | null;
29 + source_count: number;
30 + dispersion_bps: number | null;
31 + confidence: number;
32 + freshness_ms: number | null;
33 + data_status: DataStatus;
34 + market_state: MarketState | null;
35 + rights_status: string;
36 + withheld: boolean;
37 + updated_at: string;
38 + source_timestamp: string | null;
39 + session_high: number | null;
40 + session_low: number | null;
41 +}
42 +
43 +export interface Instrument {
44 + id: string;
45 + symbol: string;
46 + name: string;
47 + asset_class: string;
48 + exchange_id: string | null;
49 + mic: string | null;
50 + currency: string | null;
51 + country: string | null;
52 + company_id: string | null;
53 + security_type: string | null;
54 + base: string | null;
55 + quote?: string | null;
56 + is_active: boolean;
57 + metadata: Record<string, unknown>;
58 +}
59 +
60 +export type InstrumentWithQuote = Instrument & { quote: Quote | null; window_change_percent?: number | null };
61 +
62 +export interface ExchangeStatus {
63 + exchangeId: string;
64 + state: MarketState;
65 + localTime: string;
66 + nextTransition: { state: MarketState; at: number } | null;
67 + isHoliday: boolean;
68 + holidayName: string | null;
69 +}
70 +
71 +export interface Exchange {
72 + id: string;
73 + mic: string | null;
74 + name: string;
75 + operator: string | null;
76 + country: string;
77 + city: string | null;
78 + timezone: string;
79 + currency: string | null;
80 + website: string | null;
81 + sessions: { regular: Array<{ open: string; close: string }>; pre?: { open: string; close: string } | null; post?: { open: string; close: string } | null; weekdays?: number[]; continuous?: boolean };
82 + lat: number | null;
83 + lon: number | null;
84 + asset_classes?: string[];
85 + assetClasses?: string[];
86 + status?: ExchangeStatus;
87 +}
88 +
89 +export interface Breadth {
90 + instruments: number;
91 + advancers: number;
92 + decliners: number;
93 + unchanged: number;
94 + median_change_percent: number | null;
95 + mean_change_percent: number | null;
96 + new_session_highs: number;
97 + new_session_lows: number;
98 +}
99 +
100 +export interface MarketsOverview {
101 + indices: InstrumentWithQuote[];
102 + crypto: InstrumentWithQuote[];
103 + forex: InstrumentWithQuote[];
104 + rates: InstrumentWithQuote[];
105 + commodities: InstrumentWithQuote[];
106 + equities: InstrumentWithQuote[];
107 + gainers: InstrumentWithQuote[];
108 + losers: InstrumentWithQuote[];
109 + exchanges: Array<{ id: string; name: string; mic: string | null; country: string; city: string | null; timezone: string; lat: number | null; lon: number | null; status: ExchangeStatus; asset_classes: string[] }>;
110 + breadth: Breadth;
111 +}
112 +
113 +export interface Stats {
114 + instruments_total: number;
115 + instruments_quoted: number;
116 + instruments_live: number;
117 + exchanges_total: number;
118 + exchanges_open: number;
119 + sources_total: number;
120 + connectors_total: number;
121 + connectors_healthy: number;
122 + observations_today: number;
123 + observations_per_sec: number;
124 + observations_total: number;
125 + events_today: number;
126 + events_24h: number;
127 + events_per_min: number;
128 + filings_24h: number;
129 + median_freshness_ms: number | null;
130 + stream_clients: number;
131 + uptime_s: number;
132 +}
133 +
134 +export interface MarketEvent {
135 + id: string;
136 + type: string;
137 + timestamp: string | number;
138 + instrument_ids: string[];
139 + severity: "DEBUG" | "INFO" | "NOTICE" | "WARNING" | "CRITICAL";
140 + confidence: number;
141 + source_count: number;
142 + sources: string[];
143 + title: string;
144 + summary: string | null;
145 + data: Record<string, unknown>;
146 + confirmed_at?: string | null;
147 + instruments: Array<{ id: string; symbol?: string; name?: string; asset_class?: string; country?: string | null }>;
148 +}
149 +
150 +export interface Filing {
151 + id: string;
152 + source_id: string;
153 + cik: string | null;
154 + company_name: string;
155 + form_type: string;
156 + filed_at: string;
157 + url: string;
158 + instrument_ids: string[];
159 + metadata: Record<string, unknown>;
160 +}
161 +
162 +export interface Provenance {
163 + instrument_id: string;
164 + symbol: string;
165 + canonical_price: number | null;
166 + currency: string | null;
167 + confidence: number;
168 + dispersion_bps: number | null;
169 + independent_sources: number;
170 + freshness_ms: number | null;
171 + realtime_status: string;
172 + rights_status: string;
173 + consensus_version: string;
174 + computed_at: string;
175 + method: string;
176 + contributions: Array<{
177 + source_id: string;
178 + source: { name: string; organization: string | null; family: string | null; rights_status: string; realtime_status: string; category: string } | null;
179 + connector_id: string;
180 + value: number | null;
181 + source_timestamp: string | null;
182 + received_at: string;
183 + age_ms: number;
184 + weight: number;
185 + included: boolean;
186 + reason: string | null;
187 + realtime_status: string;
188 + rights_status: string;
189 + reliability_score: number;
190 + }>;
191 +}
192 +
193 +export interface Bar {
194 + t: number;
195 + o: number;
196 + h: number;
197 + l: number;
198 + c: number;
199 + v: number | null;
200 + n: number;
201 + p: string;
202 +}
203 +
204 +export interface Source {
205 + id: string;
206 + name: string;
207 + organization: string | null;
208 + source_type: string;
209 + homepage: string | null;
210 + jurisdiction: string | null;
211 + rights_status: string;
212 + realtime_status: string;
213 + family: string | null;
214 + category: string;
215 + enabled: boolean;
216 + connectors: number;
217 + status: string;
218 + coverage: number;
219 + last_observation: string | null;
220 + reliability_score: number | null;
221 + asset_classes: string[];
222 + description: string | null;
223 + rights_notes: string | null;
224 + terms_url: string | null;
225 +}
226 +
227 +export interface ConnectorPublic {
228 + id: string;
229 + name: string;
230 + source_id: string;
231 + organization: string | null;
232 + source_type: string;
233 + rights_status: string;
234 + realtime_status: string;
235 + asset_classes: string[];
236 + exchanges: string[];
237 + jurisdiction: string | null;
238 + supports_streaming: boolean;
239 + supports_historical: boolean;
240 + expected_latency_ms: number | null;
241 + description: string;
242 + version: string;
243 + status: string;
244 + last_message_at: string | null;
245 + messages_1m: number;
246 + messages_total: number;
247 + median_latency_ms: number | null;
248 + p95_latency_ms: number | null;
249 + parse_success_rate: number | null;
250 + instruments_covered: number;
251 + reconnects: number;
252 + reliability_score: number | null;
253 + next_poll_at: string | null;
254 +}
255 +
256 +export interface DataHealth {
257 + connectors: { total: number; healthy: number; degraded: number; recovering: number; failed: number; paused: number };
258 + healthy_ratio: number | null;
259 + quotes_public: number;
260 + quotes_realtime: number;
261 + median_freshness_ms: number | null;
262 + p95_freshness_ms: number | null;
263 + multi_source_quotes: number;
264 + mean_confidence: number | null;
265 + mean_dispersion_bps: number | null;
266 + observations_per_sec: number;
267 + queue_depth: number;
268 + by_connector: Array<{ id: string; state: string; messages_1m: number; median_latency_ms: number | null; reliability_score: number | null; instruments: number; last_message_at: string | null; last_error: string | null }>;
269 + incidents: Array<{ id: string; type: string; ts: string; severity: string; title: string; data: Record<string, unknown> }>;
270 + history: Array<{ hour: string; healthy_ratio: number; messages: number; latency_ms: number | null }>;
271 +}
272 +
273 +export interface Changes {
274 + window: string;
275 + since: string;
276 + instruments_tracked: number;
277 + instruments_moved_over_1pct: number;
278 + events: number;
279 + filings: number;
280 + events_by_type: Array<{ type: string; n: number }>;
281 + top_movers: InstrumentWithQuote[];
282 +}
283 +
284 +export interface Country {
285 + code: string;
286 + name: string;
287 + region: string;
288 + currency: string | null;
289 +}
290 +
291 +export interface SearchResult {
292 + query: string;
293 + results: Array<{ group: string; score: number; item: Record<string, unknown> }>;
294 + groups: Record<string, unknown[]>;
295 +}
296 +
297 +export interface StatusReport {
298 + components: Array<{ id: string; name: string; status: string }>;
299 + connectors: { total: number; healthy: number; degraded: number; failed: number; paused: number };
300 + version: string;
301 +}
302 +
303 +/** Stream frames */
304 +export interface StreamQuote {
305 + type: "quote";
306 + instrument_id: string;
307 + symbol: string;
308 + price: number | null;
309 + change: number | null;
310 + change_pct: number | null;
311 + bid: number | null;
312 + ask: number | null;
313 + volume: number | null;
314 + currency: string | null;
315 + timestamp: number;
316 + received: number;
317 + confidence: number;
318 + sources: number;
319 + status: string;
320 +}
321 +export interface StreamEvent {
322 + type: "event";
323 + event: MarketEvent;
324 +}
325 +export interface StreamMarketState {
326 + type: "market_state";
327 + exchangeId: string;
328 + state: string;
329 + at: number;
330 +}
331 +export type StreamMessage = StreamQuote | StreamEvent | StreamMarketState;
added apps/web/tsconfig.json +22 −0
@@ -0,0 +1,22 @@
1 +{
2 + "compilerOptions": {
3 + "target": "ES2022",
4 + "lib": ["dom", "dom.iterable", "esnext"],
5 + "allowJs": true,
6 + "skipLibCheck": true,
7 + "strict": true,
8 + "noUncheckedIndexedAccess": true,
9 + "noEmit": true,
10 + "esModuleInterop": true,
11 + "module": "esnext",
12 + "moduleResolution": "bundler",
13 + "resolveJsonModule": true,
14 + "isolatedModules": true,
15 + "jsx": "react-jsx",
16 + "incremental": true,
17 + "plugins": [{ "name": "next" }],
18 + "paths": { "@/*": ["./src/*"] }
19 + },
20 + "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts", ".next/dev/types/**/*.ts"],
21 + "exclude": ["node_modules", "qa"]
22 +}
added connectors/package.json +13 −0
@@ -0,0 +1,13 @@
1 +{
2 + "name": "@market-atlas/connectors",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "exports": {
7 + ".": "./src/index.ts"
8 + },
9 + "dependencies": {
10 + "@market-atlas/connector-sdk": "workspace:*",
11 + "@market-atlas/market-model": "workspace:*"
12 + }
13 +}
added connectors/src/_shared/crypto.ts +102 −0
@@ -0,0 +1,102 @@
1 +import type { InstrumentHint, NormalizedObservation, RealtimeStatus, RightsStatus, TimestampTrust } from "@market-atlas/market-model";
2 +import type { ProposedInstrument } from "@market-atlas/connector-sdk";
3 +
4 +export const CRYPTO_NAMES: Record<string, string> = {
5 + BTC: "Bitcoin",
6 + ETH: "Ether",
7 + SOL: "Solana",
8 + XRP: "XRP",
9 + ADA: "Cardano",
10 + DOGE: "Dogecoin",
11 + LTC: "Litecoin",
12 + LINK: "Chainlink",
13 + AVAX: "Avalanche",
14 + DOT: "Polkadot",
15 + BNB: "BNB",
16 + TRX: "TRON",
17 + MATIC: "Polygon",
18 + POL: "Polygon",
19 + UNI: "Uniswap",
20 + ATOM: "Cosmos",
21 + XLM: "Stellar",
22 + BCH: "Bitcoin Cash",
23 + ETC: "Ethereum Classic",
24 + AAVE: "Aave",
25 + NEAR: "NEAR Protocol",
26 + SUI: "Sui",
27 + TON: "Toncoin",
28 + SHIB: "Shiba Inu",
29 + PAXG: "PAX Gold",
30 +};
31 +
32 +/** Major coins observed across venues. */
33 +export const MAJOR_BASES = ["BTC", "ETH", "SOL", "XRP", "ADA", "DOGE", "LTC", "LINK", "AVAX", "DOT", "BCH", "XLM", "UNI", "AAVE", "NEAR", "SUI", "PAXG"];
34 +
35 +export function cryptoHint(base: string, quote: string, exchangeId: string): InstrumentHint {
36 + const b = base.toUpperCase();
37 + const q = quote.toUpperCase();
38 + return {
39 + assetClass: "CRYPTO",
40 + name: `${CRYPTO_NAMES[b] ?? b} / ${q}`,
41 + exchangeId,
42 + currency: q,
43 + country: "XX",
44 + base: b,
45 + quote: q,
46 + securityType: "SPOT",
47 + metadata: { featured: ["BTC", "ETH", "SOL", "XRP"].includes(b) && (q === "USD" || q === "USDT") },
48 + };
49 +}
50 +
51 +export function cryptoSeeds(pairs: Array<[string, string]>, exchangeId: string, toSymbol: (b: string, q: string) => string): ProposedInstrument[] {
52 + return pairs.map(([b, q]) => ({ symbol: toSymbol(b, q), hint: cryptoHint(b, q, exchangeId), aliases: [`${b}-${q}`, `${b}/${q}`, `${b}${q}`] }));
53 +}
54 +
55 +export interface TickerFields {
56 + last?: unknown;
57 + open?: unknown;
58 + high?: unknown;
59 + low?: unknown;
60 + bid?: unknown;
61 + ask?: unknown;
62 + bidSize?: unknown;
63 + askSize?: unknown;
64 + volume?: unknown;
65 + vwap?: unknown;
66 +}
67 +
68 +const num = (v: unknown): number | null => {
69 + if (v == null || v === "") return null;
70 + const n = typeof v === "number" ? v : Number(v);
71 + return Number.isFinite(n) ? n : null;
72 +};
73 +
74 +/** Expand a venue ticker into field observations, skipping missing/invalid values. */
75 +export function tickerObservations(
76 + symbol: string,
77 + base: string,
78 + quote: string,
79 + exchangeId: string,
80 + f: TickerFields,
81 + meta: { sourceTimestamp: number | null; timestampTrust: TimestampTrust; sequence?: number | string | null; rightsStatus: RightsStatus; realtimeStatus: RealtimeStatus },
82 +): NormalizedObservation[] {
83 + const hint = cryptoHint(base, quote, exchangeId);
84 + const out: NormalizedObservation[] = [];
85 + const push = (field: NormalizedObservation["field"], v: unknown) => {
86 + const n = num(v);
87 + if (n == null) return;
88 + if (field !== "VOLUME" && field !== "BID_SIZE" && field !== "ASK_SIZE" && n <= 0) return;
89 + out.push({ symbol, instrumentHint: hint, field, value: n, currency: quote.toUpperCase(), sourceTimestamp: meta.sourceTimestamp, timestampTrust: meta.timestampTrust, sequence: meta.sequence ?? null, rightsStatus: meta.rightsStatus, realtimeStatus: meta.realtimeStatus });
90 + };
91 + push("LAST_PRICE", f.last);
92 + push("OPEN", f.open);
93 + push("HIGH", f.high);
94 + push("LOW", f.low);
95 + push("BID", f.bid);
96 + push("ASK", f.ask);
97 + push("BID_SIZE", f.bidSize);
98 + push("ASK_SIZE", f.askSize);
99 + push("VOLUME", f.volume);
100 + push("VWAP", f.vwap);
101 + return out;
102 +}
added connectors/src/_shared/us-universe.ts +104 −0
@@ -0,0 +1,104 @@
1 +import type { InstrumentHint } from "@market-atlas/market-model";
2 +
3 +/** Curated US universe watched by delayed-quote and historical connectors: symbol → [name, venue, class]. */
4 +export const US_EQUITIES: Array<[string, string, "xnas" | "xnys" | "arcx"]> = [
5 + ["AAPL", "Apple Inc.", "xnas"],
6 + ["MSFT", "Microsoft Corporation", "xnas"],
7 + ["NVDA", "NVIDIA Corporation", "xnas"],
8 + ["AMZN", "Amazon.com, Inc.", "xnas"],
9 + ["GOOGL", "Alphabet Inc. (Class A)", "xnas"],
10 + ["GOOG", "Alphabet Inc. (Class C)", "xnas"],
11 + ["META", "Meta Platforms, Inc.", "xnas"],
12 + ["TSLA", "Tesla, Inc.", "xnas"],
13 + ["BRK.B", "Berkshire Hathaway Inc. (Class B)", "xnys"],
14 + ["AVGO", "Broadcom Inc.", "xnas"],
15 + ["JPM", "JPMorgan Chase & Co.", "xnys"],
16 + ["LLY", "Eli Lilly and Company", "xnys"],
17 + ["V", "Visa Inc.", "xnys"],
18 + ["UNH", "UnitedHealth Group Incorporated", "xnys"],
19 + ["XOM", "Exxon Mobil Corporation", "xnys"],
20 + ["MA", "Mastercard Incorporated", "xnys"],
21 + ["COST", "Costco Wholesale Corporation", "xnas"],
22 + ["HD", "The Home Depot, Inc.", "xnys"],
23 + ["PG", "The Procter & Gamble Company", "xnys"],
24 + ["JNJ", "Johnson & Johnson", "xnys"],
25 + ["ABBV", "AbbVie Inc.", "xnys"],
26 + ["WMT", "Walmart Inc.", "xnys"],
27 + ["NFLX", "Netflix, Inc.", "xnas"],
28 + ["BAC", "Bank of America Corporation", "xnys"],
29 + ["CRM", "Salesforce, Inc.", "xnys"],
30 + ["ORCL", "Oracle Corporation", "xnys"],
31 + ["KO", "The Coca-Cola Company", "xnys"],
32 + ["CVX", "Chevron Corporation", "xnys"],
33 + ["AMD", "Advanced Micro Devices, Inc.", "xnas"],
34 + ["MRK", "Merck & Co., Inc.", "xnys"],
35 + ["PEP", "PepsiCo, Inc.", "xnas"],
36 + ["TMO", "Thermo Fisher Scientific Inc.", "xnys"],
37 + ["CSCO", "Cisco Systems, Inc.", "xnas"],
38 + ["ADBE", "Adobe Inc.", "xnas"],
39 + ["MCD", "McDonald's Corporation", "xnys"],
40 + ["ACN", "Accenture plc", "xnys"],
41 + ["LIN", "Linde plc", "xnas"],
42 + ["ABT", "Abbott Laboratories", "xnys"],
43 + ["WFC", "Wells Fargo & Company", "xnys"],
44 + ["DIS", "The Walt Disney Company", "xnys"],
45 + ["IBM", "International Business Machines Corporation", "xnys"],
46 + ["GE", "GE Aerospace", "xnys"],
47 + ["CAT", "Caterpillar Inc.", "xnys"],
48 + ["INTC", "Intel Corporation", "xnas"],
49 + ["QCOM", "QUALCOMM Incorporated", "xnas"],
50 + ["TXN", "Texas Instruments Incorporated", "xnas"],
51 + ["AMGN", "Amgen Inc.", "xnas"],
52 + ["PM", "Philip Morris International Inc.", "xnys"],
53 + ["INTU", "Intuit Inc.", "xnas"],
54 + ["GS", "The Goldman Sachs Group, Inc.", "xnys"],
55 + ["MS", "Morgan Stanley", "xnys"],
56 + ["BA", "The Boeing Company", "xnys"],
57 + ["NKE", "NIKE, Inc.", "xnys"],
58 + ["PFE", "Pfizer Inc.", "xnys"],
59 + ["UBER", "Uber Technologies, Inc.", "xnys"],
60 + ["PLTR", "Palantir Technologies Inc.", "xnas"],
61 + ["SHOP", "Shopify Inc.", "xnas"],
62 + ["MU", "Micron Technology, Inc.", "xnas"],
63 + ["ARM", "Arm Holdings plc", "xnas"],
64 + ["COIN", "Coinbase Global, Inc.", "xnas"],
65 +];
66 +
67 +export const US_ETFS: Array<[string, string, "xnas" | "arcx"]> = [
68 + ["SPY", "SPDR S&P 500 ETF Trust", "arcx"],
69 + ["QQQ", "Invesco QQQ Trust", "xnas"],
70 + ["IWM", "iShares Russell 2000 ETF", "arcx"],
71 + ["DIA", "SPDR Dow Jones Industrial Average ETF Trust", "arcx"],
72 + ["VTI", "Vanguard Total Stock Market ETF", "arcx"],
73 + ["EFA", "iShares MSCI EAFE ETF", "arcx"],
74 + ["EEM", "iShares MSCI Emerging Markets ETF", "arcx"],
75 + ["GLD", "SPDR Gold Shares", "arcx"],
76 + ["SLV", "iShares Silver Trust", "arcx"],
77 + ["USO", "United States Oil Fund", "arcx"],
78 + ["TLT", "iShares 20+ Year Treasury Bond ETF", "xnas"],
79 + ["HYG", "iShares iBoxx $ High Yield Corporate Bond ETF", "arcx"],
80 + ["XLF", "Financial Select Sector SPDR Fund", "arcx"],
81 + ["XLE", "Energy Select Sector SPDR Fund", "arcx"],
82 + ["XLK", "Technology Select Sector SPDR Fund", "arcx"],
83 + ["IBIT", "iShares Bitcoin Trust ETF", "xnas"],
84 +];
85 +
86 +/** Cboe delayed-quote symbol → canonical index. */
87 +export const US_INDICES: Array<{ cboe: string; symbol: string; name: string; exchangeId: string; hfmd?: string }> = [
88 + { cboe: "_SPX", symbol: "SPX", name: "S&P 500", exchangeId: "xcbo", hfmd: "SPX" },
89 + { cboe: "_NDX", symbol: "NDX", name: "Nasdaq-100", exchangeId: "xnas", hfmd: "NDX" },
90 + { cboe: "_DJI", symbol: "DJI", name: "Dow Jones Industrial Average", exchangeId: "xnys", hfmd: "DJI" },
91 + { cboe: "_RUT", symbol: "RUT", name: "Russell 2000", exchangeId: "xcbo", hfmd: "RUT" },
92 + { cboe: "_VIX", symbol: "VIX", name: "Cboe Volatility Index", exchangeId: "xcbo", hfmd: "VIX" },
93 + { cboe: "_DJX", symbol: "DJX", name: "Dow Jones Industrial Average (1/100)", exchangeId: "xcbo" },
94 +];
95 +
96 +const MIC: Record<string, string> = { xnas: "XNAS", xnys: "XNYS", arcx: "ARCX", xcbo: "XCBO" };
97 +
98 +export function equityHint(symbol: string, name: string, venue: string, cls: "EQUITY" | "ETF" = "EQUITY"): InstrumentHint {
99 + return { assetClass: cls, name, exchangeId: venue, mic: MIC[venue] ?? null, currency: "USD", country: "US", securityType: cls === "ETF" ? "ETF" : "COMMON_STOCK", companyName: cls === "EQUITY" ? name : null, metadata: { featured: ["AAPL", "MSFT", "NVDA", "AMZN", "GOOGL", "META", "TSLA", "SPY", "QQQ"].includes(symbol) } };
100 +}
101 +
102 +export function indexHint(i: (typeof US_INDICES)[number]): InstrumentHint {
103 + return { assetClass: "INDEX", name: i.name, exchangeId: i.exchangeId, mic: MIC[i.exchangeId] ?? null, currency: "USD", country: "US", securityType: "INDEX", metadata: { featured: true } };
104 +}
added connectors/src/bank-of-canada/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# bank-of-canada-valet
2 +
3 +Bank of Canada Valet API (`/valet/observations/group/FX_RATES_DAILY/json`, `/valet/observations/<series>/json`).
4 +
5 +- **Type**: OFFICIAL_API · **Rights**: OFFICIAL_OPEN_DATA (attribution) · **Real-time**: END_OF_DAY (published ≈ 16:30 ET)
6 +- **FX**: `FX<CCY>CAD` series are CAD per 1 unit of foreign currency → inverted when the conventional pair is `CAD`-quoted the other way (e.g. `USDCAD` = value as is; `CADJPY` would be inverted).
7 +- **Rates**: `V39079` policy rate (RATE), benchmark bond yields 2y/5y/10y/long (YIELD). A LAST_PRICE copy is emitted so the generic quote machinery works.
8 +- **Independence**: separate family from the ECB → USD/CAD gets two official observations and a real dispersion figure.
added connectors/src/bank-of-canada/fixtures/fx.json +1 −0
@@ -0,0 +1 @@
1 +{"terms":{"url":"https://www.bankofcanada.ca/terms/"},"seriesDetail":{"FXUSDCAD":{"label":"USD/CAD","description":"Daily average exchange rate"},"FXEURCAD":{"label":"EUR/CAD","description":"Daily average exchange rate"}},"observations":[{"d":"2026-09-10","FXUSDCAD":{"v":"1.3870"},"FXEURCAD":{"v":"1.6060"}},{"d":"2026-09-11","FXUSDCAD":{"v":"1.3855"},"FXEURCAD":{"v":"1.6052"}}]}
added connectors/src/bank-of-canada/fixtures/rates.json +1 −0
@@ -0,0 +1 @@
1 +{"terms":{"url":"https://www.bankofcanada.ca/terms/"},"seriesDetail":{"V39079":{"label":"Target for the overnight rate"},"BD.CDN.10YR.DQ.YLD":{"label":"Benchmark bond yield: 10 year"}},"observations":[{"d":"2026-09-10","V39079":{"v":"2.25"},"BD.CDN.10YR.DQ.YLD":{"v":"3.21"}},{"d":"2026-09-11","V39079":{"v":"2.25"},"BD.CDN.10YR.DQ.YLD":{"v":"3.24"}}]}
added connectors/src/bank-of-canada/index.ts +97 −0
@@ -0,0 +1,97 @@
1 +import { defineConnector, raw, type NormalizedBatch, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +import type { InstrumentHint, NormalizedObservation } from "@market-atlas/market-model";
3 +import { zonedTimeToUtc } from "@market-atlas/market-model";
4 +import { conventionalPair, fxHint } from "../ecb-frankfurter/index.js";
5 +
6 +const BASE = "https://www.bankofcanada.ca/valet/observations";
7 +const RATE_SERIES: Record<string, { symbol: string; name: string; assetClass: "INTEREST_RATE" | "BOND"; field: "RATE" | "YIELD" }> = {
8 + V39079: { symbol: "CA_POLICY_RATE", name: "Bank of Canada target for the overnight rate", assetClass: "INTEREST_RATE", field: "RATE" },
9 + "BD.CDN.2YR.DQ.YLD": { symbol: "CA_2Y", name: "Government of Canada 2-year benchmark bond yield", assetClass: "BOND", field: "YIELD" },
10 + "BD.CDN.5YR.DQ.YLD": { symbol: "CA_5Y", name: "Government of Canada 5-year benchmark bond yield", assetClass: "BOND", field: "YIELD" },
11 + "BD.CDN.10YR.DQ.YLD": { symbol: "CA_10Y", name: "Government of Canada 10-year benchmark bond yield", assetClass: "BOND", field: "YIELD" },
12 + "BD.CDN.LONG.DQ.YLD": { symbol: "CA_30Y", name: "Government of Canada long-term benchmark bond yield", assetClass: "BOND", field: "YIELD" },
13 +};
14 +const rateHint = (s: (typeof RATE_SERIES)[string]): InstrumentHint => ({ assetClass: s.assetClass, name: s.name, exchangeId: null, currency: "CAD", country: "CA", securityType: s.assetClass === "BOND" ? "GOVERNMENT_BOND_YIELD" : "POLICY_RATE", metadata: { featured: true, unit: "percent" } });
15 +
16 +const FX_CCYS = ["USD", "EUR", "GBP", "JPY", "CHF", "AUD", "CNY", "MXN", "INR", "BRL", "HKD", "KRW", "SEK", "NOK", "SGD", "NZD", "ZAR", "TRY"];
17 +const seeds: ProposedInstrument[] = [
18 + ...Object.values(RATE_SERIES).map((s) => ({ symbol: s.symbol, hint: rateHint(s) })),
19 + ...FX_CCYS.map((c) => {
20 + const [b, q] = conventionalPair(c, "CAD");
21 + return { symbol: `${b}${q}`, hint: fxHint(b, q) };
22 + }),
23 +];
24 +
25 +/** Bank of Canada Valet API: daily FX averages (CAD per unit) + policy rate + benchmark bond yields. Independent official source for USD/CAD. */
26 +export const bankOfCanada = defineConnector({
27 + metadata: {
28 + id: "bank-of-canada-valet",
29 + name: "Bank of Canada — Valet (FX, policy rate, bond yields)",
30 + version: "1.0.0",
31 + sourceId: "bank-of-canada",
32 + organization: "Bank of Canada",
33 + sourceType: "OFFICIAL_API",
34 + jurisdiction: "CA",
35 + rightsStatus: "OFFICIAL_OPEN_DATA",
36 + realtimeStatus: "END_OF_DAY",
37 + expectedLatencyMs: null,
38 + supportsStreaming: false,
39 + supportsHistorical: true,
40 + assetClasses: ["FOREX", "INTEREST_RATE", "BOND"],
41 + exchanges: [],
42 + homepage: "https://www.bankofcanada.ca/valet/docs",
43 + description: "Bank of Canada Valet open API: daily average exchange rates (published ~16:30 ET), the target for the overnight rate and Government of Canada benchmark bond yields (2y, 5y, 10y, long).",
44 + rightsNotes: "Bank of Canada content is available under its terms of use with attribution; exchange rates are indicative daily averages.",
45 + termsUrl: "https://www.bankofcanada.ca/terms/",
46 + sourceFamily: "bank-of-canada",
47 + enabled: true,
48 + },
49 + seeds,
50 + defaultSymbols: seeds.map((s) => s.symbol),
51 + rateLimits: { "www.bankofcanada.ca": 1 },
52 + schedule: { intervalMs: 2 * 60 * 60_000 },
53 + async poll(ctx) {
54 + const out = [];
55 + const fx = await ctx.http.getJson(`${BASE}/group/FX_RATES_DAILY/json?recent=3`);
56 + if (!fx.response.notModified) out.push(raw("bank-of-canada-valet", "bank-of-canada", "fx", fx.data));
57 + const rates = await ctx.http.getJson(`${BASE}/${Object.keys(RATE_SERIES).join(",")}/json?recent=3`);
58 + if (!rates.response.notModified) out.push(raw("bank-of-canada-valet", "bank-of-canada", "rates", rates.data));
59 + return out;
60 + },
61 + normalize(r): NormalizedBatch {
62 + const p = r.payload as { observations?: Array<Record<string, { v?: string } | string>>; seriesDetail?: Record<string, { label?: string }> };
63 + if (!Array.isArray(p.observations)) return { observations: [] };
64 + const rows = [...p.observations].sort((a, b) => String(a.d).localeCompare(String(b.d)));
65 + const observations: NormalizedObservation[] = [];
66 + // Latest value per series (+ previous for PREVIOUS_CLOSE)
67 + const seriesIds = new Set(rows.flatMap((row) => Object.keys(row).filter((k) => k !== "d")));
68 + for (const sid of seriesIds) {
69 + const withValue = rows.filter((row) => typeof (row[sid] as { v?: string } | undefined)?.v === "string");
70 + const last = withValue[withValue.length - 1];
71 + const prev = withValue[withValue.length - 2];
72 + if (!last) continue;
73 + // Discontinued series (e.g. currencies the Bank stopped publishing) keep an old last value: skip anything > 10 days old.
74 + if (r.receivedAt - Date.parse(`${String(last.d)}T00:00:00Z`) > 10 * 86_400_000) continue;
75 + const emit = (row: Record<string, { v?: string } | string>, field: "LAST_PRICE" | "PREVIOUS_CLOSE" | "RATE" | "YIELD") => {
76 + const value = Number((row[sid] as { v?: string }).v);
77 + if (!Number.isFinite(value)) return;
78 + const date = String(row.d);
79 + const ts = zonedTimeToUtc(`${date}T16:30:00`, "America/Toronto");
80 + if (r.kind === "fx" && /^FX[A-Z]{3}CAD$/.test(sid)) {
81 + const foreign = sid.slice(2, 5);
82 + const [b, q] = conventionalPair(foreign, "CAD");
83 + const v = b === foreign ? value : 1 / value; // series is CAD per 1 unit of foreign
84 + observations.push({ symbol: `${b}${q}`, instrumentHint: fxHint(b, q), field: field === "RATE" || field === "YIELD" ? "LAST_PRICE" : field, value: Number(v.toPrecision(8)), currency: q, sourceTimestamp: ts, timestampTrust: "SOURCE", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", meta: { series: sid, date } });
85 + } else if (r.kind === "rates" && RATE_SERIES[sid]) {
86 + const s = RATE_SERIES[sid]!;
87 + observations.push({ symbol: s.symbol, instrumentHint: rateHint(s), field: field === "PREVIOUS_CLOSE" ? "PREVIOUS_CLOSE" : s.field, value, currency: "CAD", sourceTimestamp: ts, timestampTrust: "SOURCE", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", meta: { series: sid, date, label: p.seriesDetail?.[sid]?.label } });
88 + if (field !== "PREVIOUS_CLOSE") observations.push({ symbol: s.symbol, instrumentHint: rateHint(s), field: "LAST_PRICE", value, currency: "CAD", sourceTimestamp: ts, timestampTrust: "SOURCE", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", meta: { series: sid, date } });
89 + }
90 + };
91 + emit(last, "LAST_PRICE");
92 + if (prev) emit(prev, "PREVIOUS_CLOSE");
93 + }
94 + return { observations };
95 + },
96 + fixturesDir: "fixtures",
97 +});
added connectors/src/binance-ws/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# binance-ws
2 +
3 +Binance spot combined stream `wss://stream.binance.com:9443/stream?streams=btcusdt@miniTicker/…`.
4 +
5 +- **Type**: WEBSOCKET · **Rights**: PUBLIC_ATTRIBUTED · **Real-time**: REALTIME (event time `E`, exchange-generated)
6 +- **Pairs**: USDT-quoted majors → distinct instruments from USD pairs (`crypto_btc_usdt` vs `crypto_btc_usd`).
7 +- **Cadence**: one miniTicker per symbol per second (fixed, not per trade).
8 +- **Heartbeat**: server-initiated pings; no client ping needed.
added connectors/src/binance-ws/fixtures/miniTicker.json +1 −0
@@ -0,0 +1 @@
1 +{"e":"24hrMiniTicker","E":1789199657016,"s":"BTCUSDT","c":"77298.87000000","o":"77215.29000000","h":"79890.00000000","l":"76046.58000000","v":"18869.91575000","q":"1464678990.90518890"}
added connectors/src/binance-ws/index.ts +69 −0
@@ -0,0 +1,69 @@
1 +import { defineConnector, raw, type NormalizedBatch } from "@market-atlas/connector-sdk";
2 +import { parseTimestamp } from "@market-atlas/market-model";
3 +import { MAJOR_BASES, cryptoSeeds, tickerObservations } from "../_shared/crypto.js";
4 +
5 +const PAIRS: Array<[string, string]> = MAJOR_BASES.filter((b) => b !== "PAXG").map((b) => [b, "USDT"] as [string, string]).concat([["BNB", "USDT"], ["TRX", "USDT"], ["PAXG", "USDT"]]);
6 +const SYMBOLS = PAIRS.map(([b, q]) => `${b}${q}`);
7 +const streamUrl = (symbols: string[]) => `wss://stream.binance.com:9443/stream?streams=${symbols.map((s) => `${s.toLowerCase()}@miniTicker`).join("/")}`;
8 +
9 +/** Binance combined stream, `<symbol>@miniTicker` (1 s cadence, exchange event time). USDT-quoted pairs. */
10 +export const binanceWs = defineConnector({
11 + metadata: {
12 + id: "binance-ws",
13 + name: "Binance — miniTicker combined stream",
14 + version: "1.0.0",
15 + sourceId: "binance",
16 + organization: "Binance",
17 + sourceType: "WEBSOCKET",
18 + jurisdiction: null,
19 + rightsStatus: "PUBLIC_ATTRIBUTED",
20 + realtimeStatus: "REALTIME",
21 + expectedLatencyMs: 1000,
22 + supportsStreaming: true,
23 + supportsHistorical: false,
24 + assetClasses: ["CRYPTO"],
25 + exchanges: ["binance"],
26 + homepage: "https://developers.binance.com/docs/binance-spot-api-docs/web-socket-streams",
27 + description: "Public Binance spot combined WebSocket stream (miniTicker per symbol, pushed every second) for USDT-quoted majors. Exchange event time (E) is used as source timestamp.",
28 + rightsNotes: "Public market data displayed with attribution to Binance. Availability may vary by jurisdiction.",
29 + termsUrl: "https://www.binance.com/en/terms",
30 + sourceFamily: "binance",
31 + enabled: true,
32 + },
33 + seeds: cryptoSeeds(PAIRS, "binance", (b, q) => `${b}${q}`),
34 + defaultSymbols: SYMBOLS,
35 + async start(ctx) {
36 + const ws = ctx.openWebSocket(streamUrl(ctx.watchedSymbols()), {
37 + label: "binance",
38 + staleAfterMs: 60_000,
39 + heartbeat: null, // Binance pings the client; ws library answers pongs automatically.
40 + onMessage: (data) => {
41 + let msg: any;
42 + try {
43 + msg = JSON.parse(data);
44 + } catch {
45 + return;
46 + }
47 + if (msg?.data?.e === "24hrMiniTicker") ctx.emit(raw("binance-ws", "binance", "miniTicker", msg.data));
48 + },
49 + });
50 + ws.connect();
51 + },
52 + normalize(r): NormalizedBatch {
53 + const m = r.payload as Record<string, unknown>;
54 + if (r.kind !== "miniTicker" || typeof m.s !== "string") return { observations: [] };
55 + const pair = PAIRS.find(([b, q]) => `${b}${q}` === m.s);
56 + const base = pair?.[0] ?? (m.s.endsWith("USDT") ? m.s.slice(0, -4) : null);
57 + const quote = pair?.[1] ?? (m.s.endsWith("USDT") ? "USDT" : null);
58 + if (!base || !quote) return { observations: [] };
59 + return {
60 + observations: tickerObservations(m.s, base, quote, "binance", { last: m.c, open: m.o, high: m.h, low: m.l, volume: m.v }, {
61 + sourceTimestamp: parseTimestamp(m.E),
62 + timestampTrust: "EXCHANGE",
63 + rightsStatus: "PUBLIC_ATTRIBUTED",
64 + realtimeStatus: "REALTIME",
65 + }),
66 + };
67 + },
68 + fixturesDir: "fixtures",
69 +});
added connectors/src/cboe-delayed-quotes/README.md +11 −0
@@ -0,0 +1,11 @@
1 +# cboe-delayed-quotes
2 +
3 +`https://cdn.cboe.com/api/global/delayed_quotes/quotes/<SYMBOL>.json` — the JSON behind Cboe's public quote pages.
4 +
5 +- **Type**: XHR · **Rights**: DELAYED (attribution, 15-minute delay) · **Real-time**: DELAYED
6 +- **Universe**: 60 large caps, 16 ETFs, 6 indices (`_SPX`, `_NDX`, `_DJI`, `_RUT`, `_VIX`, `_DJX`). Index symbols are prefixed with `_` in the URL and returned as `^SPX`.
7 +- **Schedule**: adaptive — 60 s while NYSE is open/pre/post, 15 min when closed, 60 min on weekends and holidays (`exchangeId: xnys`).
8 +- **Timestamps**: `last_trade_time` is naive US/Eastern → converted with `zonedTimeToUtc`; bid/ask use the response `timestamp` (also Eastern).
9 +- **Fields**: LAST_PRICE, OPEN, HIGH, LOW, PREVIOUS_CLOSE, VOLUME, CHANGE, CHANGE_PERCENT, BID, ASK, BID_SIZE, ASK_SIZE, IMPLIED_VOLATILITY (iv30).
10 +- **Quirks**: `BRK.B` works with the dot (no `BRKB`); zero prices when a symbol never traded → dropped by the normalizer; `volume` is 0 for indices.
11 +- **Rate limit**: 1 req/s on `cdn.cboe.com` (shared bucket; the CDN returns 429 above that). Full universe every ~80 s, poll every 2 min while open.
added connectors/src/cboe-delayed-quotes/fixtures/closed_market.json +1 −0
@@ -0,0 +1 @@
1 +{"timestamp": "2026-09-13 09:00:00", "data": {"symbol": "XYZQ", "security_type": "stock", "exchange_id": 2, "current_price": 0.0, "price_change": 0.0, "price_change_percent": 0.0, "bid": 0.0, "ask": 0.0, "bid_size": 0, "ask_size": 0, "open": 0.0, "high": 0.0, "low": 0.0, "close": 0.0, "prev_day_close": 0.0, "volume": 0, "seqno": 0, "last_trade_time": null, "tick": "none"}, "symbol": "XYZQ"}
added connectors/src/cboe-delayed-quotes/fixtures/index.json +1 −0
@@ -0,0 +1 @@
1 +{"timestamp": "2026-09-12 07:53:11", "data": {"symbol": "^SPX", "security_type": "index", "exchange_id": 5, "current_price": 7656.98, "price_change": 65.28, "price_change_percent": 0.8526, "bid": 7622.71, "ask": 7690.7998, "bid_size": 1, "ask_size": 1, "open": 7636.75, "high": 7677.02, "low": 7636.75, "close": 7656.98, "prev_day_close": 7656.98, "volume": 0, "iv30": 12.055, "iv30_change": 0.0, "iv30_change_percent": 0.0, "seqno": 35613016009, "last_trade_time": "2026-09-11T16:14:59", "tick": "up"}, "symbol": "_SPX"}
added connectors/src/cboe-delayed-quotes/fixtures/quote.json +1 −0
@@ -0,0 +1 @@
1 +{"timestamp": "2026-09-12 04:40:58", "data": {"symbol": "AAPL", "security_type": "stock", "exchange_id": 2, "current_price": 332.58, "price_change": 5.7, "price_change_percent": 1.7155, "bid": 332.55, "ask": 332.6, "bid_size": 80, "ask_size": 280, "open": 327.66, "high": 336.22, "low": 326.3, "close": 332.27, "prev_day_close": 332.27, "volume": 50716865, "iv30": 23.849, "iv30_change": 0.0, "iv30_change_percent": 0.0, "seqno": 35526851304, "last_trade_time": "2026-09-11T15:59:59", "tick": "down"}, "symbol": "AAPL"}
added connectors/src/cboe-delayed-quotes/index.ts +105 −0
@@ -0,0 +1,105 @@
1 +import { defineConnector, raw, type NormalizedBatch, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +import type { NormalizedObservation, RawObservation } from "@market-atlas/market-model";
3 +import { zonedTimeToUtc } from "@market-atlas/market-model";
4 +import { US_EQUITIES, US_ETFS, US_INDICES, equityHint, indexHint } from "../_shared/us-universe.js";
5 +
6 +const BASE = "https://cdn.cboe.com/api/global/delayed_quotes/quotes";
7 +const ET = "America/New_York";
8 +
9 +const seeds: ProposedInstrument[] = [
10 + ...US_EQUITIES.map(([s, n, v]) => ({ symbol: s, hint: equityHint(s, n, v, "EQUITY"), aliases: s.includes(".") ? [s.replace(".", "-"), s.replace(".", "/"), s.replace(".", " ")] : [] })),
11 + ...US_ETFS.map(([s, n, v]) => ({ symbol: s, hint: equityHint(s, n, v, "ETF") })),
12 + ...US_INDICES.map((i) => ({ symbol: i.cboe, hint: indexHint(i), aliases: [i.symbol, `^${i.symbol}`, `.${i.symbol}`] })),
13 +];
14 +const SYMBOLS = seeds.map((s) => s.symbol);
15 +
16 +/**
17 + * Cboe public delayed quotes (JSON per symbol, the same endpoint the cboe.com quote pages use).
18 + * Adaptive polling: every 60 s per symbol during the US session, every 15 min when closed, hourly on
19 + * weekends. Requests are rate-limited centrally (3/s on cdn.cboe.com). Data is 15-minute delayed.
20 + */
21 +export const cboeDelayedQuotes = defineConnector({
22 + metadata: {
23 + id: "cboe-delayed-quotes",
24 + name: "Cboe — delayed US quotes (equities, ETFs, indices)",
25 + version: "1.0.0",
26 + sourceId: "cboe",
27 + organization: "Cboe Global Markets",
28 + sourceType: "XHR",
29 + jurisdiction: "US",
30 + rightsStatus: "DELAYED",
31 + realtimeStatus: "DELAYED",
32 + expectedLatencyMs: 15 * 60_000,
33 + supportsStreaming: false,
34 + supportsHistorical: false,
35 + assetClasses: ["EQUITY", "ETF", "INDEX"],
36 + exchanges: ["xnys", "xnas", "arcx", "xcbo"],
37 + homepage: "https://www.cboe.com/delayed_quotes/",
38 + description: "Cboe's public delayed-quote JSON (last, bid/ask + sizes, open/high/low, previous close, volume, 30-day implied volatility) for a curated US universe of large caps, ETFs and the SPX/NDX/DJI/RUT/VIX indices. 15-minute delayed.",
39 + rightsNotes: "Delayed data displayed with attribution to Cboe Global Markets; labelled DELAYED everywhere. Not real time.",
40 + termsUrl: "https://www.cboe.com/about/terms-of-use/",
41 + sourceFamily: "cboe",
42 + enabled: true,
43 + },
44 + seeds,
45 + defaultSymbols: SYMBOLS,
46 + rateLimits: { "cdn.cboe.com": 1 }, // the CDN answers 429 above ~1 req/s
47 + schedule: { openMs: 120_000, closedMs: 15 * 60_000, weekendMs: 60 * 60_000, exchangeId: "xnys" },
48 + async poll(ctx) {
49 + const out: RawObservation[] = [];
50 + const symbols = ctx.watchedSymbols();
51 + // Sequential per symbol through the shared limiter; a failed symbol never aborts the batch.
52 + await Promise.all(
53 + symbols.map(async (sym) => {
54 + try {
55 + const { data, response } = await ctx.http.getJson<Record<string, unknown>>(`${BASE}/${encodeURIComponent(sym)}.json`, { timeoutMs: 15_000 });
56 + if (response.notModified) return;
57 + out.push(raw("cboe-delayed-quotes", "cboe", "quote", data, { symbol: sym, status: response.status, ms: response.durationMs }));
58 + } catch (err) {
59 + ctx.reportError(err, { symbol: sym });
60 + }
61 + }),
62 + );
63 + return out;
64 + },
65 + normalize(r): NormalizedBatch {
66 + const p = r.payload as { timestamp?: string; symbol?: string; data?: Record<string, unknown> };
67 + const d = p?.data;
68 + if (r.kind !== "quote" || !d || typeof d.symbol !== "string") return { observations: [] };
69 + const requested = typeof p.symbol === "string" ? p.symbol : d.symbol;
70 + const index = US_INDICES.find((i) => i.cboe === requested || `^${i.symbol}` === d.symbol);
71 + const symbol = index ? index.cboe : requested;
72 + const hint = index ? indexHint(index) : undefined;
73 + // last_trade_time is exchange local time without offset ("2026-09-11T15:59:59").
74 + const tradeTs = typeof d.last_trade_time === "string" ? zonedTimeToUtc(d.last_trade_time, ET) : null;
75 + const snapshotTs = typeof p.timestamp === "string" ? zonedTimeToUtc(p.timestamp.replace(" ", "T"), ET) : null;
76 + const seq = typeof d.seqno === "number" ? d.seqno : null;
77 + const observations: NormalizedObservation[] = [];
78 + const push = (field: NormalizedObservation["field"], v: unknown, ts: number | null, trust: NormalizedObservation["timestampTrust"]) => {
79 + const n = typeof v === "number" ? v : typeof v === "string" ? Number(v) : NaN;
80 + if (!Number.isFinite(n)) return;
81 + if ((field === "BID" || field === "ASK" || field === "LAST_PRICE" || field === "OPEN" || field === "HIGH" || field === "LOW" || field === "PREVIOUS_CLOSE") && n <= 0) return;
82 + observations.push({ symbol, instrumentHint: hint, field, value: n, currency: "USD", sourceTimestamp: ts, timestampTrust: trust, sequence: seq, rightsStatus: "DELAYED", realtimeStatus: "DELAYED", meta: { security_type: d.security_type } });
83 + };
84 + push("LAST_PRICE", d.current_price, tradeTs, tradeTs ? "EXCHANGE" : "SOURCE");
85 + push("OPEN", d.open, tradeTs, "EXCHANGE");
86 + push("HIGH", d.high, tradeTs, "EXCHANGE");
87 + push("LOW", d.low, tradeTs, "EXCHANGE");
88 + push("PREVIOUS_CLOSE", d.prev_day_close, tradeTs, "EXCHANGE");
89 + push("VOLUME", d.volume, tradeTs, "EXCHANGE");
90 + push("CHANGE", d.price_change, tradeTs, "EXCHANGE");
91 + push("CHANGE_PERCENT", d.price_change_percent, tradeTs, "EXCHANGE");
92 + // Quotes (bid/ask) are as of the snapshot, not the last trade.
93 + push("BID", d.bid, snapshotTs, "SOURCE");
94 + push("ASK", d.ask, snapshotTs, "SOURCE");
95 + push("BID_SIZE", d.bid_size, snapshotTs, "SOURCE");
96 + push("ASK_SIZE", d.ask_size, snapshotTs, "SOURCE");
97 + push("IMPLIED_VOLATILITY", d.iv30, snapshotTs, "SOURCE");
98 + return { observations };
99 + },
100 + async healthCheck(ctx) {
101 + const { data } = await ctx.http.getJson<{ data?: { current_price?: number } }>(`${BASE}/SPY.json`);
102 + return { ok: typeof data?.data?.current_price === "number", detail: `SPY=${data?.data?.current_price}` };
103 + },
104 + fixturesDir: "fixtures",
105 +});
added connectors/src/coinbase-ws/README.md +10 −0
@@ -0,0 +1,10 @@
1 +# coinbase-ws
2 +
3 +Coinbase Exchange public WebSocket (`wss://ws-feed.exchange.coinbase.com`), `ticker` + `heartbeat` channels.
4 +
5 +- **Type**: WEBSOCKET · **Rights**: PUBLIC_ATTRIBUTED · **Real-time**: REALTIME (exchange timestamps)
6 +- **Instruments**: 17 USD pairs + BTC/ETH in USDT and EUR (one multiplexed connection).
7 +- **Fields**: LAST_PRICE, OPEN/HIGH/LOW (24 h), BID, ASK, BID_SIZE, ASK_SIZE, VOLUME (24 h).
8 +- **Reconnect**: managed by the SDK (jittered backoff, stale after 90 s without frames, ping every 20 s).
9 +- **Quirks**: `open_24h`/`high_24h`/`low_24h` are rolling 24 h statistics, not session values; the sequence number is per product.
10 +- **Fixtures**: `fixtures/ticker.json`, `fixtures/malformed.json`.
added connectors/src/coinbase-ws/fixtures/malformed.json +1 −0
@@ -0,0 +1 @@
1 +{"type":"ticker","product_id":"BTC-USD","price":"not-a-number","best_bid":null,"time":"garbage"}
added connectors/src/coinbase-ws/fixtures/ticker.json +1 −0
@@ -0,0 +1 @@
1 +{"type":"ticker","sequence":135978839980,"product_id":"BTC-USD","price":"77279.97","open_24h":"77199.53","volume_24h":"6678.71253229","low_24h":"76030","high_24h":"79852.22","volume_30d":"209364.48922348","best_bid":"77279.96","best_bid_size":"0.61699174","best_ask":"77280.17","best_ask_size":"0.02500000","side":"buy","time":"2026-09-12T07:54:03.482921Z","trade_id":812345678,"last_size":"0.00120000"}
added connectors/src/coinbase-ws/index.ts +75 −0
@@ -0,0 +1,75 @@
1 +import { defineConnector, raw, type NormalizedBatch } from "@market-atlas/connector-sdk";
2 +import { parseTimestamp } from "@market-atlas/market-model";
3 +import { MAJOR_BASES, cryptoSeeds, tickerObservations } from "../_shared/crypto.js";
4 +
5 +const WS_URL = "wss://ws-feed.exchange.coinbase.com";
6 +const PAIRS: Array<[string, string]> = [...MAJOR_BASES.map((b) => [b, "USD"] as [string, string]), ["BTC", "USDT"], ["ETH", "USDT"], ["BTC", "EUR"], ["ETH", "EUR"]];
7 +const SYMBOLS = PAIRS.map(([b, q]) => `${b}-${q}`);
8 +
9 +/**
10 + * Coinbase Exchange public market-data feed, `ticker` channel: one multiplexed connection for all
11 + * products, best bid/ask, last trade price, 24h stats. Exchange-generated timestamps.
12 + */
13 +export const coinbaseWs = defineConnector({
14 + metadata: {
15 + id: "coinbase-ws",
16 + name: "Coinbase Exchange — ticker feed",
17 + version: "1.0.0",
18 + sourceId: "coinbase",
19 + organization: "Coinbase Global, Inc.",
20 + sourceType: "WEBSOCKET",
21 + jurisdiction: "US",
22 + rightsStatus: "PUBLIC_ATTRIBUTED",
23 + realtimeStatus: "REALTIME",
24 + expectedLatencyMs: 300,
25 + supportsStreaming: true,
26 + supportsHistorical: false,
27 + assetClasses: ["CRYPTO"],
28 + exchanges: ["coinbase"],
29 + homepage: "https://docs.cdp.coinbase.com/exchange/docs/websocket-overview",
30 + description: "Public, unauthenticated WebSocket market-data feed of Coinbase Exchange (ticker channel). One connection carries every subscribed product.",
31 + rightsNotes: "Public market data; displayed with attribution to Coinbase Exchange. Not for commercial redistribution as a raw feed.",
32 + termsUrl: "https://www.coinbase.com/legal/market_data",
33 + sourceFamily: "coinbase",
34 + enabled: true,
35 + },
36 + seeds: cryptoSeeds(PAIRS, "coinbase", (b, q) => `${b}-${q}`),
37 + defaultSymbols: SYMBOLS,
38 + async start(ctx) {
39 + const ws = ctx.openWebSocket(WS_URL, {
40 + label: "coinbase",
41 + staleAfterMs: 90_000,
42 + heartbeat: { intervalMs: 20_000 },
43 + onOpen: (sock) => {
44 + sock.send({ type: "subscribe", product_ids: ctx.watchedSymbols(), channels: ["ticker", "heartbeat"] });
45 + },
46 + onMessage: (data) => {
47 + let msg: any;
48 + try {
49 + msg = JSON.parse(data);
50 + } catch {
51 + return;
52 + }
53 + if (msg?.type === "ticker") ctx.emit(raw("coinbase-ws", "coinbase", "ticker", msg));
54 + else if (msg?.type === "error") ctx.reportError(new Error(String(msg.message ?? "coinbase error")), { reason: msg.reason });
55 + },
56 + });
57 + ws.connect();
58 + },
59 + normalize(r): NormalizedBatch {
60 + const m = r.payload as Record<string, unknown>;
61 + if (r.kind !== "ticker" || typeof m.product_id !== "string") return { observations: [] };
62 + const [base, quote] = m.product_id.split("-");
63 + if (!base || !quote) return { observations: [] };
64 + return {
65 + observations: tickerObservations(m.product_id, base, quote, "coinbase", { last: m.price, open: m.open_24h, high: m.high_24h, low: m.low_24h, bid: m.best_bid, ask: m.best_ask, bidSize: m.best_bid_size, askSize: m.best_ask_size, volume: m.volume_24h }, {
66 + sourceTimestamp: parseTimestamp(m.time),
67 + timestampTrust: "EXCHANGE",
68 + sequence: typeof m.sequence === "number" ? m.sequence : null,
69 + rightsStatus: "PUBLIC_ATTRIBUTED",
70 + realtimeStatus: "REALTIME",
71 + }),
72 + };
73 + },
74 + fixturesDir: "fixtures",
75 +});
added connectors/src/connectors.test.ts +247 −0
@@ -0,0 +1,247 @@
1 +import { describe, expect, it } from "vitest";
2 +import { dirname, join } from "node:path";
3 +import { fileURLToPath } from "node:url";
4 +import { loadFixture, makeTestContext, normalizeFixture, fakeFetch } from "@market-atlas/connector-sdk";
5 +import { ConnectorMetadataSchema } from "@market-atlas/market-model";
6 +import { CONNECTORS, SOURCES } from "./index.js";
7 +import { coinbaseWs } from "./coinbase-ws/index.js";
8 +import { krakenWs } from "./kraken-ws/index.js";
9 +import { binanceWs } from "./binance-ws/index.js";
10 +import { okxWs } from "./okx-ws/index.js";
11 +import { cboeDelayedQuotes } from "./cboe-delayed-quotes/index.js";
12 +import { ecbFrankfurter } from "./ecb-frankfurter/index.js";
13 +import { bankOfCanada } from "./bank-of-canada/index.js";
14 +import { usTreasuryYieldCurve } from "./us-treasury-yield-curve/index.js";
15 +import { secEdgarFilings } from "./sec-edgar-filings/index.js";
16 +import { secCompanyTickers, titleCase } from "./sec-company-tickers/index.js";
17 +import { nasdaqSymbolDirectory, cleanName } from "./nasdaq-symbol-directory/index.js";
18 +import { nasdaqTradeHalts } from "./nasdaq-trade-halts/index.js";
19 +import { nasdaqMarketCalendar, parseLongDate } from "./nasdaq-market-calendar/index.js";
20 +import { hfmarketdata } from "./hfmarketdata/index.js";
21 +
22 +const here = dirname(fileURLToPath(import.meta.url));
23 +const fx = (dir: string, name: string) => loadFixture(join(here, dir, "fixtures"), name);
24 +
25 +describe("registry", () => {
26 + it("every connector has valid metadata, a known source, docs and fixtures", () => {
27 + const ids = new Set<string>();
28 + for (const c of CONNECTORS) {
29 + expect(ConnectorMetadataSchema.safeParse(c.metadata).success).toBe(true);
30 + expect(ids.has(c.metadata.id)).toBe(false);
31 + ids.add(c.metadata.id);
32 + expect(SOURCES.some((s) => s.id === c.metadata.sourceId)).toBe(true);
33 + expect(c.metadata.rightsStatus).not.toBe("UNKNOWN");
34 + expect(c.fixturesDir).toBe("fixtures");
35 + }
36 + });
37 + it("covers the V1 source-type diversity goal", () => {
38 + const types = new Set(CONNECTORS.map((c) => c.metadata.sourceType));
39 + for (const t of ["WEBSOCKET", "XHR", "OFFICIAL_API", "HTML", "RSS", "BULK_FILE", "XML"]) expect(types.has(t as never)).toBe(true);
40 + });
41 +});
42 +
43 +describe("crypto WebSocket connectors", () => {
44 + it("coinbase ticker → observations with exchange timestamp", () => {
45 + const b = normalizeFixture(coinbaseWs, "ticker", fx("coinbase-ws", "ticker.json"));
46 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
47 + expect(last.value).toBe(77279.97);
48 + expect(last.symbol).toBe("BTC-USD");
49 + expect(last.instrumentHint?.base).toBe("BTC");
50 + expect(last.sourceTimestamp).toBe(Date.parse("2026-09-12T07:54:03.482921Z"));
51 + expect(last.timestampTrust).toBe("EXCHANGE");
52 + expect(b.observations.map((o) => o.field)).toContain("BID");
53 + expect(b.observations.map((o) => o.field)).toContain("VOLUME");
54 + });
55 + it("coinbase malformed payload yields no observations and does not throw", () => {
56 + const b = normalizeFixture(coinbaseWs, "ticker", fx("coinbase-ws", "malformed.json"));
57 + expect(b.observations).toHaveLength(0);
58 + });
59 + it("kraken snapshot → connector-timed observations", () => {
60 + const b = normalizeFixture(krakenWs, "ticker", fx("kraken-ws", "ticker.json"));
61 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
62 + expect(last.value).toBe(77281.3);
63 + expect(last.sourceTimestamp).toBeNull();
64 + expect(last.timestampTrust).toBe("CONNECTOR");
65 + expect(b.observations.find((o) => o.field === "VWAP")?.value).toBe(77900.1);
66 + });
67 + it("binance miniTicker → USDT instrument", () => {
68 + const b = normalizeFixture(binanceWs, "miniTicker", fx("binance-ws", "miniTicker.json"));
69 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
70 + expect(last.value).toBeCloseTo(77298.87);
71 + expect(last.instrumentHint?.quote).toBe("USDT");
72 + expect(last.sourceTimestamp).toBe(1789199657016);
73 + });
74 + it("okx tickers → observations", () => {
75 + const b = normalizeFixture(okxWs, "tickers", fx("okx-ws", "tickers.json"));
76 + expect(b.observations.find((o) => o.field === "LAST_PRICE")?.value).toBe(77303.9);
77 + expect(b.observations.find((o) => o.field === "ASK")?.value).toBe(77303.9);
78 + expect(b.observations[0]?.sourceTimestamp).toBe(1789199660123);
79 + });
80 + it("unexpected schema (wrong kind) is ignored", () => {
81 + expect(normalizeFixture(okxWs, "trades", { arg: { channel: "trades" }, data: [{ px: "1" }] }).observations).toHaveLength(0);
82 + });
83 + it("streaming connectors open exactly one socket and subscribe on open", async () => {
84 + for (const def of [coinbaseWs, krakenWs, binanceWs, okxWs]) {
85 + const ctx = makeTestContext(def);
86 + await def.start!(ctx);
87 + expect(ctx.sockets).toHaveLength(1);
88 + ctx.sockets[0]!.close();
89 + }
90 + });
91 +});
92 +
93 +describe("cboe-delayed-quotes", () => {
94 + it("equity quote → DELAYED observations with Eastern timestamps converted to UTC", () => {
95 + const b = normalizeFixture(cboeDelayedQuotes, "quote", fx("cboe-delayed-quotes", "quote.json"));
96 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
97 + expect(last.value).toBe(332.58);
98 + expect(last.realtimeStatus).toBe("DELAYED");
99 + expect(last.rightsStatus).toBe("DELAYED");
100 + // 2026-09-11T15:59:59 America/New_York (EDT, UTC-4) → 19:59:59Z
101 + expect(new Date(last.sourceTimestamp!).toISOString()).toBe("2026-09-11T19:59:59.000Z");
102 + expect(b.observations.find((o) => o.field === "PREVIOUS_CLOSE")?.value).toBe(332.27);
103 + expect(b.observations.find((o) => o.field === "VOLUME")?.value).toBe(50716865);
104 + expect(b.observations.find((o) => o.field === "IMPLIED_VOLATILITY")?.value).toBe(23.849);
105 + });
106 + it("index quote keeps the _SPX source symbol and INDEX hint", () => {
107 + const b = normalizeFixture(cboeDelayedQuotes, "quote", fx("cboe-delayed-quotes", "index.json"));
108 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
109 + expect(last.symbol).toBe("_SPX");
110 + expect(last.instrumentHint?.assetClass).toBe("INDEX");
111 + expect(b.observations.find((o) => o.field === "VOLUME")?.value).toBe(0);
112 + });
113 + it("never-traded symbol (zeros) yields no price observations", () => {
114 + const b = normalizeFixture(cboeDelayedQuotes, "quote", fx("cboe-delayed-quotes", "closed_market.json"));
115 + expect(b.observations.filter((o) => o.field === "LAST_PRICE" || o.field === "BID")).toHaveLength(0);
116 + });
117 + it("poll tolerates a failing symbol", async () => {
118 + const ctx = makeTestContext(cboeDelayedQuotes, {
119 + symbols: ["AAPL", "BROKEN"],
120 + fetchImpl: fakeFetch({ "/AAPL.json": JSON.stringify(fx("cboe-delayed-quotes", "quote.json")), "/BROKEN.json": { status: 500, body: "boom" } }),
121 + });
122 + const raws = await cboeDelayedQuotes.poll!(ctx);
123 + expect(raws).toHaveLength(1);
124 + expect(ctx.errors).toHaveLength(1);
125 + });
126 +});
127 +
128 +describe("official sources", () => {
129 + it("frankfurter USD base → conventional pairs, latest + previous close, EUR pairs skipped", () => {
130 + const b = normalizeFixture(ecbFrankfurter, "timeseries", fx("ecb-frankfurter", "timeseries.json"));
131 + const usdcad = b.observations.filter((o) => o.symbol === "USDCAD");
132 + expect(usdcad.find((o) => o.field === "LAST_PRICE")?.value).toBe(1.3858);
133 + expect(usdcad.find((o) => o.field === "PREVIOUS_CLOSE")?.value).toBe(1.3871);
134 + expect(b.observations.some((o) => o.symbol === "EURUSD")).toBe(false); // EUR pairs come from the EUR-base call
135 + const gbpusd = b.observations.find((o) => o.symbol === "GBPUSD" && o.field === "LAST_PRICE")!;
136 + expect(gbpusd.value).toBeCloseTo(1 / 0.7403, 6);
137 + expect(gbpusd.realtimeStatus).toBe("END_OF_DAY");
138 + expect(gbpusd.rightsStatus).toBe("OFFICIAL_OPEN_DATA");
139 + });
140 + it("bank of canada fx + rates", () => {
141 + const fxb = normalizeFixture(bankOfCanada, "fx", fx("bank-of-canada", "fx.json"));
142 + expect(fxb.observations.find((o) => o.symbol === "USDCAD" && o.field === "LAST_PRICE")?.value).toBe(1.3855);
143 + expect(fxb.observations.find((o) => o.symbol === "EURCAD" && o.field === "PREVIOUS_CLOSE")?.value).toBe(1.606);
144 + const rates = normalizeFixture(bankOfCanada, "rates", fx("bank-of-canada", "rates.json"));
145 + expect(rates.observations.find((o) => o.symbol === "CA_POLICY_RATE" && o.field === "RATE")?.value).toBe(2.25);
146 + expect(rates.observations.find((o) => o.symbol === "CA_10Y" && o.field === "YIELD")?.value).toBe(3.24);
147 + expect(rates.observations.find((o) => o.symbol === "CA_10Y" && o.field === "PREVIOUS_CLOSE")?.value).toBe(3.21);
148 + });
149 + it("treasury XML → 13 tenors with yields and previous close", () => {
150 + const b = normalizeFixture(usTreasuryYieldCurve, "month", fx("us-treasury-yield-curve", "month.xml"));
151 + const us10 = b.observations.filter((o) => o.symbol === "US10Y");
152 + expect(us10.find((o) => o.field === "YIELD")?.value).toBe(4.02);
153 + expect(us10.find((o) => o.field === "PREVIOUS_CLOSE")?.value).toBe(4.05);
154 + expect(us10.find((o) => o.field === "YIELD")?.instrumentHint?.assetClass).toBe("TREASURY");
155 + expect(b.observations.some((o) => o.symbol === "US4M")).toBe(false); // absent tenor is skipped
156 + });
157 +});
158 +
159 +describe("regulatory / directory / halts", () => {
160 + it("edgar atom → filings; Form 4 reporting is not material, 8-K is", () => {
161 + const b = normalizeFixture(secEdgarFilings, "atom", fx("sec-edgar-filings", "atom.xml"));
162 + expect(b.filings).toHaveLength(2);
163 + const f4 = b.filings!.find((f) => f.formType === "4")!;
164 + expect(f4.cik).toBe("1734770");
165 + expect(f4.metadata?.material).toBe(false);
166 + const k8 = b.filings!.find((f) => f.formType === "8-K")!;
167 + expect(k8.companyName).toBe("Apple Inc.");
168 + expect(k8.cik).toBe("320193");
169 + expect(k8.metadata?.material).toBe(true);
170 + expect(k8.url).toContain("0000320193-26-000099");
171 + });
172 + it("sec directory → listed created, OTC enrich-only", () => {
173 + const b = normalizeFixture(secCompanyTickers, "directory", fx("sec-company-tickers", "directory.json"));
174 + const brk = b.instruments!.find((i) => i.symbol === "BRK.B")!;
175 + expect(brk.hint.exchangeId).toBe("xnys");
176 + expect(brk.hint.cik).toBe("1067983");
177 + expect(brk.aliases).toContain("BRK-B");
178 + expect(b.instruments!.find((i) => i.symbol === "SOTC")?.createIfMissing).toBe(false);
179 + expect(titleCase("MICROSOFT CORP")).toBe("Microsoft Corp");
180 + expect(titleCase("Apple Inc.")).toBe("Apple Inc.");
181 + });
182 + it("nasdaq directories → instruments with venues, ETF flags and no test issues", () => {
183 + const n = normalizeFixture(nasdaqSymbolDirectory, "nasdaqlisted", fx("nasdaq-symbol-directory", "nasdaqlisted.txt"));
184 + expect(n.instruments!.map((i) => i.symbol)).toEqual(["AAAP", "AACG", "AAPL", "QQQ"]);
185 + expect(n.instruments!.find((i) => i.symbol === "AAPL")?.hint.name).toBe("Apple Inc.");
186 + expect(n.instruments!.find((i) => i.symbol === "QQQ")?.hint.assetClass).toBe("ETF");
187 + expect(n.instruments!.find((i) => i.symbol === "AACG")?.hint.securityType).toBe("ADR");
188 + const o = normalizeFixture(nasdaqSymbolDirectory, "otherlisted", fx("nasdaq-symbol-directory", "otherlisted.txt"));
189 + const brk = o.instruments!.find((i) => i.symbol === "BRK.B")!;
190 + expect(brk.hint.exchangeId).toBe("xnys");
191 + expect(brk.aliases).toEqual(expect.arrayContaining(["BRK-B", "BRK B"]));
192 + expect(o.instruments!.find((i) => i.symbol === "SPY")?.hint.exchangeId).toBe("arcx");
193 + expect(o.instruments!.find((i) => i.symbol === "BAC-PB")?.hint.securityType).toBe("PREFERRED");
194 + expect(cleanName("Alcoa Corporation Common Stock ")).toBe("Alcoa Corporation");
195 + });
196 + it("halts rss → halt + resume events with Eastern → UTC conversion", () => {
197 + const b = normalizeFixture(nasdaqTradeHalts, "rss", fx("nasdaq-trade-halts", "rss.xml"));
198 + const halts = b.events!.filter((e) => e.type === "TRADING_HALT");
199 + expect(halts).toHaveLength(2);
200 + const hubc = halts.find((e) => e.symbols?.[0] === "HUBC")!;
201 + expect(new Date(hubc.timestamp).toISOString()).toBe("2026-09-11T23:50:00.000Z");
202 + expect(hubc.data?.reason).toBe("News pending");
203 + const resume = b.events!.find((e) => e.type === "TRADING_RESUME")!;
204 + expect(resume.symbols).toEqual(["ABCD"]);
205 + expect(new Date(resume.timestamp).toISOString()).toBe("2026-09-11T14:22:31.000Z");
206 + expect(b.events!.find((e) => e.symbols?.[0] === "ABCD" && e.type === "TRADING_HALT")?.severity).toBe("NOTICE");
207 + });
208 + it("calendar html → holidays for every US venue, early close parsed, change event only when changed", () => {
209 + const html = fx("nasdaq-market-calendar", "page.html") as string;
210 + const first = nasdaqMarketCalendar.normalize({ connectorId: "nasdaq-market-calendar", sourceId: "nasdaq-trader", kind: "page", payload: html, receivedAt: Date.now(), meta: { changed: true, first: true } });
211 + expect(first.events).toHaveLength(0);
212 + expect(first.holidays!.filter((h) => h.exchangeId === "xnys")).toHaveLength(7);
213 + const early = first.holidays!.find((h) => h.date === "2026-11-27" && h.exchangeId === "xnas")!;
214 + expect(early.kind).toBe("EARLY_CLOSE");
215 + expect(early.closeTime).toBe("13:00");
216 + expect(first.holidays!.find((h) => h.date === "2026-01-19")?.name).toBe("Martin Luther King, Jr. Day");
217 + const changed = nasdaqMarketCalendar.normalize({ connectorId: "nasdaq-market-calendar", sourceId: "nasdaq-trader", kind: "page", payload: html, receivedAt: Date.now(), meta: { changed: true, first: false, document_hash: "abc", changed_sections: ["x"] } });
218 + expect(changed.events).toHaveLength(1);
219 + expect(changed.events![0]!.type).toBe("DOCUMENT_CHANGED");
220 + expect(parseLongDate("December 25, 2026")).toBe("2026-12-25");
221 + });
222 +});
223 +
224 +describe("hfmarketdata-daily", () => {
225 + it("daily bars → 1d bars + END_OF_DAY reference observations", () => {
226 + const b = hfmarketdata.normalize({ connectorId: "hfmarketdata-daily", sourceId: "hfmarketdata", kind: "bars", payload: fx("hfmarketdata", "bars.json"), receivedAt: Date.now(), meta: { symbol: "AAPL" } });
227 + expect(b.bars).toHaveLength(3);
228 + expect(b.bars![0]!.ts).toBe(Date.parse("2026-09-01T00:00:00Z"));
229 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
230 + expect(last.value).toBe(328.21);
231 + expect(last.realtimeStatus).toBe("END_OF_DAY");
232 + expect(b.observations.find((o) => o.field === "PREVIOUS_CLOSE")?.value).toBe(324.96);
233 + // 2026-09-03 16:00 America/New_York → 20:00Z
234 + expect(new Date(last.sourceTimestamp!).toISOString()).toBe("2026-09-03T20:00:00.000Z");
235 + });
236 + it("continuous futures → COMMODITY instrument with contract metadata", () => {
237 + const b = hfmarketdata.normalize({ connectorId: "hfmarketdata-daily", sourceId: "hfmarketdata", kind: "continuous", payload: fx("hfmarketdata", "continuous.json"), receivedAt: Date.now(), meta: { symbol: "CL=F" } });
238 + expect(b.bars).toHaveLength(2);
239 + const last = b.observations.find((o) => o.field === "LAST_PRICE")!;
240 + expect(last.value).toBe(91.01);
241 + expect(last.instrumentHint?.assetClass).toBe("COMMODITY");
242 + expect(last.meta?.contract).toBe("CLV26");
243 + });
244 + it("unknown symbol meta yields nothing", () => {
245 + expect(hfmarketdata.normalize({ connectorId: "hfmarketdata-daily", sourceId: "hfmarketdata", kind: "bars", payload: { data: [] }, receivedAt: 0, meta: { symbol: "NOPE" } }).observations).toHaveLength(0);
246 + });
247 +});
added connectors/src/ecb-frankfurter/README.md +9 −0
@@ -0,0 +1,9 @@
1 +# ecb-frankfurter
2 +
3 +ECB euro foreign exchange reference rates redistributed by the open Frankfurter API (`https://api.frankfurter.dev/v1/{start}..?base=EUR|USD`).
4 +
5 +- **Type**: OFFICIAL_API · **Rights**: OFFICIAL_OPEN_DATA (attribution ECB) · **Real-time**: END_OF_DAY (one fixing ≈ 16:00 CET)
6 +- **Instruments**: conventional pairs (EURUSD, USDJPY, GBPUSD, USDCAD…, EUR crosses). Values derived from the fixing carry `meta.fixing_date`.
7 +- **Fields**: LAST_PRICE (latest fixing), PREVIOUS_CLOSE (previous business day).
8 +- **Consensus**: END_OF_DAY tier — superseded when a live/delayed source for the same pair exists; otherwise it is the canonical (indicative) value.
9 +- **Schedule**: hourly (the fixing changes once a day; ETag makes most polls 304).
added connectors/src/ecb-frankfurter/fixtures/timeseries.json +1 −0
@@ -0,0 +1 @@
1 +{"amount":1.0,"base":"USD","start_date":"2026-09-10","end_date":"2026-09-11","rates":{"2026-09-10":{"CAD":1.3871,"EUR":0.8631,"GBP":0.7411,"JPY":153.9,"CHF":0.8160},"2026-09-11":{"CAD":1.3858,"EUR":0.86266,"GBP":0.7403,"JPY":154.04,"CHF":0.8153}}}
added connectors/src/ecb-frankfurter/index.ts +96 −0
@@ -0,0 +1,96 @@
1 +import { defineConnector, raw, type NormalizedBatch, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +import type { InstrumentHint, NormalizedObservation } from "@market-atlas/market-model";
3 +import { zonedTimeToUtc } from "@market-atlas/market-model";
4 +
5 +const BASE = "https://api.frankfurter.dev/v1";
6 +const BASES = ["EUR", "USD"];
7 +const CURRENCY_NAMES: Record<string, string> = { EUR: "Euro", USD: "US Dollar", GBP: "British Pound", JPY: "Japanese Yen", CHF: "Swiss Franc", CAD: "Canadian Dollar", AUD: "Australian Dollar", NZD: "New Zealand Dollar", CNY: "Chinese Yuan", HKD: "Hong Kong Dollar", SGD: "Singapore Dollar", SEK: "Swedish Krona", NOK: "Norwegian Krone", DKK: "Danish Krone", PLN: "Polish Złoty", CZK: "Czech Koruna", HUF: "Hungarian Forint", MXN: "Mexican Peso", BRL: "Brazilian Real", INR: "Indian Rupee", KRW: "South Korean Won", ZAR: "South African Rand", TRY: "Turkish Lira", ILS: "Israeli Shekel", IDR: "Indonesian Rupiah", MYR: "Malaysian Ringgit", PHP: "Philippine Peso", THB: "Thai Baht", ISK: "Icelandic Króna", RON: "Romanian Leu", BGN: "Bulgarian Lev" };
8 +const FEATURED = new Set(["EURUSD", "USDJPY", "GBPUSD", "USDCHF", "USDCAD", "AUDUSD", "USDCNY", "EURGBP"]);
9 +
10 +export function fxHint(base: string, quote: string): InstrumentHint {
11 + return { assetClass: "FOREX", name: `${CURRENCY_NAMES[base] ?? base} / ${CURRENCY_NAMES[quote] ?? quote}`, exchangeId: null, currency: quote, country: "XX", base, quote, securityType: "SPOT", metadata: { featured: FEATURED.has(`${base}${quote}`) } };
12 +}
13 +
14 +/** Conventional market quoting: EUR/GBP/AUD/NZD are quoted as base against USD; USD is base against the rest. */
15 +export function conventionalPair(a: string, b: string): [string, string] {
16 + const priority = ["EUR", "GBP", "AUD", "NZD", "USD"];
17 + const ia = priority.indexOf(a);
18 + const ib = priority.indexOf(b);
19 + if (ia === -1 && ib === -1) return [a, b];
20 + if (ia === -1) return [b, a];
21 + if (ib === -1) return [a, b];
22 + return ia < ib ? [a, b] : [b, a];
23 +}
24 +
25 +const seeds: ProposedInstrument[] = Object.keys(CURRENCY_NAMES)
26 + .flatMap((c) => (c === "USD" ? [] : [conventionalPair("USD", c)]))
27 + .concat(Object.keys(CURRENCY_NAMES).flatMap((c) => (c === "EUR" || c === "USD" ? [] : [conventionalPair("EUR", c)])))
28 + .map(([b, q]) => ({ symbol: `${b}${q}`, hint: fxHint(b, q), aliases: [`${b}/${q}`, `${b}-${q}`] }));
29 +
30 +/**
31 + * ECB euro foreign exchange reference rates via the Frankfurter open API (a 1:1 redistribution of the
32 + * ECB daily fixing, ~16:00 CET). Two requests per poll (EUR and USD base, 8-day window for previous close).
33 + */
34 +export const ecbFrankfurter = defineConnector({
35 + metadata: {
36 + id: "ecb-frankfurter",
37 + name: "ECB reference rates (Frankfurter)",
38 + version: "1.0.0",
39 + sourceId: "ecb-frankfurter",
40 + organization: "European Central Bank",
41 + sourceType: "OFFICIAL_API",
42 + jurisdiction: "EU",
43 + rightsStatus: "OFFICIAL_OPEN_DATA",
44 + realtimeStatus: "END_OF_DAY",
45 + expectedLatencyMs: null,
46 + supportsStreaming: false,
47 + supportsHistorical: true,
48 + assetClasses: ["FOREX"],
49 + exchanges: [],
50 + homepage: "https://frankfurter.dev",
51 + description: "Daily ECB euro reference rates for 30 currencies, cross-computed to USD pairs. One fixing per business day (~16:00 CET); values are indicative reference rates, not tradable quotes.",
52 + rightsNotes: "ECB reference rates are free to reuse with attribution to the ECB. Frankfurter is an open-source (MIT) redistribution.",
53 + termsUrl: "https://www.ecb.europa.eu/services/disclaimer/html/index.en.html",
54 + sourceFamily: "ecb",
55 + enabled: true,
56 + },
57 + seeds,
58 + defaultSymbols: seeds.map((s) => s.symbol),
59 + rateLimits: { "api.frankfurter.dev": 1 },
60 + schedule: { intervalMs: 60 * 60_000 },
61 + async poll(ctx) {
62 + const end = new Date();
63 + const start = new Date(end.getTime() - 8 * 86_400_000).toISOString().slice(0, 10);
64 + const out = [];
65 + for (const base of BASES) {
66 + const { data, response } = await ctx.http.getJson(`${BASE}/${start}..?base=${base}`);
67 + if (response.notModified) continue;
68 + out.push(raw("ecb-frankfurter", "ecb-frankfurter", "timeseries", data, { base }));
69 + }
70 + return out;
71 + },
72 + normalize(r): NormalizedBatch {
73 + const p = r.payload as { base?: string; rates?: Record<string, Record<string, number>> };
74 + if (r.kind !== "timeseries" || !p.base || !p.rates) return { observations: [] };
75 + const dates = Object.keys(p.rates).sort();
76 + const latest = dates[dates.length - 1];
77 + const previous = dates[dates.length - 2];
78 + if (!latest) return { observations: [] };
79 + const observations: NormalizedObservation[] = [];
80 + const emit = (date: string, field: "LAST_PRICE" | "PREVIOUS_CLOSE", refDate: string) => {
81 + const ts = zonedTimeToUtc(`${date}T16:00:00`, "Europe/Berlin");
82 + for (const [ccy, rate] of Object.entries(p.rates![date] ?? {})) {
83 + if (typeof rate !== "number" || !Number.isFinite(rate) || rate <= 0) continue;
84 + // Only emit the conventional orientation; avoid duplicating EUR-quoted pairs from the USD base call.
85 + const [b, q] = conventionalPair(p.base!, ccy);
86 + if (p.base === "USD" && (b === "EUR" || q === "EUR")) continue;
87 + const value = b === p.base ? rate : 1 / rate;
88 + observations.push({ symbol: `${b}${q}`, instrumentHint: fxHint(b, q), field, value: Number(value.toPrecision(8)), currency: q, sourceTimestamp: ts, timestampTrust: "SOURCE", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", meta: { fixing_date: date, reference_date: refDate, base: p.base } });
89 + }
90 + };
91 + emit(latest, "LAST_PRICE", latest);
92 + if (previous) emit(previous, "PREVIOUS_CLOSE", latest);
93 + return { observations, stats: { dates: dates.length } };
94 + },
95 + fixturesDir: "fixtures",
96 +});
added connectors/src/hfmarketdata/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# hfmarketdata-daily
2 +
3 +HF Market Data REST API (`/v1/bars/{asset}/{ticker}?timeframe=1day`, `/v1/futures/{root}/continuous`), Bearer key `HFMD_API_KEY` (optional; raises limits).
4 +
5 +- **Type**: OFFICIAL_API · **Rights**: LICENSED (sister platform, same operator) · **Real-time**: END_OF_DAY (lake refreshed weekly)
6 +- **Universe**: curated US equities/ETFs/indices, 11 FX pairs, 10 crypto, 13 CME continuous futures (`GC=F`, `CL=F`, `SI=F`, `NG=F`, `HG=F`, `ZC=F`, `ZW=F`, `ZS=F`, `ES=F`, `NQ=F`, `ZN=F`, `6E=F`, `BTC=F`).
7 +- **Output**: `bars` (1d, producer `hfmarketdata-daily`, never overwritten by consensus-derived bars) + END_OF_DAY observations (LAST_PRICE/CLOSE/OPEN/HIGH/LOW/VOLUME/PREVIOUS_CLOSE). Superseded in consensus when a live/delayed source exists for the instrument.
8 +- **Pacing**: full universe (~120 requests) every 6 h through a 1.5 req/s bucket (≈ 80 s per poll).
added connectors/src/hfmarketdata/fixtures/bars.json +1 −0
@@ -0,0 +1 @@
1 +{"count":3,"data":[{"ticker":"AAPL","datetime":"2026-09-03","open":324.87,"high":330.81,"low":324.11,"close":328.21,"volume":37225838.0},{"ticker":"AAPL","datetime":"2026-09-02","open":326.865,"high":328.4,"low":323.53,"close":324.96,"volume":33776370.0},{"ticker":"AAPL","datetime":"2026-09-01","open":316.98,"high":327.3,"low":314.73,"close":325.13,"volume":53167388.0}]}
added connectors/src/hfmarketdata/fixtures/continuous.json +1 −0
@@ -0,0 +1 @@
1 +{"data":[{"symbol":"CLV26","datetime":"2026-09-01","open":86.31,"high":90.97,"low":86.13,"close":90.22,"volume":331106.0,"open_interest":255873.0},{"symbol":"CLV26","datetime":"2026-09-02","open":90.65,"high":92.29,"low":88.97,"close":91.01,"volume":283450.0,"open_interest":248095.0}],"meta":{"count":2,"root":"CL","interval":"1d","roll":"volume","adjust":"none","depth":1,"session":"all","timezone":"UTC"}}
added connectors/src/hfmarketdata/index.ts +141 −0
@@ -0,0 +1,141 @@
1 +import { defineConnector, raw, type NormalizedBatch, type ProposedBar, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +import type { InstrumentHint, NormalizedObservation, RawObservation } from "@market-atlas/market-model";
3 +import { zonedTimeToUtc } from "@market-atlas/market-model";
4 +import { US_EQUITIES, US_ETFS, US_INDICES, equityHint, indexHint } from "../_shared/us-universe.js";
5 +import { fxHint } from "../ecb-frankfurter/index.js";
6 +import { cryptoHint } from "../_shared/crypto.js";
7 +
8 +const BASE = "https://www.hfmarketdata.io/v1";
9 +const ET = "America/New_York";
10 +
11 +interface Target {
12 + path: string; // request path after /v1
13 + symbol: string; // Market Atlas source symbol
14 + hint: InstrumentHint;
15 + kind: "bars" | "continuous";
16 + sessionCloseLocal: string; // local close time of the daily bar
17 + timezone: string;
18 +}
19 +
20 +const FUTURES: Array<[string, string, string]> = [
21 + ["GC", "Gold futures (continuous)", "xcme"],
22 + ["SI", "Silver futures (continuous)", "xcme"],
23 + ["CL", "WTI crude oil futures (continuous)", "xcme"],
24 + ["NG", "Henry Hub natural gas futures (continuous)", "xcme"],
25 + ["HG", "Copper futures (continuous)", "xcme"],
26 + ["ZC", "Corn futures (continuous)", "xcme"],
27 + ["ZW", "Wheat futures (continuous)", "xcme"],
28 + ["ZS", "Soybean futures (continuous)", "xcme"],
29 + ["ES", "E-mini S&P 500 futures (continuous)", "xcme"],
30 + ["NQ", "E-mini Nasdaq-100 futures (continuous)", "xcme"],
31 + ["ZN", "10-year T-note futures (continuous)", "xcme"],
32 + ["6E", "Euro FX futures (continuous)", "xcme"],
33 + ["BTC", "Bitcoin futures (continuous)", "xcme"],
34 +];
35 +const FX_PAIRS = ["EURUSD", "USDJPY", "GBPUSD", "USDCHF", "USDCAD", "AUDUSD", "NZDUSD", "EURGBP", "EURJPY", "USDMXN", "USDCNH"];
36 +const CRYPTO = ["BTC", "ETH", "SOL", "XRP", "ADA", "DOGE", "LTC", "LINK", "AVAX", "DOT"];
37 +
38 +export const commodityHint = (root: string, name: string, exchangeId: string): InstrumentHint => ({
39 + assetClass: ["ES", "NQ", "ZN", "6E", "BTC"].includes(root) ? "FUTURE" : "COMMODITY",
40 + name,
41 + exchangeId,
42 + mic: "XCME",
43 + currency: "USD",
44 + country: "US",
45 + securityType: "CONTINUOUS_FUTURE",
46 + metadata: { featured: ["GC", "CL", "SI", "NG", "ES"].includes(root), roll: "volume", adjust: "none" },
47 +});
48 +
49 +const TARGETS: Target[] = [
50 + ...US_EQUITIES.map(([s, n, v]) => ({ path: `/bars/stock/${encodeURIComponent(s.replace(".", "-"))}?timeframe=1day&limit=400&order=desc`, symbol: s, hint: equityHint(s, n, v, "EQUITY"), kind: "bars" as const, sessionCloseLocal: "16:00", timezone: ET })),
51 + ...US_ETFS.map(([s, n, v]) => ({ path: `/bars/etf/${s}?timeframe=1day&limit=400&order=desc`, symbol: s, hint: equityHint(s, n, v, "ETF"), kind: "bars" as const, sessionCloseLocal: "16:00", timezone: ET })),
52 + ...US_INDICES.filter((i) => i.hfmd).map((i) => ({ path: `/bars/index/${i.hfmd}?timeframe=1day&limit=400&order=desc`, symbol: i.cboe, hint: indexHint(i), kind: "bars" as const, sessionCloseLocal: "16:00", timezone: ET })),
53 + ...FX_PAIRS.map((p) => ({ path: `/bars/fx/${p}?timeframe=1day&limit=400&order=desc`, symbol: p, hint: fxHint(p.slice(0, 3), p.slice(3)), kind: "bars" as const, sessionCloseLocal: "17:00", timezone: ET })),
54 + ...CRYPTO.map((c) => ({ path: `/bars/crypto/${c}?timeframe=1day&limit=400&order=desc`, symbol: `${c}-USD`, hint: cryptoHint(c, "USD", "coinbase"), kind: "bars" as const, sessionCloseLocal: "23:59:59", timezone: "UTC" })),
55 + ...FUTURES.map(([root, name, ex]) => ({ path: `/futures/${root}/continuous?timeframe=1day&limit=400&roll=volume&adjust=none`, symbol: `${root}=F`, hint: commodityHint(root, name, ex), kind: "continuous" as const, sessionCloseLocal: "17:00", timezone: "America/Chicago" })),
56 +];
57 +
58 +const seeds: ProposedInstrument[] = FUTURES.map(([root, name, ex]) => ({ symbol: `${root}=F`, hint: commodityHint(root, name, ex), aliases: [root, `${root}1!`] }));
59 +
60 +/**
61 + * HF Market Data (Market Atlas' sister platform, own FirstRate-derived lake): daily OHLCV history for
62 + * the curated universe + CME continuous futures (gold, oil, gas, grains, index/rates futures).
63 + * Provides chart history (bars, producer = this connector) and END_OF_DAY reference values.
64 + */
65 +export const hfmarketdata = defineConnector({
66 + metadata: {
67 + id: "hfmarketdata-daily",
68 + name: "HF Market Data — daily bars & continuous futures",
69 + version: "1.0.0",
70 + sourceId: "hfmarketdata",
71 + organization: "spboucher.ai",
72 + sourceType: "OFFICIAL_API",
73 + jurisdiction: "CA",
74 + rightsStatus: "LICENSED",
75 + realtimeStatus: "END_OF_DAY",
76 + expectedLatencyMs: null,
77 + supportsStreaming: false,
78 + supportsHistorical: true,
79 + assetClasses: ["EQUITY", "ETF", "INDEX", "FOREX", "CRYPTO", "COMMODITY", "FUTURE"],
80 + exchanges: ["xcme"],
81 + homepage: "https://www.hfmarketdata.io",
82 + description: "Daily OHLCV bars (up to 400 sessions) for the curated US universe, major FX pairs, crypto and CME continuous futures from the HF Market Data lake (updated weekly from FirstRate). Backfills chart history and provides end-of-day reference closes.",
83 + rightsNotes: "Licensed sister platform (same operator). Redistribution of daily bars permitted under the HF Market Data data licence.",
84 + termsUrl: "https://www.hfmarketdata.io/data-license",
85 + sourceFamily: "firstrate",
86 + enabled: true,
87 + },
88 + seeds,
89 + defaultSymbols: TARGETS.map((t) => t.symbol),
90 + rateLimits: { "www.hfmarketdata.io": 1.5 },
91 + schedule: { intervalMs: 6 * 60 * 60_000 },
92 + async poll(ctx) {
93 + const key = ctx.secret("HFMD_API_KEY");
94 + const headers: Record<string, string> = key ? { authorization: `Bearer ${key}` } : {};
95 + const out: RawObservation[] = [];
96 + // ~120 requests every 6 h through the 1.5 req/s bucket (≈ 80 s per poll) — far below quotas.
97 + for (const t of TARGETS) {
98 + try {
99 + const { data, response } = await ctx.http.getJson(`${BASE}${t.path}`, { headers, timeoutMs: 45_000 });
100 + if (response.notModified) continue;
101 + out.push(raw("hfmarketdata-daily", "hfmarketdata", t.kind, data, { symbol: t.symbol }));
102 + } catch (err) {
103 + ctx.reportError(err, { symbol: t.symbol });
104 + }
105 + }
106 + return out;
107 + },
108 + normalize(r): NormalizedBatch {
109 + const symbol = r.meta?.symbol as string | undefined;
110 + const target = TARGETS.find((t) => t.symbol === symbol);
111 + if (!target) return { observations: [] };
112 + const p = r.payload as { data?: Array<Record<string, unknown>> };
113 + if (!Array.isArray(p.data)) return { observations: [] };
114 + const rows = p.data
115 + .map((d) => ({ date: String(d.datetime ?? "").slice(0, 10), open: num(d.open), high: num(d.high), low: num(d.low), close: num(d.close), volume: num(d.volume), contract: typeof d.symbol === "string" ? d.symbol : null }))
116 + .filter((d) => /^\d{4}-\d{2}-\d{2}$/.test(d.date) && d.open != null && d.high != null && d.low != null && d.close != null)
117 + .sort((a, b) => a.date.localeCompare(b.date));
118 + const bars: ProposedBar[] = rows.map((d) => ({ symbol: target.symbol, instrumentHint: target.hint, resolution: "1d", ts: Date.parse(`${d.date}T00:00:00Z`), open: d.open!, high: d.high!, low: d.low!, close: d.close!, volume: d.volume }));
119 + const last = rows[rows.length - 1];
120 + const prev = rows[rows.length - 2];
121 + const observations: NormalizedObservation[] = [];
122 + if (last) {
123 + const ts = zonedTimeToUtc(`${last.date}T${target.sessionCloseLocal.length === 5 ? `${target.sessionCloseLocal}:00` : target.sessionCloseLocal}`, target.timezone);
124 + const base = { symbol: target.symbol, instrumentHint: target.hint, currency: target.hint.currency ?? "USD", sourceTimestamp: ts, timestampTrust: "SOURCE" as const, rightsStatus: "LICENSED" as const, realtimeStatus: "END_OF_DAY" as const, meta: { session: last.date, contract: last.contract } };
125 + observations.push({ ...base, field: "LAST_PRICE", value: last.close! }, { ...base, field: "CLOSE", value: last.close! }, { ...base, field: "OPEN", value: last.open! }, { ...base, field: "HIGH", value: last.high! }, { ...base, field: "LOW", value: last.low! });
126 + if (last.volume != null) observations.push({ ...base, field: "VOLUME", value: last.volume });
127 + if (prev) observations.push({ ...base, field: "PREVIOUS_CLOSE", value: prev.close!, meta: { session: prev.date } });
128 + }
129 + return { observations, bars, stats: { bars: bars.length } };
130 + },
131 + async healthCheck(ctx) {
132 + const { data } = await ctx.http.getJson<{ datasets?: unknown }>(`${BASE}/status`);
133 + return { ok: !!data?.datasets };
134 + },
135 + fixturesDir: "fixtures",
136 +});
137 +
138 +const num = (v: unknown): number | null => {
139 + const n = typeof v === "number" ? v : typeof v === "string" ? Number(v) : NaN;
140 + return Number.isFinite(n) ? n : null;
141 +};
added connectors/src/index.ts +40 −0
@@ -0,0 +1,40 @@
1 +import type { ConnectorDefinition } from "@market-atlas/connector-sdk";
2 +import { coinbaseWs } from "./coinbase-ws/index.js";
3 +import { krakenWs } from "./kraken-ws/index.js";
4 +import { binanceWs } from "./binance-ws/index.js";
5 +import { okxWs } from "./okx-ws/index.js";
6 +import { cboeDelayedQuotes } from "./cboe-delayed-quotes/index.js";
7 +import { ecbFrankfurter } from "./ecb-frankfurter/index.js";
8 +import { bankOfCanada } from "./bank-of-canada/index.js";
9 +import { usTreasuryYieldCurve } from "./us-treasury-yield-curve/index.js";
10 +import { secEdgarFilings } from "./sec-edgar-filings/index.js";
11 +import { secCompanyTickers } from "./sec-company-tickers/index.js";
12 +import { nasdaqSymbolDirectory } from "./nasdaq-symbol-directory/index.js";
13 +import { nasdaqTradeHalts } from "./nasdaq-trade-halts/index.js";
14 +import { nasdaqMarketCalendar } from "./nasdaq-market-calendar/index.js";
15 +import { hfmarketdata } from "./hfmarketdata/index.js";
16 +
17 +export { SOURCES, sourceById } from "./sources.js";
18 +
19 +/**
20 + * Production connector registry. Order matters slightly: directories first so that quote
21 + * connectors resolve onto the canonical listed instruments when everything starts cold.
22 + */
23 +export const CONNECTORS: ConnectorDefinition[] = [
24 + nasdaqSymbolDirectory,
25 + secCompanyTickers,
26 + nasdaqMarketCalendar,
27 + coinbaseWs,
28 + krakenWs,
29 + binanceWs,
30 + okxWs,
31 + cboeDelayedQuotes,
32 + ecbFrankfurter,
33 + bankOfCanada,
34 + usTreasuryYieldCurve,
35 + secEdgarFilings,
36 + nasdaqTradeHalts,
37 + hfmarketdata,
38 +];
39 +
40 +export const connectorById = (id: string) => CONNECTORS.find((c) => c.metadata.id === id);
added connectors/src/kraken-ws/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# kraken-ws
2 +
3 +Kraken WebSocket API v2 (`wss://ws.kraken.com/v2`), `ticker` channel with `event_trigger: trades`.
4 +
5 +- **Type**: WEBSOCKET · **Rights**: PUBLIC_ATTRIBUTED · **Real-time**: REALTIME
6 +- **Timestamps**: the v2 ticker payload has no event time → observations carry `timestampTrust = CONNECTOR` (receive time). Consensus weights it slightly below exchange-stamped venues.
7 +- **Heartbeat**: `{"method":"ping"}` every 25 s; the server also emits `channel: heartbeat` frames.
8 +- **Fields**: LAST_PRICE, HIGH/LOW (24 h), BID/ASK (+ sizes), VOLUME (24 h), VWAP.
added connectors/src/kraken-ws/fixtures/ticker.json +1 −0
@@ -0,0 +1 @@
1 +{"channel":"ticker","type":"snapshot","data":[{"symbol":"BTC/USD","bid":77281.2,"bid_qty":0.35,"ask":77281.3,"ask_qty":1.2,"last":77281.3,"volume":1234.567,"vwap":77900.1,"low":76001.0,"high":79880.0,"change":80.5,"change_pct":0.1}]}
added connectors/src/kraken-ws/index.ts +73 −0
@@ -0,0 +1,73 @@
1 +import { defineConnector, raw, type NormalizedBatch } from "@market-atlas/connector-sdk";
2 +import { MAJOR_BASES, cryptoSeeds, tickerObservations } from "../_shared/crypto.js";
3 +
4 +const WS_URL = "wss://ws.kraken.com/v2";
5 +const PAIRS: Array<[string, string]> = [...MAJOR_BASES.filter((b) => b !== "PAXG").map((b) => [b, "USD"] as [string, string]), ["BTC", "USDT"], ["ETH", "USDT"], ["BTC", "EUR"], ["ETH", "EUR"]];
6 +const SYMBOLS = PAIRS.map(([b, q]) => `${b}/${q}`);
7 +
8 +/** Kraken WebSocket API v2, `ticker` channel (event trigger: trades). No per-message timestamp → connector receive time. */
9 +export const krakenWs = defineConnector({
10 + metadata: {
11 + id: "kraken-ws",
12 + name: "Kraken — ticker feed (WS v2)",
13 + version: "1.0.0",
14 + sourceId: "kraken",
15 + organization: "Payward, Inc.",
16 + sourceType: "WEBSOCKET",
17 + jurisdiction: "US",
18 + rightsStatus: "PUBLIC_ATTRIBUTED",
19 + realtimeStatus: "REALTIME",
20 + expectedLatencyMs: 400,
21 + supportsStreaming: true,
22 + supportsHistorical: false,
23 + assetClasses: ["CRYPTO"],
24 + exchanges: ["kraken"],
25 + homepage: "https://docs.kraken.com/api/docs/websocket-v2/ticker",
26 + description: "Public Kraken WebSocket v2 ticker channel (best bid/ask, last trade, 24h volume/VWAP/high/low) for the major spot pairs.",
27 + rightsNotes: "Public market data displayed with attribution to Kraken.",
28 + termsUrl: "https://www.kraken.com/legal",
29 + sourceFamily: "kraken",
30 + enabled: true,
31 + },
32 + seeds: cryptoSeeds(PAIRS, "kraken", (b, q) => `${b}/${q}`),
33 + defaultSymbols: SYMBOLS,
34 + async start(ctx) {
35 + const ws = ctx.openWebSocket(WS_URL, {
36 + label: "kraken",
37 + staleAfterMs: 90_000,
38 + heartbeat: { intervalMs: 25_000, message: JSON.stringify({ method: "ping" }) },
39 + onOpen: (sock) => sock.send({ method: "subscribe", params: { channel: "ticker", symbol: ctx.watchedSymbols(), event_trigger: "trades", snapshot: true } }),
40 + onMessage: (data) => {
41 + let msg: any;
42 + try {
43 + msg = JSON.parse(data);
44 + } catch {
45 + return;
46 + }
47 + if (msg?.channel === "ticker" && Array.isArray(msg.data)) ctx.emit(raw("kraken-ws", "kraken", "ticker", msg));
48 + else if (msg?.method === "subscribe" && msg.success === false) ctx.reportError(new Error(String(msg.error ?? "subscribe failed")), { symbol: msg.result?.symbol });
49 + },
50 + });
51 + ws.connect();
52 + },
53 + normalize(r): NormalizedBatch {
54 + const m = r.payload as { channel?: string; data?: Array<Record<string, unknown>> };
55 + if (r.kind !== "ticker" || m.channel !== "ticker" || !Array.isArray(m.data)) return { observations: [] };
56 + const observations = [];
57 + for (const t of m.data) {
58 + if (typeof t.symbol !== "string") continue;
59 + const [base, quote] = t.symbol.split("/");
60 + if (!base || !quote) continue;
61 + observations.push(
62 + ...tickerObservations(t.symbol, base, quote, "kraken", { last: t.last, high: t.high, low: t.low, bid: t.bid, ask: t.ask, bidSize: t.bid_qty, askSize: t.ask_qty, volume: t.volume, vwap: t.vwap }, {
63 + sourceTimestamp: null,
64 + timestampTrust: "CONNECTOR",
65 + rightsStatus: "PUBLIC_ATTRIBUTED",
66 + realtimeStatus: "REALTIME",
67 + }),
68 + );
69 + }
70 + return { observations };
71 + },
72 + fixturesDir: "fixtures",
73 +});
added connectors/src/nasdaq-market-calendar/README.md +7 −0
@@ -0,0 +1,7 @@
1 +# nasdaq-market-calendar
2 +
3 +HTML change-detection connector on `https://www.nasdaqtrader.com/Trader.aspx?id=Calendar` (U.S. Equity and Options Markets Holiday Schedule).
4 +
5 +- **Type**: HTML · **Rights**: PUBLIC_ATTRIBUTED · every 12 h.
6 +- **Change detection**: document hash + per-section hashes (split on headings) stored in connector state; on change → `DOCUMENT_CHANGED` event listing the changed sections.
7 +- **Extraction**: table rows `Month D, YYYY | Holiday | Closed / 1:00 p.m.``holidays` for xnas, xnys, xase, arcx, bats, xcbo (CLOSED or EARLY_CLOSE with `closeTime`). Feeds the market-hours engine.
added connectors/src/nasdaq-market-calendar/fixtures/page.html +11 −0
@@ -0,0 +1,11 @@
1 +<html><body><div id="leftWideCOL"><h1>U.S. Equity and Options Markets Holiday Schedule 2026</h1>
2 +<div class="dataTable"><table><tr><th>2026</th><th>Holiday</th><th>Status</th></tr>
3 +<tr><td>January 1, 2026</td><td>New Years Day (Observed)</td><td>Closed</td></tr>
4 +<tr><td>January 19, 2026</td><td>Martin Luther King, Jr. Day</td><td>Closed</td></tr>
5 +<tr><td>April 3, 2026</td><td>Good Friday</td><td>Closed</td></tr>
6 +<tr><td>July 3, 2026</td><td>Independence Day - U.S. (Observed)</td><td>Closed</td></tr>
7 +<tr><td>November 26, 2026</td><td>Thanksgiving Day - U.S.</td><td>Closed</td></tr>
8 +<tr><td>November 27, 2026</td><td>Early Close* - U.S.</td><td>1:00 p.m.</td></tr>
9 +<tr><td>December 25, 2026</td><td>Christmas Holiday - U.S.</td><td>Closed</td></tr>
10 +</table></div>
11 +<h2>TradeDate&mdash;Settlement Date Schedule</h2><p>Text.</p></div></body></html>
added connectors/src/nasdaq-market-calendar/index.ts +98 −0
@@ -0,0 +1,98 @@
1 +import { defineConnector, extractTables, htmlFingerprints, raw, stripHtml, type NormalizedBatch, type ProposedEvent } from "@market-atlas/connector-sdk";
2 +
3 +const URL = "https://www.nasdaqtrader.com/Trader.aspx?id=Calendar";
4 +/** The Nasdaq schedule applies to all US equity and options markets. */
5 +const US_EXCHANGES = ["xnas", "xnys", "xase", "arcx", "bats", "xcbo"];
6 +const MONTHS: Record<string, number> = { january: 1, february: 2, march: 3, april: 4, may: 5, june: 6, july: 7, august: 8, september: 9, october: 10, november: 11, december: 12 };
7 +
8 +/**
9 + * HTML change-detection connector: the Nasdaq Trader "Holiday Schedule" page. Section-level
10 + * fingerprints are stored in connector state; a change emits DOCUMENT_CHANGED and the parsed
11 + * table feeds the market-hours engine (closed days and 1:00 p.m. early closes).
12 + */
13 +export const nasdaqMarketCalendar = defineConnector({
14 + metadata: {
15 + id: "nasdaq-market-calendar",
16 + name: "Nasdaq Trader — US market holiday schedule (HTML)",
17 + version: "1.0.0",
18 + sourceId: "nasdaq-trader",
19 + organization: "Nasdaq, Inc.",
20 + sourceType: "HTML",
21 + jurisdiction: "US",
22 + rightsStatus: "PUBLIC_ATTRIBUTED",
23 + realtimeStatus: "END_OF_DAY",
24 + expectedLatencyMs: null,
25 + supportsStreaming: false,
26 + supportsHistorical: false,
27 + assetClasses: ["EQUITY", "ETF", "INDEX"],
28 + exchanges: US_EXCHANGES,
29 + homepage: URL,
30 + description: "Parses the U.S. Equity and Options Markets Holiday Schedule table (date, holiday, status) and detects any change in the page (document + section hashes). Output feeds the exchange calendar for all US venues.",
31 + rightsNotes: "Public reference page; parsed for facts (dates), no content redistribution.",
32 + termsUrl: "https://www.nasdaqtrader.com/Trader.aspx?id=Terms",
33 + sourceFamily: "nasdaq",
34 + enabled: true,
35 + },
36 + rateLimits: { "www.nasdaqtrader.com": 1 },
37 + schedule: { intervalMs: 12 * 60 * 60_000 },
38 + async poll(ctx) {
39 + const res = await ctx.http.getText(URL, { timeoutMs: 30_000, headers: { accept: "text/html" } });
40 + const fp = htmlFingerprints(res.text);
41 + const prev = await ctx.state.get<{ document: string; sections: Record<string, string> }>("fingerprint");
42 + const changed = !prev || prev.document !== fp.document;
43 + const changedSections = prev ? Object.keys(fp.sections).filter((k) => prev.sections[k] !== fp.sections[k]) : Object.keys(fp.sections);
44 + await ctx.state.set("fingerprint", fp);
45 + await ctx.state.set("last_checked_at", new Date().toISOString());
46 + // Always emit the page (holidays are idempotent); mark whether the document changed so normalize can raise an event.
47 + return [raw("nasdaq-market-calendar", "nasdaq-trader", "page", res.text, { changed, first: !prev, changed_sections: changedSections.slice(0, 10), document_hash: fp.document })];
48 + },
49 + normalize(r): NormalizedBatch {
50 + if (r.kind !== "page" || typeof r.payload !== "string") return { observations: [] };
51 + const html = r.payload;
52 + const yearMatch = stripHtml(html).match(/Holiday Schedule (\d{4})/);
53 + const holidays: NonNullable<NormalizedBatch["holidays"]> = [];
54 + for (const table of extractTables(html)) {
55 + for (const row of table) {
56 + if (row.length < 3) continue;
57 + const date = parseLongDate(row[0]!);
58 + if (!date) continue;
59 + const name = row[1]!.replace(/\*+/g, "").trim();
60 + const status = row[2]!.trim();
61 + const early = status.match(/(\d{1,2})(?::(\d{2}))?\s*(a|p)\.?m\.?/i);
62 + const kind = /closed/i.test(status) ? "CLOSED" : early ? "EARLY_CLOSE" : null;
63 + if (!kind) continue;
64 + let closeTime: string | null = null;
65 + if (early) {
66 + let h = Number(early[1]);
67 + if (early[3]!.toLowerCase() === "p" && h < 12) h += 12;
68 + closeTime = `${String(h).padStart(2, "0")}:${early[2] ?? "00"}`;
69 + }
70 + for (const ex of US_EXCHANGES) holidays.push({ exchangeId: ex, date, name: kind === "EARLY_CLOSE" ? `${name.replace(/early close/i, "").trim() || "Early close"} (early close)` : name, kind, closeTime });
71 + }
72 + }
73 + const events: ProposedEvent[] = [];
74 + if (r.meta?.changed && !r.meta?.first) {
75 + events.push({
76 + type: "DOCUMENT_CHANGED",
77 + timestamp: r.receivedAt,
78 + severity: "NOTICE",
79 + confidence: 1,
80 + title: `Nasdaq Trader holiday schedule page changed${yearMatch ? ` (${yearMatch[1]})` : ""}`,
81 + summary: `Sections changed: ${(r.meta.changed_sections as string[])?.join(", ") || "n/a"}`,
82 + dedupeKey: `nasdaq-calendar:${r.meta.document_hash}`,
83 + data: { url: URL, document_hash: r.meta.document_hash, changed_sections: r.meta.changed_sections, holidays_parsed: holidays.length / US_EXCHANGES.length },
84 + });
85 + }
86 + return { observations: [], holidays, events, stats: { holidays: holidays.length / US_EXCHANGES.length } };
87 + },
88 + fixturesDir: "fixtures",
89 +});
90 +
91 +/** "January 19, 2026" → "2026-01-19" */
92 +export function parseLongDate(s: string): string | null {
93 + const m = s.trim().match(/^([A-Za-z]+)\s+(\d{1,2}),\s*(\d{4})$/);
94 + if (!m) return null;
95 + const mo = MONTHS[m[1]!.toLowerCase()];
96 + if (!mo) return null;
97 + return `${m[3]}-${String(mo).padStart(2, "0")}-${m[2]!.padStart(2, "0")}`;
98 +}
added connectors/src/nasdaq-symbol-directory/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# nasdaq-symbol-directory
2 +
3 +`nasdaqlisted.txt` + `otherlisted.txt` (pipe-delimited, daily) from Nasdaq Trader.
4 +
5 +- **Type**: BULK_FILE · **Rights**: PUBLIC_ATTRIBUTED · daily (ETag/304 aware).
6 +- **Output**: ~11,000 `instruments` proposals — symbol, cleaned name, venue (`Exchange` code N/A/P/Z/V → XNYS/XASE/ARCX/BATS/IEXG), ETF flag, security type heuristics (warrants, units, preferred, ADR…). Test issues skipped.
7 +- **Aliases**: `BRK.B``BRK-B``BRK B` (CQS/Nasdaq conventions).
8 +- **Quirk**: the last line `File Creation Time: …` is not data.
added connectors/src/nasdaq-symbol-directory/fixtures/nasdaqlisted.txt +7 −0
@@ -0,0 +1,7 @@
1 +Symbol|Security Name|Market Category|Test Issue|Financial Status|Round Lot Size|ETF|NextShares
2 +AAAP|Pacer Barings CLO Market Flex ETF|G|N|N|100|Y|N
3 +AACG|ATA Creativity Global - American Depositary Shares, each representing two common shares|S|N|N|100|N|N
4 +AAPL|Apple Inc. - Common Stock|Q|N|N|100|N|N
5 +ZTEST|Test Issue - Common Stock|Q|Y|N|100|N|N
6 +QQQ|Invesco QQQ Trust, Series 1|G|N|N|100|Y|N
7 +File Creation Time: 0912202601:30|||||||
added connectors/src/nasdaq-symbol-directory/fixtures/otherlisted.txt +7 −0
@@ -0,0 +1,7 @@
1 +ACT Symbol|Security Name|Exchange|CQS Symbol|ETF|Round Lot Size|Test Issue|NASDAQ Symbol
2 +A|Agilent Technologies, Inc. Common Stock|N|A|N|100|N|A
3 +AA|Alcoa Corporation Common Stock |N|AA|N|100|N|AA
4 +BRK.B|Berkshire Hathaway Inc. Class B Common Stock|N|BRK B|N|100|N|BRK-B
5 +SPY|SPDR S&P 500 ETF Trust|P|SPY|Y|100|N|SPY
6 +BAC-PB|Bank of America Corporation Depositary Shares, each representing a 1/1,000th interest in a share of 6.000% Non-Cumulative Preferred Stock, Series GG|N|BAC pB|N|100|N|BAC-B
7 +File Creation Time: 0912202601:30|||||||
added connectors/src/nasdaq-symbol-directory/index.ts +106 −0
@@ -0,0 +1,106 @@
1 +import { defineConnector, parseDelimited, raw, type NormalizedBatch, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +
3 +const NASDAQ_LISTED = "https://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt";
4 +const OTHER_LISTED = "https://www.nasdaqtrader.com/dynamic/SymDir/otherlisted.txt";
5 +/** otherlisted.txt "Exchange" column codes. */
6 +const OTHER_EXCHANGES: Record<string, { exchangeId: string; mic: string }> = {
7 + N: { exchangeId: "xnys", mic: "XNYS" },
8 + A: { exchangeId: "xase", mic: "XASE" },
9 + P: { exchangeId: "arcx", mic: "ARCX" },
10 + Z: { exchangeId: "bats", mic: "BATS" },
11 + V: { exchangeId: "iexg", mic: "IEXG" },
12 +};
13 +
14 +/**
15 + * Nasdaq Trader symbol directories (nasdaqlisted.txt + otherlisted.txt): the complete list of
16 + * US-listed securities with ETF flag and listing venue. Populates the instrument master daily.
17 + */
18 +export const nasdaqSymbolDirectory = defineConnector({
19 + metadata: {
20 + id: "nasdaq-symbol-directory",
21 + name: "Nasdaq Trader — US symbol directory",
22 + version: "1.0.0",
23 + sourceId: "nasdaq-trader",
24 + organization: "Nasdaq, Inc.",
25 + sourceType: "BULK_FILE",
26 + jurisdiction: "US",
27 + rightsStatus: "PUBLIC_ATTRIBUTED",
28 + realtimeStatus: "END_OF_DAY",
29 + expectedLatencyMs: null,
30 + supportsStreaming: false,
31 + supportsHistorical: false,
32 + assetClasses: ["EQUITY", "ETF"],
33 + exchanges: ["xnas", "xnys", "xase", "arcx", "bats"],
34 + homepage: "https://www.nasdaqtrader.com/trader.aspx?id=symboldirdefs",
35 + description: "Daily pipe-delimited directories of every Nasdaq-listed and other US exchange-listed security (~11,000 rows): symbol, security name, listing venue, ETF flag, test-issue flag. Seeds the US instrument master.",
36 + rightsNotes: "Public reference files published by Nasdaq Trader; displayed with attribution.",
37 + termsUrl: "https://www.nasdaqtrader.com/Trader.aspx?id=Terms",
38 + sourceFamily: "nasdaq",
39 + enabled: true,
40 + },
41 + rateLimits: { "www.nasdaqtrader.com": 1 },
42 + schedule: { intervalMs: 24 * 60 * 60_000 },
43 + async poll(ctx) {
44 + const out = [];
45 + for (const [kind, url] of [["nasdaqlisted", NASDAQ_LISTED], ["otherlisted", OTHER_LISTED]] as const) {
46 + const res = await ctx.http.getText(url, { timeoutMs: 60_000 });
47 + if (res.notModified) continue;
48 + out.push(raw("nasdaq-symbol-directory", "nasdaq-trader", kind, res.text));
49 + }
50 + return out;
51 + },
52 + normalize(r): NormalizedBatch {
53 + if (typeof r.payload !== "string") return { observations: [] };
54 + const rows = parseDelimited(r.payload, "|").filter((row) => !String(Object.values(row)[0] ?? "").startsWith("File Creation Time"));
55 + const instruments: ProposedInstrument[] = [];
56 + if (r.kind === "nasdaqlisted") {
57 + for (const row of rows) {
58 + if (row["Test Issue"] === "Y" || !row.Symbol) continue;
59 + const etf = row.ETF === "Y";
60 + instruments.push(proposal(row.Symbol!, row["Security Name"] ?? row.Symbol!, "xnas", "XNAS", etf, { market_category: row["Market Category"], financial_status: row["Financial Status"] }));
61 + }
62 + } else if (r.kind === "otherlisted") {
63 + for (const row of rows) {
64 + if (row["Test Issue"] === "Y" || !row["ACT Symbol"]) continue;
65 + const ex = OTHER_EXCHANGES[row.Exchange ?? ""];
66 + if (!ex) continue;
67 + const etf = row.ETF === "Y";
68 + instruments.push(proposal(row["ACT Symbol"]!, row["Security Name"] ?? row["ACT Symbol"]!, ex.exchangeId, ex.mic, etf, { cqs_symbol: row["CQS Symbol"], nasdaq_symbol: row["NASDAQ Symbol"] }, [row["CQS Symbol"], row["NASDAQ Symbol"]]));
69 + }
70 + }
71 + return { observations: [], instruments, stats: { rows: rows.length } };
72 + },
73 + fixturesDir: "fixtures",
74 +});
75 +
76 +function proposal(symbol: string, rawName: string, exchangeId: string, mic: string, etf: boolean, metadata: Record<string, unknown>, extraAliases: Array<string | undefined> = []): ProposedInstrument {
77 + const name = cleanName(rawName);
78 + const sym = symbol.trim();
79 + const aliases = [...new Set([sym.replace(".", "-"), sym.replace(".", "/"), ...extraAliases.filter((a): a is string => !!a && a !== sym)])].filter((a) => a !== sym);
80 + const securityType = classify(rawName);
81 + return {
82 + symbol: sym,
83 + aliases,
84 + hint: { assetClass: etf ? "ETF" : securityType === "COMMON_STOCK" || securityType === "ADR" ? "EQUITY" : "OTHER", name, exchangeId, mic, currency: "USD", country: "US", securityType: etf ? "ETF" : securityType, companyName: !etf && securityType === "COMMON_STOCK" ? name : null, metadata },
85 + };
86 +}
87 +
88 +export function cleanName(n: string): string {
89 + return n
90 + .trim()
91 + .replace(/\s+-\s+(Common Stock|Class [A-Z] Common Stock|Ordinary Shares|Common Shares|American Depositary Shares.*|Class [A-Z] Ordinary Shares?.*|Units?.*|Warrants?.*|Depositary Shares.*)$/i, "")
92 + .replace(/\s+Common Stock$/i, "")
93 + .replace(/\s{2,}/g, " ")
94 + .trim();
95 +}
96 +
97 +export function classify(n: string): string {
98 + const s = n.toLowerCase();
99 + if (/warrant/.test(s)) return "WARRANT";
100 + if (/\bunits?\b/.test(s)) return "UNIT";
101 + if (/\bright(s)?\b/.test(s)) return "RIGHT";
102 + if (/american depositary|\badr\b|\bads\b/.test(s)) return "ADR";
103 + if (/preferred|preference|depositary share/.test(s)) return "PREFERRED";
104 + if (/\bnotes?\b|debenture|bond/.test(s)) return "DEBT";
105 + return "COMMON_STOCK";
106 +}
added connectors/src/nasdaq-trade-halts/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# nasdaq-trade-halts
2 +
3 +`https://www.nasdaqtrader.com/rss.aspx?feed=tradehalts` (RSS 2.0 with `ndaq:` extension elements).
4 +
5 +- **Type**: RSS · **Rights**: PUBLIC_ATTRIBUTED · **Real-time**: REALTIME (feed TTL 1 min; polled every 60 s while Nasdaq is trading, 10 min closed, hourly weekends)
6 +- **Output**: `TRADING_HALT` (severity by reason: MWC* CRITICAL, LULD/T5 NOTICE, others WARNING) and `TRADING_RESUME` events, deduplicated on symbol + halt date/time.
7 +- **Timestamps**: `HaltDate` (MM/DD/YYYY) + `HaltTime` (HH:MM:SS.mmm) in US/Eastern → UTC.
8 +- **Reason codes**: mapped in `REASON_CODES` (T1 news pending, LUDP volatility pause, H10 SEC suspension…).
added connectors/src/nasdaq-trade-halts/fixtures/rss.xml +41 −0
@@ -0,0 +1,41 @@
1 +<?xml version="1.0" encoding="utf-8"?>
2 +<rss version="2.0" xmlns:ndaq="http://www.nasdaqtrader.com/">
3 + <channel>
4 + <title>NASDAQTrader.com</title>
5 + <link>http://www.nasdaqtrader.com</link>
6 + <description>NASDAQ Trade Halts</description>
7 + <pubDate>Sat, 12 Sep 2026 07:53:39 GMT</pubDate>
8 + <ttl>1</ttl>
9 + <ndaq:numItems>2</ndaq:numItems>
10 + <item>
11 + <title>HUBC</title>
12 + <pubDate>Fri, 11 Sep 2026 04:00:00 GMT</pubDate>
13 + <ndaq:HaltDate>09/11/2026</ndaq:HaltDate>
14 + <ndaq:HaltTime>19:50:00.000</ndaq:HaltTime>
15 + <ndaq:IssueSymbol>HUBC</ndaq:IssueSymbol>
16 + <ndaq:IssueName>Hub Cyber Security Ltd. Ord</ndaq:IssueName>
17 + <ndaq:Market>NASDAQ</ndaq:Market>
18 + <ndaq:ReasonCode>T1</ndaq:ReasonCode>
19 + <ndaq:PauseThresholdPrice />
20 + <ndaq:ResumptionDate />
21 + <ndaq:ResumptionQuoteTime />
22 + <ndaq:ResumptionTradeTime />
23 + <description><![CDATA[<table>...</table>]]></description>
24 + </item>
25 + <item>
26 + <title>ABCD</title>
27 + <pubDate>Fri, 11 Sep 2026 04:00:00 GMT</pubDate>
28 + <ndaq:HaltDate>09/11/2026</ndaq:HaltDate>
29 + <ndaq:HaltTime>10:12:31.512</ndaq:HaltTime>
30 + <ndaq:IssueSymbol>ABCD</ndaq:IssueSymbol>
31 + <ndaq:IssueName>ABCD Corp Common Stock</ndaq:IssueName>
32 + <ndaq:Market>NYSE</ndaq:Market>
33 + <ndaq:ReasonCode>LUDP</ndaq:ReasonCode>
34 + <ndaq:PauseThresholdPrice>12.34</ndaq:PauseThresholdPrice>
35 + <ndaq:ResumptionDate>09/11/2026</ndaq:ResumptionDate>
36 + <ndaq:ResumptionQuoteTime>10:17:31.000</ndaq:ResumptionQuoteTime>
37 + <ndaq:ResumptionTradeTime>10:22:31.000</ndaq:ResumptionTradeTime>
38 + <description><![CDATA[<table>...</table>]]></description>
39 + </item>
40 + </channel>
41 +</rss>
added connectors/src/nasdaq-trade-halts/index.ts +120 −0
@@ -0,0 +1,120 @@
1 +import { defineConnector, parseFeed, raw, type NormalizedBatch, type ProposedEvent } from "@market-atlas/connector-sdk";
2 +import { zonedTimeToUtc } from "@market-atlas/market-model";
3 +
4 +const FEED = "https://www.nasdaqtrader.com/rss.aspx?feed=tradehalts";
5 +const ET = "America/New_York";
6 +/** Nasdaq halt reason codes (subset; unknown codes are passed through verbatim). */
7 +export const REASON_CODES: Record<string, string> = {
8 + T1: "News pending",
9 + T2: "News released",
10 + T5: "Single stock trading pause in effect",
11 + T6: "Extraordinary market activity",
12 + T8: "ETF halt",
13 + T12: "Additional information requested by Nasdaq",
14 + H4: "Non-compliance with listing requirements",
15 + H9: "Not current in required filings",
16 + H10: "SEC trading suspension",
17 + H11: "Regulatory concern",
18 + O1: "Operations halt",
19 + LUDP: "Volatility trading pause (LULD)",
20 + LUDS: "Volatility trading pause (LULD, straddle)",
21 + MWC1: "Market-wide circuit breaker level 1",
22 + MWC2: "Market-wide circuit breaker level 2",
23 + MWC3: "Market-wide circuit breaker level 3",
24 + IPO1: "IPO not yet trading",
25 + M1: "Corporate action",
26 + M2: "Quotation not available",
27 + D: "Security deletion",
28 +};
29 +
30 +/** Nasdaq Trader trade-halts RSS (TTL 1 min): halts and resumptions across US listed markets → TRADING_HALT / TRADING_RESUME events. */
31 +export const nasdaqTradeHalts = defineConnector({
32 + metadata: {
33 + id: "nasdaq-trade-halts",
34 + name: "Nasdaq Trader — trading halts feed",
35 + version: "1.0.0",
36 + sourceId: "nasdaq-trader",
37 + organization: "Nasdaq, Inc.",
38 + sourceType: "RSS",
39 + jurisdiction: "US",
40 + rightsStatus: "PUBLIC_ATTRIBUTED",
41 + realtimeStatus: "REALTIME",
42 + expectedLatencyMs: 60_000,
43 + supportsStreaming: false,
44 + supportsHistorical: false,
45 + assetClasses: ["EQUITY", "ETF"],
46 + exchanges: ["xnas", "xnys", "xase", "arcx", "bats"],
47 + homepage: "https://www.nasdaqtrader.com/trader.aspx?id=TradeHalts",
48 + description: "Current-day trading halts published by Nasdaq (all US listing markets): symbol, halt time, reason code, LULD threshold price and resumption times. Polled every minute during trading hours.",
49 + rightsNotes: "Public feed; displayed with attribution to Nasdaq Trader.",
50 + termsUrl: "https://www.nasdaqtrader.com/Trader.aspx?id=Terms",
51 + sourceFamily: "nasdaq",
52 + enabled: true,
53 + },
54 + rateLimits: { "www.nasdaqtrader.com": 1 },
55 + schedule: { openMs: 60_000, closedMs: 10 * 60_000, weekendMs: 60 * 60_000, exchangeId: "xnas" },
56 + async poll(ctx) {
57 + const res = await ctx.http.getText(FEED, { timeoutMs: 20_000, headers: { accept: "application/rss+xml, application/xml" } });
58 + if (res.notModified) return [];
59 + return [raw("nasdaq-trade-halts", "nasdaq-trader", "rss", res.text)];
60 + },
61 + normalize(r): NormalizedBatch {
62 + if (r.kind !== "rss" || typeof r.payload !== "string") return { observations: [] };
63 + const feed = parseFeed(r.payload);
64 + const events: ProposedEvent[] = [];
65 + for (const item of feed.items) {
66 + const x = item.extra as Record<string, unknown>;
67 + const t = (k: string) => {
68 + const v = x[`ndaq:${k}`];
69 + return typeof v === "string" ? v.trim() : typeof v === "number" ? String(v) : "";
70 + };
71 + const symbol = t("IssueSymbol") || item.title.trim();
72 + const haltDate = t("HaltDate"); // MM/DD/YYYY
73 + const haltTime = t("HaltTime"); // HH:MM:SS.mmm
74 + if (!symbol || !haltDate || !haltTime) continue;
75 + const ts = toUtc(haltDate, haltTime);
76 + if (!ts) continue;
77 + const reason = t("ReasonCode");
78 + const market = t("Market");
79 + const name = t("IssueName");
80 + const key = `${symbol}|${haltDate}|${haltTime}`;
81 + events.push({
82 + type: "TRADING_HALT",
83 + symbols: [symbol],
84 + timestamp: ts,
85 + severity: reason.startsWith("MWC") ? "CRITICAL" : reason.startsWith("LUD") || reason === "T5" ? "NOTICE" : "WARNING",
86 + confidence: 1,
87 + title: `${symbol}: trading halt (${REASON_CODES[reason] ?? (reason || "reason not stated")})`,
88 + summary: name ? `${name} · ${market}` : market || null,
89 + dedupeKey: `halt:${key}`,
90 + data: { symbol, market, reason_code: reason, reason: REASON_CODES[reason] ?? null, issue_name: name, pause_threshold_price: t("PauseThresholdPrice") || null, halt_date: haltDate, halt_time: haltTime },
91 + });
92 + const resDate = t("ResumptionDate");
93 + const resTrade = t("ResumptionTradeTime");
94 + if (resDate && resTrade) {
95 + const rts = toUtc(resDate, resTrade);
96 + if (rts)
97 + events.push({
98 + type: "TRADING_RESUME",
99 + symbols: [symbol],
100 + timestamp: rts,
101 + severity: "INFO",
102 + confidence: 1,
103 + title: `${symbol}: trading resumed`,
104 + summary: name ? `${name} · ${market}` : market || null,
105 + dedupeKey: `resume:${key}|${resDate}|${resTrade}`,
106 + data: { symbol, market, reason_code: reason, resumption_quote_time: t("ResumptionQuoteTime") || null, resumption_trade_time: resTrade, halt_time: `${haltDate} ${haltTime}` },
107 + });
108 + }
109 + }
110 + return { observations: [], events, stats: { items: feed.items.length } };
111 + },
112 + fixturesDir: "fixtures",
113 +});
114 +
115 +function toUtc(mdy: string, hms: string): number | null {
116 + const m = mdy.match(/^(\d{2})\/(\d{2})\/(\d{4})$/);
117 + const h = hms.match(/^(\d{2}):(\d{2}):(\d{2})(?:\.(\d{1,3}))?$/);
118 + if (!m || !h) return null;
119 + return zonedTimeToUtc(`${m[3]}-${m[1]}-${m[2]}T${h[1]}:${h[2]}:${h[3]}${h[4] ? `.${h[4]}` : ""}`, ET);
120 +}
added connectors/src/okx-ws/README.md +7 −0
@@ -0,0 +1,7 @@
1 +# okx-ws
2 +
3 +OKX public WebSocket v5 (`wss://ws.okx.com:8443/ws/v5/public`), `tickers` channel.
4 +
5 +- **Type**: WEBSOCKET · **Rights**: PUBLIC_ATTRIBUTED · **Real-time**: REALTIME (exchange `ts`)
6 +- **Heartbeat**: the server closes idle connections after 30 s → client sends the literal text `ping` every 20 s and ignores `pong`.
7 +- **Fields**: LAST_PRICE, OPEN/HIGH/LOW (24 h), BID/ASK (+ sizes), VOLUME (24 h, base currency).
added connectors/src/okx-ws/fixtures/tickers.json +1 −0
@@ -0,0 +1 @@
1 +{"arg":{"channel":"tickers","instId":"BTC-USDT"},"data":[{"instType":"SPOT","instId":"BTC-USDT","last":"77303.9","lastSz":"0.00005174","askPx":"77303.9","askSz":"0.46749191","bidPx":"77303.8","bidSz":"1.40269327","open24h":"77218","high24h":"79896.3","low24h":"76001","sodUtc0":"77223.5","sodUtc8":"77500.1","volCcy24h":"1234567.8","vol24h":"15900.42","ts":"1789199660123"}]}
added connectors/src/okx-ws/index.ts +75 −0
@@ -0,0 +1,75 @@
1 +import { defineConnector, raw, type NormalizedBatch } from "@market-atlas/connector-sdk";
2 +import { parseTimestamp } from "@market-atlas/market-model";
3 +import { MAJOR_BASES, cryptoSeeds, tickerObservations } from "../_shared/crypto.js";
4 +
5 +const WS_URL = "wss://ws.okx.com:8443/ws/v5/public";
6 +const PAIRS: Array<[string, string]> = MAJOR_BASES.filter((b) => b !== "PAXG").map((b) => [b, "USDT"] as [string, string]).concat([["TON", "USDT"], ["TRX", "USDT"]]);
7 +const SYMBOLS = PAIRS.map(([b, q]) => `${b}-${q}`);
8 +
9 +/** OKX public WebSocket v5, `tickers` channel. Requires a text "ping" at least every 30 s. */
10 +export const okxWs = defineConnector({
11 + metadata: {
12 + id: "okx-ws",
13 + name: "OKX — tickers channel",
14 + version: "1.0.0",
15 + sourceId: "okx",
16 + organization: "OKX",
17 + sourceType: "WEBSOCKET",
18 + jurisdiction: null,
19 + rightsStatus: "PUBLIC_ATTRIBUTED",
20 + realtimeStatus: "REALTIME",
21 + expectedLatencyMs: 500,
22 + supportsStreaming: true,
23 + supportsHistorical: false,
24 + assetClasses: ["CRYPTO"],
25 + exchanges: ["okx"],
26 + homepage: "https://www.okx.com/docs-v5/en/#public-data-websocket-tickers-channel",
27 + description: "Public OKX v5 WebSocket tickers channel (last, bid/ask with sizes, 24h open/high/low/volume) for USDT spot majors. Exchange timestamp `ts`.",
28 + rightsNotes: "Public market data displayed with attribution to OKX.",
29 + termsUrl: "https://www.okx.com/help/terms-of-service",
30 + sourceFamily: "okx",
31 + enabled: true,
32 + },
33 + seeds: cryptoSeeds(PAIRS, "okx", (b, q) => `${b}-${q}`),
34 + defaultSymbols: SYMBOLS,
35 + async start(ctx) {
36 + const ws = ctx.openWebSocket(WS_URL, {
37 + label: "okx",
38 + staleAfterMs: 60_000,
39 + heartbeat: { intervalMs: 20_000, message: "ping" },
40 + onOpen: (sock) => sock.send({ op: "subscribe", args: ctx.watchedSymbols().map((instId) => ({ channel: "tickers", instId })) }),
41 + onMessage: (data) => {
42 + if (data === "pong") return;
43 + let msg: any;
44 + try {
45 + msg = JSON.parse(data);
46 + } catch {
47 + return;
48 + }
49 + if (msg?.arg?.channel === "tickers" && Array.isArray(msg.data)) ctx.emit(raw("okx-ws", "okx", "tickers", msg));
50 + else if (msg?.event === "error") ctx.reportError(new Error(String(msg.msg ?? "okx error")), { code: msg.code });
51 + },
52 + });
53 + ws.connect();
54 + },
55 + normalize(r): NormalizedBatch {
56 + const m = r.payload as { arg?: { channel?: string }; data?: Array<Record<string, unknown>> };
57 + if (r.kind !== "tickers" || m.arg?.channel !== "tickers" || !Array.isArray(m.data)) return { observations: [] };
58 + const observations = [];
59 + for (const t of m.data) {
60 + if (typeof t.instId !== "string") continue;
61 + const [base, quote] = t.instId.split("-");
62 + if (!base || !quote) continue;
63 + observations.push(
64 + ...tickerObservations(t.instId, base, quote, "okx", { last: t.last, open: t.open24h, high: t.high24h, low: t.low24h, bid: t.bidPx, ask: t.askPx, bidSize: t.bidSz, askSize: t.askSz, volume: t.vol24h }, {
65 + sourceTimestamp: parseTimestamp(t.ts),
66 + timestampTrust: "EXCHANGE",
67 + rightsStatus: "PUBLIC_ATTRIBUTED",
68 + realtimeStatus: "REALTIME",
69 + }),
70 + );
71 + }
72 + return { observations };
73 + },
74 + fixturesDir: "fixtures",
75 +});
added connectors/src/sec-company-tickers/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# sec-company-tickers
2 +
3 +`https://www.sec.gov/files/company_tickers_exchange.json` — SEC registrants with ticker, exchange (Nasdaq/NYSE/CBOE/OTC) and CIK.
4 +
5 +- **Type**: BULK_FILE · **Rights**: OFFICIAL_OPEN_DATA · daily.
6 +- **Output**: `instruments` proposals. Listed tickers (Nasdaq/NYSE/CBOE) may be created (`eq_us_<mic>_<symbol>`); OTC or unknown venues only enrich existing instruments (`createIfMissing: false`).
7 +- **Effect**: every US equity gets a `companies` row with CIK → EDGAR filings link to instruments.
8 +- **Quirk**: SEC uses `BRK-B`; Market Atlas canonical symbol is `BRK.B` with the dash form kept as alias.
added connectors/src/sec-company-tickers/fixtures/directory.json +1 −0
@@ -0,0 +1 @@
1 +{"fields":["cik","name","ticker","exchange"],"data":[[1045810,"NVIDIA CORP","NVDA","Nasdaq"],[320193,"Apple Inc.","AAPL","Nasdaq"],[1067983,"BERKSHIRE HATHAWAY INC","BRK-B","NYSE"],[19617,"JPMORGAN CHASE & CO","JPM","NYSE"],[1234567,"SOME OTC HOLDINGS INC","SOTC","OTC"],[7654321,"NO EXCHANGE CO","NOEX",null]]}
added connectors/src/sec-company-tickers/index.ts +77 −0
@@ -0,0 +1,77 @@
1 +import { defineConnector, raw, type NormalizedBatch, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +
3 +const URL = "https://www.sec.gov/files/company_tickers_exchange.json";
4 +const EXCHANGE_MAP: Record<string, { exchangeId: string; mic: string } | null> = {
5 + Nasdaq: { exchangeId: "xnas", mic: "XNAS" },
6 + NYSE: { exchangeId: "xnys", mic: "XNYS" },
7 + CBOE: { exchangeId: "bats", mic: "BATS" },
8 + OTC: null,
9 +};
10 +
11 +/** SEC company ↔ ticker ↔ exchange directory (bulk JSON, refreshed daily). Enriches instruments with CIK + company; creates listed US equities. */
12 +export const secCompanyTickers = defineConnector({
13 + metadata: {
14 + id: "sec-company-tickers",
15 + name: "SEC — company tickers & exchanges directory",
16 + version: "1.0.0",
17 + sourceId: "sec-edgar",
18 + organization: "U.S. Securities and Exchange Commission",
19 + sourceType: "BULK_FILE",
20 + jurisdiction: "US",
21 + rightsStatus: "OFFICIAL_OPEN_DATA",
22 + realtimeStatus: "END_OF_DAY",
23 + expectedLatencyMs: null,
24 + supportsStreaming: false,
25 + supportsHistorical: false,
26 + assetClasses: ["EQUITY"],
27 + exchanges: ["xnas", "xnys", "bats"],
28 + homepage: "https://www.sec.gov/files/company_tickers_exchange.json",
29 + description: "Bulk directory of ~10,000 SEC registrants with ticker, listing exchange and CIK. Used to resolve filings to instruments and to attach companies (CIK) to US equities.",
30 + rightsNotes: "U.S. government data (public domain).",
31 + termsUrl: "https://www.sec.gov/os/accessing-edgar-data",
32 + sourceFamily: "sec",
33 + enabled: true,
34 + },
35 + rateLimits: { "www.sec.gov": 2 },
36 + schedule: { intervalMs: 24 * 60 * 60_000 },
37 + async poll(ctx) {
38 + const res = await ctx.http.getText(URL, { timeoutMs: 60_000, headers: { accept: "application/json" } });
39 + if (res.notModified) return [];
40 + return [raw("sec-company-tickers", "sec-edgar", "directory", JSON.parse(res.text))];
41 + },
42 + normalize(r): NormalizedBatch {
43 + const p = r.payload as { fields?: string[]; data?: Array<[number, string, string, string | null]> };
44 + if (r.kind !== "directory" || !Array.isArray(p.data) || !Array.isArray(p.fields)) return { observations: [] };
45 + const idx = (n: string) => p.fields!.indexOf(n);
46 + const iCik = idx("cik"), iName = idx("name"), iTicker = idx("ticker"), iEx = idx("exchange");
47 + if ([iCik, iName, iTicker, iEx].some((i) => i < 0)) return { observations: [] };
48 + const instruments: ProposedInstrument[] = [];
49 + for (const row of p.data) {
50 + const ticker = String(row[iTicker] ?? "").trim();
51 + const name = String(row[iName] ?? "").trim();
52 + const cik = String(row[iCik] ?? "").trim();
53 + const ex = row[iEx] ? EXCHANGE_MAP[String(row[iEx])] : null;
54 + if (!ticker || !name || !cik) continue;
55 + const listed = ex != null;
56 + instruments.push({
57 + symbol: ticker.replace("-", "."),
58 + aliases: ticker.includes("-") ? [ticker] : [],
59 + // Listed names may be created (id derives from the exchange); OTC/unknown venues only enrich existing rows.
60 + createIfMissing: listed,
61 + hint: { assetClass: "EQUITY", name: titleCase(name), exchangeId: ex?.exchangeId ?? null, mic: ex?.mic ?? null, currency: "USD", country: "US", securityType: "COMMON_STOCK", companyName: titleCase(name), cik },
62 + });
63 + }
64 + return { observations: [], instruments, stats: { rows: p.data.length } };
65 + },
66 + fixturesDir: "fixtures",
67 +});
68 +
69 +/** "MICROSOFT CORP" → "Microsoft Corp" while preserving already mixed-case names like "Apple Inc.". */
70 +export function titleCase(s: string): string {
71 + if (s !== s.toUpperCase()) return s;
72 + return s
73 + .toLowerCase()
74 + .replace(/(^|[\s\-/(&])([a-z])/g, (m, pre: string, c: string) => pre + c.toUpperCase())
75 + .replace(/\b(Llc|Inc|Corp|Ltd|Plc|Co|Sa|Nv|Ag|Lp)\b/g, (w) => w)
76 + .replace(/\b(Etf|Reit|Spac|Usa|Us|Adr|Ii|Iii|Iv)\b/g, (w) => w.toUpperCase());
77 +}
added connectors/src/sec-edgar-filings/README.md +8 −0
@@ -0,0 +1,8 @@
1 +# sec-edgar-filings
2 +
3 +EDGAR latest-filings Atom feed (`browse-edgar?action=getcurrent&count=100&output=atom`).
4 +
5 +- **Type**: RSS/Atom · **Rights**: OFFICIAL_OPEN_DATA (public domain) · **Real-time**: REALTIME (feed updates continuously, polled every 2 min)
6 +- **Output**: `filings` (accession number, CIK, form type, filer, URL, size) → `filings` table; material forms (see `MATERIAL_FORMS`) become `FILING_PUBLISHED` events. Form 4 "Reporting" entries (insiders) are stored but produce no event.
7 +- **Linking**: filings join instruments via `companies.cik` populated by `sec-company-tickers`.
8 +- **Politeness**: identifying User-Agent with contact e-mail (SEC requirement), 2 req/s bucket, conditional requests.
added connectors/src/sec-edgar-filings/fixtures/atom.xml +25 −0
@@ -0,0 +1,25 @@
1 +<?xml version="1.0" encoding="ISO-8859-1" ?>
2 +<feed xmlns="http://www.w3.org/2005/Atom">
3 +<title>Latest Filings - Sat, 12 Sep 2026 03:53:59 EDT</title>
4 +<updated>2026-09-12T03:53:59-04:00</updated>
5 +<entry>
6 +<title>4 - Ault &amp; Company, Inc. (0001734770) (Reporting)</title>
7 +<link rel="alternate" type="text/html" href="https://www.sec.gov/Archives/edgar/data/1734770/000121465926011648/0001214659-26-011648-index.htm"/>
8 +<summary type="html">
9 + &lt;b&gt;Filed:&lt;/b&gt; 2026-09-11 &lt;b&gt;AccNo:&lt;/b&gt; 0001214659-26-011648 &lt;b&gt;Size:&lt;/b&gt; 12 KB
10 +</summary>
11 +<updated>2026-09-11T21:59:18-04:00</updated>
12 +<category scheme="https://www.sec.gov/" label="form type" term="4"/>
13 +<id>urn:tag:sec.gov,2008:accession-number=0001214659-26-011648</id>
14 +</entry>
15 +<entry>
16 +<title>8-K - Apple Inc. (0000320193) (Filer)</title>
17 +<link rel="alternate" type="text/html" href="https://www.sec.gov/Archives/edgar/data/320193/000032019326000099/0000320193-26-000099-index.htm"/>
18 +<summary type="html">
19 + &lt;b&gt;Filed:&lt;/b&gt; 2026-09-11 &lt;b&gt;AccNo:&lt;/b&gt; 0000320193-26-000099 &lt;b&gt;Size:&lt;/b&gt; 145 KB
20 +</summary>
21 +<updated>2026-09-11T16:31:02-04:00</updated>
22 +<category scheme="https://www.sec.gov/" label="form type" term="8-K"/>
23 +<id>urn:tag:sec.gov,2008:accession-number=0000320193-26-000099</id>
24 +</entry>
25 +</feed>
added connectors/src/sec-edgar-filings/index.ts +73 −0
@@ -0,0 +1,73 @@
1 +import { defineConnector, parseFeed, raw, stripHtml, type NormalizedBatch } from "@market-atlas/connector-sdk";
2 +import { parseTimestamp } from "@market-atlas/market-model";
3 +
4 +const FEED = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&type=&company=&dateb=&owner=include&start=0&count=100&output=atom";
5 +/** Forms that produce a public FILING_PUBLISHED event; the rest is stored but not surfaced as an event. */
6 +export const MATERIAL_FORMS = new Set(["8-K", "8-K/A", "10-K", "10-K/A", "10-Q", "10-Q/A", "6-K", "20-F", "40-F", "S-1", "S-1/A", "F-1", "424B4", "SC 13D", "SC 13D/A", "SC 13G", "SC 13G/A", "DEF 14A", "DEFM14A", "425", "SC TO-T", "S-4", "NT 10-K", "NT 10-Q"]);
7 +
8 +/**
9 + * SEC EDGAR "latest filings" Atom feed (public, no key; SEC asks for an identifying User-Agent
10 + * and ≤ 10 req/s — Market Atlas polls once every 2 minutes). Each entry → one Filing + a
11 + * FILING_PUBLISHED event for material forms.
12 + */
13 +export const secEdgarFilings = defineConnector({
14 + metadata: {
15 + id: "sec-edgar-filings",
16 + name: "SEC EDGAR — latest filings feed",
17 + version: "1.0.0",
18 + sourceId: "sec-edgar",
19 + organization: "U.S. Securities and Exchange Commission",
20 + sourceType: "RSS",
21 + jurisdiction: "US",
22 + rightsStatus: "OFFICIAL_OPEN_DATA",
23 + realtimeStatus: "REALTIME",
24 + expectedLatencyMs: 120_000,
25 + supportsStreaming: false,
26 + supportsHistorical: false,
27 + assetClasses: ["EQUITY"],
28 + exchanges: [],
29 + homepage: "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent",
30 + description: "Real-time list of documents filed on EDGAR (all forms). Each filing is stored with accession number, form type, filer name and CIK, and linked to instruments through the SEC company/ticker directory. Material forms (8-K, 10-K/Q, 6-K, S-1, 13D/G…) become FILING_PUBLISHED events.",
31 + rightsNotes: "U.S. government data (public domain). SEC fair-access policy: identifying User-Agent, max 10 requests/second.",
32 + termsUrl: "https://www.sec.gov/os/accessing-edgar-data",
33 + sourceFamily: "sec",
34 + enabled: true,
35 + },
36 + rateLimits: { "www.sec.gov": 2 },
37 + schedule: { intervalMs: 2 * 60_000 },
38 + async poll(ctx) {
39 + const res = await ctx.http.getText(FEED, { timeoutMs: 30_000, headers: { accept: "application/atom+xml, application/xml" } });
40 + if (res.notModified) return [];
41 + return [raw("sec-edgar-filings", "sec-edgar", "atom", res.text, { status: res.status })];
42 + },
43 + normalize(r): NormalizedBatch {
44 + if (r.kind !== "atom" || typeof r.payload !== "string") return { observations: [] };
45 + const feed = parseFeed(r.payload);
46 + const filings: NonNullable<NormalizedBatch["filings"]> = [];
47 + for (const item of feed.items) {
48 + // Title: "4 - Ault & Company, Inc. (0001734770) (Reporting)"
49 + const m = item.title.match(/^(.+?)\s+-\s+(.+?)\s+\((\d{10})\)\s*(?:\((\w+)\))?$/);
50 + const formType = (item.categories.find((c) => c.label === "form type")?.term ?? m?.[1] ?? "").trim();
51 + const companyName = (m?.[2] ?? item.title).trim();
52 + const cik = m?.[3] ? m[3].replace(/^0+/, "") : null;
53 + const role = m?.[4] ?? null;
54 + const accn = item.id?.match(/accession-number=([\d-]+)/)?.[1] ?? item.link?.match(/(\d{10}-\d{2}-\d{6})/)?.[1] ?? null;
55 + if (!accn || !formType) continue;
56 + const summary = item.summary ? stripHtml(item.summary) : "";
57 + const filedDate = summary.match(/Filed:\s*(\d{4}-\d{2}-\d{2})/)?.[1] ?? null;
58 + const filedAt = parseTimestamp(item.updated ?? item.published) ?? (filedDate ? Date.parse(filedDate) : null);
59 + if (!filedAt) continue;
60 + filings.push({
61 + id: `${accn}:${cik ?? "x"}`,
62 + cik,
63 + companyName,
64 + formType,
65 + filedAt,
66 + url: item.link ?? `https://www.sec.gov/Archives/edgar/data/${cik ?? ""}/${accn.replace(/-/g, "")}/${accn}-index.htm`,
67 + metadata: { accession_number: accn, role, filed_date: filedDate, size: summary.match(/Size:\s*([\d.]+\s*\w+)/)?.[1] ?? null, material: MATERIAL_FORMS.has(formType) && role !== "Reporting" },
68 + });
69 + }
70 + return { observations: [], filings, stats: { entries: feed.items.length } };
71 + },
72 + fixturesDir: "fixtures",
73 +});
added connectors/src/sources.ts +22 −0
@@ -0,0 +1,22 @@
1 +import type { SourceRecord } from "@market-atlas/market-model";
2 +
3 +/**
4 + * Source registry. A source is an organization/endpoint family; a connector is one technical
5 + * integration of a source. `family` groups sources believed to share an upstream so the consensus
6 + * engine does not count them as independent confirmations.
7 + */
8 +export const SOURCES: SourceRecord[] = [
9 + { id: "coinbase", name: "Coinbase Exchange", organization: "Coinbase Global, Inc.", sourceType: "WEBSOCKET", homepage: "https://exchange.coinbase.com", jurisdiction: "US", rightsStatus: "PUBLIC_ATTRIBUTED", realtimeStatus: "REALTIME", family: "coinbase", category: "EXCHANGE", enabled: true },
10 + { id: "kraken", name: "Kraken", organization: "Payward, Inc.", sourceType: "WEBSOCKET", homepage: "https://www.kraken.com", jurisdiction: "US", rightsStatus: "PUBLIC_ATTRIBUTED", realtimeStatus: "REALTIME", family: "kraken", category: "EXCHANGE", enabled: true },
11 + { id: "binance", name: "Binance", organization: "Binance", sourceType: "WEBSOCKET", homepage: "https://www.binance.com", jurisdiction: null, rightsStatus: "PUBLIC_ATTRIBUTED", realtimeStatus: "REALTIME", family: "binance", category: "EXCHANGE", enabled: true },
12 + { id: "okx", name: "OKX", organization: "OKX", sourceType: "WEBSOCKET", homepage: "https://www.okx.com", jurisdiction: null, rightsStatus: "PUBLIC_ATTRIBUTED", realtimeStatus: "REALTIME", family: "okx", category: "EXCHANGE", enabled: true },
13 + { id: "cboe", name: "Cboe Global Markets — delayed quotes", organization: "Cboe Global Markets", sourceType: "XHR", homepage: "https://www.cboe.com", jurisdiction: "US", rightsStatus: "DELAYED", realtimeStatus: "DELAYED", family: "cboe", category: "EXCHANGE", enabled: true },
14 + { id: "ecb-frankfurter", name: "ECB euro foreign exchange reference rates (via Frankfurter)", organization: "European Central Bank", sourceType: "OFFICIAL_API", homepage: "https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html", jurisdiction: "EU", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", family: "ecb", category: "CENTRAL_BANK", enabled: true },
15 + { id: "bank-of-canada", name: "Bank of Canada Valet API", organization: "Bank of Canada", sourceType: "OFFICIAL_API", homepage: "https://www.bankofcanada.ca/valet/docs", jurisdiction: "CA", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", family: "bank-of-canada", category: "CENTRAL_BANK", enabled: true },
16 + { id: "us-treasury", name: "U.S. Department of the Treasury — daily par yield curve", organization: "U.S. Department of the Treasury", sourceType: "XML", homepage: "https://home.treasury.gov/resource-center/data-chart-center/interest-rates", jurisdiction: "US", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "END_OF_DAY", family: "us-treasury", category: "GOVERNMENT", enabled: true },
17 + { id: "sec-edgar", name: "SEC EDGAR", organization: "U.S. Securities and Exchange Commission", sourceType: "RSS", homepage: "https://www.sec.gov/edgar", jurisdiction: "US", rightsStatus: "OFFICIAL_OPEN_DATA", realtimeStatus: "REALTIME", family: "sec", category: "REGULATOR", enabled: true },
18 + { id: "nasdaq-trader", name: "Nasdaq Trader", organization: "Nasdaq, Inc.", sourceType: "BULK_FILE", homepage: "https://www.nasdaqtrader.com", jurisdiction: "US", rightsStatus: "PUBLIC_ATTRIBUTED", realtimeStatus: "REALTIME", family: "nasdaq", category: "EXCHANGE", enabled: true },
19 + { id: "hfmarketdata", name: "HF Market Data", organization: "spboucher.ai", sourceType: "OFFICIAL_API", homepage: "https://www.hfmarketdata.io", jurisdiction: "CA", rightsStatus: "LICENSED", realtimeStatus: "END_OF_DAY", family: "firstrate", category: "OFFICIAL_DATASET", enabled: true },
20 +];
21 +
22 +export const sourceById = (id: string) => SOURCES.find((s) => s.id === id);
added connectors/src/us-treasury-yield-curve/README.md +7 −0
@@ -0,0 +1,7 @@
1 +# us-treasury-yield-curve
2 +
3 +Official Treasury XML (OData Atom) feed: `…/interest-rates/pages/xml?data=daily_treasury_yield_curve&field_tdr_date_value_month=YYYYMM`.
4 +
5 +- **Type**: XML · **Rights**: OFFICIAL_OPEN_DATA (public domain) · **Real-time**: END_OF_DAY (posted ≈ 15:30–18:00 ET)
6 +- **Instruments**: 13 tenors `US1M … US30Y` (TREASURY, unit percent). Fields: YIELD (+ LAST_PRICE copy), PREVIOUS_CLOSE.
7 +- **Quirks**: the month file only contains the current month → the previous month is fetched when fewer than 2 rows exist. Tenors that did not exist (e.g. 4-month before 2022) are simply absent.
added connectors/src/us-treasury-yield-curve/fixtures/month.xml +41 −0
@@ -0,0 +1,41 @@
1 +<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
2 +<feed xml:base="https://home.treasury.gov/resource-center/data-chart-center/interest-rates/pages/xml" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
3 +<title type="text">DailyTreasuryYieldCurveRateData</title>
4 +<updated>2026-09-12T02:01:04Z</updated>
5 +<entry>
6 +<id>https://home.treasury.gov/resource-center/data-chart-center/interest-rates/pages/xml-item?data=daily_treasury_yield_curve&amp;id=307</id>
7 +<content type="application/xml">
8 +<m:properties>
9 +<d:Id m:type="Edm.Int32">307</d:Id>
10 +<d:NEW_DATE m:type="Edm.DateTime">2026-09-10T00:00:00</d:NEW_DATE>
11 +<d:BC_1MONTH m:type="Edm.Double">3.85</d:BC_1MONTH>
12 +<d:BC_2MONTH m:type="Edm.Double">3.86</d:BC_2MONTH>
13 +<d:BC_3MONTH m:type="Edm.Double">3.80</d:BC_3MONTH>
14 +<d:BC_6MONTH m:type="Edm.Double">3.66</d:BC_6MONTH>
15 +<d:BC_1YEAR m:type="Edm.Double">3.50</d:BC_1YEAR>
16 +<d:BC_2YEAR m:type="Edm.Double">3.48</d:BC_2YEAR>
17 +<d:BC_5YEAR m:type="Edm.Double">3.62</d:BC_5YEAR>
18 +<d:BC_10YEAR m:type="Edm.Double">4.05</d:BC_10YEAR>
19 +<d:BC_30YEAR m:type="Edm.Double">4.70</d:BC_30YEAR>
20 +</m:properties>
21 +</content>
22 +</entry>
23 +<entry>
24 +<id>https://home.treasury.gov/resource-center/data-chart-center/interest-rates/pages/xml-item?data=daily_treasury_yield_curve&amp;id=308</id>
25 +<content type="application/xml">
26 +<m:properties>
27 +<d:Id m:type="Edm.Int32">308</d:Id>
28 +<d:NEW_DATE m:type="Edm.DateTime">2026-09-11T00:00:00</d:NEW_DATE>
29 +<d:BC_1MONTH m:type="Edm.Double">3.84</d:BC_1MONTH>
30 +<d:BC_2MONTH m:type="Edm.Double">3.85</d:BC_2MONTH>
31 +<d:BC_3MONTH m:type="Edm.Double">3.79</d:BC_3MONTH>
32 +<d:BC_6MONTH m:type="Edm.Double">3.64</d:BC_6MONTH>
33 +<d:BC_1YEAR m:type="Edm.Double">3.47</d:BC_1YEAR>
34 +<d:BC_2YEAR m:type="Edm.Double">3.45</d:BC_2YEAR>
35 +<d:BC_5YEAR m:type="Edm.Double">3.60</d:BC_5YEAR>
36 +<d:BC_10YEAR m:type="Edm.Double">4.02</d:BC_10YEAR>
37 +<d:BC_30YEAR m:type="Edm.Double">4.68</d:BC_30YEAR>
38 +</m:properties>
39 +</content>
40 +</entry>
41 +</feed>
added connectors/src/us-treasury-yield-curve/index.ts +106 −0
@@ -0,0 +1,106 @@
1 +import { defineConnector, parseXml, raw, type NormalizedBatch, type ProposedInstrument } from "@market-atlas/connector-sdk";
2 +import type { InstrumentHint, NormalizedObservation } from "@market-atlas/market-model";
3 +import { zonedTimeToUtc } from "@market-atlas/market-model";
4 +
5 +const URL = "https://home.treasury.gov/resource-center/data-chart-center/interest-rates/pages/xml?data=daily_treasury_yield_curve&field_tdr_date_value_month=";
6 +const TENORS: Array<[string, string, string]> = [
7 + ["BC_1MONTH", "US1M", "US Treasury 1-month"],
8 + ["BC_2MONTH", "US2M", "US Treasury 2-month"],
9 + ["BC_3MONTH", "US3M", "US Treasury 3-month"],
10 + ["BC_4MONTH", "US4M", "US Treasury 4-month"],
11 + ["BC_6MONTH", "US6M", "US Treasury 6-month"],
12 + ["BC_1YEAR", "US1Y", "US Treasury 1-year"],
13 + ["BC_2YEAR", "US2Y", "US Treasury 2-year"],
14 + ["BC_3YEAR", "US3Y", "US Treasury 3-year"],
15 + ["BC_5YEAR", "US5Y", "US Treasury 5-year"],
16 + ["BC_7YEAR", "US7Y", "US Treasury 7-year"],
17 + ["BC_10YEAR", "US10Y", "US Treasury 10-year"],
18 + ["BC_20YEAR", "US20Y", "US Treasury 20-year"],
19 + ["BC_30YEAR", "US30Y", "US Treasury 30-year"],
20 +];
21 +const hint = (symbol: string, name: string): InstrumentHint => ({ assetClass: "TREASURY", name: `${name} par yield`, exchangeId: null, currency: "USD", country: "US", securityType: "PAR_YIELD", metadata: { featured: ["US2Y", "US10Y", "US30Y", "US3M"].includes(symbol), unit: "percent" } });
22 +const seeds: ProposedInstrument[] = TENORS.map(([, s, n]) => ({ symbol: s, hint: hint(s, n), aliases: [s.replace("US", "US "), `${s.slice(2)} treasury`] }));
23 +
24 +const month = (d: Date) => `${d.getUTCFullYear()}${String(d.getUTCMonth() + 1).padStart(2, "0")}`;
25 +
26 +/** U.S. Treasury daily par yield curve (official XML/OData feed). Publishes once per business day after ~15:30 ET. */
27 +export const usTreasuryYieldCurve = defineConnector({
28 + metadata: {
29 + id: "us-treasury-yield-curve",
30 + name: "U.S. Treasury — daily par yield curve",
31 + version: "1.0.0",
32 + sourceId: "us-treasury",
33 + organization: "U.S. Department of the Treasury",
34 + sourceType: "XML",
35 + jurisdiction: "US",
36 + rightsStatus: "OFFICIAL_OPEN_DATA",
37 + realtimeStatus: "END_OF_DAY",
38 + expectedLatencyMs: null,
39 + supportsStreaming: false,
40 + supportsHistorical: true,
41 + assetClasses: ["TREASURY"],
42 + exchanges: [],
43 + homepage: "https://home.treasury.gov/resource-center/data-chart-center/interest-rates/TextView?type=daily_treasury_yield_curve",
44 + description: "Daily Treasury par yield curve rates (1 month → 30 years) from the official Treasury XML feed. One value per tenor per business day.",
45 + rightsNotes: "U.S. federal government work — public domain.",
46 + termsUrl: "https://home.treasury.gov/subfooter/terms-of-use",
47 + sourceFamily: "us-treasury",
48 + enabled: true,
49 + },
50 + seeds,
51 + defaultSymbols: seeds.map((s) => s.symbol),
52 + rateLimits: { "home.treasury.gov": 0.5 },
53 + schedule: { intervalMs: 60 * 60_000 },
54 + async poll(ctx) {
55 + const now = new Date();
56 + const out = [];
57 + const cur = await ctx.http.getText(URL + month(now), { timeoutMs: 30_000 });
58 + if (!cur.notModified) out.push(raw("us-treasury-yield-curve", "us-treasury", "month", cur.text, { month: month(now) }));
59 + // Early in the month the current file may hold < 2 rows → also read the previous month for PREVIOUS_CLOSE.
60 + if ((cur.text.match(/<entry>/g) ?? []).length < 2) {
61 + const prevMonth = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() - 1, 1));
62 + const prev = await ctx.http.getText(URL + month(prevMonth), { timeoutMs: 30_000 });
63 + out.unshift(raw("us-treasury-yield-curve", "us-treasury", "month", prev.text, { month: month(prevMonth) }));
64 + }
65 + return out;
66 + },
67 + normalize(r): NormalizedBatch {
68 + if (r.kind !== "month" || typeof r.payload !== "string") return { observations: [] };
69 + const doc = parseXml<any>(r.payload);
70 + const entries: any[] = doc?.feed?.entry ? (Array.isArray(doc.feed.entry) ? doc.feed.entry : [doc.feed.entry]) : [];
71 + const rows = entries
72 + .map((e) => e?.content?.["m:properties"])
73 + .filter(Boolean)
74 + .map((p: Record<string, unknown>) => ({ date: text(p["d:NEW_DATE"]), values: Object.fromEntries(TENORS.map(([k]) => [k, num(p[`d:${k}`])])) }))
75 + .filter((row) => row.date)
76 + .sort((a, b) => a.date!.localeCompare(b.date!));
77 + const last = rows[rows.length - 1];
78 + const prev = rows[rows.length - 2];
79 + if (!last) return { observations: [] };
80 + const observations: NormalizedObservation[] = [];
81 + for (const [key, symbol, name] of TENORS) {
82 + const v = last.values[key];
83 + if (v == null) continue;
84 + const ts = zonedTimeToUtc(`${last.date!.slice(0, 10)}T15:30:00`, "America/New_York");
85 + const base = { symbol, instrumentHint: hint(symbol, name), currency: "USD", sourceTimestamp: ts, timestampTrust: "SOURCE" as const, rightsStatus: "OFFICIAL_OPEN_DATA" as const, realtimeStatus: "END_OF_DAY" as const, meta: { date: last.date!.slice(0, 10) } };
86 + observations.push({ ...base, field: "YIELD", value: v }, { ...base, field: "LAST_PRICE", value: v });
87 + const pv = prev?.values[key];
88 + if (pv != null) observations.push({ ...base, field: "PREVIOUS_CLOSE", value: pv, meta: { date: prev!.date!.slice(0, 10) } });
89 + }
90 + return { observations, stats: { rows: rows.length } };
91 + },
92 + fixturesDir: "fixtures",
93 +});
94 +
95 +function text(v: unknown): string | null {
96 + if (v == null) return null;
97 + if (typeof v === "string") return v;
98 + if (typeof v === "object" && typeof (v as Record<string, unknown>)["#text"] === "string") return (v as Record<string, string>)["#text"]!;
99 + return null;
100 +}
101 +function num(v: unknown): number | null {
102 + const t = text(v);
103 + if (t == null || t === "") return null;
104 + const n = Number(t);
105 + return Number.isFinite(n) ? n : null;
106 +}
added deploy/market-atlas.mld.json +114 −0
@@ -0,0 +1,114 @@
1 +{
2 + "app": "market-atlas",
3 + "label": "Market Atlas — the live map of global markets",
4 + "domain": "www.market-atlas.co",
5 + "port": 8380,
6 + "health_path": "/v1/health",
7 + "dir": "~/apps/market-atlas",
8 + "extra_paths": [],
9 + "sync_excludes": [
10 + ".git/",
11 + ".env",
12 + ".env.*",
13 + "!.env.example",
14 + "node_modules/",
15 + "apps/api/dist/",
16 + "apps/web/.next/",
17 + "apps/web/.next-*/",
18 + "apps/web/next-env.d.ts",
19 + "apps/web/qa/screens/",
20 + "*.tsbuildinfo",
21 + "data/",
22 + "/tmp/",
23 + "logs/",
24 + "coverage/",
25 + ".DS_Store",
26 + ".claude/",
27 + "deploy/.admin-token",
28 + "deploy/.hfmd-key",
29 + "deploy/rendered/"
30 + ],
31 + "requires": {
32 + "runtimes": ["pm2", "node", "pnpm", "postgresql@17"],
33 + "ram_gb": 6,
34 + "ports": [8380, 8382]
35 + },
36 + "ram_mb_observed": 1500,
37 + "size_mb": 30,
38 + "placement": {
39 + "pin": "M2U64",
40 + "prefer": null,
41 + "avoid": ["M3U96b", "M1M32"],
42 + "reason": "Mac Studio M2 Ultra 24 c / 64 Go : Postgres 17 Homebrew déjà présent (rôle/db market_atlas créés le 2026-09-12), 40 Go libres, raccordé au tunnel wg1 (10.67.0.12) ; node 25 + pnpm 9 + PM2"
43 + },
44 + "processes": [
45 + {
46 + "name": "market-atlas-core",
47 + "manager": "pm2",
48 + "script": "/opt/homebrew/bin/node",
49 + "args": ["apps/api/dist/main.js"],
50 + "interpreter": null,
51 + "cwd": "{{HOME}}/apps/market-atlas",
52 + "env": {
53 + "APP_ENV": "production",
54 + "NODE_ENV": "production",
55 + "MA_SITE_URL": "https://www.market-atlas.co",
56 + "MA_API_HOST": "0.0.0.0",
57 + "MA_API_PORT": "8380",
58 + "MA_ROLE": "all",
59 + "MA_WEB_UPSTREAM": "http://127.0.0.1:8382",
60 + "DATABASE_URL": "postgres://market_atlas:market_atlas@127.0.0.1:5432/market_atlas",
61 + "MA_DATA_DIR": "{{HOME}}/market-atlas-data",
62 + "MA_ADMIN_TOKEN": "{{ADMIN_TOKEN}}",
63 + "MA_CONTACT_EMAIL": "spbou4@icloud.com",
64 + "HFMD_API_KEY": "{{HFMD_KEY}}",
65 + "MA_LOG_JSON": "1",
66 + "MA_RAW_SAMPLE_RATE": "0.02",
67 + "MA_OBSERVATION_RETENTION_DAYS": "45",
68 + "PG_DUMP": "/opt/homebrew/opt/postgresql@17/bin/pg_dump",
69 + "PATH": "/opt/homebrew/opt/postgresql@17/bin:/opt/homebrew/bin:/usr/bin:/bin"
70 + },
71 + "cron_restart": null,
72 + "autorestart": true,
73 + "max_memory_restart": "3G"
74 + },
75 + {
76 + "name": "market-atlas-web",
77 + "manager": "pm2",
78 + "script": "/opt/homebrew/bin/node",
79 + "args": ["node_modules/next/dist/bin/next", "start", "-p", "8382", "-H", "127.0.0.1"],
80 + "interpreter": null,
81 + "cwd": "{{HOME}}/apps/market-atlas/apps/web",
82 + "env": {
83 + "NODE_ENV": "production",
84 + "API_URL": "http://127.0.0.1:8380",
85 + "NEXT_PUBLIC_SITE_URL": "https://www.market-atlas.co",
86 + "NEXT_TELEMETRY_DISABLED": "1"
87 + },
88 + "cron_restart": null,
89 + "autorestart": true,
90 + "max_memory_restart": "2G"
91 + }
92 + ],
93 + "ngrok": null,
94 + "launchd": [],
95 + "env_overrides": {},
96 + "hooks": {
97 + "post_sync": [
98 + "set -o pipefail; mkdir -p $HOME/market-atlas-data/{raw,archive,backups,logs,cache} && echo ' data dirs ok'",
99 + "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm install --frozen-lockfile --silent && echo ' deps ok'",
100 + "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm --filter @market-atlas/api build 2>&1 | tail -2 && echo ' api build ok'",
101 + "set -o pipefail; export PATH=\"/opt/homebrew/opt/postgresql@17/bin:/opt/homebrew/bin:$PATH\"; DATABASE_URL=postgres://market_atlas:market_atlas@127.0.0.1:5432/market_atlas MA_DATA_DIR=$HOME/market-atlas-data MA_LOG_JSON=0 node apps/api/dist/cli.js migrate | tail -1 && DATABASE_URL=postgres://market_atlas:market_atlas@127.0.0.1:5432/market_atlas MA_DATA_DIR=$HOME/market-atlas-data MA_LOG_JSON=0 node apps/api/dist/cli.js seed | tail -1 && echo ' migrate + seed ok'",
102 + "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; cd apps/web && API_URL=http://127.0.0.1:8380 NEXT_PUBLIC_SITE_URL=https://www.market-atlas.co NEXT_TELEMETRY_DISABLED=1 pnpm build 2>&1 | tail -3 && echo ' web build ok'"
103 + ],
104 + "post_start": []
105 + },
106 + "notes": "v0.1.0 (2026-09-12) : monorepo TypeScript. market-atlas-core = Fastify sur 0.0.0.0:8380 (edge public : REST /v1, WebSocket /v1/stream, SSE /v1/sse ; tout autre chemin proxyfié vers Next 127.0.0.1:8382) + workers in-process (14 connecteurs : Coinbase/Kraken/Binance/OKX WS, Cboe delayed, ECB/Frankfurter, Banque du Canada, US Treasury, SEC EDGAR filings + directory, Nasdaq Trader symbol directory/halts/calendar, hfmarketdata), consensus multi-sources, moteur d'événements, barres, archive brute ~/market-atlas-data. Postgres 17 Homebrew local (db/rôle market_atlas, observations partitionnées par jour, archive gzip après 45 j, pg_dump métadonnées 04:00). Secrets : MA_ADMIN_TOKEN (laptop deploy/.admin-token) et HFMD_API_KEY (deploy/.hfmd-key). Console : https://www.market-atlas.co/admin.",
107 + "tunnel": {
108 + "domain": "www.market-atlas.co",
109 + "gateway": "BHS64",
110 + "redirects": [],
111 + "websocket": true,
112 + "note": "DNS GoDaddy A www → 51.161.112.61 posé par l'utilisateur le 2026-09-12. Apex market-atlas.co : ajouter A @ → 51.161.112.61 puis redirects: [\"market-atlas.co\"]."
113 + }
114 +}
added deploy/render-manifest.sh +19 −0
@@ -0,0 +1,19 @@
1 +#!/bin/bash
2 +# Render deploy/market-atlas.mld.json with the real secrets and push it to the mld gateway (M1M32:~/dispatch/apps/).
3 +# Usage: deploy/render-manifest.sh [--push]
4 +# Secrets (git-ignored): deploy/.admin-token (generated if missing), deploy/.hfmd-key (hfmarketdata.io API key; optional, keyless limits otherwise).
5 +set -euo pipefail
6 +cd "$(dirname "$0")/.."
7 +TOKEN_FILE=deploy/.admin-token
8 +HFMD_FILE=deploy/.hfmd-key
9 +[ -s "$TOKEN_FILE" ] || { openssl rand -hex 24 > "$TOKEN_FILE"; echo "generated $TOKEN_FILE"; }
10 +TOKEN=$(tr -d '\n' < "$TOKEN_FILE")
11 +HFMD_KEY=""
12 +[ -s "$HFMD_FILE" ] && HFMD_KEY=$(tr -d '\n' < "$HFMD_FILE") || echo "warning: $HFMD_FILE missing — hfmarketdata connector will use keyless limits"
13 +mkdir -p deploy/rendered
14 +sed -e "s/{{ADMIN_TOKEN}}/$TOKEN/g" -e "s#{{HFMD_KEY}}#$HFMD_KEY#g" deploy/market-atlas.mld.json > deploy/rendered/market-atlas.json
15 +python3 -c "import json; json.load(open('deploy/rendered/market-atlas.json')); print('manifest ok')"
16 +if [ "${1:-}" = "--push" ]; then
17 + scp -q deploy/rendered/market-atlas.json M1M32:~/dispatch/apps/market-atlas.json
18 + ssh M1M32 'chmod 600 ~/dispatch/apps/market-atlas.json && echo "pushed to M1M32:~/dispatch/apps/market-atlas.json"'
19 +fi
added docs/API.md +44 −0
@@ -0,0 +1,44 @@
1 +# Market Atlas API v1
2 +
3 +Base: `https://www.market-atlas.co/v1` (same origin as the web app; in development `http://127.0.0.1:8391/v1`).
4 +Envelope: `{ "data": …, "meta": { "request_id", "timestamp", … } }`; errors: `{ "error": { "code", "message" } }`.
5 +Public rate limit: 240 requests/min per IP (burst 60), `429` + `retry-after`. Admin routes need `x-ma-admin-token`.
6 +
7 +Quote object (`publicQuote`): `instrument_id, symbol, name, asset_class, exchange_id, country, price, open, high, low, previous_close,
8 +change, change_percent, volume, bid, ask, currency, source_count, dispersion_bps, confidence, freshness_ms, data_status, market_state,
9 +rights_status, withheld, updated_at, source_timestamp, session_high, session_low`.
10 +`data_status``REALTIME | DELAYED | END_OF_DAY | AT_CLOSE | STALE | WITHHELD` — never present STALE/AT_CLOSE as live.
11 +
12 +| Method & path | Purpose |
13 +|---|---|
14 +| `GET /health` | liveness (db, connectors healthy, quotes cached) |
15 +| `GET /status` | public status components (website, api, stream, ingestion, source groups) |
16 +| `GET /stats` | homepage telemetry — real counters only |
17 +| `GET /metrics` | Prometheus exposition |
18 +| `GET /instruments?asset_class&exchange&country&q&quoted=1&sort&limit&offset` | instrument list with quotes |
19 +| `GET /instruments/:idOrSymbol` | instrument + company + exchange status + quote + aliases + sources + related (`NASDAQ:AAPL` form accepted) |
20 +| `GET /quotes/:idOrSymbol` · `GET /quotes?ids=a,b` | canonical quotes |
21 +| `GET /quotes/:id/provenance` | **Why this price?** — contributions, weights, inclusion reasons, method, versions |
22 +| `GET /history/:id?resolution=1m|5m|15m|1h|1d&from&to&limit` | bars (`p` = producer: `consensus` or a historical connector id) |
23 +| `GET /events?type&instrument&country&asset_class&severity&since&before&limit` · `GET /events/:id` | canonical market events |
24 +| `GET /filings?form&cik&q&limit&offset` | regulatory filings |
25 +| `GET /search?q` | hybrid search (exact/prefix/alias/name), grouped by asset class / exchange / country |
26 +| `GET /markets` | global overview: indices, crypto, forex, rates, commodities, equities, gainers/losers, exchanges with status, breadth |
27 +| `GET /breadth?scope=world|US|xnas` | advancers/decliners/median return |
28 +| `GET /changes?window=1m|5m|15m|1h|1d` | what changed: movers > 1 %, events by type, filings |
29 +| `GET /exchanges` · `GET /exchanges/:id` | venues, status/next transition, holidays, breadth, movers, events |
30 +| `GET /countries` · `GET /countries/:code` | country atlas page data |
31 +| `GET /compare?ids=a,b,c&resolution=1d&limit` | normalized series, return/vol/drawdown, correlations |
32 +| `GET /sources` · `GET /connectors` · `GET /data-health` | source directory, public connector table, data-health dashboard |
33 +| `GET /stream` (WebSocket) · `GET /sse?channels=` | live stream — see below |
34 +| `GET /admin/overview|connectors|connectors/:id|schema-changes|divergence|storage|raw?ref=` · `POST /admin/connectors/:id/pause|resume|restart|test` · `POST /admin/connectors/:id/schema-changes/:changeId/ack` · `POST /admin/discovery {url}` | admin console |
35 +
36 +## Stream protocol
37 +
38 +Client → server: `{"action":"subscribe","channels":["quotes:AAPL","quotes:crypto_btc_usd","quotes:*","quotes:class:CRYPTO","events:*","events:US","events:type:TRADING_HALT","market:xnas","tape"]}`,
39 +`{"action":"unsubscribe",…}`, `{"action":"ping","id":1}`.
40 +
41 +Server → client: `hello`, `subscriptions`, `heartbeat` (25 s), `pong`, `error`, and batches at ≤ 10 Hz per client:
42 +`{"type":"batch","seq":12,"ts":…,"messages":[{"type":"quote","instrument_id","symbol","price","change","change_pct","bid","ask","volume","currency","timestamp","received","confidence","sources","status"}, {"type":"event","event":{…}}, {"type":"market_state","exchangeId","state","at"}]}`.
43 +`quotes:*` is throttled to one quote per instrument per 500 ms; explicit `quotes:<id>` subscriptions receive every recomputation.
44 +`seq` is per client; a gap means the client missed batches (backpressure drops oldest). Withheld (non-redistributable) values are never streamed.
added docs/ARCHITECTURE.md +77 −0
@@ -0,0 +1,77 @@
1 +# Market Atlas — architecture (v0.1, 2026-09-12)
2 +
3 +Market Atlas is a market **observation infrastructure**: a connector network observes public/official sources,
4 +normalizes their outputs into one observation model, resolves instruments, builds a multi-source consensus,
5 +derives events, keeps the history, and redistributes the result through one web app, a REST API and a stream.
6 +
7 +```
8 + sources (WS / XHR / official APIs / RSS / XML / HTML / bulk files)
9 + └─ connectors/ ConnectorDefinition (metadata + rights + normalize()) [L0 raw archived, sampled]
10 + └─ ConnectorManager lifecycle · adaptive polling · rate limits · circuit breaker · schema drift
11 + └─ Pipeline validate → resolve instrument → Observation (L3) → ObservationWriter (Postgres, daily partitions)
12 + ├─ ConsensusEngine weighted median · source families · outliers · freshness → CanonicalQuote
13 + │ └─ QuoteStore hot cache + debounced upsert · bus "canonical.quote"
14 + ├─ EventEngine price change · session high/low · volatility · divergence · halts · filings · market open/close
15 + ├─ BarAggregator live 1m bars → SQL rollups 5m/15m/1h/1d (+ backfill bars from historical connectors)
16 + └─ HealthEngine state · latency · parse rate · reliability score (0–100) · minute snapshots
17 + MarketBus (in-process, topic names = spec) → API (Fastify): REST /v1 · WebSocket /v1/stream · SSE /v1/sse · Prometheus /v1/metrics
18 + apps/web (Next.js) consumes the API and the stream; in production the API process is the public edge and proxies to Next.
19 +```
20 +
21 +## Packages
22 +
23 +| Path | Role |
24 +|---|---|
25 +| `packages/market-model` | Vocabularies (asset classes, fields, rights/real-time statuses, event types), types (Instrument, Observation, CanonicalQuote, MarketEvent…), stable id builders, zod schemas, Intl-only time-zone utilities. |
26 +| `packages/connector-sdk` | `defineConnector`, `ConnectorContext`, polite `HttpClient` (per-host token buckets, ETag/If-Modified-Since, jittered retries, redacted URLs), `ManagedWebSocket` (reconnect, heartbeat, stale detection), parsers (RSS/Atom, XML, delimited, numbers, HTML tables, change-detection fingerprints), schema fingerprints, SSRF guard, offline test helpers. |
27 +| `connectors/` | One folder per connector: `index.ts`, `fixtures/`, `README.md`; `sources.ts` = source registry with families; `connectors.test.ts`. |
28 +| `apps/api` | Runtime: config, Postgres access + forward-only SQL migrations (`infra/migrations`), core engines, connector manager, scheduler, REST/WS/SSE, admin API, discovery prototype, CLI (`ma`). Bundled with esbuild to `dist/`. |
29 +| `apps/web` | Next.js 16 app (see `docs/FRONTEND.md`). |
30 +
31 +## Data layers
32 +
33 +- **L0 raw**`MA_DATA_DIR/raw/<day>/<connector>/<sha>.json.gz` (redacted, content-addressed). Polled/bulk payloads always; streaming frames sampled (`MA_RAW_SAMPLE_RATE`). `raw_ref` on observations points back. `ma replay <day>` re-normalizes offline.
34 +- **L1/L2** — connector `normalize()` (pure, deterministic, no network) → `NormalizedObservation` with source symbol + instrument hint, rights & real-time status, timestamp trust.
35 +- **L3**`observations` (partitioned by `received_at` day; fingerprint dedupes reconnect replays; `normalizer_version`). Retention `MA_OBSERVATION_RETENTION_DAYS` then gzip NDJSON archive in `MA_DATA_DIR/archive` (never silently deleted; `observation_archives` index).
36 +- **L4**`canonical_quotes`, `bars`, `market_events`, `filings`, `connector_health`, `fact_changes`, `document_snapshots`.
37 +
38 +## Consensus (apps/api/src/core/consensus.ts)
39 +
40 +Per instrument and field, the newest observation of each source is kept (out-of-order values ignored). A value is *fresh* within a
41 +window depending on its real-time class (REALTIME 15 s, DELAYED 30 min, INDICATIVE 1 h, END_OF_DAY 3 d; 4 d for live classes while the
42 +venue is closed). Weight = source reliability × timestamp trust × real-time class × freshness decay × official bonus. One vote per
43 +**source family** (sources believed to share an upstream count once). When live classes exist, END_OF_DAY/INDICATIVE values are
44 +*superseded*. Outliers (> 2 % from the weighted median with ≥ 3 candidates) are excluded. Output: weighted median, dispersion (bps),
45 +independent-family count, freshness, confidence (agreement + redundancy + freshness + reliability, capped at 0.995), and the full
46 +contribution list with inclusion reasons — exposed as **"Why this price?"** (`GET /v1/quotes/:id/provenance`).
47 +
48 +Rights guard: a canonical value built from any non-redistributable source is `INTERNAL_ONLY` and withheld from public responses
49 +(metadata still visible). `PUBLICLY_REDISTRIBUTABLE` in `market-model` is the single source of truth.
50 +
51 +## Market hours (core/calendar.ts)
52 +
53 +Exchanges carry IANA time zone, sessions (regular/pre/post, multi-session for Asia, weekday sets for Gulf markets, `continuous` for
54 +crypto venues) and holidays (seed + HTML connector). States PRE/OPEN/POST/CLOSED with DST handled by Intl. Used by adaptive polling,
55 +consensus freshness, MARKET_OPEN/CLOSE events, session resets and the world map.
56 +
57 +## Events (core/events.ts)
58 +
59 +Derived from canonical quotes with per-instrument adaptive baselines (EWMA of tick moves): PRICE_CHANGE (≥ max(0.5 %, 8× typical
60 +tick)), SESSION_HIGH/LOW (after history, ≥ 0.1 % improvement, cooldown), VOLATILITY_SPIKE (30-tick realized vol ≥ 4× baseline),
61 +SOURCE_DIVERGENCE (> 50 bps between included sources). Connector-proposed events (halts, filings, document changes) and system events
62 +(SOURCE_FAILURE, SCHEMA_DRIFT, MARKET_OPEN/CLOSE) go through the same dedupe (fingerprint, 24 h) and confirmation counting.
63 +
64 +## Resilience
65 +
66 +- Connector failures: jittered exponential backoff, FAILED after 3 consecutive poll failures (SOURCE_FAILURE event), auto-restart.
67 +- Schema drift: structural fingerprint per payload kind; new shapes are recorded (`connector_schema_changes`); repeated shapes that
68 + normalize to nothing pause the connector (SCHEMA_DRIFT event) rather than emitting wrong values.
69 +- Backpressure: bounded observation queue (200 k) — only REALTIME ticks are dropped under pressure, never events; batched inserts.
70 +- Graceful shutdown flushes writers, quotes, bars, events, raw archive.
71 +- Split roles are possible (`MA_ROLE=api|worker`) — the bus is in-process today; topic names match the spec for a future NATS/Redpanda.
72 +
73 +## Security
74 +
75 +Admin API behind `x-ma-admin-token` (timing-safe compare); public REST rate-limited per IP (240/min); security headers; strict
76 +parameter validation (zod); SSRF guard on discovery URLs (private ranges, localhost, cluster hostnames, DNS resolution check);
77 +secrets redacted in logs and raw archives; no stack traces in responses.
added docs/CONNECTORS.md +51 −0
@@ -0,0 +1,51 @@
1 +# Writing a Market Atlas connector
2 +
3 +A connector is one technical integration of a **source** (`connectors/src/sources.ts`). It declares metadata (type, rights,
4 +real-time class, family), optionally seeds instruments, and implements either a streaming lifecycle (`start/stop`) or a polling
5 +one (`poll` + `schedule`), plus a **pure** `normalize(raw)`.
6 +
7 +```ts
8 +export const example = defineConnector({
9 + metadata: { id: "example-xhr", name: "…", version: "1.0.0", sourceId: "example", organization: "…", sourceType: "XHR",
10 + jurisdiction: "US", rightsStatus: "PUBLIC_ATTRIBUTED", realtimeStatus: "DELAYED", expectedLatencyMs: 900_000,
11 + supportsStreaming: false, supportsHistorical: false, assetClasses: ["EQUITY"], exchanges: ["xnys"], homepage: "https://…",
12 + description: "What it observes, how often, what is delayed.", rightsNotes: "Attribution…", termsUrl: "https://…",
13 + sourceFamily: "example", enabled: true },
14 + seeds: [{ symbol: "ABC", hint: { assetClass: "EQUITY", name: "ABC Corp", exchangeId: "xnys", mic: "XNYS", currency: "USD", country: "US" } }],
15 + defaultSymbols: ["ABC"],
16 + rateLimits: { "api.example.com": 2 }, // requests/second, shared bucket
17 + schedule: { openMs: 60_000, closedMs: 900_000, weekendMs: 3_600_000, exchangeId: "xnys" }, // or { intervalMs }
18 + async poll(ctx) { const { data, response } = await ctx.http.getJson(url); return response.notModified ? [] : [raw(id, sourceId, "quote", data)]; },
19 + normalize(r) { /* payload → { observations, events?, instruments?, filings?, holidays?, bars? } */ },
20 + fixturesDir: "fixtures",
21 +});
22 +```
23 +
24 +Rules (enforced by tests and the runtime):
25 +
26 +1. **Metadata is complete**`ConnectorMetadataSchema`; `rightsStatus` must not be UNKNOWN for production connectors; describe delays honestly.
27 +2. **normalize() is pure** — no network, no clock other than `raw.receivedAt`; deterministic so raw payloads can be replayed (`ma replay`).
28 +3. **Timestamps** — always convert to UTC ms; declare `timestampTrust` (EXCHANGE / SOURCE / CONNECTOR). Naive local times → `zonedTimeToUtc(local, tz)`.
29 +4. **Symbols** — emit the source-native symbol; provide an `instrumentHint` so the resolver can create the instrument deterministically (`makeInstrumentId`). Aliases (`BRK.B` / `BRK-B`) are declared in seeds/proposals.
30 +5. **Never sleep** in connector code; use `rateLimits` and the shared `ctx.http`. One WebSocket carries many symbols.
31 +6. **Fixtures + tests** — at least one valid payload, one malformed/unexpected payload; tests run offline (`makeTestContext`, `fakeFetch`).
32 +7. **README** per connector: endpoint, type, rights, real-time class, fields, quirks, schedule.
33 +8. Register in `connectors/src/index.ts` and, for a new source, in `sources.ts` (family = upstream provider when known).
34 +
35 +Side outputs of `normalize()`:
36 +
37 +| key | consumer |
38 +|---|---|
39 +| `observations` | pipeline → observations, consensus, quotes, events |
40 +| `events` (`ProposedEvent`, `dedupeKey`) | event engine (halts, document changes…) |
41 +| `instruments` (`ProposedInstrument`, `createIfMissing`) | instrument master (directories) |
42 +| `filings` (`metadata.material`) | `filings` table + FILING_PUBLISHED for material forms |
43 +| `holidays` | market-hours engine (`exchange_holidays`) |
44 +| `bars` | historical backfill (`bars`, producer = connector id) |
45 +
46 +Production connectors (v0.1): coinbase-ws, kraken-ws, binance-ws, okx-ws (WEBSOCKET) · cboe-delayed-quotes (XHR) · ecb-frankfurter,
47 +bank-of-canada-valet, hfmarketdata-daily (OFFICIAL_API) · us-treasury-yield-curve (XML) · sec-edgar-filings, nasdaq-trade-halts (RSS) ·
48 +sec-company-tickers, nasdaq-symbol-directory (BULK_FILE) · nasdaq-market-calendar (HTML change detection). See each `README.md`.
49 +
50 +Onboarding workflow for a *discovered* source: `POST /v1/admin/discovery {url}` → candidate report (endpoints, embedded state, price/symbol-like
51 +fields) → manual rights review → connector + fixtures → staging (`enabled: false` or `MA_DISABLED_CONNECTORS`) → production.
added docs/DEPLOY.md +50 −0
@@ -0,0 +1,50 @@
1 +# Deploying Market Atlas on MacLustr
2 +
3 +Production runs on **M2U64** (Mac Studio M2 Ultra, 24 c / 64 GB) behind the MacLustr Tunnel (BHS64 Caddy → WireGuard wg1 10.67.0.12).
4 +Everything goes through the gateway M1M32 and `mld` (`~/Desktop/cluster-skill/mld`).
5 +
6 +```
7 +Internet → GoDaddy DNS (A www.market-atlas.co → 51.161.112.61)
8 + → BHS64 Caddy (TLS, WebSocket passthrough) → wg1 → M2U64:8380 market-atlas-core (Fastify edge: /v1 REST + WS + SSE, workers, connectors)
9 + └─ proxies every non-/v1 path → 127.0.0.1:8382 market-atlas-web (Next.js)
10 + └─ Postgres 17 Homebrew (db/role market_atlas) · ~/market-atlas-data (raw, archive, backups, logs)
11 +```
12 +
13 +Node prerequisites (present on M2U64 on 2026-09-12): node 25, pnpm 9, pm2, `postgresql@17` (brew service), role/db `market_atlas`
14 +(password `market_atlas`, localhost only). Data dirs: `~/market-atlas-data/{raw,archive,backups,logs,cache}`.
15 +
16 +## Release procedure (from the laptop)
17 +
18 +```bash
19 +cd ~/Desktop/Projets/apps-web/market-atlas
20 +pnpm typecheck && pnpm test # 1. types + 58+ unit/connector/consensus/calendar tests
21 +deploy/render-manifest.sh --push # 2. manifest with MA_ADMIN_TOKEN + HFMD_API_KEY → M1M32:~/dispatch/apps/market-atlas.json
22 +~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/market-atlas market-atlas # 3. laptop → gateway staging (sync_excludes)
23 +~/Desktop/cluster-skill/mld deploy market-atlas --node M2U64 # 4. rsync → node, hooks (pnpm install, esbuild, migrate+seed, next build), PM2, health, tunnel route, registry
24 +curl -s https://www.market-atlas.co/v1/health # 5. public check
25 +```
26 +
27 +The `post_sync` hooks build the API bundle (`apps/api/dist`), run migrations + seeds and build the web app. `main.js` also migrates and
28 +seeds at start (idempotent). `mld heal` (every 5 min on M1M32) restarts anything missing after a power cut and keeps the Caddy route
29 +pointing at the current node.
30 +
31 +## Operations
32 +
33 +```bash
34 +~/Desktop/cluster-skill/mld status | grep market-atlas
35 +ssh M2U64 'pm2 logs market-atlas-core --lines 100 --nostream'
36 +ssh M2U64 'cd ~/apps/market-atlas && export PATH=/opt/homebrew/opt/postgresql@17/bin:/opt/homebrew/bin:$PATH DATABASE_URL=postgres://market_atlas:market_atlas@127.0.0.1:5432/market_atlas MA_DATA_DIR=$HOME/market-atlas-data && node apps/api/dist/cli.js status'
37 +ssh M2U64 '… node apps/api/dist/cli.js run cboe-delayed-quotes' # one poll, printed
38 +ssh M2U64 '… node apps/api/dist/cli.js replay 2026-09-12' # re-normalize archived raw payloads offline
39 +ssh M2U64 '… node apps/api/dist/cli.js backup' # pg_dump of metadata → ~/market-atlas-data/backups (also nightly ~04:00)
40 +curl -s https://www.market-atlas.co/v1/stats | python3 -m json.tool # live counters
41 +```
42 +
43 +Admin console: `https://www.market-atlas.co/admin` (token = `deploy/.admin-token`). Connector actions (pause/resume/restart/test),
44 +schema-change acknowledgements, storage/partitions, discovery form. Environment variables are listed in `.env.example`.
45 +
46 +Retention: observations partitions older than `MA_OBSERVATION_RETENTION_DAYS` (45) are exported to `~/market-atlas-data/archive/<year>/…ndjson.gz`
47 +then dropped (index in `observation_archives`). Bars, events, filings, quotes are permanent. Off-node copies of `backups/` and `archive/`
48 +should be scheduled to M1M32 (`rsync -a M2U64:~/market-atlas-data/backups/ M1M32:~/backups/market-atlas/`).
49 +
50 +Apex domain: add `A market-atlas.co → 51.161.112.61` at GoDaddy, then set `"redirects": ["market-atlas.co"]` in the manifest and redeploy.
added infra/migrations/0001_init.sql +346 −0
@@ -0,0 +1,346 @@
1 +-- Market Atlas — initial schema (PostgreSQL 17)
2 +-- Metadata (instruments, exchanges, sources, connectors, events) + time-series (observations partitioned by day, bars).
3 +
4 +create table if not exists schema_migrations (
5 + version text primary key,
6 + applied_at timestamptz not null default now()
7 +);
8 +
9 +create table if not exists countries (
10 + code text primary key,
11 + name text not null,
12 + region text not null,
13 + currency text
14 +);
15 +
16 +create table if not exists exchanges (
17 + id text primary key,
18 + mic text,
19 + name text not null,
20 + operator text,
21 + country text not null references countries(code),
22 + city text,
23 + timezone text not null,
24 + currency text,
25 + website text,
26 + sessions jsonb not null default '{"regular":[]}'::jsonb,
27 + lat double precision,
28 + lon double precision,
29 + asset_classes text[] not null default '{}',
30 + created_at timestamptz not null default now(),
31 + updated_at timestamptz not null default now()
32 +);
33 +create index if not exists exchanges_country_idx on exchanges(country);
34 +
35 +create table if not exists exchange_holidays (
36 + exchange_id text not null references exchanges(id) on delete cascade,
37 + date date not null,
38 + name text not null,
39 + kind text not null default 'CLOSED',
40 + close_time text,
41 + source_id text,
42 + primary key (exchange_id, date)
43 +);
44 +
45 +create table if not exists companies (
46 + id text primary key,
47 + name text not null,
48 + cik text,
49 + country text,
50 + sector text,
51 + industry text,
52 + website text,
53 + created_at timestamptz not null default now(),
54 + updated_at timestamptz not null default now()
55 +);
56 +create index if not exists companies_cik_idx on companies(cik);
57 +
58 +create table if not exists instruments (
59 + id text primary key,
60 + symbol text not null,
61 + name text not null,
62 + asset_class text not null,
63 + exchange_id text references exchanges(id),
64 + mic text,
65 + currency text,
66 + country text,
67 + company_id text references companies(id),
68 + isin text,
69 + security_type text,
70 + base text,
71 + quote text,
72 + is_active boolean not null default true,
73 + metadata jsonb not null default '{}'::jsonb,
74 + search_text text generated always as (lower(symbol || ' ' || name)) stored,
75 + created_at timestamptz not null default now(),
76 + updated_at timestamptz not null default now()
77 +);
78 +create index if not exists instruments_symbol_idx on instruments(upper(symbol));
79 +create index if not exists instruments_class_idx on instruments(asset_class);
80 +create index if not exists instruments_exchange_idx on instruments(exchange_id);
81 +create index if not exists instruments_country_idx on instruments(country);
82 +create index if not exists instruments_company_idx on instruments(company_id);
83 +create index if not exists instruments_search_idx on instruments using gin (to_tsvector('simple', search_text));
84 +
85 +create table if not exists symbol_aliases (
86 + alias text not null,
87 + source_id text not null default '*',
88 + instrument_id text not null references instruments(id) on delete cascade,
89 + primary key (alias, source_id)
90 +);
91 +create index if not exists symbol_aliases_instrument_idx on symbol_aliases(instrument_id);
92 +
93 +create table if not exists sources (
94 + id text primary key,
95 + name text not null,
96 + organization text,
97 + source_type text not null,
98 + homepage text,
99 + jurisdiction text,
100 + rights_status text not null,
101 + realtime_status text not null,
102 + family text,
103 + category text not null default 'PUBLIC_MARKET_SOURCE',
104 + enabled boolean not null default true,
105 + created_at timestamptz not null default now(),
106 + updated_at timestamptz not null default now()
107 +);
108 +
109 +create table if not exists connectors (
110 + id text primary key,
111 + source_id text not null references sources(id),
112 + name text not null,
113 + version text not null,
114 + source_type text not null,
115 + rights_status text not null,
116 + realtime_status text not null,
117 + enabled boolean not null default true,
118 + paused boolean not null default false,
119 + metadata jsonb not null default '{}'::jsonb,
120 + state jsonb not null default '{}'::jsonb, -- connector-private persisted state (cursors, seen ids)
121 + schema_fingerprints jsonb not null default '{}'::jsonb, -- kind -> [fingerprints]
122 + status text not null default 'STARTING',
123 + last_message_at timestamptz,
124 + last_success_at timestamptz,
125 + last_error_at timestamptz,
126 + last_error text,
127 + messages_total bigint not null default 0,
128 + errors_total bigint not null default 0,
129 + reconnects bigint not null default 0,
130 + reliability_score double precision,
131 + created_at timestamptz not null default now(),
132 + updated_at timestamptz not null default now()
133 +);
134 +
135 +create table if not exists connector_versions (
136 + connector_id text not null references connectors(id) on delete cascade,
137 + version text not null,
138 + first_seen_at timestamptz not null default now(),
139 + metadata jsonb not null default '{}'::jsonb,
140 + primary key (connector_id, version)
141 +);
142 +
143 +create table if not exists connector_schema_changes (
144 + id bigserial primary key,
145 + connector_id text not null references connectors(id) on delete cascade,
146 + kind text not null,
147 + old_fingerprint text,
148 + new_fingerprint text not null,
149 + sample jsonb,
150 + detected_at timestamptz not null default now(),
151 + acknowledged boolean not null default false
152 +);
153 +create index if not exists connector_schema_changes_idx on connector_schema_changes(connector_id, detected_at desc);
154 +
155 +-- Time series of connector health snapshots (1 per connector per minute).
156 +create table if not exists connector_health (
157 + ts timestamptz not null,
158 + connector_id text not null,
159 + status text not null,
160 + messages_1m integer not null default 0,
161 + errors_1m integer not null default 0,
162 + median_latency_ms integer,
163 + p95_latency_ms integer,
164 + parse_success_rate double precision,
165 + instruments_covered integer,
166 + reconnects integer not null default 0,
167 + reliability_score double precision,
168 + primary key (connector_id, ts)
169 +);
170 +create index if not exists connector_health_ts_idx on connector_health(ts);
171 +
172 +-- L3 observations, partitioned by received day. Values are kept at source precision.
173 +create table if not exists observations (
174 + received_at timestamptz not null,
175 + source_ts timestamptz,
176 + instrument_id text not null,
177 + field text not null,
178 + value double precision not null,
179 + currency text,
180 + source_id text not null,
181 + connector_id text not null,
182 + rights_status text not null,
183 + realtime_status text not null,
184 + timestamp_trust text not null,
185 + confidence real,
186 + latency_ms bigint,
187 + sequence text,
188 + fingerprint text not null,
189 + raw_ref text,
190 + normalizer_version text not null,
191 + meta jsonb
192 +) partition by range (received_at);
193 +create index if not exists observations_instr_idx on observations(instrument_id, field, received_at desc);
194 +create index if not exists observations_source_idx on observations(source_id, received_at desc);
195 +create index if not exists observations_fp_idx on observations(fingerprint);
196 +
197 +create table if not exists canonical_quotes (
198 + instrument_id text primary key references instruments(id) on delete cascade,
199 + symbol text not null,
200 + price double precision,
201 + open double precision,
202 + high double precision,
203 + low double precision,
204 + previous_close double precision,
205 + change double precision,
206 + change_percent double precision,
207 + volume double precision,
208 + bid double precision,
209 + ask double precision,
210 + currency text,
211 + source_count integer not null default 0,
212 + dispersion_bps double precision,
213 + confidence double precision not null default 0,
214 + freshness_ms integer,
215 + realtime_status text not null default 'UNKNOWN',
216 + rights_status text not null default 'UNKNOWN',
217 + updated_at timestamptz not null,
218 + source_ts timestamptz,
219 + session_high double precision,
220 + session_low double precision,
221 + contributions jsonb not null default '[]'::jsonb,
222 + consensus_version text not null
223 +);
224 +create index if not exists canonical_quotes_updated_idx on canonical_quotes(updated_at desc);
225 +create index if not exists canonical_quotes_change_idx on canonical_quotes(change_percent);
226 +
227 +create table if not exists bars (
228 + instrument_id text not null,
229 + resolution text not null,
230 + ts timestamptz not null,
231 + open double precision not null,
232 + high double precision not null,
233 + low double precision not null,
234 + close double precision not null,
235 + volume double precision,
236 + source_count integer not null default 1,
237 + producer text not null,
238 + version text not null,
239 + primary key (instrument_id, resolution, ts)
240 +);
241 +create index if not exists bars_res_ts_idx on bars(resolution, ts desc);
242 +
243 +create table if not exists market_events (
244 + id text primary key,
245 + type text not null,
246 + ts timestamptz not null,
247 + instrument_ids text[] not null default '{}',
248 + entity_ids text[] not null default '{}',
249 + severity text not null,
250 + confidence double precision not null,
251 + source_count integer not null default 1,
252 + sources text[] not null default '{}',
253 + title text not null,
254 + summary text,
255 + data jsonb not null default '{}'::jsonb,
256 + fingerprint text not null unique,
257 + supporting_observations text[] not null default '{}',
258 + first_seen_at timestamptz not null default now(),
259 + confirmed_at timestamptz
260 +);
261 +create index if not exists market_events_ts_idx on market_events(ts desc);
262 +create index if not exists market_events_type_idx on market_events(type, ts desc);
263 +create index if not exists market_events_instr_idx on market_events using gin (instrument_ids);
264 +
265 +create table if not exists filings (
266 + id text primary key,
267 + source_id text not null,
268 + cik text,
269 + company_name text not null,
270 + form_type text not null,
271 + filed_at timestamptz not null,
272 + url text not null,
273 + instrument_ids text[] not null default '{}',
274 + metadata jsonb not null default '{}'::jsonb,
275 + created_at timestamptz not null default now()
276 +);
277 +create index if not exists filings_filed_idx on filings(filed_at desc);
278 +create index if not exists filings_cik_idx on filings(cik);
279 +create index if not exists filings_form_idx on filings(form_type);
280 +
281 +create table if not exists corporate_actions (
282 + id text primary key,
283 + instrument_id text references instruments(id),
284 + company_id text references companies(id),
285 + type text not null,
286 + announced_at timestamptz,
287 + ex_date date,
288 + effective_date date,
289 + payable_date date,
290 + value double precision,
291 + currency text,
292 + ratio text,
293 + source_id text not null,
294 + url text,
295 + metadata jsonb not null default '{}'::jsonb,
296 + created_at timestamptz not null default now()
297 +);
298 +
299 +-- Slowly-changing facts diff log (market cap, shares, profile…).
300 +create table if not exists fact_changes (
301 + id bigserial primary key,
302 + entity_type text not null,
303 + entity_id text not null,
304 + field text not null,
305 + old_value jsonb,
306 + new_value jsonb,
307 + detected_at timestamptz not null default now(),
308 + effective_at timestamptz,
309 + source_id text
310 +);
311 +create index if not exists fact_changes_entity_idx on fact_changes(entity_type, entity_id, detected_at desc);
312 +
313 +-- Content fingerprints for change detection on documents/pages.
314 +create table if not exists document_snapshots (
315 + connector_id text not null,
316 + url text not null,
317 + document_hash text not null,
318 + section_hashes jsonb not null default '{}'::jsonb,
319 + fetched_at timestamptz not null default now(),
320 + raw_ref text,
321 + primary key (connector_id, url)
322 +);
323 +
324 +create table if not exists system_counters (
325 + key text primary key,
326 + value bigint not null default 0,
327 + updated_at timestamptz not null default now()
328 +);
329 +
330 +create table if not exists observation_archives (
331 + partition_name text primary key,
332 + day date not null,
333 + rows_archived bigint not null,
334 + path text not null,
335 + bytes bigint not null,
336 + archived_at timestamptz not null default now()
337 +);
338 +
339 +-- Daily counts used by the homepage telemetry (observations today, etc.).
340 +create table if not exists daily_stats (
341 + day date primary key,
342 + observations bigint not null default 0,
343 + events bigint not null default 0,
344 + filings bigint not null default 0,
345 + updated_at timestamptz not null default now()
346 +);
added package.json +28 −0
@@ -0,0 +1,28 @@
1 +{
2 + "name": "market-atlas",
3 + "version": "0.1.0",
4 + "private": true,
5 + "description": "Market Atlas — the live map of global markets (www.market-atlas.co)",
6 + "type": "module",
7 + "engines": {
8 + "node": ">=22"
9 + },
10 + "scripts": {
11 + "build": "pnpm --filter @market-atlas/api build && pnpm --filter @market-atlas/web build",
12 + "build:api": "pnpm --filter @market-atlas/api build",
13 + "build:web": "pnpm --filter @market-atlas/web build",
14 + "typecheck": "tsc -p tsconfig.json --noEmit && pnpm --filter @market-atlas/web typecheck",
15 + "test": "vitest run",
16 + "test:watch": "vitest",
17 + "dev:api": "pnpm --filter @market-atlas/api dev",
18 + "dev:web": "pnpm --filter @market-atlas/web dev",
19 + "migrate": "pnpm --filter @market-atlas/api migrate",
20 + "seed": "pnpm --filter @market-atlas/api seed",
21 + "ma": "pnpm --filter @market-atlas/api cli"
22 + },
23 + "devDependencies": {
24 + "@types/node": "^24.0.0",
25 + "typescript": "^5.9.3",
26 + "vitest": "^3.2.4"
27 + }
28 +}
added packages/connector-sdk/package.json +18 −0
@@ -0,0 +1,18 @@
1 +{
2 + "name": "@market-atlas/connector-sdk",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "exports": {
7 + ".": "./src/index.ts"
8 + },
9 + "dependencies": {
10 + "@market-atlas/market-model": "workspace:*",
11 + "fast-xml-parser": "^5.2.5",
12 + "ws": "^8.18.3",
13 + "zod": "^3.24.1"
14 + },
15 + "devDependencies": {
16 + "@types/ws": "^8.18.1"
17 + }
18 +}
added packages/connector-sdk/src/define.ts +21 −0
@@ -0,0 +1,21 @@
1 +import { ConnectorMetadataSchema, type RawObservation } from "@market-atlas/market-model";
2 +import type { ConnectorDefinition } from "./types.js";
3 +
4 +/** Validates metadata eagerly so a mis-declared connector fails at load time, not at 3 a.m. */
5 +export function defineConnector(def: ConnectorDefinition): ConnectorDefinition {
6 + const parsed = ConnectorMetadataSchema.safeParse(def.metadata);
7 + if (!parsed.success) {
8 + throw new Error(`connector "${def.metadata?.id ?? "?"}" has invalid metadata: ${parsed.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`).join("; ")}`);
9 + }
10 + if (!def.start && !def.poll) throw new Error(`connector "${def.metadata.id}" must implement start() or poll()`);
11 + if (def.poll && !def.schedule) throw new Error(`connector "${def.metadata.id}" polls but declares no schedule`);
12 + if (def.metadata.rightsStatus === "UNKNOWN") {
13 + // Allowed to exist, but the runtime will refuse to expose its values publicly.
14 + }
15 + return def;
16 +}
17 +
18 +/** Helper for connectors to build raw observations consistently. */
19 +export function raw(connectorId: string, sourceId: string, kind: string, payload: unknown, meta?: Record<string, unknown>): RawObservation {
20 + return { connectorId, sourceId, kind, payload, receivedAt: Date.now(), ...(meta ? { meta } : {}) };
21 +}
added packages/connector-sdk/src/fingerprint.ts +48 −0
@@ -0,0 +1,48 @@
1 +import { createHash } from "node:crypto";
2 +
3 +export function sha256(input: string | Buffer): string {
4 + return createHash("sha256").update(input).digest("hex");
5 +}
6 +
7 +export function shortHash(input: string, len = 16): string {
8 + return sha256(input).slice(0, len);
9 +}
10 +
11 +/**
12 + * Structural fingerprint of a payload: the sorted set of key paths with primitive types,
13 + * ignoring values and array indexes. Used to detect schema drift on public endpoints.
14 + */
15 +export function schemaFingerprint(payload: unknown, maxDepth = 6): string {
16 + const paths = new Set<string>();
17 + const walk = (v: unknown, path: string, depth: number) => {
18 + if (depth > maxDepth) return;
19 + if (Array.isArray(v)) {
20 + if (v.length === 0) paths.add(`${path}[]`);
21 + // Sample first 3 elements — enough to capture the shape.
22 + for (const item of v.slice(0, 3)) walk(item, `${path}[]`, depth + 1);
23 + return;
24 + }
25 + if (v && typeof v === "object") {
26 + for (const [k, val] of Object.entries(v as Record<string, unknown>)) walk(val, path ? `${path}.${k}` : k, depth + 1);
27 + return;
28 + }
29 + paths.add(`${path}:${v === null ? "null" : typeof v}`);
30 + };
31 + walk(payload, "", 0);
32 + return shortHash([...paths].sort().join("|"), 20);
33 +}
34 +
35 +/** Deterministic fingerprint for deduplicating observations replayed after a reconnect. */
36 +export function observationFingerprint(parts: {
37 + sourceId: string;
38 + instrumentId: string;
39 + field: string;
40 + value: number;
41 + sourceTimestamp: number | null;
42 + sequence?: number | string | null;
43 +}): string {
44 + return shortHash(
45 + `${parts.sourceId}|${parts.instrumentId}|${parts.field}|${parts.value}|${parts.sourceTimestamp ?? ""}|${parts.sequence ?? ""}`,
46 + 24,
47 + );
48 +}
added packages/connector-sdk/src/http.ts +198 −0
@@ -0,0 +1,198 @@
1 +import { isIP } from "node:net";
2 +import { lookup } from "node:dns/promises";
3 +import { RateLimiter, backoffMs } from "./ratelimit.js";
4 +import { redactUrl } from "./redact.js";
5 +
6 +export interface HttpRequestOptions {
7 + method?: "GET" | "POST" | "HEAD";
8 + headers?: Record<string, string>;
9 + body?: string;
10 + timeoutMs?: number;
11 + /** Use conditional requests (ETag / If-Modified-Since) keyed by URL. Default true for GET. */
12 + conditional?: boolean;
13 + retries?: number;
14 + /** Accept 304 as success and return cached body. */
15 + acceptNotModified?: boolean;
16 +}
17 +
18 +export interface HttpResponse {
19 + status: number;
20 + ok: boolean;
21 + notModified: boolean;
22 + headers: Record<string, string>;
23 + text: string;
24 + url: string;
25 + durationMs: number;
26 + fromCache: boolean;
27 +}
28 +
29 +export class HttpError extends Error {
30 + constructor(
31 + message: string,
32 + public status: number,
33 + public url: string,
34 + public retryAfterMs: number | null = null,
35 + ) {
36 + super(message);
37 + this.name = "HttpError";
38 + }
39 +}
40 +
41 +interface CacheEntry {
42 + etag: string | null;
43 + lastModified: string | null;
44 + body: string;
45 + headers: Record<string, string>;
46 +}
47 +
48 +export interface HttpClientOptions {
49 + userAgent: string;
50 + limiter?: RateLimiter;
51 + defaultTimeoutMs?: number;
52 + /** Called for every request (metrics). */
53 + onRequest?: (info: { host: string; status: number; durationMs: number; ok: boolean }) => void;
54 + fetchImpl?: typeof fetch;
55 +}
56 +
57 +/**
58 + * Polite HTTP client: central per-host rate limiter, ETag/If-Modified-Since cache,
59 + * bounded retries with jittered backoff, secret-free URLs in errors, identified User-Agent.
60 + */
61 +export class HttpClient {
62 + private cache = new Map<string, CacheEntry>();
63 + readonly limiter: RateLimiter;
64 + private fetchImpl: typeof fetch;
65 +
66 + constructor(private opts: HttpClientOptions) {
67 + this.limiter = opts.limiter ?? new RateLimiter();
68 + this.fetchImpl = opts.fetchImpl ?? fetch;
69 + }
70 +
71 + async request(url: string, options: HttpRequestOptions = {}): Promise<HttpResponse> {
72 + const method = options.method ?? "GET";
73 + const u = new URL(url);
74 + const retries = options.retries ?? 2;
75 + const conditional = options.conditional ?? method === "GET";
76 + let attempt = 0;
77 + for (;;) {
78 + await this.limiter.acquire(u.host);
79 + const started = Date.now();
80 + const headers: Record<string, string> = {
81 + "user-agent": this.opts.userAgent,
82 + accept: "application/json, text/xml, application/xml, text/html, text/csv, text/plain;q=0.9, */*;q=0.8",
83 + "accept-encoding": "gzip, br",
84 + ...(options.headers ?? {}),
85 + };
86 + const cached = conditional ? this.cache.get(url) : undefined;
87 + if (cached?.etag) headers["if-none-match"] = cached.etag;
88 + if (cached?.lastModified) headers["if-modified-since"] = cached.lastModified;
89 +
90 + const ctrl = new AbortController();
91 + const timer = setTimeout(() => ctrl.abort(), options.timeoutMs ?? this.opts.defaultTimeoutMs ?? 20_000);
92 + try {
93 + const res = await this.fetchImpl(url, { method, headers, body: options.body, signal: ctrl.signal, redirect: "follow" });
94 + const durationMs = Date.now() - started;
95 + const resHeaders: Record<string, string> = {};
96 + res.headers.forEach((v, k) => (resHeaders[k] = v));
97 + this.opts.onRequest?.({ host: u.host, status: res.status, durationMs, ok: res.ok || res.status === 304 });
98 +
99 + if (res.status === 304 && cached) {
100 + return { status: 304, ok: true, notModified: true, headers: resHeaders, text: cached.body, url, durationMs, fromCache: true };
101 + }
102 + if (res.status === 429 || res.status >= 500) {
103 + const ra = res.headers.get("retry-after");
104 + const retryAfterMs = ra ? (Number.isFinite(Number(ra)) ? Number(ra) * 1000 : Math.max(0, Date.parse(ra) - Date.now())) : null;
105 + if (attempt < retries) {
106 + attempt++;
107 + await sleep(retryAfterMs ?? backoffMs(attempt, 750, 30_000));
108 + continue;
109 + }
110 + throw new HttpError(`HTTP ${res.status} from ${redactUrl(url)}`, res.status, redactUrl(url), retryAfterMs);
111 + }
112 + const text = await res.text();
113 + if (!res.ok) throw new HttpError(`HTTP ${res.status} from ${redactUrl(url)}`, res.status, redactUrl(url));
114 + if (conditional) {
115 + const etag = res.headers.get("etag");
116 + const lastModified = res.headers.get("last-modified");
117 + if (etag || lastModified) this.cache.set(url, { etag, lastModified, body: text, headers: resHeaders });
118 + }
119 + return { status: res.status, ok: true, notModified: false, headers: resHeaders, text, url, durationMs, fromCache: false };
120 + } catch (err) {
121 + if (err instanceof HttpError) throw err;
122 + if (attempt < retries) {
123 + attempt++;
124 + await sleep(backoffMs(attempt, 500, 15_000));
125 + continue;
126 + }
127 + const msg = err instanceof Error ? err.message : String(err);
128 + throw new HttpError(`request failed: ${msg} (${redactUrl(url)})`, 0, redactUrl(url));
129 + } finally {
130 + clearTimeout(timer);
131 + }
132 + }
133 + }
134 +
135 + async getText(url: string, options?: HttpRequestOptions): Promise<HttpResponse> {
136 + return this.request(url, { ...options, method: "GET" });
137 + }
138 +
139 + async getJson<T = unknown>(url: string, options?: HttpRequestOptions): Promise<{ data: T; response: HttpResponse }> {
140 + const response = await this.request(url, { ...options, method: "GET", headers: { accept: "application/json", ...(options?.headers ?? {}) } });
141 + try {
142 + return { data: JSON.parse(response.text) as T, response };
143 + } catch {
144 + throw new HttpError(`invalid JSON from ${redactUrl(url)}`, response.status, redactUrl(url));
145 + }
146 + }
147 +
148 + cacheSize(): number {
149 + return this.cache.size;
150 + }
151 +}
152 +
153 +export function sleep(ms: number): Promise<void> {
154 + return new Promise((r) => setTimeout(r, ms));
155 +}
156 +
157 +const PRIVATE_V4 = [
158 + /^10\./,
159 + /^127\./,
160 + /^0\./,
161 + /^169\.254\./,
162 + /^172\.(1[6-9]|2\d|3[01])\./,
163 + /^192\.168\./,
164 + /^100\.(6[4-9]|[7-9]\d|1[01]\d|12[0-7])\./,
165 + /^22[4-9]\./,
166 + /^2[3-5]\d\./,
167 +];
168 +
169 +export function isPrivateAddress(ip: string): boolean {
170 + const v = isIP(ip);
171 + if (v === 4) return PRIVATE_V4.some((re) => re.test(ip));
172 + if (v === 6) {
173 + const low = ip.toLowerCase();
174 + return low === "::1" || low === "::" || low.startsWith("fc") || low.startsWith("fd") || low.startsWith("fe80") || low.startsWith("::ffff:");
175 + }
176 + return true;
177 +}
178 +
179 +/**
180 + * SSRF guard for user-supplied URLs (discovery): public http(s) only, no private/link-local/metadata
181 + * targets, resolved addresses checked too. Throws on violation.
182 + */
183 +export async function assertPublicUrl(input: string): Promise<URL> {
184 + const u = new URL(input);
185 + if (u.protocol !== "http:" && u.protocol !== "https:") throw new Error("only http(s) URLs are allowed");
186 + if (u.username || u.password) throw new Error("credentials in URL are not allowed");
187 + const host = u.hostname.toLowerCase();
188 + if (host === "localhost" || host.endsWith(".local") || host.endsWith(".internal") || host.endsWith(".maclustr.io") || host === "metadata.google.internal")
189 + throw new Error("internal hostnames are not allowed");
190 + if (isIP(host)) {
191 + if (isPrivateAddress(host)) throw new Error("private addresses are not allowed");
192 + return u;
193 + }
194 + const addrs = await lookup(host, { all: true });
195 + if (!addrs.length) throw new Error("hostname does not resolve");
196 + for (const a of addrs) if (isPrivateAddress(a.address)) throw new Error("hostname resolves to a private address");
197 + return u;
198 +}
added packages/connector-sdk/src/index.ts +9 −0
@@ -0,0 +1,9 @@
1 +export * from "./types.js";
2 +export * from "./define.js";
3 +export * from "./http.js";
4 +export * from "./ratelimit.js";
5 +export * from "./ws.js";
6 +export * from "./fingerprint.js";
7 +export * from "./parse.js";
8 +export * from "./redact.js";
9 +export * from "./testing.js";
added packages/connector-sdk/src/parse.ts +181 −0
@@ -0,0 +1,181 @@
1 +import { XMLParser } from "fast-xml-parser";
2 +import { sha256 } from "./fingerprint.js";
3 +
4 +const xmlParser = new XMLParser({
5 + ignoreAttributes: false,
6 + attributeNamePrefix: "@_",
7 + textNodeName: "#text",
8 + parseTagValue: false,
9 + trimValues: true,
10 + cdataPropName: "#cdata",
11 + removeNSPrefix: false,
12 +});
13 +
14 +export function parseXml<T = Record<string, unknown>>(text: string): T {
15 + return xmlParser.parse(text) as T;
16 +}
17 +
18 +export interface FeedItem {
19 + id: string | null;
20 + title: string;
21 + link: string | null;
22 + published: string | null;
23 + updated: string | null;
24 + summary: string | null;
25 + categories: Array<{ term: string; label?: string; scheme?: string }>;
26 + /** Any namespaced or custom element (e.g. ndaq:HaltDate) kept verbatim. */
27 + extra: Record<string, unknown>;
28 +}
29 +
30 +const asArray = <T>(v: T | T[] | undefined | null): T[] => (v == null ? [] : Array.isArray(v) ? v : [v]);
31 +const text = (v: unknown): string | null => {
32 + if (v == null) return null;
33 + if (typeof v === "string") return v;
34 + if (typeof v === "number") return String(v);
35 + if (typeof v === "object") {
36 + const o = v as Record<string, unknown>;
37 + if (typeof o["#cdata"] === "string") return o["#cdata"] as string;
38 + if (typeof o["#text"] === "string") return o["#text"] as string;
39 + if (typeof o["@_href"] === "string") return o["@_href"] as string;
40 + }
41 + return null;
42 +};
43 +
44 +/** Parse RSS 2.0 or Atom into a common item list. */
45 +export function parseFeed(xml: string): { title: string | null; items: FeedItem[] } {
46 + const doc = parseXml<Record<string, any>>(xml);
47 + if (doc.feed) {
48 + const feed = doc.feed;
49 + const items = asArray(feed.entry).map((e: Record<string, unknown>) => {
50 + const links = asArray(e.link as any);
51 + const alt = links.find((l: any) => l?.["@_rel"] === "alternate") ?? links[0];
52 + return {
53 + id: text(e.id),
54 + title: text(e.title) ?? "",
55 + link: alt ? text(alt) : null,
56 + published: text(e.published),
57 + updated: text(e.updated),
58 + summary: text(e.summary) ?? text(e.content),
59 + categories: asArray(e.category as any).map((c: any) => ({ term: String(c?.["@_term"] ?? ""), label: c?.["@_label"], scheme: c?.["@_scheme"] })),
60 + extra: e as Record<string, unknown>,
61 + } satisfies FeedItem;
62 + });
63 + return { title: text(feed.title), items };
64 + }
65 + const channel = doc.rss?.channel ?? doc.channel;
66 + if (channel) {
67 + const items = asArray(channel.item).map((e: Record<string, unknown>) => ({
68 + id: text(e.guid) ?? text(e.link),
69 + title: text(e.title) ?? "",
70 + link: text(e.link),
71 + published: text(e.pubDate),
72 + updated: null,
73 + summary: text(e.description),
74 + categories: asArray(e.category as any).map((c: any) => ({ term: text(c) ?? "" })),
75 + extra: e as Record<string, unknown>,
76 + }));
77 + return { title: text(channel.title), items };
78 + }
79 + return { title: null, items: [] };
80 +}
81 +
82 +/** Delimited text (pipe/comma/tab) → array of records keyed by header. Handles simple quoted fields. */
83 +export function parseDelimited(textInput: string, delimiter = ","): Record<string, string>[] {
84 + const lines = textInput.replace(/\r\n?/g, "\n").split("\n").filter((l) => l.length > 0);
85 + if (!lines.length) return [];
86 + const split = (line: string): string[] => {
87 + const out: string[] = [];
88 + let cur = "";
89 + let q = false;
90 + for (let i = 0; i < line.length; i++) {
91 + const ch = line[i]!;
92 + if (ch === '"') {
93 + if (q && line[i + 1] === '"') {
94 + cur += '"';
95 + i++;
96 + } else q = !q;
97 + } else if (ch === delimiter && !q) {
98 + out.push(cur);
99 + cur = "";
100 + } else cur += ch;
101 + }
102 + out.push(cur);
103 + return out;
104 + };
105 + const header = split(lines[0]!).map((h) => h.trim());
106 + const rows: Record<string, string>[] = [];
107 + for (const line of lines.slice(1)) {
108 + const cells = split(line);
109 + if (cells.length < 2) continue;
110 + const rec: Record<string, string> = {};
111 + header.forEach((h, i) => (rec[h] = (cells[i] ?? "").trim()));
112 + rows.push(rec);
113 + }
114 + return rows;
115 +}
116 +
117 +/** Locale-tolerant number parser: "1,234.5", "1 234,5", "(12.3)", "12.3%", "N/A" → number|null. */
118 +export function parseNumber(v: unknown): number | null {
119 + if (v == null) return null;
120 + if (typeof v === "number") return Number.isFinite(v) ? v : null;
121 + if (typeof v !== "string") return null;
122 + let s = v.trim();
123 + if (!s || /^(n\/?a|null|undefined|-||–)$/i.test(s)) return null;
124 + let neg = false;
125 + if (/^\(.*\)$/.test(s)) {
126 + neg = true;
127 + s = s.slice(1, -1);
128 + }
129 + s = s.replace(/[%$€£¥\s]/g, "").replace(/[+]/g, "");
130 + if (/^-?\d{1,3}(\.\d{3})+(,\d+)?$/.test(s)) s = s.replace(/\./g, "").replace(",", ".");
131 + else if (/^-?\d+,\d{1,2}$/.test(s) || /^-?\d+,\d{4,}$/.test(s)) s = s.replace(",", "."); // "12,5" decimal comma; "1,000" stays thousands
132 + else s = s.replace(/,/g, "");
133 + const n = Number(s);
134 + if (!Number.isFinite(n)) return null;
135 + return neg ? -n : n;
136 +}
137 +
138 +export function stripHtml(html: string): string {
139 + return html
140 + .replace(/<script[\s\S]*?<\/script>/gi, " ")
141 + .replace(/<style[\s\S]*?<\/style>/gi, " ")
142 + .replace(/<[^>]+>/g, " ")
143 + .replace(/&nbsp;/g, " ")
144 + .replace(/&amp;/g, "&")
145 + .replace(/&lt;/g, "<")
146 + .replace(/&gt;/g, ">")
147 + .replace(/&#39;|&apos;/g, "'")
148 + .replace(/&quot;/g, '"')
149 + .replace(/\s+/g, " ")
150 + .trim();
151 +}
152 +
153 +/** Extract simple HTML tables as arrays of cell texts (no nested tables). */
154 +export function extractTables(html: string): string[][][] {
155 + const tables: string[][][] = [];
156 + const tableRe = /<table[\s\S]*?<\/table>/gi;
157 + for (const t of html.match(tableRe) ?? []) {
158 + const rows: string[][] = [];
159 + for (const tr of t.match(/<tr[\s\S]*?<\/tr>/gi) ?? []) {
160 + const cells = (tr.match(/<t[hd][^>]*>[\s\S]*?<\/t[hd]>/gi) ?? []).map((c) => stripHtml(c));
161 + if (cells.length) rows.push(cells);
162 + }
163 + if (rows.length) tables.push(rows);
164 + }
165 + return tables;
166 +}
167 +
168 +/**
169 + * Change-detection fingerprints for HTML documents: whole normalized text + per-section hashes
170 + * (split on headings) so that a page can be compared section by section.
171 + */
172 +export function htmlFingerprints(html: string): { document: string; sections: Record<string, string> } {
173 + const normalized = stripHtml(html).toLowerCase();
174 + const sections: Record<string, string> = {};
175 + const parts = html.split(/<h[1-4][^>]*>/i);
176 + parts.forEach((p, i) => {
177 + const title = stripHtml(p.split(/<\/h[1-4]>/i)[0] ?? "").slice(0, 60) || `section-${i}`;
178 + sections[title] = sha256(stripHtml(p).toLowerCase()).slice(0, 16);
179 + });
180 + return { document: sha256(normalized).slice(0, 24), sections };
181 +}
added packages/connector-sdk/src/ratelimit.ts +87 −0
@@ -0,0 +1,87 @@
1 +/** Token bucket with async acquisition. Never busy-waits. */
2 +export class TokenBucket {
3 + private tokens: number;
4 + private last: number;
5 + private queue: Array<() => void> = [];
6 + private timer: NodeJS.Timeout | null = null;
7 +
8 + constructor(
9 + public ratePerSec: number,
10 + public burst: number = Math.max(1, Math.ceil(ratePerSec)),
11 + ) {
12 + this.tokens = burst;
13 + this.last = Date.now();
14 + }
15 +
16 + private refill() {
17 + const now = Date.now();
18 + const delta = (now - this.last) / 1000;
19 + this.tokens = Math.min(this.burst, this.tokens + delta * this.ratePerSec);
20 + this.last = now;
21 + }
22 +
23 + tryTake(): boolean {
24 + this.refill();
25 + if (this.tokens >= 1) {
26 + this.tokens -= 1;
27 + return true;
28 + }
29 + return false;
30 + }
31 +
32 + acquire(): Promise<void> {
33 + if (this.tryTake()) return Promise.resolve();
34 + return new Promise((resolve) => {
35 + this.queue.push(resolve);
36 + this.schedule();
37 + });
38 + }
39 +
40 + private schedule() {
41 + if (this.timer) return;
42 + const waitMs = Math.max(5, ((1 - this.tokens) / this.ratePerSec) * 1000);
43 + this.timer = setTimeout(() => {
44 + this.timer = null;
45 + while (this.queue.length && this.tryTake()) this.queue.shift()!();
46 + if (this.queue.length) this.schedule();
47 + }, waitMs);
48 + }
49 +
50 + pending(): number {
51 + return this.queue.length;
52 + }
53 +}
54 +
55 +/** Central limiter keyed by host. Connectors never sleep on their own. */
56 +export class RateLimiter {
57 + private buckets = new Map<string, TokenBucket>();
58 + constructor(private defaults: { ratePerSec: number; burst?: number } = { ratePerSec: 2 }) {}
59 +
60 + configure(host: string, ratePerSec: number, burst?: number) {
61 + this.buckets.set(host.toLowerCase(), new TokenBucket(ratePerSec, burst));
62 + }
63 +
64 + bucket(host: string): TokenBucket {
65 + const key = host.toLowerCase();
66 + let b = this.buckets.get(key);
67 + if (!b) {
68 + b = new TokenBucket(this.defaults.ratePerSec, this.defaults.burst);
69 + this.buckets.set(key, b);
70 + }
71 + return b;
72 + }
73 +
74 + acquire(host: string): Promise<void> {
75 + return this.bucket(host).acquire();
76 + }
77 +
78 + snapshot(): Array<{ host: string; ratePerSec: number; pending: number }> {
79 + return [...this.buckets.entries()].map(([host, b]) => ({ host, ratePerSec: b.ratePerSec, pending: b.pending() }));
80 + }
81 +}
82 +
83 +/** Exponential backoff with full jitter. */
84 +export function backoffMs(attempt: number, baseMs = 1000, maxMs = 5 * 60_000): number {
85 + const exp = Math.min(maxMs, baseMs * 2 ** Math.max(0, attempt));
86 + return Math.floor(Math.random() * exp);
87 +}
added packages/connector-sdk/src/redact.ts +22 −0
@@ -0,0 +1,22 @@
1 +const SECRET_KEYS = /^(authorization|cookie|set-cookie|x-api-key|api[-_]?key|token|access[-_]?token|refresh[-_]?token|session(id)?|csrf|x-csrf-token|password|secret|private[-_]?key)$/i;
2 +const SECRET_QUERY = /([?&](api_?key|token|key|secret|session|sig|signature|auth)=)[^&#]+/gi;
3 +
4 +/** Removes secrets from a URL query string. */
5 +export function redactUrl(url: string): string {
6 + return url.replace(SECRET_QUERY, "$1[redacted]");
7 +}
8 +
9 +/** Deep-copies an object removing secret-looking keys (headers, cookies, tokens). */
10 +export function redactObject<T>(value: T, depth = 0): T {
11 + if (depth > 12) return value;
12 + if (Array.isArray(value)) return value.map((v) => redactObject(v, depth + 1)) as T;
13 + if (value && typeof value === "object") {
14 + const out: Record<string, unknown> = {};
15 + for (const [k, v] of Object.entries(value as Record<string, unknown>)) {
16 + out[k] = SECRET_KEYS.test(k) ? "[redacted]" : redactObject(v, depth + 1);
17 + }
18 + return out as T;
19 + }
20 + if (typeof value === "string" && value.length > 20 && /^(Bearer|Basic)\s/i.test(value)) return "[redacted]" as T;
21 + return value;
22 +}
added packages/connector-sdk/src/sdk.test.ts +97 −0
@@ -0,0 +1,97 @@
1 +import { describe, expect, it } from "vitest";
2 +import { HttpClient, RateLimiter, TokenBucket, assertPublicUrl, extractTables, htmlFingerprints, isPrivateAddress, parseDelimited, parseFeed, parseNumber, redactObject, redactUrl, schemaFingerprint } from "./index.js";
3 +import { fakeFetch } from "./testing.js";
4 +
5 +describe("rate limiter", () => {
6 + it("token bucket paces requests", async () => {
7 + const b = new TokenBucket(50, 2);
8 + const t0 = Date.now();
9 + for (let i = 0; i < 6; i++) await b.acquire();
10 + expect(Date.now() - t0).toBeGreaterThanOrEqual(60); // 4 extra tokens at 50/s ≈ 80 ms
11 + });
12 + it("limiter keys by host", () => {
13 + const l = new RateLimiter({ ratePerSec: 1 });
14 + l.configure("a.example", 10);
15 + expect(l.bucket("A.EXAMPLE").ratePerSec).toBe(10);
16 + expect(l.bucket("b.example").ratePerSec).toBe(1);
17 + });
18 +});
19 +
20 +describe("http client", () => {
21 + it("uses ETag for conditional requests and returns cached body on 304", async () => {
22 + let calls = 0;
23 + const fetchImpl = (async (_url: string | URL | Request, init?: RequestInit) => {
24 + calls++;
25 + const h = (init?.headers ?? {}) as Record<string, string>;
26 + if (h["if-none-match"] === '"v1"') return new Response(null, { status: 304 });
27 + return new Response('{"a":1}', { status: 200, headers: { etag: '"v1"' } });
28 + }) as typeof fetch;
29 + const c = new HttpClient({ userAgent: "t", limiter: new RateLimiter({ ratePerSec: 1000 }), fetchImpl });
30 + const r1 = await c.getJson<{ a: number }>("https://x.example/data");
31 + const r2 = await c.getJson<{ a: number }>("https://x.example/data");
32 + expect(r1.data.a).toBe(1);
33 + expect(r2.response.notModified).toBe(true);
34 + expect(r2.data.a).toBe(1);
35 + expect(calls).toBe(2);
36 + });
37 + it("retries 5xx then throws HttpError with redacted URL", async () => {
38 + const c = new HttpClient({ userAgent: "t", limiter: new RateLimiter({ ratePerSec: 1000 }), fetchImpl: fakeFetch({ boom: { status: 503, body: "x" } }) });
39 + await expect(c.getText("https://x.example/boom?api_key=SECRET", { retries: 1 })).rejects.toThrow(/HTTP 503.*api_key=\[redacted\]/);
40 + });
41 +});
42 +
43 +describe("ssrf guard", () => {
44 + it("blocks private and internal targets", async () => {
45 + await expect(assertPublicUrl("http://127.0.0.1/")).rejects.toThrow();
46 + await expect(assertPublicUrl("http://10.0.0.5/x")).rejects.toThrow();
47 + await expect(assertPublicUrl("http://localhost:8080/")).rejects.toThrow();
48 + await expect(assertPublicUrl("ftp://example.com/")).rejects.toThrow();
49 + await expect(assertPublicUrl("http://169.254.169.254/latest/meta-data")).rejects.toThrow();
50 + await expect(assertPublicUrl("https://m3u96a.maclustr.io/")).rejects.toThrow();
51 + expect(isPrivateAddress("192.168.2.1")).toBe(true);
52 + expect(isPrivateAddress("8.8.8.8")).toBe(false);
53 + expect(isPrivateAddress("fd00::1")).toBe(true);
54 + });
55 +});
56 +
57 +describe("parsers", () => {
58 + it("schema fingerprint ignores values but not shape", () => {
59 + const a = schemaFingerprint({ sym: "AAPL", lp: "1.0", v: 1, ts: 2 });
60 + const b = schemaFingerprint({ sym: "MSFT", lp: "2.0", v: 9, ts: 3 });
61 + const c = schemaFingerprint({ ticker: "AAPL", lastPrice: 1, eventTime: 2 });
62 + expect(a).toBe(b);
63 + expect(a).not.toBe(c);
64 + });
65 + it("parses numbers in several locales", () => {
66 + expect(parseNumber("1,234.5")).toBe(1234.5);
67 + expect(parseNumber("1.234,5")).toBe(1234.5);
68 + expect(parseNumber("(12.3)")).toBe(-12.3);
69 + expect(parseNumber("12.3%")).toBe(12.3);
70 + expect(parseNumber("N/A")).toBeNull();
71 + expect(parseNumber("$1,000")).toBe(1000);
72 + });
73 + it("parses pipe-delimited and quoted csv", () => {
74 + const rows = parseDelimited('a|b\n1|"x|y"\n2|z', "|");
75 + expect(rows).toEqual([{ a: "1", b: "x|y" }, { a: "2", b: "z" }]);
76 + });
77 + it("parses RSS and Atom into items", () => {
78 + const rss = parseFeed('<rss><channel><title>T</title><item><title>A</title><link>http://a</link><guid>g1</guid><pubDate>Fri, 11 Sep 2026 04:00:00 GMT</pubDate></item></channel></rss>');
79 + expect(rss.items[0]).toMatchObject({ title: "A", link: "http://a", id: "g1" });
80 + const atom = parseFeed('<feed xmlns="http://www.w3.org/2005/Atom"><title>F</title><entry><title>E</title><id>i1</id><link rel="alternate" href="http://e"/><updated>2026-09-11T00:00:00Z</updated><category term="8-K" label="form type"/></entry></feed>');
81 + expect(atom.items[0]).toMatchObject({ title: "E", link: "http://e", id: "i1" });
82 + expect(atom.items[0]!.categories[0]!.term).toBe("8-K");
83 + });
84 + it("extracts tables and fingerprints html sections", () => {
85 + const html = "<h1>Title</h1><table><tr><th>a</th><th>b</th></tr><tr><td>1</td><td>2</td></tr></table><h2>Other</h2><p>x</p>";
86 + expect(extractTables(html)).toEqual([[["a", "b"], ["1", "2"]]]);
87 + const fp1 = htmlFingerprints(html);
88 + const fp2 = htmlFingerprints(html.replace("<p>x</p>", "<p>y</p>"));
89 + expect(fp1.document).not.toBe(fp2.document);
90 + expect(fp1.sections["Title"]).toBe(fp2.sections["Title"]);
91 + expect(fp1.sections["Other"]).not.toBe(fp2.sections["Other"]);
92 + });
93 + it("redacts secrets", () => {
94 + expect(redactUrl("https://x/y?token=abc&z=1")).toBe("https://x/y?token=[redacted]&z=1");
95 + expect(redactObject({ headers: { Authorization: "Bearer xyz", accept: "*/*" }, cookie: "a=b", price: 1 })).toEqual({ headers: { Authorization: "[redacted]", accept: "*/*" }, cookie: "[redacted]", price: 1 });
96 + });
97 +});
added packages/connector-sdk/src/testing.ts +91 −0
@@ -0,0 +1,91 @@
1 +import { readFileSync } from "node:fs";
2 +import { join } from "node:path";
3 +import type { RawObservation } from "@market-atlas/market-model";
4 +import { NormalizedObservationSchema } from "@market-atlas/market-model";
5 +import { HttpClient } from "./http.js";
6 +import { RateLimiter } from "./ratelimit.js";
7 +import type { ConnectorContext, ConnectorDefinition, ConnectorLogger, NormalizedBatch } from "./types.js";
8 +import { ManagedWebSocket, type ManagedWebSocketOptions } from "./ws.js";
9 +
10 +/** Load a fixture file (JSON parsed when the extension is .json, raw text otherwise). */
11 +export function loadFixture(dir: string, name: string): unknown {
12 + const text = readFileSync(join(dir, name), "utf8");
13 + return name.endsWith(".json") ? JSON.parse(text) : text;
14 +}
15 +
16 +export function rawFromFixture(def: ConnectorDefinition, kind: string, payload: unknown, receivedAt = 1_789_200_000_000): RawObservation {
17 + return { connectorId: def.metadata.id, sourceId: def.metadata.sourceId, kind, payload, receivedAt };
18 +}
19 +
20 +/** Runs normalize() on a payload and validates every observation against the canonical schema. */
21 +export function normalizeFixture(def: ConnectorDefinition, kind: string, payload: unknown): NormalizedBatch {
22 + const batch = def.normalize(rawFromFixture(def, kind, payload));
23 + for (const o of batch.observations) {
24 + const res = NormalizedObservationSchema.safeParse(o);
25 + if (!res.success) throw new Error(`invalid observation from ${def.metadata.id}: ${res.error.message}`);
26 + }
27 + return batch;
28 +}
29 +
30 +const silentLogger: ConnectorLogger = { debug() {}, info() {}, warn() {}, error() {} };
31 +
32 +/** Offline connector context for tests: no network unless a fetchImpl is provided. */
33 +export function makeTestContext(
34 + def: ConnectorDefinition,
35 + opts: { fetchImpl?: typeof fetch; marketOpen?: boolean; secrets?: Record<string, string>; symbols?: string[] } = {},
36 +): ConnectorContext & { emitted: RawObservation[]; errors: unknown[]; sockets: ManagedWebSocket[] } {
37 + const emitted: RawObservation[] = [];
38 + const errors: unknown[] = [];
39 + const sockets: ManagedWebSocket[] = [];
40 + const store = new Map<string, unknown>();
41 + const http = new HttpClient({
42 + userAgent: "MarketAtlas-test/0.1",
43 + limiter: new RateLimiter({ ratePerSec: 1000 }),
44 + fetchImpl:
45 + opts.fetchImpl ??
46 + (async () => {
47 + throw new Error("network disabled in tests");
48 + }),
49 + });
50 + return {
51 + connectorId: def.metadata.id,
52 + logger: silentLogger,
53 + http,
54 + state: {
55 + async get(k) {
56 + return store.get(k) as never;
57 + },
58 + async set(k, v) {
59 + store.set(k, v);
60 + },
61 + },
62 + emit(r) {
63 + emitted.push(...(Array.isArray(r) ? r : [r]));
64 + },
65 + openWebSocket(url: string, o: ManagedWebSocketOptions) {
66 + const s = new ManagedWebSocket(url, o);
67 + sockets.push(s);
68 + return s;
69 + },
70 + isMarketOpen: () => opts.marketOpen ?? true,
71 + watchedSymbols: () => opts.symbols ?? def.defaultSymbols ?? [],
72 + reportError: (e) => errors.push(e),
73 + secret: (n) => opts.secrets?.[n],
74 + now: () => Date.now(),
75 + emitted,
76 + errors,
77 + sockets,
78 + };
79 +}
80 +
81 +/** Build a fake fetch that serves canned responses by URL substring. */
82 +export function fakeFetch(routes: Record<string, string | { status?: number; body: string; headers?: Record<string, string> }>): typeof fetch {
83 + return (async (input: string | URL | Request) => {
84 + const url = typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
85 + const key = Object.keys(routes).find((k) => url.includes(k));
86 + if (!key) return new Response("not found", { status: 404 });
87 + const r = routes[key]!;
88 + const spec = typeof r === "string" ? { body: r } : r;
89 + return new Response(spec.body, { status: spec.status ?? 200, headers: spec.headers ?? {} });
90 + }) as typeof fetch;
91 +}
added packages/connector-sdk/src/types.ts +133 −0
@@ -0,0 +1,133 @@
1 +import type {
2 + ConnectorMetadata,
3 + EventType,
4 + Filing,
5 + HolidaySpec,
6 + Instrument,
7 + InstrumentHint,
8 + NormalizedObservation,
9 + RawObservation,
10 + Severity,
11 +} from "@market-atlas/market-model";
12 +import type { HttpClient } from "./http.js";
13 +import type { ManagedWebSocket, ManagedWebSocketOptions } from "./ws.js";
14 +
15 +export interface ConnectorLogger {
16 + debug(msg: string, data?: Record<string, unknown>): void;
17 + info(msg: string, data?: Record<string, unknown>): void;
18 + warn(msg: string, data?: Record<string, unknown>): void;
19 + error(msg: string, data?: Record<string, unknown>): void;
20 +}
21 +
22 +/** An event proposed by a connector (halts, filings…). The event engine dedupes and canonicalizes it. */
23 +export interface ProposedEvent {
24 + type: EventType;
25 + /** Source-native symbols the event refers to (resolved to instrument ids by the pipeline). */
26 + symbols?: string[];
27 + instrumentIds?: string[];
28 + entityIds?: string[];
29 + timestamp: number;
30 + severity?: Severity;
31 + confidence?: number;
32 + title: string;
33 + summary?: string | null;
34 + data?: Record<string, unknown>;
35 + /** Stable key for deduplication across sources (e.g. accession number, halt id). */
36 + dedupeKey: string;
37 +}
38 +
39 +/** Instrument master record proposed by a bulk/directory connector. */
40 +export interface ProposedInstrument {
41 + symbol: string;
42 + hint: InstrumentHint;
43 + aliases?: string[];
44 + isActive?: boolean;
45 + /** When false, only enrich an existing instrument (company, CIK…) — never create one. Default true. */
46 + createIfMissing?: boolean;
47 +}
48 +
49 +/** Historical bar proposed by a connector with historical support (backfill). */
50 +export interface ProposedBar {
51 + symbol: string;
52 + instrumentHint?: InstrumentHint;
53 + resolution: "1m" | "5m" | "15m" | "1h" | "1d";
54 + ts: number;
55 + open: number;
56 + high: number;
57 + low: number;
58 + close: number;
59 + volume: number | null;
60 +}
61 +
62 +export interface NormalizedBatch {
63 + observations: NormalizedObservation[];
64 + events?: ProposedEvent[];
65 + instruments?: ProposedInstrument[];
66 + bars?: ProposedBar[];
67 + filings?: Array<Omit<Filing, "instrumentIds" | "sourceId"> & { symbols?: string[] }>;
68 + holidays?: Array<Omit<HolidaySpec, "sourceId">>;
69 + /** Free-form counters surfaced in health (e.g. rows parsed). */
70 + stats?: Record<string, number>;
71 +}
72 +
73 +export interface AdaptiveSchedule {
74 + /** Interval while the reference exchange is open (ms). */
75 + openMs: number;
76 + /** Interval while closed on a weekday (ms). */
77 + closedMs: number;
78 + /** Interval on weekends/holidays (ms). */
79 + weekendMs: number;
80 + /** Exchange id used to evaluate market state (default: none → openMs always). */
81 + exchangeId?: string;
82 +}
83 +
84 +export type ScheduleSpec = { intervalMs: number } | AdaptiveSchedule;
85 +
86 +export interface ConnectorState {
87 + get<T = unknown>(key: string): Promise<T | undefined>;
88 + set(key: string, value: unknown): Promise<void>;
89 +}
90 +
91 +export interface ConnectorContext {
92 + readonly connectorId: string;
93 + readonly logger: ConnectorLogger;
94 + readonly http: HttpClient;
95 + readonly state: ConnectorState;
96 + /** Push raw observations into the ingest bus (streaming connectors). */
97 + emit(raw: RawObservation | RawObservation[]): void;
98 + /** Open a managed WebSocket (reconnect, heartbeat, backoff handled by the runtime). */
99 + openWebSocket(url: string, opts: ManagedWebSocketOptions): ManagedWebSocket;
100 + /** Market state helper for adaptive behaviour. */
101 + isMarketOpen(exchangeId: string): boolean;
102 + /** Instruments currently watched for this connector (symbols in source-native form). */
103 + watchedSymbols(): string[];
104 + /** Report an operational error (health tracking + backoff decisions). */
105 + reportError(err: unknown, context?: Record<string, unknown>): void;
106 + /** Resolve an environment secret by name (never logged). */
107 + secret(name: string): string | undefined;
108 + now(): number;
109 +}
110 +
111 +export interface ConnectorDefinition {
112 + metadata: ConnectorMetadata;
113 + /** Instruments this connector seeds when first enabled (source-native symbols → hints). */
114 + seeds?: ProposedInstrument[];
115 + /** Symbols to watch by default (source-native). */
116 + defaultSymbols?: string[];
117 + /** Streaming lifecycle. */
118 + start?(ctx: ConnectorContext): Promise<void>;
119 + stop?(ctx: ConnectorContext): Promise<void>;
120 + /** Polling lifecycle. */
121 + poll?(ctx: ConnectorContext): Promise<RawObservation[]>;
122 + schedule?: ScheduleSpec;
123 + /** Pure, deterministic mapping from a raw payload to Market Atlas vocabulary. Must never call the network. */
124 + normalize(raw: RawObservation): NormalizedBatch;
125 + /** Optional active probe (used by the admin "run test" action). */
126 + healthCheck?(ctx: ConnectorContext): Promise<{ ok: boolean; detail?: string }>;
127 + /** Static per-host rate limits requested by the connector (requests per second). */
128 + rateLimits?: Record<string, number>;
129 + /** Fixtures directory (relative to the connector), documented for tests. */
130 + fixturesDir?: string;
131 +}
132 +
133 +export type { ConnectorMetadata, Instrument, InstrumentHint, NormalizedObservation, RawObservation };
added packages/connector-sdk/src/ws.ts +146 −0
@@ -0,0 +1,146 @@
1 +import WebSocket from "ws";
2 +import { backoffMs } from "./ratelimit.js";
3 +
4 +export interface ManagedWebSocketOptions {
5 + /** Called on every (re)connect; send subscriptions here. */
6 + onOpen?: (ws: ManagedWebSocket) => void;
7 + onMessage: (data: string, ws: ManagedWebSocket) => void;
8 + onClose?: (code: number, reason: string) => void;
9 + onError?: (err: Error) => void;
10 + /** Consider the connection stale after this many ms without any frame (default 60 s). */
11 + staleAfterMs?: number;
12 + /** Ping payload sent when idle (WebSocket ping frame is used when null). */
13 + heartbeat?: { intervalMs: number; message?: string } | null;
14 + headers?: Record<string, string>;
15 + maxBackoffMs?: number;
16 + label?: string;
17 +}
18 +
19 +type Listener = (event: "connected" | "reconnecting" | "closed" | "stale", detail?: Record<string, unknown>) => void;
20 +
21 +/**
22 + * Reconnecting WebSocket with heartbeat and stale detection.
23 + * One connection may carry many instruments — never open one socket per symbol.
24 + */
25 +export class ManagedWebSocket {
26 + private ws: WebSocket | null = null;
27 + private attempt = 0;
28 + private closedByUser = false;
29 + private lastFrameAt = 0;
30 + private staleTimer: NodeJS.Timeout | null = null;
31 + private heartbeatTimer: NodeJS.Timeout | null = null;
32 + private reconnectTimer: NodeJS.Timeout | null = null;
33 + private listeners: Listener[] = [];
34 + reconnects = 0;
35 + connectedAt: number | null = null;
36 +
37 + constructor(
38 + public readonly url: string,
39 + private opts: ManagedWebSocketOptions,
40 + ) {}
41 +
42 + on(listener: Listener) {
43 + this.listeners.push(listener);
44 + }
45 +
46 + private emit(...args: Parameters<Listener>) {
47 + for (const l of this.listeners) l(...args);
48 + }
49 +
50 + connect(): void {
51 + this.closedByUser = false;
52 + this.open();
53 + }
54 +
55 + private open() {
56 + if (this.closedByUser) return;
57 + const ws = new WebSocket(this.url, { headers: this.opts.headers, handshakeTimeout: 15_000 });
58 + this.ws = ws;
59 + ws.on("open", () => {
60 + this.attempt = 0;
61 + this.connectedAt = Date.now();
62 + this.lastFrameAt = Date.now();
63 + this.emit("connected");
64 + this.armStale();
65 + this.armHeartbeat();
66 + this.opts.onOpen?.(this);
67 + });
68 + ws.on("message", (data) => {
69 + this.lastFrameAt = Date.now();
70 + try {
71 + this.opts.onMessage(typeof data === "string" ? data : data.toString("utf8"), this);
72 + } catch (err) {
73 + this.opts.onError?.(err instanceof Error ? err : new Error(String(err)));
74 + }
75 + });
76 + ws.on("pong", () => (this.lastFrameAt = Date.now()));
77 + ws.on("error", (err) => this.opts.onError?.(err));
78 + ws.on("close", (code, reason) => {
79 + this.clearTimers();
80 + this.ws = null;
81 + this.opts.onClose?.(code, reason.toString());
82 + if (!this.closedByUser) this.scheduleReconnect();
83 + else this.emit("closed");
84 + });
85 + }
86 +
87 + private scheduleReconnect() {
88 + if (this.reconnectTimer) return;
89 + const wait = backoffMs(this.attempt, 1000, this.opts.maxBackoffMs ?? 120_000);
90 + this.attempt++;
91 + this.reconnects++;
92 + this.emit("reconnecting", { attempt: this.attempt, waitMs: wait });
93 + this.reconnectTimer = setTimeout(() => {
94 + this.reconnectTimer = null;
95 + this.open();
96 + }, wait);
97 + }
98 +
99 + private armStale() {
100 + const staleAfter = this.opts.staleAfterMs ?? 60_000;
101 + if (this.staleTimer) clearInterval(this.staleTimer);
102 + this.staleTimer = setInterval(() => {
103 + if (Date.now() - this.lastFrameAt > staleAfter) {
104 + this.emit("stale", { silentMs: Date.now() - this.lastFrameAt });
105 + this.ws?.terminate();
106 + }
107 + }, Math.max(5000, Math.floor(staleAfter / 3)));
108 + }
109 +
110 + private armHeartbeat() {
111 + const hb = this.opts.heartbeat;
112 + if (hb === null) return;
113 + const intervalMs = hb?.intervalMs ?? 25_000;
114 + this.heartbeatTimer = setInterval(() => {
115 + if (this.ws?.readyState !== WebSocket.OPEN) return;
116 + if (hb?.message) this.ws.send(hb.message);
117 + else this.ws.ping();
118 + }, intervalMs);
119 + }
120 +
121 + private clearTimers() {
122 + if (this.staleTimer) clearInterval(this.staleTimer);
123 + if (this.heartbeatTimer) clearInterval(this.heartbeatTimer);
124 + this.staleTimer = null;
125 + this.heartbeatTimer = null;
126 + }
127 +
128 + send(data: string | object): boolean {
129 + if (this.ws?.readyState !== WebSocket.OPEN) return false;
130 + this.ws.send(typeof data === "string" ? data : JSON.stringify(data));
131 + return true;
132 + }
133 +
134 + isOpen(): boolean {
135 + return this.ws?.readyState === WebSocket.OPEN;
136 + }
137 +
138 + close(): void {
139 + this.closedByUser = true;
140 + if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
141 + this.reconnectTimer = null;
142 + this.clearTimers();
143 + this.ws?.close();
144 + this.ws = null;
145 + }
146 +}
added packages/market-model/package.json +12 −0
@@ -0,0 +1,12 @@
1 +{
2 + "name": "@market-atlas/market-model",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "exports": {
7 + ".": "./src/index.ts"
8 + },
9 + "dependencies": {
10 + "zod": "^3.24.1"
11 + }
12 +}
added packages/market-model/src/enums.ts +170 −0
@@ -0,0 +1,170 @@
1 +// Canonical vocabularies of Market Atlas. Every boundary (bus, DB, API, UI) uses these strings verbatim.
2 +
3 +export const ASSET_CLASSES = [
4 + "EQUITY",
5 + "ETF",
6 + "ETN",
7 + "INDEX",
8 + "CRYPTO",
9 + "FOREX",
10 + "COMMODITY",
11 + "FUTURE",
12 + "OPTION",
13 + "BOND",
14 + "TREASURY",
15 + "INTEREST_RATE",
16 + "MUTUAL_FUND",
17 + "ADR",
18 + "GDR",
19 + "REIT",
20 + "WARRANT",
21 + "CFD",
22 + "OTHER",
23 +] as const;
24 +export type AssetClass = (typeof ASSET_CLASSES)[number];
25 +
26 +export const RIGHTS_STATUSES = [
27 + "PUBLIC_OPEN",
28 + "PUBLIC_ATTRIBUTED",
29 + "PUBLIC_RESTRICTED_REDISTRIBUTION",
30 + "OFFICIAL_OPEN_DATA",
31 + "LICENSED",
32 + "DELAYED",
33 + "INDICATIVE",
34 + "RESEARCH_ONLY",
35 + "INTERNAL_ONLY",
36 + "UNKNOWN",
37 + "DISABLED",
38 +] as const;
39 +export type RightsStatus = (typeof RIGHTS_STATUSES)[number];
40 +
41 +/** Rights statuses whose raw values may be redistributed publicly by Market Atlas. */
42 +export const PUBLICLY_REDISTRIBUTABLE: ReadonlySet<RightsStatus> = new Set<RightsStatus>([
43 + "PUBLIC_OPEN",
44 + "PUBLIC_ATTRIBUTED",
45 + "OFFICIAL_OPEN_DATA",
46 + "LICENSED",
47 + "DELAYED",
48 + "INDICATIVE",
49 +]);
50 +
51 +export const REALTIME_STATUSES = ["REALTIME", "DELAYED", "END_OF_DAY", "INDICATIVE", "STALE", "UNKNOWN"] as const;
52 +export type RealtimeStatus = (typeof REALTIME_STATUSES)[number];
53 +
54 +export const SOURCE_TYPES = [
55 + "WEBSOCKET",
56 + "SSE",
57 + "XHR",
58 + "FETCH",
59 + "GRAPHQL",
60 + "HTML",
61 + "RSS",
62 + "CSV",
63 + "XML",
64 + "PDF",
65 + "OFFICIAL_API",
66 + "BULK_FILE",
67 +] as const;
68 +export type SourceType = (typeof SOURCE_TYPES)[number];
69 +
70 +export const FIELDS = [
71 + "LAST_PRICE",
72 + "OPEN",
73 + "HIGH",
74 + "LOW",
75 + "PREVIOUS_CLOSE",
76 + "CLOSE",
77 + "BID",
78 + "ASK",
79 + "BID_SIZE",
80 + "ASK_SIZE",
81 + "VOLUME",
82 + "VWAP",
83 + "MARKET_CAP",
84 + "CHANGE",
85 + "CHANGE_PERCENT",
86 + "OPEN_INTEREST",
87 + "YIELD",
88 + "RATE",
89 + "NAV",
90 + "52_WEEK_HIGH",
91 + "52_WEEK_LOW",
92 + "IMPLIED_VOLATILITY",
93 +] as const;
94 +export type Field = (typeof FIELDS)[number];
95 +
96 +export const TIMESTAMP_TRUSTS = ["EXCHANGE", "SOURCE", "CONNECTOR", "UNKNOWN"] as const;
97 +export type TimestampTrust = (typeof TIMESTAMP_TRUSTS)[number];
98 +
99 +export const CONNECTOR_STATES = [
100 + "HEALTHY",
101 + "DEGRADED",
102 + "STALE",
103 + "RECONNECTING",
104 + "FAILED",
105 + "PAUSED",
106 + "DISABLED",
107 + "STARTING",
108 +] as const;
109 +export type ConnectorState = (typeof CONNECTOR_STATES)[number];
110 +
111 +export const MARKET_STATES = ["PRE", "OPEN", "HALTED", "CLOSED", "POST", "AUCTION", "UNKNOWN"] as const;
112 +export type MarketState = (typeof MARKET_STATES)[number];
113 +
114 +export const EVENT_TYPES = [
115 + "PRICE_CHANGE",
116 + "SESSION_HIGH",
117 + "SESSION_LOW",
118 + "PRICE_BREAKOUT",
119 + "PRICE_GAP",
120 + "VOLUME_SPIKE",
121 + "VOLATILITY_SPIKE",
122 + "SPREAD_WIDENING",
123 + "SPREAD_COMPRESSION",
124 + "LIQUIDITY_CHANGE",
125 + "MARKET_OPEN",
126 + "MARKET_CLOSE",
127 + "TRADING_HALT",
128 + "TRADING_RESUME",
129 + "CORPORATE_ACTION",
130 + "DIVIDEND_DECLARED",
131 + "DIVIDEND_CHANGED",
132 + "STOCK_SPLIT",
133 + "EARNINGS_RELEASE",
134 + "GUIDANCE_CHANGE",
135 + "FILING_PUBLISHED",
136 + "INSIDER_TRANSACTION",
137 + "IPO_STATUS_CHANGE",
138 + "INDEX_CONSTITUENT_CHANGE",
139 + "RATING_CHANGE",
140 + "RATE_DECISION",
141 + "REFERENCE_RATE_PUBLISHED",
142 + "DOCUMENT_CHANGED",
143 + "SOURCE_FAILURE",
144 + "SOURCE_RECOVERY",
145 + "SOURCE_DIVERGENCE",
146 + "SCHEMA_DRIFT",
147 + "INSTRUMENT_LISTED",
148 + "INSTRUMENT_DELISTED",
149 +] as const;
150 +export type EventType = (typeof EVENT_TYPES)[number];
151 +
152 +export const SEVERITIES = ["DEBUG", "INFO", "NOTICE", "WARNING", "CRITICAL"] as const;
153 +export type Severity = (typeof SEVERITIES)[number];
154 +
155 +export const BAR_RESOLUTIONS = ["1m", "5m", "15m", "1h", "1d"] as const;
156 +export type BarResolution = (typeof BAR_RESOLUTIONS)[number];
157 +
158 +export const RESOLUTION_MS: Record<BarResolution, number> = {
159 + "1m": 60_000,
160 + "5m": 300_000,
161 + "15m": 900_000,
162 + "1h": 3_600_000,
163 + "1d": 86_400_000,
164 +};
165 +
166 +export const CANONICAL_VERSIONS = {
167 + normalizer: "1.0",
168 + consensus: "1.0",
169 + events: "1.0",
170 +} as const;
added packages/market-model/src/ids.ts +72 −0
@@ -0,0 +1,72 @@
1 +import type { AssetClass, InstrumentHint } from "./index.js";
2 +
3 +const slug = (s: string) =>
4 + s
5 + .toLowerCase()
6 + .normalize("NFKD")
7 + .replace(/[̀-ͯ]/g, "")
8 + .replace(/[^a-z0-9]+/g, "_")
9 + .replace(/^_+|_+$/g, "");
10 +
11 +/**
12 + * Canonical symbol form used inside instrument ids and for alias matching.
13 + * "BRK.B", "BRK-B", "BRK/B", "BRK B" → "brk_b".
14 + */
15 +export function canonicalSymbol(symbol: string): string {
16 + return slug(symbol.trim());
17 +}
18 +
19 +/** Deterministic stable ids; never database sequences. */
20 +export function makeInstrumentId(assetClass: AssetClass, hint: { symbol: string } & Partial<InstrumentHint>): string {
21 + const sym = canonicalSymbol(hint.symbol);
22 + switch (assetClass) {
23 + case "CRYPTO":
24 + return `crypto_${slug(hint.base ?? sym)}_${slug(hint.quote ?? "usd")}`;
25 + case "FOREX":
26 + return `fx_${slug(hint.base ?? sym.slice(0, 3))}_${slug(hint.quote ?? sym.slice(3, 6))}`;
27 + case "INDEX":
28 + return `index_${slug(hint.country ?? "xx")}_${sym.replace(/^[_^.]+/, "")}`;
29 + case "TREASURY":
30 + case "INTEREST_RATE":
31 + case "BOND":
32 + return `rate_${slug(hint.country ?? "xx")}_${sym}`;
33 + case "COMMODITY":
34 + case "FUTURE":
35 + return `${assetClass === "FUTURE" ? "fut" : "cmd"}_${slug(hint.exchangeId ?? "xx")}_${sym}`;
36 + case "ETF":
37 + case "ETN":
38 + case "REIT":
39 + case "ADR":
40 + case "GDR":
41 + case "EQUITY":
42 + default: {
43 + const prefix =
44 + assetClass === "ETF" || assetClass === "ETN"
45 + ? "etf"
46 + : assetClass === "MUTUAL_FUND"
47 + ? "fund"
48 + : assetClass === "OPTION"
49 + ? "opt"
50 + : "eq";
51 + const venue = slug(hint.mic ?? hint.exchangeId ?? "xxxx");
52 + return `${prefix}_${slug(hint.country ?? "xx")}_${venue}_${sym}`;
53 + }
54 + }
55 +}
56 +
57 +export function makeCompanyId(name: string): string {
58 + return slug(name).replace(/_/g, "-").slice(0, 80);
59 +}
60 +
61 +export function makeEventId(ts: number, fingerprint: string): string {
62 + return `evt_${ts.toString(36)}_${fingerprint.slice(0, 12)}`;
63 +}
64 +
65 +/** Split "BTC-USD" / "BTC/USD" / "BTCUSD" (6 letters) into base and quote. */
66 +export function splitPair(symbol: string): { base: string; quote: string } | null {
67 + const m = symbol.toUpperCase().match(/^([A-Z0-9]{2,10})[-_/]([A-Z0-9]{2,6})$/);
68 + if (m) return { base: m[1]!, quote: m[2]!, };
69 + const compact = symbol.toUpperCase();
70 + if (/^[A-Z]{6}$/.test(compact)) return { base: compact.slice(0, 3), quote: compact.slice(3) };
71 + return null;
72 +}
added packages/market-model/src/index.ts +5 −0
@@ -0,0 +1,5 @@
1 +export * from "./enums.js";
2 +export * from "./types.js";
3 +export * from "./ids.js";
4 +export * from "./schemas.js";
5 +export * from "./time.js";
added packages/market-model/src/model.test.ts +44 −0
@@ -0,0 +1,44 @@
1 +import { describe, expect, it } from "vitest";
2 +import { canonicalSymbol, makeInstrumentId, parseTimestamp, splitPair, tzOffsetMs, zonedParts, zonedTimeToUtc } from "./index.js";
3 +
4 +describe("ids", () => {
5 + it("canonicalizes symbol variants to one form", () => {
6 + for (const v of ["BRK.B", "BRK-B", "BRK/B", "BRK B", "brk.b"]) expect(canonicalSymbol(v)).toBe("brk_b");
7 + });
8 + it("builds stable ids per asset class", () => {
9 + expect(makeInstrumentId("EQUITY", { symbol: "AAPL", mic: "XNAS", country: "US" })).toBe("eq_us_xnas_aapl");
10 + expect(makeInstrumentId("CRYPTO", { symbol: "BTC-USD", base: "BTC", quote: "USD" })).toBe("crypto_btc_usd");
11 + expect(makeInstrumentId("FOREX", { symbol: "EURUSD", base: "EUR", quote: "USD" })).toBe("fx_eur_usd");
12 + expect(makeInstrumentId("INDEX", { symbol: "_SPX", country: "US" })).toBe("index_us_spx");
13 + expect(makeInstrumentId("TREASURY", { symbol: "US10Y", country: "US" })).toBe("rate_us_us10y");
14 + expect(makeInstrumentId("ETF", { symbol: "SPY", mic: "ARCX", country: "US" })).toBe("etf_us_arcx_spy");
15 + });
16 + it("splits pairs", () => {
17 + expect(splitPair("BTC-USD")).toEqual({ base: "BTC", quote: "USD" });
18 + expect(splitPair("EURUSD")).toEqual({ base: "EUR", quote: "USD" });
19 + expect(splitPair("AAPL")).toBeNull();
20 + });
21 +});
22 +
23 +describe("time", () => {
24 + it("converts Eastern wall clock to UTC across DST", () => {
25 + expect(new Date(zonedTimeToUtc("2026-09-11T15:59:59", "America/New_York")!).toISOString()).toBe("2026-09-11T19:59:59.000Z"); // EDT
26 + expect(new Date(zonedTimeToUtc("2026-01-15T16:00:00", "America/New_York")!).toISOString()).toBe("2026-01-15T21:00:00.000Z"); // EST
27 + expect(new Date(zonedTimeToUtc("2026-09-11", "UTC")!).toISOString()).toBe("2026-09-11T00:00:00.000Z");
28 + });
29 + it("computes offsets and parts", () => {
30 + expect(tzOffsetMs(Date.UTC(2026, 6, 1), "Asia/Tokyo")).toBe(9 * 3_600_000);
31 + const p = zonedParts(Date.UTC(2026, 8, 11, 19, 59, 59), "America/New_York");
32 + expect(p.date).toBe("2026-09-11");
33 + expect(p.time).toBe("15:59");
34 + expect(p.weekday).toBe(5);
35 + });
36 + it("parses timestamps in s, ms, µs, ns and ISO", () => {
37 + expect(parseTimestamp(1789199657)).toBe(1789199657000);
38 + expect(parseTimestamp(1789199657016)).toBe(1789199657016);
39 + expect(parseTimestamp("1789199660123")).toBe(1789199660123);
40 + expect(parseTimestamp(1789199657016000)).toBe(1789199657016);
41 + expect(parseTimestamp("2026-09-12T07:54:03.482921Z")).toBe(Date.parse("2026-09-12T07:54:03.482Z"));
42 + expect(parseTimestamp("garbage")).toBeNull();
43 + });
44 +});
added packages/market-model/src/schemas.ts +91 −0
@@ -0,0 +1,91 @@
1 +import { z } from "zod";
2 +import {
3 + ASSET_CLASSES,
4 + EVENT_TYPES,
5 + FIELDS,
6 + REALTIME_STATUSES,
7 + RIGHTS_STATUSES,
8 + SEVERITIES,
9 + SOURCE_TYPES,
10 + TIMESTAMP_TRUSTS,
11 +} from "./enums.js";
12 +
13 +export const InstrumentHintSchema = z.object({
14 + assetClass: z.enum(ASSET_CLASSES),
15 + name: z.string().optional(),
16 + exchangeId: z.string().nullable().optional(),
17 + mic: z.string().nullable().optional(),
18 + currency: z.string().nullable().optional(),
19 + country: z.string().nullable().optional(),
20 + base: z.string().nullable().optional(),
21 + quote: z.string().nullable().optional(),
22 + securityType: z.string().nullable().optional(),
23 + companyName: z.string().nullable().optional(),
24 + cik: z.string().nullable().optional(),
25 + metadata: z.record(z.unknown()).optional(),
26 +});
27 +
28 +export const NormalizedObservationSchema = z.object({
29 + symbol: z.string().min(1).max(64),
30 + instrumentId: z.string().optional(),
31 + instrumentHint: InstrumentHintSchema.optional(),
32 + field: z.enum(FIELDS),
33 + value: z.number().finite(),
34 + currency: z.string().nullable().optional(),
35 + sourceTimestamp: z.number().int().nullable(),
36 + timestampTrust: z.enum(TIMESTAMP_TRUSTS),
37 + sequence: z.union([z.number(), z.string()]).nullable().optional(),
38 + rightsStatus: z.enum(RIGHTS_STATUSES),
39 + realtimeStatus: z.enum(REALTIME_STATUSES),
40 + confidence: z.number().min(0).max(1).optional(),
41 + meta: z.record(z.unknown()).optional(),
42 +});
43 +
44 +export const MarketEventSchema = z.object({
45 + version: z.literal(1),
46 + id: z.string(),
47 + type: z.enum(EVENT_TYPES),
48 + instrumentIds: z.array(z.string()),
49 + entityIds: z.array(z.string()),
50 + timestamp: z.number().int(),
51 + severity: z.enum(SEVERITIES),
52 + confidence: z.number().min(0).max(1),
53 + sourceCount: z.number().int().nonnegative(),
54 + sources: z.array(z.string()),
55 + title: z.string(),
56 + summary: z.string().nullable(),
57 + data: z.record(z.unknown()),
58 + fingerprint: z.string(),
59 + supportingObservations: z.array(z.string()),
60 +});
61 +
62 +export const ConnectorMetadataSchema = z.object({
63 + id: z.string().regex(/^[a-z0-9][a-z0-9-]{1,60}$/),
64 + name: z.string(),
65 + version: z.string(),
66 + sourceId: z.string(),
67 + organization: z.string().nullable(),
68 + sourceType: z.enum(SOURCE_TYPES),
69 + jurisdiction: z.string().nullable(),
70 + rightsStatus: z.enum(RIGHTS_STATUSES),
71 + realtimeStatus: z.enum(REALTIME_STATUSES),
72 + expectedLatencyMs: z.number().nullable(),
73 + supportsStreaming: z.boolean(),
74 + supportsHistorical: z.boolean(),
75 + assetClasses: z.array(z.enum(ASSET_CLASSES)),
76 + exchanges: z.array(z.string()),
77 + homepage: z.string().url().nullable(),
78 + description: z.string(),
79 + rightsNotes: z.string().nullable(),
80 + termsUrl: z.string().url().nullable(),
81 + sourceFamily: z.string().nullable(),
82 + enabled: z.boolean(),
83 +});
84 +
85 +/** WebSocket client → gateway messages. */
86 +export const StreamClientMessageSchema = z.discriminatedUnion("action", [
87 + z.object({ action: z.literal("subscribe"), channels: z.array(z.string().max(120)).max(500) }),
88 + z.object({ action: z.literal("unsubscribe"), channels: z.array(z.string().max(120)).max(500) }),
89 + z.object({ action: z.literal("ping"), id: z.union([z.string(), z.number()]).optional() }),
90 +]);
91 +export type StreamClientMessage = z.infer<typeof StreamClientMessageSchema>;
added packages/market-model/src/time.ts +113 −0
@@ -0,0 +1,113 @@
1 +// Time-zone utilities without external dependencies (Intl only).
2 +
3 +const dtfCache = new Map<string, Intl.DateTimeFormat>();
4 +
5 +function dtf(timeZone: string): Intl.DateTimeFormat {
6 + let f = dtfCache.get(timeZone);
7 + if (!f) {
8 + f = new Intl.DateTimeFormat("en-US", {
9 + timeZone,
10 + hourCycle: "h23",
11 + year: "numeric",
12 + month: "2-digit",
13 + day: "2-digit",
14 + hour: "2-digit",
15 + minute: "2-digit",
16 + second: "2-digit",
17 + weekday: "short",
18 + });
19 + dtfCache.set(timeZone, f);
20 + }
21 + return f;
22 +}
23 +
24 +export interface ZonedParts {
25 + year: number;
26 + month: number; // 1-12
27 + day: number;
28 + hour: number;
29 + minute: number;
30 + second: number;
31 + weekday: number; // 0 = Sunday
32 + date: string; // YYYY-MM-DD
33 + time: string; // HH:MM
34 +}
35 +
36 +const WEEKDAYS: Record<string, number> = { Sun: 0, Mon: 1, Tue: 2, Wed: 3, Thu: 4, Fri: 5, Sat: 6 };
37 +
38 +export function zonedParts(ms: number, timeZone: string): ZonedParts {
39 + const parts = dtf(timeZone).formatToParts(new Date(ms));
40 + const get = (t: string) => parts.find((p) => p.type === t)?.value ?? "";
41 + const year = Number(get("year"));
42 + const month = Number(get("month"));
43 + const day = Number(get("day"));
44 + const hour = Number(get("hour")) % 24;
45 + const minute = Number(get("minute"));
46 + const second = Number(get("second"));
47 + const pad = (n: number) => String(n).padStart(2, "0");
48 + return {
49 + year,
50 + month,
51 + day,
52 + hour,
53 + minute,
54 + second,
55 + weekday: WEEKDAYS[get("weekday")] ?? 0,
56 + date: `${year}-${pad(month)}-${pad(day)}`,
57 + time: `${pad(hour)}:${pad(minute)}`,
58 + };
59 +}
60 +
61 +/** Offset (ms) of `timeZone` relative to UTC at instant `ms`. */
62 +export function tzOffsetMs(ms: number, timeZone: string): number {
63 + const p = zonedParts(ms, timeZone);
64 + const asUtc = Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute, p.second);
65 + return asUtc - Math.floor(ms / 1000) * 1000;
66 +}
67 +
68 +/**
69 + * Interpret a wall-clock time in `timeZone` as a UTC instant.
70 + * Accepts "YYYY-MM-DD", "YYYY-MM-DDTHH:MM[:SS[.sss]]" or "YYYY-MM-DD HH:MM[:SS]".
71 + */
72 +export function zonedTimeToUtc(local: string, timeZone: string): number | null {
73 + const m = local
74 + .trim()
75 + .match(/^(\d{4})-(\d{2})-(\d{2})(?:[T ](\d{2}):(\d{2})(?::(\d{2})(?:\.(\d{1,3}))?)?)?$/);
76 + if (!m) return null;
77 + const [_, y, mo, d, h = "0", mi = "0", s = "0", frac = "0"] = m;
78 + const guess = Date.UTC(+y!, +mo! - 1, +d!, +h, +mi, +s, +frac.padEnd(3, "0"));
79 + // Two-pass correction handles DST edges well enough for market data.
80 + const off1 = tzOffsetMs(guess, timeZone);
81 + const utc1 = guess - off1;
82 + const off2 = tzOffsetMs(utc1, timeZone);
83 + return guess - off2;
84 +}
85 +
86 +/** Local wall-clock "HH:MM" of `timeZone` → ms since local midnight. */
87 +export function hmToMs(hm: string): number {
88 + const [h, m] = hm.split(":").map(Number);
89 + return ((h ?? 0) * 60 + (m ?? 0)) * 60_000;
90 +}
91 +
92 +/** Parse a variety of timestamp encodings; returns ms epoch or null. */
93 +export function parseTimestamp(input: unknown): number | null {
94 + if (input == null) return null;
95 + if (typeof input === "number") {
96 + if (!Number.isFinite(input)) return null;
97 + if (input > 1e17) return Math.floor(input / 1e6); // ns
98 + if (input > 1e14) return Math.floor(input / 1e3); // µs
99 + if (input > 1e11) return Math.floor(input); // ms
100 + return Math.floor(input * 1000); // s
101 + }
102 + if (typeof input === "string") {
103 + const t = input.trim();
104 + if (/^\d+(\.\d+)?$/.test(t)) return parseTimestamp(Number(t));
105 + const ms = Date.parse(t);
106 + return Number.isFinite(ms) ? ms : null;
107 + }
108 + return null;
109 +}
110 +
111 +export function floorTo(ms: number, bucketMs: number): number {
112 + return Math.floor(ms / bucketMs) * bucketMs;
113 +}
added packages/market-model/src/types.ts +331 −0
@@ -0,0 +1,331 @@
1 +import type {
2 + AssetClass,
3 + BarResolution,
4 + ConnectorState,
5 + EventType,
6 + Field,
7 + MarketState,
8 + RealtimeStatus,
9 + RightsStatus,
10 + Severity,
11 + SourceType,
12 + TimestampTrust,
13 +} from "./enums.js";
14 +
15 +/** Stable public identifier, e.g. `eq_us_xnas_aapl`, `crypto_btc_usd`, `fx_eur_usd`, `index_us_spx`. */
16 +export type InstrumentId = string;
17 +
18 +export interface Exchange {
19 + id: string; // e.g. "xnas"
20 + mic: string | null;
21 + name: string;
22 + operator: string | null;
23 + country: string; // ISO-3166 alpha-2
24 + city: string | null;
25 + timezone: string; // IANA
26 + currency: string | null;
27 + website: string | null;
28 + /** Regular sessions in local time, e.g. [{"open":"09:30","close":"16:00"}] plus optional pre/post. */
29 + sessions: TradingSessionSpec;
30 + lat: number | null;
31 + lon: number | null;
32 + assetClasses: AssetClass[];
33 +}
34 +
35 +export interface TradingSessionSpec {
36 + regular: Array<{ open: string; close: string }>;
37 + pre?: { open: string; close: string } | null;
38 + post?: { open: string; close: string } | null;
39 + /** 0 = Sunday … 6 = Saturday. Defaults to Mon–Fri. */
40 + weekdays?: number[];
41 + /** Always open (crypto). */
42 + continuous?: boolean;
43 +}
44 +
45 +export interface Country {
46 + code: string;
47 + name: string;
48 + region: string;
49 + currency: string | null;
50 +}
51 +
52 +export interface Company {
53 + id: string; // e.g. "apple-inc"
54 + name: string;
55 + cik: string | null;
56 + country: string | null;
57 + sector: string | null;
58 + industry: string | null;
59 + website: string | null;
60 +}
61 +
62 +export interface Instrument {
63 + id: InstrumentId;
64 + symbol: string;
65 + name: string;
66 + assetClass: AssetClass;
67 + exchangeId: string | null;
68 + mic: string | null;
69 + currency: string | null;
70 + country: string | null;
71 + companyId: string | null;
72 + isin: string | null;
73 + securityType: string | null;
74 + /** Base/quote for pairs (crypto, FX). */
75 + base: string | null;
76 + quote: string | null;
77 + isActive: boolean;
78 + metadata: Record<string, unknown>;
79 +}
80 +
81 +export interface SymbolAlias {
82 + alias: string;
83 + sourceId: string | null; // null = global alias
84 + instrumentId: InstrumentId;
85 +}
86 +
87 +export interface ConnectorMetadata {
88 + id: string;
89 + name: string;
90 + version: string;
91 + sourceId: string;
92 + organization: string | null;
93 + sourceType: SourceType;
94 + jurisdiction: string | null;
95 + rightsStatus: RightsStatus;
96 + realtimeStatus: RealtimeStatus;
97 + /** Expected source latency in ms (null when unknown). */
98 + expectedLatencyMs: number | null;
99 + supportsStreaming: boolean;
100 + supportsHistorical: boolean;
101 + assetClasses: AssetClass[];
102 + exchanges: string[];
103 + homepage: string | null;
104 + /** Human documentation shown in the admin console and /sources (no endpoint secrets). */
105 + description: string;
106 + /** Data-usage notes: attribution requirements, delays, terms URL. */
107 + rightsNotes: string | null;
108 + termsUrl: string | null;
109 + /** Family used by the consensus engine to discount non-independent sources sharing an upstream. */
110 + sourceFamily: string | null;
111 + enabled: boolean;
112 +}
113 +
114 +export interface SourceRecord {
115 + id: string;
116 + name: string;
117 + organization: string | null;
118 + sourceType: SourceType;
119 + homepage: string | null;
120 + jurisdiction: string | null;
121 + rightsStatus: RightsStatus;
122 + realtimeStatus: RealtimeStatus;
123 + family: string | null;
124 + category: SourceCategory;
125 + enabled: boolean;
126 +}
127 +
128 +export type SourceCategory =
129 + | "EXCHANGE"
130 + | "REGULATOR"
131 + | "CENTRAL_BANK"
132 + | "GOVERNMENT"
133 + | "ISSUER"
134 + | "PUBLIC_MARKET_SOURCE"
135 + | "OFFICIAL_DATASET"
136 + | "NEWS"
137 + | "INTERNAL";
138 +
139 +/** L0 — what a connector received, before any interpretation. */
140 +export interface RawObservation {
141 + connectorId: string;
142 + sourceId: string;
143 + /** Wall clock when the connector received the payload (ms epoch). */
144 + receivedAt: number;
145 + /** Logical payload kind used for schema fingerprints (e.g. "ticker", "trade", "quote", "file"). */
146 + kind: string;
147 + payload: unknown;
148 + /** Optional transport metadata (URL without secrets, status, content-type…). */
149 + meta?: Record<string, unknown>;
150 +}
151 +
152 +/** L2 — one value mapped to Market Atlas vocabulary, still expressed with the source's own symbol. */
153 +export interface NormalizedObservation {
154 + /** Source-native symbol (e.g. "BTC-USD", "AAPL", "_SPX", "BC_10YEAR"). */
155 + symbol: string;
156 + /** Optional canonical id when the connector can resolve it deterministically. */
157 + instrumentId?: InstrumentId;
158 + /** Hints for the resolver when the instrument does not exist yet. */
159 + instrumentHint?: InstrumentHint;
160 + field: Field;
161 + value: number;
162 + currency?: string | null;
163 + /** Source event time (ms epoch) — null when the source does not provide one. */
164 + sourceTimestamp: number | null;
165 + timestampTrust: TimestampTrust;
166 + sequence?: number | string | null;
167 + rightsStatus: RightsStatus;
168 + realtimeStatus: RealtimeStatus;
169 + /** Optional per-observation confidence override (0..1). */
170 + confidence?: number;
171 + /** Extra provenance (e.g. period date for EOD values). */
172 + meta?: Record<string, unknown>;
173 +}
174 +
175 +export interface InstrumentHint {
176 + assetClass: AssetClass;
177 + name?: string;
178 + exchangeId?: string | null;
179 + mic?: string | null;
180 + currency?: string | null;
181 + country?: string | null;
182 + base?: string | null;
183 + quote?: string | null;
184 + securityType?: string | null;
185 + companyName?: string | null;
186 + cik?: string | null;
187 + metadata?: Record<string, unknown>;
188 +}
189 +
190 +/** L3 — resolved and validated observation persisted in the observation store. */
191 +export interface Observation extends Omit<NormalizedObservation, "instrumentHint" | "instrumentId"> {
192 + observationId: string;
193 + instrumentId: InstrumentId;
194 + sourceId: string;
195 + connectorId: string;
196 + receivedAt: number;
197 + latencyMs: number | null;
198 + rawRef: string | null;
199 + normalizerVersion: string;
200 +}
201 +
202 +export interface SourceContribution {
203 + sourceId: string;
204 + connectorId: string;
205 + value: number;
206 + sourceTimestamp: number | null;
207 + receivedAt: number;
208 + ageMs: number;
209 + weight: number;
210 + included: boolean;
211 + reason?: string; // why excluded (stale, outlier…)
212 + realtimeStatus: RealtimeStatus;
213 + rightsStatus: RightsStatus;
214 +}
215 +
216 +export interface CanonicalQuote {
217 + instrumentId: InstrumentId;
218 + symbol: string;
219 + price: number | null;
220 + open: number | null;
221 + high: number | null;
222 + low: number | null;
223 + previousClose: number | null;
224 + change: number | null;
225 + changePercent: number | null;
226 + volume: number | null;
227 + bid: number | null;
228 + ask: number | null;
229 + currency: string | null;
230 + /** Number of independent source families supporting `price`. */
231 + sourceCount: number;
232 + dispersionBps: number | null;
233 + confidence: number;
234 + /** Age of the newest included observation at computation time. */
235 + freshnessMs: number | null;
236 + realtimeStatus: RealtimeStatus;
237 + rightsStatus: RightsStatus;
238 + updatedAt: number;
239 + sourceTimestamp: number | null;
240 + contributions: SourceContribution[];
241 + consensusVersion: string;
242 + /** Session high/low observed by Market Atlas itself (derived). */
243 + sessionHigh: number | null;
244 + sessionLow: number | null;
245 +}
246 +
247 +export interface MarketEvent {
248 + version: 1;
249 + id: string;
250 + type: EventType;
251 + instrumentIds: InstrumentId[];
252 + entityIds: string[];
253 + timestamp: number;
254 + severity: Severity;
255 + confidence: number;
256 + sourceCount: number;
257 + sources: string[];
258 + title: string;
259 + summary: string | null;
260 + data: Record<string, unknown>;
261 + fingerprint: string;
262 + /** Hash of the observations that support the event, when applicable. */
263 + supportingObservations: string[];
264 +}
265 +
266 +export interface Bar {
267 + instrumentId: InstrumentId;
268 + resolution: BarResolution;
269 + ts: number; // bar open time (ms epoch)
270 + open: number;
271 + high: number;
272 + low: number;
273 + close: number;
274 + volume: number | null;
275 + sourceCount: number;
276 + /** Which producer wrote the bar: "consensus" (live) or a connector id (historical backfill). */
277 + producer: string;
278 + version: string;
279 +}
280 +
281 +export interface Filing {
282 + id: string; // accession number or source-native id
283 + sourceId: string;
284 + cik: string | null;
285 + companyName: string;
286 + formType: string;
287 + filedAt: number;
288 + url: string;
289 + instrumentIds: InstrumentId[];
290 + metadata: Record<string, unknown>;
291 +}
292 +
293 +export interface ConnectorHealth {
294 + connectorId: string;
295 + state: ConnectorState;
296 + lastMessageAt: number | null;
297 + lastSuccessAt: number | null;
298 + lastErrorAt: number | null;
299 + lastError: string | null;
300 + messages1m: number;
301 + messagesTotal: number;
302 + errorsTotal: number;
303 + reconnects: number;
304 + medianLatencyMs: number | null;
305 + p95LatencyMs: number | null;
306 + parseSuccessRate: number | null;
307 + instrumentsCovered: number;
308 + schemaFingerprints: string[];
309 + reliabilityScore: number | null;
310 + startedAt: number | null;
311 + nextPollAt: number | null;
312 +}
313 +
314 +export interface ExchangeStatus {
315 + exchangeId: string;
316 + state: MarketState;
317 + localTime: string;
318 + nextTransition: { state: MarketState; at: number } | null;
319 + isHoliday: boolean;
320 + holidayName: string | null;
321 +}
322 +
323 +export interface HolidaySpec {
324 + exchangeId: string;
325 + date: string; // YYYY-MM-DD
326 + name: string;
327 + /** "CLOSED" or "EARLY_CLOSE" (with closeTime local "13:00"). */
328 + kind: "CLOSED" | "EARLY_CLOSE";
329 + closeTime?: string | null;
330 + sourceId: string;
331 +}
added pnpm-lock.yaml +3122 −0
@@ -0,0 +1,3122 @@
1 +lockfileVersion: '9.0'
2 +
3 +settings:
4 + autoInstallPeers: true
5 + excludeLinksFromLockfile: false
6 +
7 +importers:
8 +
9 + .:
10 + devDependencies:
11 + '@types/node':
12 + specifier: ^24.0.0
13 + version: 24.13.4
14 + typescript:
15 + specifier: ^5.9.3
16 + version: 5.9.3
17 + vitest:
18 + specifier: ^3.2.4
19 + version: 3.2.7(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
20 +
21 + apps/api:
22 + dependencies:
23 + '@fastify/cors':
24 + specifier: ^11.0.1
25 + version: 11.3.0
26 + '@fastify/http-proxy':
27 + specifier: ^11.1.2
28 + version: 11.6.2
29 + '@fastify/websocket':
30 + specifier: ^11.1.0
31 + version: 11.3.0
32 + '@market-atlas/connector-sdk':
33 + specifier: workspace:*
34 + version: link:../../packages/connector-sdk
35 + '@market-atlas/connectors':
36 + specifier: workspace:*
37 + version: link:../../connectors
38 + '@market-atlas/market-model':
39 + specifier: workspace:*
40 + version: link:../../packages/market-model
41 + fastify:
42 + specifier: ^5.4.0
43 + version: 5.12.3
44 + pg:
45 + specifier: ^8.16.3
46 + version: 8.23.0
47 + pino:
48 + specifier: ^9.7.0
49 + version: 9.14.0
50 + ws:
51 + specifier: ^8.18.3
52 + version: 8.21.3
53 + zod:
54 + specifier: ^3.24.1
55 + version: 3.25.76
56 + devDependencies:
57 + '@types/pg':
58 + specifier: ^8.15.4
59 + version: 8.23.1
60 + '@types/ws':
61 + specifier: ^8.18.1
62 + version: 8.18.1
63 + esbuild:
64 + specifier: ^0.25.9
65 + version: 0.25.12
66 + tsx:
67 + specifier: ^4.20.5
68 + version: 4.23.13
69 +
70 + apps/web:
71 + dependencies:
72 + d3-geo:
73 + specifier: ^3.1.1
74 + version: 3.1.1
75 + geist:
76 + specifier: ^1.5.1
77 + version: 1.7.2(next@16.3.4(@types/node@24.13.4)(react-dom@19.2.8(react@19.2.8))(react@19.2.8))
78 + lightweight-charts:
79 + specifier: ^5.0.8
80 + version: 5.2.1
81 + lucide-react:
82 + specifier: ^1.0.0
83 + version: 1.44.0(react@19.2.8)
84 + next:
85 + specifier: 16.3.4
86 + version: 16.3.4(@types/node@24.13.4)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
87 + react:
88 + specifier: 19.2.8
89 + version: 19.2.8
90 + react-dom:
91 + specifier: 19.2.8
92 + version: 19.2.8(react@19.2.8)
93 + server-only:
94 + specifier: ^0.0.1
95 + version: 0.0.1
96 + topojson-client:
97 + specifier: ^3.1.0
98 + version: 3.1.0
99 + world-atlas:
100 + specifier: ^2.0.2
101 + version: 2.0.2
102 + devDependencies:
103 + '@tailwindcss/postcss':
104 + specifier: ^4
105 + version: 4.3.3
106 + '@types/d3-geo':
107 + specifier: ^3.1.0
108 + version: 3.1.1
109 + '@types/node':
110 + specifier: ^24.0.0
111 + version: 24.13.4
112 + '@types/react':
113 + specifier: ^19
114 + version: 19.3.0
115 + '@types/react-dom':
116 + specifier: ^19
117 + version: 19.3.0(@types/react@19.3.0)
118 + '@types/topojson-client':
119 + specifier: ^3.1.5
120 + version: 3.1.5
121 + tailwindcss:
122 + specifier: ^4
123 + version: 4.3.3
124 + typescript:
125 + specifier: ^5.9.3
126 + version: 5.9.3
127 +
128 + connectors:
129 + dependencies:
130 + '@market-atlas/connector-sdk':
131 + specifier: workspace:*
132 + version: link:../packages/connector-sdk
133 + '@market-atlas/market-model':
134 + specifier: workspace:*
135 + version: link:../packages/market-model
136 +
137 + packages/connector-sdk:
138 + dependencies:
139 + '@market-atlas/market-model':
140 + specifier: workspace:*
141 + version: link:../market-model
142 + fast-xml-parser:
143 + specifier: ^5.2.5
144 + version: 5.11.1
145 + ws:
146 + specifier: ^8.18.3
147 + version: 8.21.3
148 + zod:
149 + specifier: ^3.24.1
150 + version: 3.25.76
151 + devDependencies:
152 + '@types/ws':
153 + specifier: ^8.18.1
154 + version: 8.18.1
155 +
156 + packages/market-model:
157 + dependencies:
158 + zod:
159 + specifier: ^3.24.1
160 + version: 3.25.76
161 +
162 +packages:
163 +
164 + '@alloc/quick-lru@5.3.0':
165 + resolution: {integrity: sha512-U4+70Pc5ZS9osnCBCE5Jha/ciHM+Yp+CNMNC/7HvYbNRk1Ldd+f7qO65W5qfhu/TCv+/ozljlXXe9Nj8419DMA==}
166 + engines: {node: '>=10'}
167 +
168 + '@emnapi/runtime@1.11.3':
169 + resolution: {integrity: sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==}
170 +
171 + '@esbuild/aix-ppc64@0.25.12':
172 + resolution: {integrity: sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==}
173 + engines: {node: '>=18'}
174 + cpu: [ppc64]
175 + os: [aix]
176 +
177 + '@esbuild/aix-ppc64@0.28.2':
178 + resolution: {integrity: sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==}
179 + engines: {node: '>=18'}
180 + cpu: [ppc64]
181 + os: [aix]
182 +
183 + '@esbuild/android-arm64@0.25.12':
184 + resolution: {integrity: sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==}
185 + engines: {node: '>=18'}
186 + cpu: [arm64]
187 + os: [android]
188 +
189 + '@esbuild/android-arm64@0.28.2':
190 + resolution: {integrity: sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==}
191 + engines: {node: '>=18'}
192 + cpu: [arm64]
193 + os: [android]
194 +
195 + '@esbuild/android-arm@0.25.12':
196 + resolution: {integrity: sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==}
197 + engines: {node: '>=18'}
198 + cpu: [arm]
199 + os: [android]
200 +
201 + '@esbuild/android-arm@0.28.2':
202 + resolution: {integrity: sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==}
203 + engines: {node: '>=18'}
204 + cpu: [arm]
205 + os: [android]
206 +
207 + '@esbuild/android-x64@0.25.12':
208 + resolution: {integrity: sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==}
209 + engines: {node: '>=18'}
210 + cpu: [x64]
211 + os: [android]
212 +
213 + '@esbuild/android-x64@0.28.2':
214 + resolution: {integrity: sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==}
215 + engines: {node: '>=18'}
216 + cpu: [x64]
217 + os: [android]
218 +
219 + '@esbuild/darwin-arm64@0.25.12':
220 + resolution: {integrity: sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==}
221 + engines: {node: '>=18'}
222 + cpu: [arm64]
223 + os: [darwin]
224 +
225 + '@esbuild/darwin-arm64@0.28.2':
226 + resolution: {integrity: sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==}
227 + engines: {node: '>=18'}
228 + cpu: [arm64]
229 + os: [darwin]
230 +
231 + '@esbuild/darwin-x64@0.25.12':
232 + resolution: {integrity: sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==}
233 + engines: {node: '>=18'}
234 + cpu: [x64]
235 + os: [darwin]
236 +
237 + '@esbuild/darwin-x64@0.28.2':
238 + resolution: {integrity: sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==}
239 + engines: {node: '>=18'}
240 + cpu: [x64]
241 + os: [darwin]
242 +
243 + '@esbuild/freebsd-arm64@0.25.12':
244 + resolution: {integrity: sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==}
245 + engines: {node: '>=18'}
246 + cpu: [arm64]
247 + os: [freebsd]
248 +
249 + '@esbuild/freebsd-arm64@0.28.2':
250 + resolution: {integrity: sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==}
251 + engines: {node: '>=18'}
252 + cpu: [arm64]
253 + os: [freebsd]
254 +
255 + '@esbuild/freebsd-x64@0.25.12':
256 + resolution: {integrity: sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==}
257 + engines: {node: '>=18'}
258 + cpu: [x64]
259 + os: [freebsd]
260 +
261 + '@esbuild/freebsd-x64@0.28.2':
262 + resolution: {integrity: sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==}
263 + engines: {node: '>=18'}
264 + cpu: [x64]
265 + os: [freebsd]
266 +
267 + '@esbuild/linux-arm64@0.25.12':
268 + resolution: {integrity: sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==}
269 + engines: {node: '>=18'}
270 + cpu: [arm64]
271 + os: [linux]
272 +
273 + '@esbuild/linux-arm64@0.28.2':
274 + resolution: {integrity: sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==}
275 + engines: {node: '>=18'}
276 + cpu: [arm64]
277 + os: [linux]
278 +
279 + '@esbuild/linux-arm@0.25.12':
280 + resolution: {integrity: sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==}
281 + engines: {node: '>=18'}
282 + cpu: [arm]
283 + os: [linux]
284 +
285 + '@esbuild/linux-arm@0.28.2':
286 + resolution: {integrity: sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==}
287 + engines: {node: '>=18'}
288 + cpu: [arm]
289 + os: [linux]
290 +
291 + '@esbuild/linux-ia32@0.25.12':
292 + resolution: {integrity: sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==}
293 + engines: {node: '>=18'}
294 + cpu: [ia32]
295 + os: [linux]
296 +
297 + '@esbuild/linux-ia32@0.28.2':
298 + resolution: {integrity: sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==}
299 + engines: {node: '>=18'}
300 + cpu: [ia32]
301 + os: [linux]
302 +
303 + '@esbuild/linux-loong64@0.25.12':
304 + resolution: {integrity: sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==}
305 + engines: {node: '>=18'}
306 + cpu: [loong64]
307 + os: [linux]
308 +
309 + '@esbuild/linux-loong64@0.28.2':
310 + resolution: {integrity: sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==}
311 + engines: {node: '>=18'}
312 + cpu: [loong64]
313 + os: [linux]
314 +
315 + '@esbuild/linux-mips64el@0.25.12':
316 + resolution: {integrity: sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==}
317 + engines: {node: '>=18'}
318 + cpu: [mips64el]
319 + os: [linux]
320 +
321 + '@esbuild/linux-mips64el@0.28.2':
322 + resolution: {integrity: sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==}
323 + engines: {node: '>=18'}
324 + cpu: [mips64el]
325 + os: [linux]
326 +
327 + '@esbuild/linux-ppc64@0.25.12':
328 + resolution: {integrity: sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==}
329 + engines: {node: '>=18'}
330 + cpu: [ppc64]
331 + os: [linux]
332 +
333 + '@esbuild/linux-ppc64@0.28.2':
334 + resolution: {integrity: sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==}
335 + engines: {node: '>=18'}
336 + cpu: [ppc64]
337 + os: [linux]
338 +
339 + '@esbuild/linux-riscv64@0.25.12':
340 + resolution: {integrity: sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==}
341 + engines: {node: '>=18'}
342 + cpu: [riscv64]
343 + os: [linux]
344 +
345 + '@esbuild/linux-riscv64@0.28.2':
346 + resolution: {integrity: sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==}
347 + engines: {node: '>=18'}
348 + cpu: [riscv64]
349 + os: [linux]
350 +
351 + '@esbuild/linux-s390x@0.25.12':
352 + resolution: {integrity: sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==}
353 + engines: {node: '>=18'}
354 + cpu: [s390x]
355 + os: [linux]
356 +
357 + '@esbuild/linux-s390x@0.28.2':
358 + resolution: {integrity: sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==}
359 + engines: {node: '>=18'}
360 + cpu: [s390x]
361 + os: [linux]
362 +
363 + '@esbuild/linux-x64@0.25.12':
364 + resolution: {integrity: sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==}
365 + engines: {node: '>=18'}
366 + cpu: [x64]
367 + os: [linux]
368 +
369 + '@esbuild/linux-x64@0.28.2':
370 + resolution: {integrity: sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==}
371 + engines: {node: '>=18'}
372 + cpu: [x64]
373 + os: [linux]
374 +
375 + '@esbuild/netbsd-arm64@0.25.12':
376 + resolution: {integrity: sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==}
377 + engines: {node: '>=18'}
378 + cpu: [arm64]
379 + os: [netbsd]
380 +
381 + '@esbuild/netbsd-arm64@0.28.2':
382 + resolution: {integrity: sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==}
383 + engines: {node: '>=18'}
384 + cpu: [arm64]
385 + os: [netbsd]
386 +
387 + '@esbuild/netbsd-x64@0.25.12':
388 + resolution: {integrity: sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==}
389 + engines: {node: '>=18'}
390 + cpu: [x64]
391 + os: [netbsd]
392 +
393 + '@esbuild/netbsd-x64@0.28.2':
394 + resolution: {integrity: sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==}
395 + engines: {node: '>=18'}
396 + cpu: [x64]
397 + os: [netbsd]
398 +
399 + '@esbuild/openbsd-arm64@0.25.12':
400 + resolution: {integrity: sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==}
401 + engines: {node: '>=18'}
402 + cpu: [arm64]
403 + os: [openbsd]
404 +
405 + '@esbuild/openbsd-arm64@0.28.2':
406 + resolution: {integrity: sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==}
407 + engines: {node: '>=18'}
408 + cpu: [arm64]
409 + os: [openbsd]
410 +
411 + '@esbuild/openbsd-x64@0.25.12':
412 + resolution: {integrity: sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==}
413 + engines: {node: '>=18'}
414 + cpu: [x64]
415 + os: [openbsd]
416 +
417 + '@esbuild/openbsd-x64@0.28.2':
418 + resolution: {integrity: sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==}
419 + engines: {node: '>=18'}
420 + cpu: [x64]
421 + os: [openbsd]
422 +
423 + '@esbuild/openharmony-arm64@0.25.12':
424 + resolution: {integrity: sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==}
425 + engines: {node: '>=18'}
426 + cpu: [arm64]
427 + os: [openharmony]
428 +
429 + '@esbuild/openharmony-arm64@0.28.2':
430 + resolution: {integrity: sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==}
431 + engines: {node: '>=18'}
432 + cpu: [arm64]
433 + os: [openharmony]
434 +
435 + '@esbuild/sunos-x64@0.25.12':
436 + resolution: {integrity: sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==}
437 + engines: {node: '>=18'}
438 + cpu: [x64]
439 + os: [sunos]
440 +
441 + '@esbuild/sunos-x64@0.28.2':
442 + resolution: {integrity: sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==}
443 + engines: {node: '>=18'}
444 + cpu: [x64]
445 + os: [sunos]
446 +
447 + '@esbuild/win32-arm64@0.25.12':
448 + resolution: {integrity: sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==}
449 + engines: {node: '>=18'}
450 + cpu: [arm64]
451 + os: [win32]
452 +
453 + '@esbuild/win32-arm64@0.28.2':
454 + resolution: {integrity: sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==}
455 + engines: {node: '>=18'}
456 + cpu: [arm64]
457 + os: [win32]
458 +
459 + '@esbuild/win32-ia32@0.25.12':
460 + resolution: {integrity: sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==}
461 + engines: {node: '>=18'}
462 + cpu: [ia32]
463 + os: [win32]
464 +
465 + '@esbuild/win32-ia32@0.28.2':
466 + resolution: {integrity: sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==}
467 + engines: {node: '>=18'}
468 + cpu: [ia32]
469 + os: [win32]
470 +
471 + '@esbuild/win32-x64@0.25.12':
472 + resolution: {integrity: sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==}
473 + engines: {node: '>=18'}
474 + cpu: [x64]
475 + os: [win32]
476 +
477 + '@esbuild/win32-x64@0.28.2':
478 + resolution: {integrity: sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==}
479 + engines: {node: '>=18'}
480 + cpu: [x64]
481 + os: [win32]
482 +
483 + '@fastify/ajv-compiler@4.0.6':
484 + resolution: {integrity: sha512-NtuzM0SfaMJbGlnjr9LWQUN5LzgSrbB8tf/wRZNas+4E1O/Nmzl53e7ruT61HDZyRCJGC6FxIogmNZO1c5ETBA==}
485 +
486 + '@fastify/cors@11.3.0':
487 + resolution: {integrity: sha512-ggQGua+xHv1MvePbPr0v//xLYEsCXbWspquXCJS9Ot5YoRXq8J8ZWzHnxDBVnbtXosvistXo6LtNzOJswf64Fw==}
488 +
489 + '@fastify/error@4.2.0':
490 + resolution: {integrity: sha512-RSo3sVDXfHskiBZKBPRgnQTtIqpi/7zhJOEmAxCiBcM7d0uwdGdxLlsCaLzGs8v8NnxIRlfG0N51p5yFaOentQ==}
491 +
492 + '@fastify/fast-json-stringify-compiler@5.1.0':
493 + resolution: {integrity: sha512-PxcYtKLbQ8Z+yApiqjK8FwxIwvEj38k2OiLc17u8dkJSlmfi2wHHPaSnaoqBPQqtvF8YVsDgDpP2snDCfFrpfw==}
494 +
495 + '@fastify/forwarded@3.0.2':
496 + resolution: {integrity: sha512-NE8HgKLgYejV9lDpqkEFaDKMLYelJBVfHekhB0UKvX0ghagXRJqg68feg8er1NPXxG4N9i6vPxzt8E+3wHfcmA==}
497 +
498 + '@fastify/http-proxy@11.6.2':
499 + resolution: {integrity: sha512-SR/rDXL3RihOOPYx0Han41G2ksXvAj9WpSy9MqjdOL7U1s4EbUXLKkEi7S8zM/ockDLXjwVuT2qB8d5unL3XKg==}
500 +
501 + '@fastify/merge-json-schemas@0.2.1':
502 + resolution: {integrity: sha512-OA3KGBCy6KtIvLf8DINC5880o5iBlDX4SxzLQS8HorJAbqluzLRn80UXU0bxZn7UOFhFgpRJDasfwn9nG4FG4A==}
503 +
504 + '@fastify/proxy-addr@5.1.0':
505 + resolution: {integrity: sha512-INS+6gh91cLUjB+PVHfu1UqcB76Sqtpyp7bnL+FYojhjygvOPA9ctiD/JDKsyD9Xgu4hUhCSJBPig/w7duNajw==}
506 +
507 + '@fastify/reply-from@12.6.5':
508 + resolution: {integrity: sha512-/4MP+iF6T2+UBEmVYveMU2Tbx3lR1Q7xsDo8GAGjNeQrLF/JqNRlOMhOksh8fXBA47KI8ifsIlUXYdMOUXGZNQ==}
509 +
510 + '@fastify/websocket@11.3.0':
511 + resolution: {integrity: sha512-g89ag4BCcD9YP5wBZXixzoLnuf5j89p/sXFcfpCiv2pdEkYYukBEoK3heVzqsp0EAtszVDc2BBZG0KZqeAShIA==}
512 +
513 + '@img/colour@1.1.0':
514 + resolution: {integrity: sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==}
515 + engines: {node: '>=18'}
516 +
517 + '@img/sharp-darwin-arm64@0.35.4':
518 + resolution: {integrity: sha512-Uhfl4V4lhP2nbUVF9+hyH1+luj86f1gUFeo8ALYxFoULoU+G87D43BfeMP8XHsk9boxAnCY/bf2EHwhA7MuGsA==}
519 + engines: {node: '>=20.9.0'}
520 + cpu: [arm64]
521 + os: [darwin]
522 +
523 + '@img/sharp-darwin-x64@0.35.4':
524 + resolution: {integrity: sha512-hWniXY3bG5qKpkKrAwPe4y+VTPmf086YQAnkxWh7uA1YrlRouWGa0M0Mxj3ZjnXFkv7/TD1bTy9lGUK26vRvWw==}
525 + engines: {node: '>=20.9.0'}
526 + cpu: [x64]
527 + os: [darwin]
528 +
529 + '@img/sharp-freebsd-wasm32@0.35.4':
530 + resolution: {integrity: sha512-lIsKw/BU+kjB4eZjxrYrZmwOJYi3Ajrv66iAlBmUPyKc3HpnloevB1g3wxGD9P/5BbQ1brBGl65VRRrCvQDEqA==}
531 + engines: {node: '>=20.9.0'}
532 + os: [freebsd]
533 +
534 + '@img/sharp-libvips-darwin-arm64@1.3.3':
535 + resolution: {integrity: sha512-suTBPTDGrI9WodccaDdwZItTSaBYASlBk1NSfElSHrUfzu3szG6lvIF58+WiFvnfzuK8ZBFS5zE00PxqxnRiPg==}
536 + cpu: [arm64]
537 + os: [darwin]
538 +
539 + '@img/sharp-libvips-darwin-x64@1.3.3':
540 + resolution: {integrity: sha512-FVJZ5mITMobmXIz/hPDTw0EintTW5H3WfrxwLqEqjiIihlu+hVRyGrFQ60xl0Lxn7Bt3zdpevPaQi0HEzqz9fw==}
541 + cpu: [x64]
542 + os: [darwin]
543 +
544 + '@img/sharp-libvips-linux-arm64@1.3.3':
545 + resolution: {integrity: sha512-0DaL0A6Xu6sQSQFwe4iVCrKWU2cCTItnRsYsCdxAMm9NF6twAA9BKnoqy4hqz4+azQ0JHuA26qiUKsf1XJ/v5A==}
546 + cpu: [arm64]
547 + os: [linux]
548 + libc: [glibc]
549 +
550 + '@img/sharp-libvips-linux-arm@1.3.3':
551 + resolution: {integrity: sha512-3rbU4vqXXc3hY/OiXdl52xZvT0F1yEngWfvqudtPJg/KkyiaQw2DRsFrNzpmLvfavbwOq3qXn36GP8obHRULQA==}
552 + cpu: [arm]
553 + os: [linux]
554 + libc: [glibc]
555 +
556 + '@img/sharp-libvips-linux-ppc64@1.3.3':
557 + resolution: {integrity: sha512-cdn1OvUBwsXhbC0zSzJnNzf5MZ/mTrobawDvNXBTxe8VtqKAm0sRuEY2Evzovb/w9JMk4TvRxqt1mekSuJz64w==}
558 + cpu: [ppc64]
559 + os: [linux]
560 + libc: [glibc]
561 +
562 + '@img/sharp-libvips-linux-riscv64@1.3.3':
563 + resolution: {integrity: sha512-HjPVx7yKz+0lqdhDlTw1tt90wamBoxhiXpvl1XZpJLiHH4RCJ5yDTqH+VlYPv2fwFs89JFw4c1IexYOcQUi4IQ==}
564 + cpu: [riscv64]
565 + os: [linux]
566 + libc: [glibc]
567 +
568 + '@img/sharp-libvips-linux-s390x@1.3.3':
569 + resolution: {integrity: sha512-neWLh+3yCNThxnfy3c4BbVBeGgt9aftno+XbT56iK28RgeDs3UOFWviLWlUu0bArYVYJaFDK+RRohbicUNCm8Q==}
570 + cpu: [s390x]
571 + os: [linux]
572 + libc: [glibc]
573 +
574 + '@img/sharp-libvips-linux-x64@1.3.3':
575 + resolution: {integrity: sha512-4vKmvAst9nrowcqquKFAyZJUDolUaIp8uRiN0mWFguJ1IplC9/pitXtlnnlU4aa/eJw3J7i67V+pwUL+wZGdsA==}
576 + cpu: [x64]
577 + os: [linux]
578 + libc: [glibc]
579 +
580 + '@img/sharp-libvips-linuxmusl-arm64@1.3.3':
581 + resolution: {integrity: sha512-Y9kQaLMuNoB0bPYOOdcZMaseNrFpPodIWWMrx+CZyydf2xn68j9WYc6sWWRrDwNkzCQjKYfc68L7jKjGlHMibw==}
582 + cpu: [arm64]
583 + os: [linux]
584 + libc: [musl]
585 +
586 + '@img/sharp-libvips-linuxmusl-x64@1.3.3':
587 + resolution: {integrity: sha512-fj8Mv0HHfD1Rr+4I68+3agJynxDWtBFgicTbSOb9Bke6pIwzGcJ+RX/yHjmiEGFMCavY/dxvem7MyNaJF+wDiw==}
588 + cpu: [x64]
589 + os: [linux]
590 + libc: [musl]
591 +
592 + '@img/sharp-linux-arm64@0.35.4':
593 + resolution: {integrity: sha512-De4jpEnAU8Hd5oT0j1G3uL4ZvTuipVMn7YC6vPaJhy6/7EwEae0SVAoBrUMYQbkLGDm85taVWwuPc1a44LTzCQ==}
594 + engines: {node: '>=20.9.0'}
595 + cpu: [arm64]
596 + os: [linux]
597 + libc: [glibc]
598 +
599 + '@img/sharp-linux-arm@0.35.4':
600 + resolution: {integrity: sha512-7OAS8gI0EReKGVN2HssHlM6umJgxF5VI3xN0p9FA91p/YO+ou5hiNghLdZ5BEHztwaaK5+bLKRf8x/o2L2nk9A==}
601 + engines: {node: '>=20.9.0'}
602 + cpu: [arm]
603 + os: [linux]
604 + libc: [glibc]
605 +
606 + '@img/sharp-linux-ppc64@0.35.4':
607 + resolution: {integrity: sha512-2oYZJeIl4kCcMGk4ouZVjnkCtFrpQFlNEtJ6GbxzhHQchwH0NH/qEb9ykmOl29dqwMq+JhFdZn+1ak2FKhI9fQ==}
608 + engines: {node: '>=20.9.0'}
609 + cpu: [ppc64]
610 + os: [linux]
611 + libc: [glibc]
612 +
613 + '@img/sharp-linux-riscv64@0.35.4':
614 + resolution: {integrity: sha512-cPbNChoRURAWdebDIHSenxRpgEdy7JkPydSnUxRm9VvKD7m0/xVaR/8Fzlu81pk5nHEvHH87UZUA7cTtwnbJSA==}
615 + engines: {node: '>=20.9.0'}
616 + cpu: [riscv64]
617 + os: [linux]
618 + libc: [glibc]
619 +
620 + '@img/sharp-linux-s390x@0.35.4':
621 + resolution: {integrity: sha512-RY0JFY8Fd6RonCBtHz+DvadaPkXDSI1AUn6yWL9TipqkZ1vY8w8evqdgyDFnkm4/K1ve1TvZiaePP5oSd4+WVQ==}
622 + engines: {node: '>=20.9.0'}
623 + cpu: [s390x]
624 + os: [linux]
625 + libc: [glibc]
626 +
627 + '@img/sharp-linux-x64@0.35.4':
628 + resolution: {integrity: sha512-9qvvEAuk8k89TfWUoX2htWjbAMX8p+NxCppjpcg5k6xMsjhBQPTsoIh36h9Qde4WRuGpJeYnOjdosDn/cnv+OA==}
629 + engines: {node: '>=20.9.0'}
630 + cpu: [x64]
631 + os: [linux]
632 + libc: [glibc]
633 +
634 + '@img/sharp-linuxmusl-arm64@0.35.4':
635 + resolution: {integrity: sha512-KB5jxpfWQTr0nc3xdHtWChdbifHrBGsd2SM62Eyxrl8afikm+f5qGBU75SJIZBT/S1MC8XyacdlXBMSWq6OURA==}
636 + engines: {node: '>=20.9.0'}
637 + cpu: [arm64]
638 + os: [linux]
639 + libc: [musl]
640 +
641 + '@img/sharp-linuxmusl-x64@0.35.4':
642 + resolution: {integrity: sha512-f+eZJZIQNEEd26RPSW+76chwOf1XtA2Y/O+5ocVyLliHkeih3e+jhLVBdNTd2rS3IbNXK8+ug93Vf5ZXtF5Lxg==}
643 + engines: {node: '>=20.9.0'}
644 + cpu: [x64]
645 + os: [linux]
646 + libc: [musl]
647 +
648 + '@img/sharp-wasm32@0.35.4':
649 + resolution: {integrity: sha512-zQnl4Kwp7Q6NHsENtU2T/00Zi+w3AQNwz3+UaTyVBy2FpXrzXzGjndpK61onhZjRtRpQXxCTeqw19bVyXOh7jA==}
650 + engines: {node: '>=20.9.0'}
651 +
652 + '@img/sharp-webcontainers-wasm32@0.35.4':
653 + resolution: {integrity: sha512-ESfNkywmCfPNyaZjxooddJQiQ+l/nTpGEOGthxiLnIHXC/CmcBixnfwUleX9mCz9ovrUUvKMap/pm8RYbzfwaA==}
654 + engines: {node: '>=20.9.0'}
655 + cpu: [wasm32]
656 +
657 + '@img/sharp-win32-arm64@0.35.4':
658 + resolution: {integrity: sha512-iNdlBX9gLVvqe2I3uIJSIKTq6wckP/DYxZtcqxm09x5Gi24DnFBmPAWZmr60ZyYMG0xlzo6goG3670ar+RXvRw==}
659 + engines: {node: '>=20.9.0'}
660 + cpu: [arm64]
661 + os: [win32]
662 +
663 + '@img/sharp-win32-ia32@0.35.4':
664 + resolution: {integrity: sha512-kqRsbaa5CS6KHlpxnN7WhE6vAAugXyZButpRdvDWetlv6Qv4N9WTcrWzF7tXfB9T7MsoadqdI8hmwLq6UlLvtw==}
665 + engines: {node: ^20.9.0}
666 + cpu: [ia32]
667 + os: [win32]
668 +
669 + '@img/sharp-win32-x64@0.35.4':
670 + resolution: {integrity: sha512-XtmnYhBcrORsJ4XJngyzr/EWP0hRZLAZRFaApdKuviyqF78+ylxh2y06ZmtULAMOnObJ3ucpN0AcwSWnMowTRg==}
671 + engines: {node: '>=20.9.0'}
672 + cpu: [x64]
673 + os: [win32]
674 +
675 + '@jridgewell/gen-mapping@0.3.13':
676 + resolution: {integrity: sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==}
677 +
678 + '@jridgewell/remapping@2.3.5':
679 + resolution: {integrity: sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ==}
680 +
681 + '@jridgewell/resolve-uri@3.1.2':
682 + resolution: {integrity: sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==}
683 + engines: {node: '>=6.0.0'}
684 +
685 + '@jridgewell/sourcemap-codec@1.6.0':
686 + resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
687 +
688 + '@jridgewell/trace-mapping@0.3.31':
689 + resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==}
690 +
691 + '@napi-rs/lzma-linux-x64-gnu@1.5.1':
692 + resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
693 + engines: {node: ^22.20 || ^24.12 || >=25}
694 + cpu: [x64]
695 + os: [linux]
696 + libc: [glibc]
697 +
698 + '@next/env@16.3.4':
699 + resolution: {integrity: sha512-cjWZnUUa6jZq2kFaNe/ZyJdZonOZ/QoN0Zka2nz/FLOrfx14pQuM9c5RaSVkWMqgdt4ksgPAMWPyHSs/CyV48Q==}
700 +
701 + '@next/swc-darwin-arm64@16.3.4':
702 + resolution: {integrity: sha512-iBr3I5LZNk5/bgl5//iTgD2tcym14MX0Xo7fD//u9dYAEgGzza1y9oywluPtf74YnOswVdH1908aK9xVz7zQTw==}
703 + engines: {node: '>= 10'}
704 + cpu: [arm64]
705 + os: [darwin]
706 +
707 + '@next/swc-darwin-x64@16.3.4':
708 + resolution: {integrity: sha512-2dpiSyl2Jw/NrBPaU2MAKGSa+2MR82pJIn4Sm5Rjr+gxAeuh0z158Su3Z2O8zn7UNNq+ej4bToed6RcRN/Lydg==}
709 + engines: {node: '>= 10'}
710 + cpu: [x64]
711 + os: [darwin]
712 +
713 + '@next/swc-linux-arm64-gnu@16.3.4':
714 + resolution: {integrity: sha512-+t+U8HZT+fApePCS5h89CSH3datz29MkzyfCn+6fpsZBG/oiEOhINcb9rtkv6sdpToLGFn2e6146NzaKCXkqrA==}
715 + engines: {node: '>= 10'}
716 + cpu: [arm64]
717 + os: [linux]
718 + libc: [glibc]
719 +
720 + '@next/swc-linux-arm64-musl@16.3.4':
721 + resolution: {integrity: sha512-mx03GNs1ocQA5JQ4FxDMmIsNkdrZh8cuezKCrId28e5/gIPU/l7Kcy2+vmCCzdjnnmXJy+iOAu+7K0QppO6Urg==}
722 + engines: {node: '>= 10'}
723 + cpu: [arm64]
724 + os: [linux]
725 + libc: [musl]
726 +
727 + '@next/swc-linux-x64-gnu@16.3.4':
728 + resolution: {integrity: sha512-YIhGY6fSMfha52bnVxnzc9zaVBzJg+cqQTOD8tXIBSx4fuv0pVMxQTE0PaS59YhnMOiYiG09IMwxJAf/CFm/Dw==}
729 + engines: {node: '>= 10'}
730 + cpu: [x64]
731 + os: [linux]
732 + libc: [glibc]
733 +
734 + '@next/swc-linux-x64-musl@16.3.4':
735 + resolution: {integrity: sha512-+eaaX6axpDb0yF1GCpiERe6njplvdC+nks/fKfcHu3XPGRrald8P3/X7yv7QLdjA51knnxwl9pxdIJsg+w1L+Q==}
736 + engines: {node: '>= 10'}
737 + cpu: [x64]
738 + os: [linux]
739 + libc: [musl]
740 +
741 + '@next/swc-win32-arm64-msvc@16.3.4':
742 + resolution: {integrity: sha512-0jcXW7Xs/uzICrmgV3MhDYDeRy++1CqnpDIerlPIqYO4bhzB4WNbX/aRnQclustsAyTkFKB0z6rbcjmNg5tR8A==}
743 + engines: {node: '>= 10'}
744 + cpu: [arm64]
745 + os: [win32]
746 +
747 + '@next/swc-win32-x64-msvc@16.3.4':
748 + resolution: {integrity: sha512-vvBzwu1pYQCp92maZCFCIw/XgOTMR5tur9GjakwIo2cmwRTMKajRZZDS9+e4KsUZWKu1E007WUeAFXRRjZeuzw==}
749 + engines: {node: '>= 10'}
750 + cpu: [x64]
751 + os: [win32]
752 +
753 + '@nodable/entities@3.0.0':
754 + resolution: {integrity: sha512-8L9xFeTYKhm49xfIypoe2W5wV1m/3Z58kT+7kR9A8OyFxcPduI4VmxaUMQyKYrRjUoLLSXv6EKKID5Tvj9cUVw==}
755 +
756 + '@pinojs/redact@0.4.0':
757 + resolution: {integrity: sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==}
758 +
759 + '@rollup/rollup-android-arm-eabi@4.63.1':
760 + resolution: {integrity: sha512-UZ8sUxPTiHWYX9QNdJedb1kDZSpS1t/VPWBWGSgqHNi9w3Cu6IXvu2mzbhiTiPvtrqgTQJ+zqiAq2iPIPilpaQ==}
761 + cpu: [arm]
762 + os: [android]
763 +
764 + '@rollup/rollup-android-arm64@4.63.1':
765 + resolution: {integrity: sha512-cQ4nFQABN5cDvDpbvJ7bMStCpnaVxynZrRMfUJYgxcIk9Sh54FIO1vtfkg0B69REjER77ioZ/ov+eAApx/KmLQ==}
766 + cpu: [arm64]
767 + os: [android]
768 +
769 + '@rollup/rollup-darwin-arm64@4.63.1':
770 + resolution: {integrity: sha512-FQNqd1lRy/0QhDk3xeRIkSBiCpXCiDnZO3YLVdcDKN1UBiKToNftCzcXYNLshmPDUMlu2TdeS8tGcsU6f3YF1Q==}
771 + cpu: [arm64]
772 + os: [darwin]
773 +
774 + '@rollup/rollup-darwin-x64@4.63.1':
775 + resolution: {integrity: sha512-pvD16V939D3CloK0+qikpGaxiPrDUXTe7Y5cWOMkMSy7m1cawa8EGy/kXYi/G/cKAC4HDAbSnzCIk1WmsoOKXg==}
776 + cpu: [x64]
777 + os: [darwin]
778 +
779 + '@rollup/rollup-freebsd-arm64@4.63.1':
780 + resolution: {integrity: sha512-pcFGeL2345VwdTnJhA6zLbew+YgWB0qBG2+dMtXjCicf6+rm6kO6cOoh5VnTe0ZMrMRgRyuHmCJxZWrIdzYuOw==}
781 + cpu: [arm64]
782 + os: [freebsd]
783 +
784 + '@rollup/rollup-freebsd-x64@4.63.1':
785 + resolution: {integrity: sha512-mRJlqSRulVzcKq/LKA6ICSIc3K/l4fzlVn/gePn2nXIHy8seRi5z/eeRE0d/XMBxcMldiXtQTSpRj0tkkC3g8Q==}
786 + cpu: [x64]
787 + os: [freebsd]
788 +
789 + '@rollup/rollup-linux-arm-gnueabihf@4.63.1':
790 + resolution: {integrity: sha512-YDUNvVM85TI3g/1OpnqKP1h4NeW/j64DfWMf+G3M809xNk1bJSnpFp4sh83NpmVE5DXnkh8ULor4LTVZKoYLHw==}
791 + cpu: [arm]
792 + os: [linux]
793 + libc: [glibc]
794 +
795 + '@rollup/rollup-linux-arm-musleabihf@4.63.1':
796 + resolution: {integrity: sha512-7Mcn71p9ZuQFAj+h+dhQXy/yeLePRS2yKRnmW1DijA9thKO5qap0GNOIQK4yQ6iP3SU0Mrb/yWo8h8vgRba8lw==}
797 + cpu: [arm]
798 + os: [linux]
799 + libc: [musl]
800 +
801 + '@rollup/rollup-linux-arm64-gnu@4.63.1':
802 + resolution: {integrity: sha512-4YiLQTX6U4CSl0L9cluep9A9W6UmTfqBDc2/CH6wlu54pl4E7Jn3cOD8oxzvBDEGk/JMKgJ47C8g+radF7mwvg==}
803 + cpu: [arm64]
804 + os: [linux]
805 + libc: [glibc]
806 +
807 + '@rollup/rollup-linux-arm64-musl@4.63.1':
808 + resolution: {integrity: sha512-2ra8F7w8OquwZN9z2/fKFnli69wa8PLwaVzRMIPGb13ByMJwC28Fbp8YcVGoUhlYMTt7j5j9bNgpysrN2UM+vw==}
809 + cpu: [arm64]
810 + os: [linux]
811 + libc: [musl]
812 +
813 + '@rollup/rollup-linux-loong64-gnu@4.63.1':
814 + resolution: {integrity: sha512-Sy20ncyhjmBP0Ml+UvQbimjlk6VFgjW5uNP+qqwHB00mTE8Bl2C1TuHTlRwK2YoXeZbee5lP2XevBWVkAQAtSQ==}
815 + cpu: [loong64]
816 + os: [linux]
817 + libc: [glibc]
818 +
819 + '@rollup/rollup-linux-loong64-musl@4.63.1':
820 + resolution: {integrity: sha512-noITLp8oNjYliPnGWmLyelIHwULGqbHloQHGw1rtxbWhTuWooRpnZarZQJ1y9EUC4szuCusCc+HEpUtxpIwYvA==}
821 + cpu: [loong64]
822 + os: [linux]
823 + libc: [musl]
824 +
825 + '@rollup/rollup-linux-ppc64-gnu@4.63.1':
826 + resolution: {integrity: sha512-hlxxXd+F1mWiAcaFR7Sv9ZQT6m6UfI8+Vy/kFJzztq2pDMU/0wZ9sish0iszNZvsQDo8Gc0i5yuFEOz5dDf6fA==}
827 + cpu: [ppc64]
828 + os: [linux]
829 + libc: [glibc]
830 +
831 + '@rollup/rollup-linux-ppc64-musl@4.63.1':
832 + resolution: {integrity: sha512-EF7OpqQTQ/BvGqLzUi4rEHuagCV9MugAUXSHemwPW5vxZ75RR+jxO/2j95Ph2dalMpFHSVECjRoioHZgA9zOYA==}
833 + cpu: [ppc64]
834 + os: [linux]
835 + libc: [musl]
836 +
837 + '@rollup/rollup-linux-riscv64-gnu@4.63.1':
838 + resolution: {integrity: sha512-wQO3JesW9PRkwlabQ27y7sPfVOOTLRG73I4F2UYHG5PXun3J9U3y+b7ezVKSYbsvSKGQ1k1cq8Qlun4C9kLt3w==}
839 + cpu: [riscv64]
840 + os: [linux]
841 + libc: [glibc]
842 +
843 + '@rollup/rollup-linux-riscv64-musl@4.63.1':
844 + resolution: {integrity: sha512-ouAGwhO6wHRXdnOVCOsB0tRFkA7nhNB2Nwax6oECXN0YiN8EYUTBAOudADOB1PI+yDL61TeNx/u7MVCzksNbkQ==}
845 + cpu: [riscv64]
846 + os: [linux]
847 + libc: [musl]
848 +
849 + '@rollup/rollup-linux-s390x-gnu@4.63.1':
850 + resolution: {integrity: sha512-q2R38Sn+1J8RxhfJ+T54wSWmyKXWec+9jgDfqO2AtArEqHO5R2aeayp5H5OYLr5UYDVGsVaZPEFUooMhYCdz5A==}
851 + cpu: [s390x]
852 + os: [linux]
853 + libc: [glibc]
854 +
855 + '@rollup/rollup-linux-x64-gnu@4.63.1':
856 + resolution: {integrity: sha512-gfI5T24WLLuFfSKw7Go/zDXjAAV0fny0swTaDv+WjK7vqcw4cRhFfdsyKL1n+ukI+ooBxn3bVQnyrn06WpI50w==}
857 + cpu: [x64]
858 + os: [linux]
859 + libc: [glibc]
860 +
861 + '@rollup/rollup-linux-x64-musl@4.63.1':
862 + resolution: {integrity: sha512-4h6XqthmB4Hspji84wvgk+ElodTsGj+dbZqHJHHtKxj4mYq0ANSEEPX9ys3moJueqsRjwpaJYH7874Itwnj2ow==}
863 + cpu: [x64]
864 + os: [linux]
865 + libc: [musl]
866 +
867 + '@rollup/rollup-openbsd-x64@4.63.1':
868 + resolution: {integrity: sha512-dlfCOa87o1VAYegLQ9EKilx2JCeRofiyPGhTCmqnuXZ6bMPiycO1rq1+sKoulAp7pGLIsTIw+1x5R+zgh5LhhA==}
869 + cpu: [x64]
870 + os: [openbsd]
871 +
872 + '@rollup/rollup-openharmony-arm64@4.63.1':
873 + resolution: {integrity: sha512-cjkLbOlfcm3QGhMM1J5zaZjsw1GggbN6rw9UTSSRrPrR1KkcXnN7Uq9rPw34xImQ9VOY9GN+6u2Zj80B9ptkcw==}
874 + cpu: [arm64]
875 + os: [openharmony]
876 +
877 + '@rollup/rollup-win32-arm64-msvc@4.63.1':
878 + resolution: {integrity: sha512-Li1KdUnWGE4N3e1F/B4RTB1ms+nG4WBgjByO46pkeBVX/2UBsY53xf5vK9WygVmnH3RwncIST7lkSdLSY6P9lg==}
879 + cpu: [arm64]
880 + os: [win32]
881 +
882 + '@rollup/rollup-win32-ia32-msvc@4.63.1':
883 + resolution: {integrity: sha512-t4ZYOSoLTgwhuFMrmTMLx/+i1DQVK7HYqMc6kY46EApwi8X0nIVphzdNoThU3xt6n+N5urG1/gxBdCaKDLavfg==}
884 + cpu: [ia32]
885 + os: [win32]
886 +
887 + '@rollup/rollup-win32-x64-gnu@4.63.1':
888 + resolution: {integrity: sha512-RgroPfMmKlD1RzSDxvwgcPiy2HNQKoYV7OmwIXDsk73uKW5t6B/V8KIy27SMv/FNXFo/oSBtWc9J0X7t91ezZg==}
889 + cpu: [x64]
890 + os: [win32]
891 +
892 + '@rollup/rollup-win32-x64-msvc@4.63.1':
893 + resolution: {integrity: sha512-at8QVep6S3h5Y6gSbdGU06bRY5WJkf6WUduM9YtvYMbYhB1MOFfUgc6kehitQXzOtMSaT70q7f9ydPhpqu821w==}
894 + cpu: [x64]
895 + os: [win32]
896 +
897 + '@swc/helpers@0.5.23':
898 + resolution: {integrity: sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==}
899 +
900 + '@tailwindcss/node@4.3.3':
901 + resolution: {integrity: sha512-/T8IKEsf9VTU6tLjgC7+sv2mOPtQxzE2jMw7u4Tt40Tx+QSZxpzh95/H6cMKoja9XuW7iMdLJYBB0o9G1CaAgg==}
902 +
903 + '@tailwindcss/oxide-android-arm64@4.3.3':
904 + resolution: {integrity: sha512-Y85A2gmPSkl5Ve5qR86GL4HT509cFqQh1aes9p3sSkyTPwt0Pppf3GkwGe4JPACcRYjgJIEhQgM6dBClnr0NYw==}
905 + engines: {node: '>= 20'}
906 + cpu: [arm64]
907 + os: [android]
908 +
909 + '@tailwindcss/oxide-darwin-arm64@4.3.3':
910 + resolution: {integrity: sha512-BiaWatpBcERQFDlOjRDpIVXuFK5PJez5SA4JMg6VYZdBYU+qKfV/vqjcIs+IYmtitf1xYQZTwXvU/8y4lfZUGw==}
911 + engines: {node: '>= 20'}
912 + cpu: [arm64]
913 + os: [darwin]
914 +
915 + '@tailwindcss/oxide-darwin-x64@4.3.3':
916 + resolution: {integrity: sha512-fAeUqfV5ndhxRwai8cXGzdLvul9utWOmeTkv69unv4ZXixjn61Z+p9lCWdwOwA3TYboG3BwdVuN/RDjhBRl0mw==}
917 + engines: {node: '>= 20'}
918 + cpu: [x64]
919 + os: [darwin]
920 +
921 + '@tailwindcss/oxide-freebsd-x64@4.3.3':
922 + resolution: {integrity: sha512-iyf5bV6+wnAlflVeEy7R25dupxTNECZN5QMI0qNT6eT+EgaGdZcKhGkr5SdoaWiLJ3spLqIY9VCeSGrwmtg4kw==}
923 + engines: {node: '>= 20'}
924 + cpu: [x64]
925 + os: [freebsd]
926 +
927 + '@tailwindcss/oxide-linux-arm-gnueabihf@4.3.3':
928 + resolution: {integrity: sha512-aAYUprJAJQWWbRrPvtjdroZ56Md+JM8pMiopS6xGEwDfLhqj+2ver2p4nU4Mb3CRqcMmNBjo8KkUgcxhkzVQGQ==}
929 + engines: {node: '>= 20'}
930 + cpu: [arm]
931 + os: [linux]
932 +
933 + '@tailwindcss/oxide-linux-arm64-gnu@4.3.3':
934 + resolution: {integrity: sha512-nDxldcEENOxZRzC2uu9jrutZdAAQtb+8WWDCSnWL1zvBk1+FN+x6MtDViPB5AJMfttVCUhehGWus3XBPgatM/w==}
935 + engines: {node: '>= 20'}
936 + cpu: [arm64]
937 + os: [linux]
938 + libc: [glibc]
939 +
940 + '@tailwindcss/oxide-linux-arm64-musl@4.3.3':
941 + resolution: {integrity: sha512-Md44bD6veX/PC5iyF8cDVnw4HBIANZepRZZ7a8DQOvkfo5WUBwcp6iAuCUz23u+4SUkhJlD3eL7hNdW8ezd/kA==}
942 + engines: {node: '>= 20'}
943 + cpu: [arm64]
944 + os: [linux]
945 + libc: [musl]
946 +
947 + '@tailwindcss/oxide-linux-x64-gnu@4.3.3':
948 + resolution: {integrity: sha512-tx7us1muwOKAKWao2v/GaafFeQboE6aj88vC6ziN2NCGcRm8gWUhwjzg+YdVB1e4boAtdtma4L43onunI6NS4w==}
949 + engines: {node: '>= 20'}
950 + cpu: [x64]
951 + os: [linux]
952 + libc: [glibc]
953 +
954 + '@tailwindcss/oxide-linux-x64-musl@4.3.3':
955 + resolution: {integrity: sha512-SJxX60smvHgasZoBy11dX6YRjXJFovwWBoedhbQPOBzgFWBHGB+TVPWB9BxzR7TTxU8FQZAI2AyiNCMzFm8Img==}
956 + engines: {node: '>= 20'}
957 + cpu: [x64]
958 + os: [linux]
959 + libc: [musl]
960 +
961 + '@tailwindcss/oxide-wasm32-wasi@4.3.3':
962 + resolution: {integrity: sha512-jx1+rPhY/5Ympkktd656HBWEBLxP7dH06losBLjjf5vgCODXvi9KhtftWcMIwTFIDqBr7cRnQkdLnAG+IOlGvQ==}
963 + engines: {node: '>=14.0.0'}
964 + cpu: [wasm32]
965 + bundledDependencies:
966 + - '@napi-rs/wasm-runtime'
967 + - '@emnapi/core'
968 + - '@emnapi/runtime'
969 + - '@tybys/wasm-util'
970 + - '@emnapi/wasi-threads'
971 + - tslib
972 +
973 + '@tailwindcss/oxide-win32-arm64-msvc@4.3.3':
974 + resolution: {integrity: sha512-3rc292Ca2ceK6Ulcc/bAVnTs/3nDtoPhyEKlgPv+yQJQi/JS/AMJlqzxvlDacL1nekbrcf6bTqp/jV4qgnPxNQ==}
975 + engines: {node: '>= 20'}
976 + cpu: [arm64]
977 + os: [win32]
978 +
979 + '@tailwindcss/oxide-win32-x64-msvc@4.3.3':
980 + resolution: {integrity: sha512-yJ0pwIVc/nYeGoV02WtsN8KYyLQv7kyI2wDnkezyJlGGjkd4QLwDGAwl47YpPJeuI0M0ObaXGSPjvWDPeTPggw==}
981 + engines: {node: '>= 20'}
982 + cpu: [x64]
983 + os: [win32]
984 +
985 + '@tailwindcss/oxide@4.3.3':
986 + resolution: {integrity: sha512-krXjAikiaFSPaK/FkAQT5UTx3VormQaiZ5hBFlJZ9UFQGB/rwg1MZIhHAG9smMQRTdyJxP6Qt5MwMtdyU5FWrA==}
987 + engines: {node: '>= 20'}
988 +
989 + '@tailwindcss/postcss@4.3.3':
990 + resolution: {integrity: sha512-JTSZZGQi1AyKirbLN3azmjVzef92tcX7h+iSqPdaeStyFpGpDlKvvpxeOE8njhbUanbRwr3z8DyzhICWnMtQeg==}
991 +
992 + '@types/chai@5.2.3':
993 + resolution: {integrity: sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==}
994 +
995 + '@types/d3-geo@3.1.1':
996 + resolution: {integrity: sha512-65Emv9fQiQQqphLlRkuQ5ypPsOmWPhtBGCMv61JDPEPMvsx+gzhGf74yw1a78xFKPj6zw4AgQICJoQv0vK9M2w==}
997 +
998 + '@types/deep-eql@4.0.2':
999 + resolution: {integrity: sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==}
1000 +
1001 + '@types/estree@1.0.9':
1002 + resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
1003 +
1004 + '@types/geojson@7946.0.16':
1005 + resolution: {integrity: sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg==}
1006 +
1007 + '@types/node@24.13.4':
1008 + resolution: {integrity: sha512-YJ7EqCstVTzIr0fMr7qul/977en+pQHrfmuKIo6Zr9i75Be21dr3MovcfvGtyvi2HAUrRerWps5sMO9I7WaxDw==}
1009 +
1010 + '@types/pg@8.23.1':
1011 + resolution: {integrity: sha512-fKVHpikPdg4GKks3JuLEhvwSyvwzF23hnabPy6DD8ljVbC7+6J5dQzdv4arV6jqq57djnMgs1HKBxX4P8aBI3A==}
1012 +
1013 + '@types/react-dom@19.3.0':
1014 + resolution: {integrity: sha512-ZI7bU42mZXXKHn/qNLEw2IrbiINU7X5+vfgdixBHkCNpYWXjKgfQ/P+uyGb5CjOLB9UcnTeg3rylQtV2hym44Q==}
1015 + peerDependencies:
1016 + '@types/react': ^19.3.0
1017 +
1018 + '@types/react@19.3.0':
1019 + resolution: {integrity: sha512-N0rFCuH9YoxG9/m61l9MfpJKfmLOVU0em7ipIz6TRgSSkvReLB9vL85GB+yr8Bs5leqpvg96JSwF4ZS1s4viQg==}
1020 +
1021 + '@types/topojson-client@3.1.5':
1022 + resolution: {integrity: sha512-C79rySTyPxnQNNguTZNI1Ct4D7IXgvyAs3p9HPecnl6mNrJ5+UhvGNYcZfpROYV2lMHI48kJPxwR+F9C6c7nmw==}
1023 +
1024 + '@types/topojson-specification@1.0.5':
1025 + resolution: {integrity: sha512-C7KvcQh+C2nr6Y2Ub4YfgvWvWCgP2nOQMtfhlnwsRL4pYmmwzBS7HclGiS87eQfDOU/DLQpX6GEscviaz4yLIQ==}
1026 +
1027 + '@types/ws@8.18.1':
1028 + resolution: {integrity: sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg==}
1029 +
1030 + '@vitest/expect@3.2.7':
1031 + resolution: {integrity: sha512-E8eBXaKibuvH2pSZErOjdVb5vF4PbKYcrnluBTYxEk1l/VhhwZg1kZQsdtjq+CsF5CFydf2Rdkz7jDHKSisi3w==}
1032 +
1033 + '@vitest/mocker@3.2.7':
1034 + resolution: {integrity: sha512-Trr0hYO9CM3Wj6ksWHRhK9IZpIY6wTMO5u/MqXurMxT57sWBaOPEtP3Oq60ihZuh5JsiagKfz95OcxdEP6dBrA==}
1035 + peerDependencies:
1036 + msw: ^2.4.9
1037 + vite: ^5.0.0 || ^6.0.0 || ^7.0.0-0
1038 + peerDependenciesMeta:
1039 + msw:
1040 + optional: true
1041 + vite:
1042 + optional: true
1043 +
1044 + '@vitest/pretty-format@3.2.7':
1045 + resolution: {integrity: sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA==}
1046 +
1047 + '@vitest/runner@3.2.7':
1048 + resolution: {integrity: sha512-sB9y4ovltoQP+WaUPwmSxO9WIg9Ig694Di5PalVPsYHklAdE027mehpWF2SQSVq+k6sFgaivbTjTJwZLSHbedA==}
1049 +
1050 + '@vitest/snapshot@3.2.7':
1051 + resolution: {integrity: sha512-7C+MwShwtBSI5Buwoyg3s/iY1eHL9PKAf+O1wVh/TdnjXUtkoL/9YQtre90i4MtNXM6edP1wJ2zOBpfCyhIS7g==}
1052 +
1053 + '@vitest/spy@3.2.7':
1054 + resolution: {integrity: sha512-Q2eQGI6d2L/hBtZ0qNuKcAGid68XK6cv1xsoaIma6PaJhHPoqcEJhYpXZ/5myCMqkNgtP6UKuBhbc0nHKnrkuQ==}
1055 +
1056 + '@vitest/utils@3.2.7':
1057 + resolution: {integrity: sha512-x6BDOd7dyo3PFLY3I9/HJ25X/6OurhGXk2/B9gOZNPF7XDVjeBK4k01lQE5uvDpbuheErh91qYuE1E2OEjK3Rw==}
1058 +
1059 + abstract-logging@2.0.1:
1060 + resolution: {integrity: sha512-2BjRTZxTPvheOvGbBslFSYOUkr+SjPtOnrLP33f+VIWLzezQpZcqVg7ja3L4dBXmzzgwT+a029jRx5PCi3JuiA==}
1061 +
1062 + ajv-formats@3.0.1:
1063 + resolution: {integrity: sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==}
1064 + peerDependencies:
1065 + ajv: ^8.0.0
1066 + peerDependenciesMeta:
1067 + ajv:
1068 + optional: true
1069 +
1070 + ajv@8.20.0:
1071 + resolution: {integrity: sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==}
1072 +
1073 + anynum@1.0.1:
1074 + resolution: {integrity: sha512-N6//FLET/tXYNM/F6ABca1oH6fWB+KlTt909Le28WMDBk8oaT4vY17DCrwg2MvmuqUKt3Ni4N5dGJ/EoBgcO6A==}
1075 +
1076 + assertion-error@2.0.1:
1077 + resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
1078 + engines: {node: '>=12'}
1079 +
1080 + atomic-sleep@1.0.0:
1081 + resolution: {integrity: sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==}
1082 + engines: {node: '>=8.0.0'}
1083 +
1084 + avvio@9.3.0:
1085 + resolution: {integrity: sha512-g2tQ7LE7oOSqDfwEm3M+ZCMTJc7KiZCdJ4UwyZJb5ckTKyYu50OYmvv0mCFXPuYXoM4zkSt8zM9XQ9KCvxA74A==}
1086 +
1087 + baseline-browser-mapping@2.11.22:
1088 + resolution: {integrity: sha512-pWc4w51fBFd7mav43/zKRC+RI6f4yfzQoVlfvE8dECePyfkn1bzLp01Fj0QACcyCZyFhiEMyD2qScfKRWgWibA==}
1089 + engines: {node: '>=6.0.0'}
1090 + hasBin: true
1091 +
1092 + cac@6.7.14:
1093 + resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
1094 + engines: {node: '>=8'}
1095 +
1096 + caniuse-lite@1.0.30001810:
1097 + resolution: {integrity: sha512-TITQPUkaz+aVk5GL6NhOdwk1aEaNTSDPsGFWrTuhKGtjTF70jL/Oht2W4c6rXUe5fu7Ie19VIahAXHIIiWWNeg==}
1098 +
1099 + chai@5.3.3:
1100 + resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
1101 + engines: {node: '>=18'}
1102 +
1103 + check-error@2.1.3:
1104 + resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
1105 + engines: {node: '>= 16'}
1106 +
1107 + client-only@0.0.1:
1108 + resolution: {integrity: sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA==}
1109 +
1110 + commander@2.20.3:
1111 + resolution: {integrity: sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ==}
1112 +
1113 + cookie@1.1.1:
1114 + resolution: {integrity: sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==}
1115 + engines: {node: '>=18'}
1116 +
1117 + csstype@3.2.3:
1118 + resolution: {integrity: sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==}
1119 +
1120 + d3-array@3.2.4:
1121 + resolution: {integrity: sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==}
1122 + engines: {node: '>=12'}
1123 +
1124 + d3-geo@3.1.1:
1125 + resolution: {integrity: sha512-637ln3gXKXOwhalDzinUgY83KzNWZRKbYubaG+fGVuc/dxO64RRljtCTnf5ecMyE1RIdtqpkVcq0IbtU2S8j2Q==}
1126 + engines: {node: '>=12'}
1127 +
1128 + debug@4.4.3:
1129 + resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
1130 + engines: {node: '>=6.0'}
1131 + peerDependencies:
1132 + supports-color: '*'
1133 + peerDependenciesMeta:
1134 + supports-color:
1135 + optional: true
1136 +
1137 + deep-eql@5.0.2:
1138 + resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
1139 + engines: {node: '>=6'}
1140 +
1141 + dequal@2.0.3:
1142 + resolution: {integrity: sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==}
1143 + engines: {node: '>=6'}
1144 +
1145 + detect-libc@2.1.2:
1146 + resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==}
1147 + engines: {node: '>=8'}
1148 +
1149 + duplexify@4.1.3:
1150 + resolution: {integrity: sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA==}
1151 +
1152 + end-of-stream@1.4.5:
1153 + resolution: {integrity: sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==}
1154 +
1155 + enhanced-resolve@5.24.5:
1156 + resolution: {integrity: sha512-L1l8TNvomm6UVW5B253AGxQagSQr+vGwhMlrrfRS2qmhx46AMpMVJKQYLvWYbysTMY8VoicOvzHzoHMbyzB+4A==}
1157 + engines: {node: '>=10.13.0'}
1158 +
1159 + es-module-lexer@1.7.0:
1160 + resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
1161 +
1162 + esbuild@0.25.12:
1163 + resolution: {integrity: sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==}
1164 + engines: {node: '>=18'}
1165 + hasBin: true
1166 +
1167 + esbuild@0.28.2:
1168 + resolution: {integrity: sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==}
1169 + engines: {node: '>=18'}
1170 + hasBin: true
1171 +
1172 + estree-walker@3.0.3:
1173 + resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
1174 +
1175 + expect-type@1.4.0:
1176 + resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
1177 + engines: {node: '>=12.0.0'}
1178 +
1179 + fancy-canvas@2.1.0:
1180 + resolution: {integrity: sha512-nifxXJ95JNLFR2NgRV4/MxVP45G9909wJTEKz5fg/TZS20JJZA6hfgRVh/bC9bwl2zBtBNcYPjiBE4njQHVBwQ==}
1181 +
1182 + fast-content-type-parse@3.0.0:
1183 + resolution: {integrity: sha512-ZvLdcY8P+N8mGQJahJV5G4U88CSvT1rP8ApL6uETe88MBXrBHAkZlSEySdUlyztF7ccb+Znos3TFqaepHxdhBg==}
1184 +
1185 + fast-decode-uri-component@1.0.1:
1186 + resolution: {integrity: sha512-WKgKWg5eUxvRZGwW8FvfbaH7AXSh2cL+3j5fMGzUMCxWBJ3dV3a7Wz8y2f/uQ0e3B6WmodD3oS54jTQ9HVTIIg==}
1187 +
1188 + fast-deep-equal@3.1.3:
1189 + resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==}
1190 +
1191 + fast-json-stringify@7.0.1:
1192 + resolution: {integrity: sha512-eRSayARSbbwlBjpP4vnTTIRD5QPcIrmihPxDeN1DtKnHPg66UuJLx+8hlK1kaFdjvzyQ/dzALoi4vwAQ+T+iZA==}
1193 +
1194 + fast-querystring@1.1.2:
1195 + resolution: {integrity: sha512-g6KuKWmFXc0fID8WWH0jit4g0AGBoJhCkJMb1RmbsSEUNvQ+ZC8D6CUZ+GtF8nMzSPXnhiePyyqqipzNNEnHjg==}
1196 +
1197 + fast-uri@3.1.7:
1198 + resolution: {integrity: sha512-dOvZVzjdZdz7phd9v6jCbwxrBW3fK6n8Rc0CtdmM4bumzMnxywBYhuph6J819RRw/ku+rLbelwfMunktuzVVHg==}
1199 +
1200 + fast-uri@4.1.4:
1201 + resolution: {integrity: sha512-dODXrIxlS9JSdgAnhIUKOosKV1oMtU2VtVw87QRaHzyl5jxO290Ii5tEZfCfzfWNHi3jKWwBSdQj0qIyshdZdQ==}
1202 +
1203 + fast-xml-builder@1.3.1:
1204 + resolution: {integrity: sha512-pIM/1n3ntFXKYrUZwW7QCK0gAW7XY+wzj1YMIV3tLDvPj/V+zTGJK5e3/4WJfwj0qWw2ElNXiTixda/R+3YSug==}
1205 +
1206 + fast-xml-parser@5.11.1:
1207 + resolution: {integrity: sha512-TBw6K/fxoQGGjCmZDw9w/ZwP3uDcnTM4YH/g+PFRWr8sbe5idXtxNN6vITh4+1ruCZaho6uBFurElsA7F0zzgw==}
1208 + hasBin: true
1209 +
1210 + fastify-plugin@6.0.0:
1211 + resolution: {integrity: sha512-fZOty7z3O7vOliF6d8bHE3wiEh1KcNnKEQensSgTk9C1DvN6nRLS++XVd86v33Hw/8u9Un8A1zDrQ8ujcQDHEg==}
1212 +
1213 + fastify@5.12.3:
1214 + resolution: {integrity: sha512-reZ8wce5VNCcufIt9AVtzZa3L4u1j8esikn7OEgHWLVpRpL5R7Y2+Xzj70OUkv5zDfzUAxXZT6cu4Rt0zr3EKA==}
1215 +
1216 + fastq@1.20.3:
1217 + resolution: {integrity: sha512-XKv5nnLs6nLF71NgiKJLIZFLkPyIEuOselLG7ujZnGrRfQK8HpvY+WqKhAJUAdLomwVHErVS4LfxFlPq0/FTAw==}
1218 +
1219 + fdir@6.5.0:
1220 + resolution: {integrity: sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==}
1221 + engines: {node: '>=12.0.0'}
1222 + peerDependencies:
1223 + picomatch: ^3 || ^4
1224 + peerDependenciesMeta:
1225 + picomatch:
1226 + optional: true
1227 +
1228 + find-my-way@9.9.0:
1229 + resolution: {integrity: sha512-sJsgZ1sQH2UDuowPuMKg8az7Qc8F0jnj+SKkFWU/+T0xcFlgV5skgXOGUqmQzOdmW6ALA7AhJINWx3qFBkbLHA==}
1230 + engines: {node: '>=20'}
1231 +
1232 + fsevents@2.3.3:
1233 + resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
1234 + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
1235 + os: [darwin]
1236 +
1237 + geist@1.7.2:
1238 + resolution: {integrity: sha512-Gu5lDFa3pLRyoBlBPf0QIFHVdWAnpco7fS1bJm41jyLPFoguBgiubseUN2oLXMgqZ7uxAxDoXcHMhCY/fOTTgg==}
1239 + peerDependencies:
1240 + next: '>=13.2.0'
1241 +
1242 + graceful-fs@4.2.11:
1243 + resolution: {integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==}
1244 +
1245 + inherits@2.0.4:
1246 + resolution: {integrity: sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==}
1247 +
1248 + internmap@2.0.3:
1249 + resolution: {integrity: sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==}
1250 + engines: {node: '>=12'}
1251 +
1252 + ipaddr.js@2.5.0:
1253 + resolution: {integrity: sha512-aq+t5NAc+cS6rZQQVWC2x98CPqGtKKTMDd4Gaodv0wShnItdKg/51djkGJ1hqH+Oy0ivDftCbSLCQob8zso01w==}
1254 + engines: {node: '>= 10'}
1255 +
1256 + is-unsafe@2.0.2:
1257 + resolution: {integrity: sha512-HgbIHPBH0KHHCcjLfGsCvhtPTVxjaAZlXjwdz7/GQC40SjSe4sfQsar8J5VFo8JOSbarkpV0OLG95bbaNd9aAQ==}
1258 +
1259 + jiti@2.7.0:
1260 + resolution: {integrity: sha512-AC/7JofJvZGrrneWNaEnJeOLUx+JlGt7tNa0wZiRPT4MY1wmfKjt2+6O2p2uz2+skll8OZZmJMNqeke7kKbNgQ==}
1261 + hasBin: true
1262 +
1263 + js-tokens@9.0.1:
1264 + resolution: {integrity: sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==}
1265 +
1266 + json-schema-ref-resolver@3.0.0:
1267 + resolution: {integrity: sha512-hOrZIVL5jyYFjzk7+y7n5JDzGlU8rfWDuYyHwGa2WA8/pcmMHezp2xsVwxrebD/Q9t8Nc5DboieySDpCp4WG4A==}
1268 +
1269 + json-schema-traverse@1.0.0:
1270 + resolution: {integrity: sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==}
1271 +
1272 + light-my-request@6.6.0:
1273 + resolution: {integrity: sha512-CHYbu8RtboSIoVsHZ6Ye4cj4Aw/yg2oAFimlF7mNvfDV192LR7nDiKtSIfCuLT7KokPSTn/9kfVLm5OGN0A28A==}
1274 +
1275 + lightningcss-android-arm64@1.32.0:
1276 + resolution: {integrity: sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg==}
1277 + engines: {node: '>= 12.0.0'}
1278 + cpu: [arm64]
1279 + os: [android]
1280 +
1281 + lightningcss-darwin-arm64@1.32.0:
1282 + resolution: {integrity: sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ==}
1283 + engines: {node: '>= 12.0.0'}
1284 + cpu: [arm64]
1285 + os: [darwin]
1286 +
1287 + lightningcss-darwin-x64@1.32.0:
1288 + resolution: {integrity: sha512-U+QsBp2m/s2wqpUYT/6wnlagdZbtZdndSmut/NJqlCcMLTWp5muCrID+K5UJ6jqD2BFshejCYXniPDbNh73V8w==}
1289 + engines: {node: '>= 12.0.0'}
1290 + cpu: [x64]
1291 + os: [darwin]
1292 +
1293 + lightningcss-freebsd-x64@1.32.0:
1294 + resolution: {integrity: sha512-JCTigedEksZk3tHTTthnMdVfGf61Fky8Ji2E4YjUTEQX14xiy/lTzXnu1vwiZe3bYe0q+SpsSH/CTeDXK6WHig==}
1295 + engines: {node: '>= 12.0.0'}
1296 + cpu: [x64]
1297 + os: [freebsd]
1298 +
1299 + lightningcss-linux-arm-gnueabihf@1.32.0:
1300 + resolution: {integrity: sha512-x6rnnpRa2GL0zQOkt6rts3YDPzduLpWvwAF6EMhXFVZXD4tPrBkEFqzGowzCsIWsPjqSK+tyNEODUBXeeVHSkw==}
1301 + engines: {node: '>= 12.0.0'}
1302 + cpu: [arm]
1303 + os: [linux]
1304 +
1305 + lightningcss-linux-arm64-gnu@1.32.0:
1306 + resolution: {integrity: sha512-0nnMyoyOLRJXfbMOilaSRcLH3Jw5z9HDNGfT/gwCPgaDjnx0i8w7vBzFLFR1f6CMLKF8gVbebmkUN3fa/kQJpQ==}
1307 + engines: {node: '>= 12.0.0'}
1308 + cpu: [arm64]
1309 + os: [linux]
1310 + libc: [glibc]
1311 +
1312 + lightningcss-linux-arm64-musl@1.32.0:
1313 + resolution: {integrity: sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg==}
1314 + engines: {node: '>= 12.0.0'}
1315 + cpu: [arm64]
1316 + os: [linux]
1317 + libc: [musl]
1318 +
1319 + lightningcss-linux-x64-gnu@1.32.0:
1320 + resolution: {integrity: sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA==}
1321 + engines: {node: '>= 12.0.0'}
1322 + cpu: [x64]
1323 + os: [linux]
1324 + libc: [glibc]
1325 +
1326 + lightningcss-linux-x64-musl@1.32.0:
1327 + resolution: {integrity: sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg==}
1328 + engines: {node: '>= 12.0.0'}
1329 + cpu: [x64]
1330 + os: [linux]
1331 + libc: [musl]
1332 +
1333 + lightningcss-win32-arm64-msvc@1.32.0:
1334 + resolution: {integrity: sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw==}
1335 + engines: {node: '>= 12.0.0'}
1336 + cpu: [arm64]
1337 + os: [win32]
1338 +
1339 + lightningcss-win32-x64-msvc@1.32.0:
1340 + resolution: {integrity: sha512-Amq9B/SoZYdDi1kFrojnoqPLxYhQ4Wo5XiL8EVJrVsB8ARoC1PWW6VGtT0WKCemjy8aC+louJnjS7U18x3b06Q==}
1341 + engines: {node: '>= 12.0.0'}
1342 + cpu: [x64]
1343 + os: [win32]
1344 +
1345 + lightningcss@1.32.0:
1346 + resolution: {integrity: sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ==}
1347 + engines: {node: '>= 12.0.0'}
1348 +
1349 + lightweight-charts@5.2.1:
1350 + resolution: {integrity: sha512-IVwoK1RLFiLPubaKIjNbtjWLnpPMqiABSrTay6whmNa8L1+19292VtHJ+BWyPUuLCwF0tcQlhEWd1CLB2a1nsQ==}
1351 +
1352 + loupe@3.2.1:
1353 + resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
1354 +
1355 + lucide-react@1.44.0:
1356 + resolution: {integrity: sha512-2egNApH4hX4j/qdCgRublh88+9u3mEhz9iSlW5ckm4kaQEqZbXbMr0l5u5JZLy8nmWRx2dbHGQkEDYz6C9aCgw==}
1357 + peerDependencies:
1358 + react: ^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0
1359 +
1360 + magic-string@0.30.21:
1361 + resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
1362 +
1363 + ms@2.1.3:
1364 + resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
1365 +
1366 + nanoid@3.3.19:
1367 + resolution: {integrity: sha512-Y2tUNy4ouw6tq5oDSKeQYGOyhkUBhNOcGV/02KC+6kd9eDGqdZd++mjMiIDilrBYvjEnCYvVtsuHCuP+okSfug==}
1368 + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
1369 + hasBin: true
1370 +
1371 + next@16.3.4:
1372 + resolution: {integrity: sha512-/Ztf6CeRH+ejEXUrYtqI4gkS66eFIHuSwqi60RgcpWKodxFZx2/dqVCMKBwILfAHXQ+F1b1vAudgj3mnxqtoIA==}
1373 + engines: {node: '>=20.9.0'}
1374 + hasBin: true
1375 + peerDependencies:
1376 + '@opentelemetry/api': ^1.1.0
1377 + '@playwright/test': ^1.51.1
1378 + babel-plugin-react-compiler: '*'
1379 + react: ^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0
1380 + react-dom: ^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0
1381 + sass: ^1.3.0
1382 + peerDependenciesMeta:
1383 + '@opentelemetry/api':
1384 + optional: true
1385 + '@playwright/test':
1386 + optional: true
1387 + babel-plugin-react-compiler:
1388 + optional: true
1389 + sass:
1390 + optional: true
1391 +
1392 + on-exit-leak-free@2.1.2:
1393 + resolution: {integrity: sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==}
1394 + engines: {node: '>=14.0.0'}
1395 +
1396 + once@1.4.0:
1397 + resolution: {integrity: sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==}
1398 +
1399 + path-expression-matcher@1.6.2:
1400 + resolution: {integrity: sha512-enSlaiat05iasnzmgNxRj8reFdj3puY2QpNgP1aPIaVfT6nn9ICuPoFlKHk8EN22HcwewshO+mN2DGbkCEOtqQ==}
1401 + engines: {node: '>=14.0.0'}
1402 +
1403 + pathe@2.0.3:
1404 + resolution: {integrity: sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==}
1405 +
1406 + pathval@2.0.1:
1407 + resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
1408 + engines: {node: '>= 14.16'}
1409 +
1410 + pg-cloudflare@1.4.0:
1411 + resolution: {integrity: sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==}
1412 +
1413 + pg-connection-string@2.14.0:
1414 + resolution: {integrity: sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==}
1415 +
1416 + pg-int8@1.0.1:
1417 + resolution: {integrity: sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==}
1418 + engines: {node: '>=4.0.0'}
1419 +
1420 + pg-pool@3.14.0:
1421 + resolution: {integrity: sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==}
1422 + peerDependencies:
1423 + pg: '>=8.0'
1424 +
1425 + pg-protocol@1.16.0:
1426 + resolution: {integrity: sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==}
1427 +
1428 + pg-types@2.2.0:
1429 + resolution: {integrity: sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==}
1430 + engines: {node: '>=4'}
1431 +
1432 + pg@8.23.0:
1433 + resolution: {integrity: sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==}
1434 + engines: {node: '>= 16.0.0'}
1435 + peerDependencies:
1436 + pg-native: '>=3.0.1'
1437 + peerDependenciesMeta:
1438 + pg-native:
1439 + optional: true
1440 +
1441 + pgpass@1.0.5:
1442 + resolution: {integrity: sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==}
1443 +
1444 + picocolors@1.1.1:
1445 + resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
1446 +
1447 + picomatch@4.0.7:
1448 + resolution: {integrity: sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==}
1449 + engines: {node: '>=12'}
1450 +
1451 + pino-abstract-transport@2.0.0:
1452 + resolution: {integrity: sha512-F63x5tizV6WCh4R6RHyi2Ml+M70DNRXt/+HANowMflpgGFMAym/VKm6G7ZOQRjqN7XbGxK1Lg9t6ZrtzOaivMw==}
1453 +
1454 + pino-std-serializers@7.1.0:
1455 + resolution: {integrity: sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw==}
1456 +
1457 + pino@9.14.0:
1458 + resolution: {integrity: sha512-8OEwKp5juEvb/MjpIc4hjqfgCNysrS94RIOMXYvpYCdm/jglrKEiAYmiumbmGhCvs+IcInsphYDFwqrjr7398w==}
1459 + hasBin: true
1460 +
1461 + postcss@8.5.23:
1462 + resolution: {integrity: sha512-g50586zr4bZmwFiTlflMu8E0bDTb5I5gertgwAKmsdUlTQIhZtunzUlD1WSzwcVWPoAVpsrA6vlfCD7oXvRwgg==}
1463 + engines: {node: ^10 || ^12 || >=14}
1464 +
1465 + postcss@8.5.28:
1466 + resolution: {integrity: sha512-RRuzqDtt5Y9h3quz5hWhK+TPnsmVs6WwSU6LkJMeY4HstUEDuYTG8UJSdawMRzmzAtV+KEoG8N3Qg2qLy5vM/A==}
1467 + engines: {node: ^10 || ^12 || >=14}
1468 +
1469 + postgres-array@2.0.0:
1470 + resolution: {integrity: sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==}
1471 + engines: {node: '>=4'}
1472 +
1473 + postgres-bytea@1.0.1:
1474 + resolution: {integrity: sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==}
1475 + engines: {node: '>=0.10.0'}
1476 +
1477 + postgres-date@1.0.7:
1478 + resolution: {integrity: sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==}
1479 + engines: {node: '>=0.10.0'}
1480 +
1481 + postgres-interval@1.2.0:
1482 + resolution: {integrity: sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==}
1483 + engines: {node: '>=0.10.0'}
1484 +
1485 + process-warning@4.0.1:
1486 + resolution: {integrity: sha512-3c2LzQ3rY9d0hc1emcsHhfT9Jwz0cChib/QN89oME2R451w5fy3f0afAhERFZAwrbDU43wk12d0ORBpDVME50Q==}
1487 +
1488 + process-warning@5.1.0:
1489 + resolution: {integrity: sha512-jQSaVHsPgtyw60e1rQ/A+/ArPEj/S8pS/vFnyGa/gYFXrKk/6RuDkoqVDQ5NI5MmS01698ltlAk0NoDBNLujRw==}
1490 +
1491 + quick-format-unescaped@4.0.4:
1492 + resolution: {integrity: sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==}
1493 +
1494 + react-dom@19.2.8:
1495 + resolution: {integrity: sha512-rVprimfGBG3DR+Tq0IQG2DT5PxKth1WIGDmj5yPmlzr4YBe7uyE+Du4oVqTDXZSHGGGXRtTJEGSSePyQCMBglQ==}
1496 + peerDependencies:
1497 + react: ^19.2.8
1498 +
1499 + react@19.2.8:
1500 + resolution: {integrity: sha512-PWaYA1L/q9u2u7xYQi+Y3L3Yfnie7XyLeaJICV1MGD6LprsBxcAqGjYyr0eY3p+QdsA+x/Irkt4Qif8D63+Sbw==}
1501 + engines: {node: '>=0.10.0'}
1502 +
1503 + readable-stream@3.6.2:
1504 + resolution: {integrity: sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==}
1505 + engines: {node: '>= 6'}
1506 +
1507 + real-require@0.2.0:
1508 + resolution: {integrity: sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==}
1509 + engines: {node: '>= 12.13.0'}
1510 +
1511 + require-from-string@2.0.2:
1512 + resolution: {integrity: sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==}
1513 + engines: {node: '>=0.10.0'}
1514 +
1515 + ret@0.5.0:
1516 + resolution: {integrity: sha512-I1XxrZSQ+oErkRR4jYbAyEEu2I0avBvvMM5JN+6EBprOGRCs63ENqZ3vjavq8fBw2+62G5LF5XelKwuJpcvcxw==}
1517 + engines: {node: '>=10'}
1518 +
1519 + reusify@1.1.0:
1520 + resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==}
1521 + engines: {iojs: '>=1.0.0', node: '>=0.10.0'}
1522 +
1523 + rfdc@1.4.1:
1524 + resolution: {integrity: sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==}
1525 +
1526 + rollup@4.63.1:
1527 + resolution: {integrity: sha512-3Df9jsstwhccuEfmAMi9l8XUh/GOkVObmFTU7CCVBysEbcOZLl84jCtaAZMcPiMz2EGKsATzQcU+Xr3n/wU6cg==}
1528 + engines: {node: '>=18.0.0', npm: '>=8.0.0'}
1529 + hasBin: true
1530 +
1531 + safe-buffer@5.2.1:
1532 + resolution: {integrity: sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==}
1533 +
1534 + safe-regex2@5.1.1:
1535 + resolution: {integrity: sha512-mOSBvHGDZMuIEZMdOz/aCEYDCv0E7nfcNsIhUF+/P+xC7Hyf3FkvymqgPbg9D1EdSGu+uKbJgy09K/RKKc7kJA==}
1536 + hasBin: true
1537 +
1538 + safe-stable-stringify@2.5.0:
1539 + resolution: {integrity: sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==}
1540 + engines: {node: '>=10'}
1541 +
1542 + scheduler@0.27.0:
1543 + resolution: {integrity: sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==}
1544 +
1545 + secure-json-parse@4.1.0:
1546 + resolution: {integrity: sha512-l4KnYfEyqYJxDwlNVyRfO2E4NTHfMKAWdUuA8J0yve2Dz/E/PdBepY03RvyJpssIpRFwJoCD55wA+mEDs6ByWA==}
1547 +
1548 + semver@7.8.5:
1549 + resolution: {integrity: sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==}
1550 + engines: {node: '>=10'}
1551 + hasBin: true
1552 +
1553 + server-only@0.0.1:
1554 + resolution: {integrity: sha512-qepMx2JxAa5jjfzxG79yPPq+8BuFToHd1hm7kI+Z4zAq1ftQiP7HcxMhDDItrbtwVeLg/cY2JnKnrcFkmiswNA==}
1555 +
1556 + set-cookie-parser@2.7.2:
1557 + resolution: {integrity: sha512-oeM1lpU/UvhTxw+g3cIfxXHyJRc/uidd3yK1P242gzHds0udQBYzs3y8j4gCCW+ZJ7ad0yctld8RYO+bdurlvw==}
1558 +
1559 + sharp@0.35.4:
1560 + resolution: {integrity: sha512-n++8XWcj+jCOr2IOl7h8LbKnGBDY4aPbmprMONBNFdn0ImXqpGVv5zliDs0V9HbmbCQLpbuo2ej9rAoOQTvMDA==}
1561 + engines: {node: '>=20.9.0'}
1562 + peerDependencies:
1563 + '@types/node': '*'
1564 + peerDependenciesMeta:
1565 + '@types/node':
1566 + optional: true
1567 +
1568 + siginfo@2.0.0:
1569 + resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
1570 +
1571 + sonic-boom@4.2.1:
1572 + resolution: {integrity: sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==}
1573 +
1574 + source-map-js@1.2.1:
1575 + resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==}
1576 + engines: {node: '>=0.10.0'}
1577 +
1578 + split2@4.2.0:
1579 + resolution: {integrity: sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==}
1580 + engines: {node: '>= 10.x'}
1581 +
1582 + stackback@0.0.2:
1583 + resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
1584 +
1585 + std-env@3.10.0:
1586 + resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
1587 +
1588 + stream-shift@1.0.3:
1589 + resolution: {integrity: sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ==}
1590 +
1591 + string_decoder@1.3.0:
1592 + resolution: {integrity: sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==}
1593 +
1594 + strip-literal@3.1.0:
1595 + resolution: {integrity: sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==}
1596 +
1597 + strnum@2.4.2:
1598 + resolution: {integrity: sha512-rDG3Ah4TV0k1hWvLSzkZtMmLN9+eS+h3knq4MP6A42Y3Yh5qGNnOUs1jJkoSr8FG5dsL28c7KgkIBzSEykqtuw==}
1599 +
1600 + styled-jsx@5.1.6:
1601 + resolution: {integrity: sha512-qSVyDTeMotdvQYoHWLNGwRFJHC+i+ZvdBRYosOFgC+Wg1vx4frN2/RG/NA7SYqqvKNLf39P2LSRA2pu6n0XYZA==}
1602 + engines: {node: '>= 12.0.0'}
1603 + peerDependencies:
1604 + '@babel/core': '*'
1605 + babel-plugin-macros: '*'
1606 + react: '>= 16.8.0 || 17.x.x || ^18.0.0-0 || ^19.0.0-0'
1607 + peerDependenciesMeta:
1608 + '@babel/core':
1609 + optional: true
1610 + babel-plugin-macros:
1611 + optional: true
1612 +
1613 + tailwindcss@4.3.3:
1614 + resolution: {integrity: sha512-gOhV3P7ufE62QDGg1zVaTgCR+EtPv92k2nIhVcVKcLmxT1sUBsQGhnZj175j+MqRt4zLF7ic+sCYjfhxMxj7YQ==}
1615 +
1616 + tapable@2.3.3:
1617 + resolution: {integrity: sha512-uxc/zpqFg6x7C8vOE7lh6Lbda8eEL9zmVm/PLeTPBRhh1xCgdWaQ+J1CUieGpIfm2HdtsUpRv+HshiasBMcc6A==}
1618 + engines: {node: '>=6'}
1619 +
1620 + thread-stream@3.2.0:
1621 + resolution: {integrity: sha512-zLBvqpwr4Esa0kRjcrzGU6zL25lePWaCLMx0RQFrmteozIfeNdaMLpG5U7PeHzvlFkAWaRKA9/KVW4F60iB+qw==}
1622 +
1623 + tinybench@2.9.0:
1624 + resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
1625 +
1626 + tinyexec@0.3.2:
1627 + resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
1628 +
1629 + tinyglobby@0.2.17:
1630 + resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==}
1631 + engines: {node: '>=12.0.0'}
1632 +
1633 + tinypool@1.1.1:
1634 + resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
1635 + engines: {node: ^18.0.0 || >=20.0.0}
1636 +
1637 + tinyrainbow@2.0.0:
1638 + resolution: {integrity: sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==}
1639 + engines: {node: '>=14.0.0'}
1640 +
1641 + tinyspy@4.0.6:
1642 + resolution: {integrity: sha512-u8KszXvGfU68hVcZpRHKG28T0krMuv2G5nDhiHaMLen/gIuFEgIJhaJuO69qjnXg5paSrbPMFfx3brNuN8eVSg==}
1643 + engines: {node: '>=14.0.0'}
1644 +
1645 + toad-cache@3.7.4:
1646 + resolution: {integrity: sha512-m1TdR/rvT7kgGJZhspNtXdsdYk0fddFpJJFlG5s+UkPFo6lkLoZ3YLOaovPYjq1R75NP5JfeTlSHaOsE09peCg==}
1647 + engines: {node: '>=20'}
1648 +
1649 + topojson-client@3.1.0:
1650 + resolution: {integrity: sha512-605uxS6bcYxGXw9qi62XyrV6Q3xwbndjachmNxu8HWTtVPxZfEJN9fd/SZS1Q54Sn2y0TMyMxFj/cJINqGHrKw==}
1651 + hasBin: true
1652 +
1653 + tslib@2.8.1:
1654 + resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==}
1655 +
1656 + tsx@4.23.13:
1657 + resolution: {integrity: sha512-BL5MGkRln6aDYhb0xbQlEAGw743BaZYWdbWtdJOBriYJboKgUUYCadFp2/FpBBZquBC/ezNBn7wMMPx7FDZUDw==}
1658 + engines: {node: '>=18.0.0'}
1659 + hasBin: true
1660 +
1661 + typescript@5.9.3:
1662 + resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
1663 + engines: {node: '>=14.17'}
1664 + hasBin: true
1665 +
1666 + undici-types@7.18.2:
1667 + resolution: {integrity: sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==}
1668 +
1669 + undici@7.29.1:
1670 + resolution: {integrity: sha512-RYONW2MeafgYlkVOKYKkA/Ag7BmXqgIWCa8t1m0JcxrQg9pI9lEqRhAOruOBCbAohOa/gkCF+iPi9hrgvTzu6Q==}
1671 + engines: {node: '>=20.18.1'}
1672 +
1673 + util-deprecate@1.0.2:
1674 + resolution: {integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==}
1675 +
1676 + vite-node@3.2.4:
1677 + resolution: {integrity: sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==}
1678 + engines: {node: ^18.0.0 || ^20.0.0 || >=22.0.0}
1679 + hasBin: true
1680 +
1681 + vite@7.3.6:
1682 + resolution: {integrity: sha512-4XP60spRGjSZFf1qYH+dJIkK2znL3zQfl9KkOV9MkkRR/3Dls0dxaBsQPTloEc5BLXWPL9vsOxopxyKoMmDueg==}
1683 + engines: {node: ^20.19.0 || >=22.12.0}
1684 + hasBin: true
1685 + peerDependencies:
1686 + '@types/node': ^20.19.0 || >=22.12.0
1687 + jiti: '>=1.21.0'
1688 + less: ^4.0.0
1689 + lightningcss: ^1.21.0
1690 + sass: ^1.70.0
1691 + sass-embedded: ^1.70.0
1692 + stylus: '>=0.54.8'
1693 + sugarss: ^5.0.0
1694 + terser: ^5.16.0
1695 + tsx: ^4.8.1
1696 + yaml: ^2.4.2
1697 + peerDependenciesMeta:
1698 + '@types/node':
1699 + optional: true
1700 + jiti:
1701 + optional: true
1702 + less:
1703 + optional: true
1704 + lightningcss:
1705 + optional: true
1706 + sass:
1707 + optional: true
1708 + sass-embedded:
1709 + optional: true
1710 + stylus:
1711 + optional: true
1712 + sugarss:
1713 + optional: true
1714 + terser:
1715 + optional: true
1716 + tsx:
1717 + optional: true
1718 + yaml:
1719 + optional: true
1720 +
1721 + vitest@3.2.7:
1722 + resolution: {integrity: sha512-KrxIJ62Fd89gfysR4WotlgZABiz2dqFPgqGzX7s+CwsqLFomRH7777ZcrOD6+WVAh7khPQP41A+BKbpcJFrdEg==}
1723 + engines: {node: ^18.0.0 || ^20.0.0 || >=22.0.0}
1724 + hasBin: true
1725 + peerDependencies:
1726 + '@edge-runtime/vm': '*'
1727 + '@types/debug': ^4.1.12
1728 + '@types/node': ^18.0.0 || ^20.0.0 || >=22.0.0
1729 + '@vitest/browser': 3.2.7
1730 + '@vitest/ui': 3.2.7
1731 + happy-dom: '*'
1732 + jsdom: '*'
1733 + peerDependenciesMeta:
1734 + '@edge-runtime/vm':
1735 + optional: true
1736 + '@types/debug':
1737 + optional: true
1738 + '@types/node':
1739 + optional: true
1740 + '@vitest/browser':
1741 + optional: true
1742 + '@vitest/ui':
1743 + optional: true
1744 + happy-dom:
1745 + optional: true
1746 + jsdom:
1747 + optional: true
1748 +
1749 + why-is-node-running@2.3.0:
1750 + resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
1751 + engines: {node: '>=8'}
1752 + hasBin: true
1753 +
1754 + world-atlas@2.0.2:
1755 + resolution: {integrity: sha512-IXfV0qwlKXpckz1FhwXVwKRjiIhOnWttOskm5CtxMsjgE/MXAYRHWJqgXOpM8IkcPBoXnyTU5lFHcYa5ChG0LQ==}
1756 +
1757 + wrappy@1.0.2:
1758 + resolution: {integrity: sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==}
1759 +
1760 + ws@8.21.3:
1761 + resolution: {integrity: sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==}
1762 + engines: {node: '>=10.0.0'}
1763 + peerDependencies:
1764 + bufferutil: ^4.0.1
1765 + utf-8-validate: '>=5.0.2'
1766 + peerDependenciesMeta:
1767 + bufferutil:
1768 + optional: true
1769 + utf-8-validate:
1770 + optional: true
1771 +
1772 + xml-naming@0.3.0:
1773 + resolution: {integrity: sha512-ghig2TBE/H11aOVgmahA3MhimvkBr6JIYknH/Dhdk10nXwdbIqBJsbfMxpvFPG8bAw77gN29aQWvKpmVoPlvPQ==}
1774 + engines: {node: '>=16.0.0'}
1775 +
1776 + xtend@4.0.2:
1777 + resolution: {integrity: sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==}
1778 + engines: {node: '>=0.4'}
1779 +
1780 + zod@3.25.76:
1781 + resolution: {integrity: sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==}
1782 +
1783 +snapshots:
1784 +
1785 + '@alloc/quick-lru@5.3.0': {}
1786 +
1787 + '@emnapi/runtime@1.11.3':
1788 + dependencies:
1789 + tslib: 2.8.1
1790 + optional: true
1791 +
1792 + '@esbuild/aix-ppc64@0.25.12':
1793 + optional: true
1794 +
1795 + '@esbuild/aix-ppc64@0.28.2':
1796 + optional: true
1797 +
1798 + '@esbuild/android-arm64@0.25.12':
1799 + optional: true
1800 +
1801 + '@esbuild/android-arm64@0.28.2':
1802 + optional: true
1803 +
1804 + '@esbuild/android-arm@0.25.12':
1805 + optional: true
1806 +
1807 + '@esbuild/android-arm@0.28.2':
1808 + optional: true
1809 +
1810 + '@esbuild/android-x64@0.25.12':
1811 + optional: true
1812 +
1813 + '@esbuild/android-x64@0.28.2':
1814 + optional: true
1815 +
1816 + '@esbuild/darwin-arm64@0.25.12':
1817 + optional: true
1818 +
1819 + '@esbuild/darwin-arm64@0.28.2':
1820 + optional: true
1821 +
1822 + '@esbuild/darwin-x64@0.25.12':
1823 + optional: true
1824 +
1825 + '@esbuild/darwin-x64@0.28.2':
1826 + optional: true
1827 +
1828 + '@esbuild/freebsd-arm64@0.25.12':
1829 + optional: true
1830 +
1831 + '@esbuild/freebsd-arm64@0.28.2':
1832 + optional: true
1833 +
1834 + '@esbuild/freebsd-x64@0.25.12':
1835 + optional: true
1836 +
1837 + '@esbuild/freebsd-x64@0.28.2':
1838 + optional: true
1839 +
1840 + '@esbuild/linux-arm64@0.25.12':
1841 + optional: true
1842 +
1843 + '@esbuild/linux-arm64@0.28.2':
1844 + optional: true
1845 +
1846 + '@esbuild/linux-arm@0.25.12':
1847 + optional: true
1848 +
1849 + '@esbuild/linux-arm@0.28.2':
1850 + optional: true
1851 +
1852 + '@esbuild/linux-ia32@0.25.12':
1853 + optional: true
1854 +
1855 + '@esbuild/linux-ia32@0.28.2':
1856 + optional: true
1857 +
1858 + '@esbuild/linux-loong64@0.25.12':
1859 + optional: true
1860 +
1861 + '@esbuild/linux-loong64@0.28.2':
1862 + optional: true
1863 +
1864 + '@esbuild/linux-mips64el@0.25.12':
1865 + optional: true
1866 +
1867 + '@esbuild/linux-mips64el@0.28.2':
1868 + optional: true
1869 +
1870 + '@esbuild/linux-ppc64@0.25.12':
1871 + optional: true
1872 +
1873 + '@esbuild/linux-ppc64@0.28.2':
1874 + optional: true
1875 +
1876 + '@esbuild/linux-riscv64@0.25.12':
1877 + optional: true
1878 +
1879 + '@esbuild/linux-riscv64@0.28.2':
1880 + optional: true
1881 +
1882 + '@esbuild/linux-s390x@0.25.12':
1883 + optional: true
1884 +
1885 + '@esbuild/linux-s390x@0.28.2':
1886 + optional: true
1887 +
1888 + '@esbuild/linux-x64@0.25.12':
1889 + optional: true
1890 +
1891 + '@esbuild/linux-x64@0.28.2':
1892 + optional: true
1893 +
1894 + '@esbuild/netbsd-arm64@0.25.12':
1895 + optional: true
1896 +
1897 + '@esbuild/netbsd-arm64@0.28.2':
1898 + optional: true
1899 +
1900 + '@esbuild/netbsd-x64@0.25.12':
1901 + optional: true
1902 +
1903 + '@esbuild/netbsd-x64@0.28.2':
1904 + optional: true
1905 +
1906 + '@esbuild/openbsd-arm64@0.25.12':
1907 + optional: true
1908 +
1909 + '@esbuild/openbsd-arm64@0.28.2':
1910 + optional: true
1911 +
1912 + '@esbuild/openbsd-x64@0.25.12':
1913 + optional: true
1914 +
1915 + '@esbuild/openbsd-x64@0.28.2':
1916 + optional: true
1917 +
1918 + '@esbuild/openharmony-arm64@0.25.12':
1919 + optional: true
1920 +
1921 + '@esbuild/openharmony-arm64@0.28.2':
1922 + optional: true
1923 +
1924 + '@esbuild/sunos-x64@0.25.12':
1925 + optional: true
1926 +
1927 + '@esbuild/sunos-x64@0.28.2':
1928 + optional: true
1929 +
1930 + '@esbuild/win32-arm64@0.25.12':
1931 + optional: true
1932 +
1933 + '@esbuild/win32-arm64@0.28.2':
1934 + optional: true
1935 +
1936 + '@esbuild/win32-ia32@0.25.12':
1937 + optional: true
1938 +
1939 + '@esbuild/win32-ia32@0.28.2':
1940 + optional: true
1941 +
1942 + '@esbuild/win32-x64@0.25.12':
1943 + optional: true
1944 +
1945 + '@esbuild/win32-x64@0.28.2':
1946 + optional: true
1947 +
1948 + '@fastify/ajv-compiler@4.0.6':
1949 + dependencies:
1950 + ajv: 8.20.0
1951 + ajv-formats: 3.0.1(ajv@8.20.0)
1952 + fast-uri: 4.1.4
1953 +
1954 + '@fastify/cors@11.3.0':
1955 + dependencies:
1956 + fastify-plugin: 6.0.0
1957 + toad-cache: 3.7.4
1958 +
1959 + '@fastify/error@4.2.0': {}
1960 +
1961 + '@fastify/fast-json-stringify-compiler@5.1.0':
1962 + dependencies:
1963 + fast-json-stringify: 7.0.1
1964 +
1965 + '@fastify/forwarded@3.0.2': {}
1966 +
1967 + '@fastify/http-proxy@11.6.2':
1968 + dependencies:
1969 + '@fastify/reply-from': 12.6.5
1970 + fast-querystring: 1.1.2
1971 + fastify-plugin: 6.0.0
1972 + ws: 8.21.3
1973 + transitivePeerDependencies:
1974 + - bufferutil
1975 + - utf-8-validate
1976 +
1977 + '@fastify/merge-json-schemas@0.2.1':
1978 + dependencies:
1979 + dequal: 2.0.3
1980 +
1981 + '@fastify/proxy-addr@5.1.0':
1982 + dependencies:
1983 + '@fastify/forwarded': 3.0.2
1984 + ipaddr.js: 2.5.0
1985 +
1986 + '@fastify/reply-from@12.6.5':
1987 + dependencies:
1988 + '@fastify/error': 4.2.0
1989 + end-of-stream: 1.4.5
1990 + fast-content-type-parse: 3.0.0
1991 + fast-querystring: 1.1.2
1992 + fastify-plugin: 6.0.0
1993 + toad-cache: 3.7.4
1994 + undici: 7.29.1
1995 +
1996 + '@fastify/websocket@11.3.0':
1997 + dependencies:
1998 + duplexify: 4.1.3
1999 + fastify-plugin: 6.0.0
2000 + ws: 8.21.3
2001 + transitivePeerDependencies:
2002 + - bufferutil
2003 + - utf-8-validate
2004 +
2005 + '@img/colour@1.1.0':
2006 + optional: true
2007 +
2008 + '@img/sharp-darwin-arm64@0.35.4':
2009 + optionalDependencies:
2010 + '@img/sharp-libvips-darwin-arm64': 1.3.3
2011 + optional: true
2012 +
2013 + '@img/sharp-darwin-x64@0.35.4':
2014 + optionalDependencies:
2015 + '@img/sharp-libvips-darwin-x64': 1.3.3
2016 + optional: true
2017 +
2018 + '@img/sharp-freebsd-wasm32@0.35.4':
2019 + dependencies:
2020 + '@img/sharp-wasm32': 0.35.4
2021 + optional: true
2022 +
2023 + '@img/sharp-libvips-darwin-arm64@1.3.3':
2024 + optional: true
2025 +
2026 + '@img/sharp-libvips-darwin-x64@1.3.3':
2027 + optional: true
2028 +
2029 + '@img/sharp-libvips-linux-arm64@1.3.3':
2030 + optional: true
2031 +
2032 + '@img/sharp-libvips-linux-arm@1.3.3':
2033 + optional: true
2034 +
2035 + '@img/sharp-libvips-linux-ppc64@1.3.3':
2036 + optional: true
2037 +
2038 + '@img/sharp-libvips-linux-riscv64@1.3.3':
2039 + optional: true
2040 +
2041 + '@img/sharp-libvips-linux-s390x@1.3.3':
2042 + optional: true
2043 +
2044 + '@img/sharp-libvips-linux-x64@1.3.3':
2045 + optional: true
2046 +
2047 + '@img/sharp-libvips-linuxmusl-arm64@1.3.3':
2048 + optional: true
2049 +
2050 + '@img/sharp-libvips-linuxmusl-x64@1.3.3':
2051 + optional: true
2052 +
2053 + '@img/sharp-linux-arm64@0.35.4':
2054 + optionalDependencies:
2055 + '@img/sharp-libvips-linux-arm64': 1.3.3
2056 + optional: true
2057 +
2058 + '@img/sharp-linux-arm@0.35.4':
2059 + optionalDependencies:
2060 + '@img/sharp-libvips-linux-arm': 1.3.3
2061 + optional: true
2062 +
2063 + '@img/sharp-linux-ppc64@0.35.4':
2064 + optionalDependencies:
2065 + '@img/sharp-libvips-linux-ppc64': 1.3.3
2066 + optional: true
2067 +
2068 + '@img/sharp-linux-riscv64@0.35.4':
2069 + optionalDependencies:
2070 + '@img/sharp-libvips-linux-riscv64': 1.3.3
2071 + optional: true
2072 +
2073 + '@img/sharp-linux-s390x@0.35.4':
2074 + optionalDependencies:
2075 + '@img/sharp-libvips-linux-s390x': 1.3.3
2076 + optional: true
2077 +
2078 + '@img/sharp-linux-x64@0.35.4':
2079 + optionalDependencies:
2080 + '@img/sharp-libvips-linux-x64': 1.3.3
2081 + optional: true
2082 +
2083 + '@img/sharp-linuxmusl-arm64@0.35.4':
2084 + optionalDependencies:
2085 + '@img/sharp-libvips-linuxmusl-arm64': 1.3.3
2086 + optional: true
2087 +
2088 + '@img/sharp-linuxmusl-x64@0.35.4':
2089 + optionalDependencies:
2090 + '@img/sharp-libvips-linuxmusl-x64': 1.3.3
2091 + optional: true
2092 +
2093 + '@img/sharp-wasm32@0.35.4':
2094 + dependencies:
2095 + '@emnapi/runtime': 1.11.3
2096 + optional: true
2097 +
2098 + '@img/sharp-webcontainers-wasm32@0.35.4':
2099 + dependencies:
2100 + '@img/sharp-wasm32': 0.35.4
2101 + optional: true
2102 +
2103 + '@img/sharp-win32-arm64@0.35.4':
2104 + optional: true
2105 +
2106 + '@img/sharp-win32-ia32@0.35.4':
2107 + optional: true
2108 +
2109 + '@img/sharp-win32-x64@0.35.4':
2110 + optional: true
2111 +
2112 + '@jridgewell/gen-mapping@0.3.13':
2113 + dependencies:
2114 + '@jridgewell/sourcemap-codec': 1.6.0
2115 + '@jridgewell/trace-mapping': 0.3.31
2116 +
2117 + '@jridgewell/remapping@2.3.5':
2118 + dependencies:
2119 + '@jridgewell/gen-mapping': 0.3.13
2120 + '@jridgewell/trace-mapping': 0.3.31
2121 +
2122 + '@jridgewell/resolve-uri@3.1.2': {}
2123 +
2124 + '@jridgewell/sourcemap-codec@1.6.0': {}
2125 +
2126 + '@jridgewell/trace-mapping@0.3.31':
2127 + dependencies:
2128 + '@jridgewell/resolve-uri': 3.1.2
2129 + '@jridgewell/sourcemap-codec': 1.6.0
2130 +
2131 + '@napi-rs/lzma-linux-x64-gnu@1.5.1':
2132 + optional: true
2133 +
2134 + '@next/env@16.3.4': {}
2135 +
2136 + '@next/swc-darwin-arm64@16.3.4':
2137 + optional: true
2138 +
2139 + '@next/swc-darwin-x64@16.3.4':
2140 + optional: true
2141 +
2142 + '@next/swc-linux-arm64-gnu@16.3.4':
2143 + optional: true
2144 +
2145 + '@next/swc-linux-arm64-musl@16.3.4':
2146 + optional: true
2147 +
2148 + '@next/swc-linux-x64-gnu@16.3.4':
2149 + optional: true
2150 +
2151 + '@next/swc-linux-x64-musl@16.3.4':
2152 + optional: true
2153 +
2154 + '@next/swc-win32-arm64-msvc@16.3.4':
2155 + optional: true
2156 +
2157 + '@next/swc-win32-x64-msvc@16.3.4':
2158 + optional: true
2159 +
2160 + '@nodable/entities@3.0.0': {}
2161 +
2162 + '@pinojs/redact@0.4.0': {}
2163 +
2164 + '@rollup/rollup-android-arm-eabi@4.63.1':
2165 + optional: true
2166 +
2167 + '@rollup/rollup-android-arm64@4.63.1':
2168 + optional: true
2169 +
2170 + '@rollup/rollup-darwin-arm64@4.63.1':
2171 + optional: true
2172 +
2173 + '@rollup/rollup-darwin-x64@4.63.1':
2174 + optional: true
2175 +
2176 + '@rollup/rollup-freebsd-arm64@4.63.1':
2177 + optional: true
2178 +
2179 + '@rollup/rollup-freebsd-x64@4.63.1':
2180 + optional: true
2181 +
2182 + '@rollup/rollup-linux-arm-gnueabihf@4.63.1':
2183 + optional: true
2184 +
2185 + '@rollup/rollup-linux-arm-musleabihf@4.63.1':
2186 + optional: true
2187 +
2188 + '@rollup/rollup-linux-arm64-gnu@4.63.1':
2189 + optional: true
2190 +
2191 + '@rollup/rollup-linux-arm64-musl@4.63.1':
2192 + optional: true
2193 +
2194 + '@rollup/rollup-linux-loong64-gnu@4.63.1':
2195 + optional: true
2196 +
2197 + '@rollup/rollup-linux-loong64-musl@4.63.1':
2198 + optional: true
2199 +
2200 + '@rollup/rollup-linux-ppc64-gnu@4.63.1':
2201 + optional: true
2202 +
2203 + '@rollup/rollup-linux-ppc64-musl@4.63.1':
2204 + optional: true
2205 +
2206 + '@rollup/rollup-linux-riscv64-gnu@4.63.1':
2207 + optional: true
2208 +
2209 + '@rollup/rollup-linux-riscv64-musl@4.63.1':
2210 + optional: true
2211 +
2212 + '@rollup/rollup-linux-s390x-gnu@4.63.1':
2213 + optional: true
2214 +
2215 + '@rollup/rollup-linux-x64-gnu@4.63.1':
2216 + optional: true
2217 +
2218 + '@rollup/rollup-linux-x64-musl@4.63.1':
2219 + optional: true
2220 +
2221 + '@rollup/rollup-openbsd-x64@4.63.1':
2222 + optional: true
2223 +
2224 + '@rollup/rollup-openharmony-arm64@4.63.1':
2225 + optional: true
2226 +
2227 + '@rollup/rollup-win32-arm64-msvc@4.63.1':
2228 + optional: true
2229 +
2230 + '@rollup/rollup-win32-ia32-msvc@4.63.1':
2231 + optional: true
2232 +
2233 + '@rollup/rollup-win32-x64-gnu@4.63.1':
2234 + optional: true
2235 +
2236 + '@rollup/rollup-win32-x64-msvc@4.63.1':
2237 + optional: true
2238 +
2239 + '@swc/helpers@0.5.23':
2240 + dependencies:
2241 + tslib: 2.8.1
2242 +
2243 + '@tailwindcss/node@4.3.3':
2244 + dependencies:
2245 + '@jridgewell/remapping': 2.3.5
2246 + enhanced-resolve: 5.24.5
2247 + jiti: 2.7.0
2248 + lightningcss: 1.32.0
2249 + magic-string: 0.30.21
2250 + source-map-js: 1.2.1
2251 + tailwindcss: 4.3.3
2252 +
2253 + '@tailwindcss/oxide-android-arm64@4.3.3':
2254 + optional: true
2255 +
2256 + '@tailwindcss/oxide-darwin-arm64@4.3.3':
2257 + optional: true
2258 +
2259 + '@tailwindcss/oxide-darwin-x64@4.3.3':
2260 + optional: true
2261 +
2262 + '@tailwindcss/oxide-freebsd-x64@4.3.3':
2263 + optional: true
2264 +
2265 + '@tailwindcss/oxide-linux-arm-gnueabihf@4.3.3':
2266 + optional: true
2267 +
2268 + '@tailwindcss/oxide-linux-arm64-gnu@4.3.3':
2269 + optional: true
2270 +
2271 + '@tailwindcss/oxide-linux-arm64-musl@4.3.3':
2272 + optional: true
2273 +
2274 + '@tailwindcss/oxide-linux-x64-gnu@4.3.3':
2275 + optional: true
2276 +
2277 + '@tailwindcss/oxide-linux-x64-musl@4.3.3':
2278 + optional: true
2279 +
2280 + '@tailwindcss/oxide-wasm32-wasi@4.3.3':
2281 + optional: true
2282 +
2283 + '@tailwindcss/oxide-win32-arm64-msvc@4.3.3':
2284 + optional: true
2285 +
2286 + '@tailwindcss/oxide-win32-x64-msvc@4.3.3':
2287 + optional: true
2288 +
2289 + '@tailwindcss/oxide@4.3.3':
2290 + optionalDependencies:
2291 + '@tailwindcss/oxide-android-arm64': 4.3.3
2292 + '@tailwindcss/oxide-darwin-arm64': 4.3.3
2293 + '@tailwindcss/oxide-darwin-x64': 4.3.3
2294 + '@tailwindcss/oxide-freebsd-x64': 4.3.3
2295 + '@tailwindcss/oxide-linux-arm-gnueabihf': 4.3.3
2296 + '@tailwindcss/oxide-linux-arm64-gnu': 4.3.3
2297 + '@tailwindcss/oxide-linux-arm64-musl': 4.3.3
2298 + '@tailwindcss/oxide-linux-x64-gnu': 4.3.3
2299 + '@tailwindcss/oxide-linux-x64-musl': 4.3.3
2300 + '@tailwindcss/oxide-wasm32-wasi': 4.3.3
2301 + '@tailwindcss/oxide-win32-arm64-msvc': 4.3.3
2302 + '@tailwindcss/oxide-win32-x64-msvc': 4.3.3
2303 +
2304 + '@tailwindcss/postcss@4.3.3':
2305 + dependencies:
2306 + '@alloc/quick-lru': 5.3.0
2307 + '@tailwindcss/node': 4.3.3
2308 + '@tailwindcss/oxide': 4.3.3
2309 + postcss: 8.5.28
2310 + tailwindcss: 4.3.3
2311 +
2312 + '@types/chai@5.2.3':
2313 + dependencies:
2314 + '@types/deep-eql': 4.0.2
2315 + assertion-error: 2.0.1
2316 +
2317 + '@types/d3-geo@3.1.1':
2318 + dependencies:
2319 + '@types/geojson': 7946.0.16
2320 +
2321 + '@types/deep-eql@4.0.2': {}
2322 +
2323 + '@types/estree@1.0.9': {}
2324 +
2325 + '@types/geojson@7946.0.16': {}
2326 +
2327 + '@types/node@24.13.4':
2328 + dependencies:
2329 + undici-types: 7.18.2
2330 +
2331 + '@types/pg@8.23.1':
2332 + dependencies:
2333 + '@types/node': 24.13.4
2334 + pg-protocol: 1.16.0
2335 + pg-types: 2.2.0
2336 +
2337 + '@types/react-dom@19.3.0(@types/react@19.3.0)':
2338 + dependencies:
2339 + '@types/react': 19.3.0
2340 +
2341 + '@types/react@19.3.0':
2342 + dependencies:
2343 + csstype: 3.2.3
2344 +
2345 + '@types/topojson-client@3.1.5':
2346 + dependencies:
2347 + '@types/geojson': 7946.0.16
2348 + '@types/topojson-specification': 1.0.5
2349 +
2350 + '@types/topojson-specification@1.0.5':
2351 + dependencies:
2352 + '@types/geojson': 7946.0.16
2353 +
2354 + '@types/ws@8.18.1':
2355 + dependencies:
2356 + '@types/node': 24.13.4
2357 +
2358 + '@vitest/expect@3.2.7':
2359 + dependencies:
2360 + '@types/chai': 5.2.3
2361 + '@vitest/spy': 3.2.7
2362 + '@vitest/utils': 3.2.7
2363 + chai: 5.3.3
2364 + tinyrainbow: 2.0.0
2365 +
2366 + '@vitest/mocker@3.2.7(vite@7.3.6(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13))':
2367 + dependencies:
2368 + '@vitest/spy': 3.2.7
2369 + estree-walker: 3.0.3
2370 + magic-string: 0.30.21
2371 + optionalDependencies:
2372 + vite: 7.3.6(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
2373 +
2374 + '@vitest/pretty-format@3.2.7':
2375 + dependencies:
2376 + tinyrainbow: 2.0.0
2377 +
2378 + '@vitest/runner@3.2.7':
2379 + dependencies:
2380 + '@vitest/utils': 3.2.7
2381 + pathe: 2.0.3
2382 + strip-literal: 3.1.0
2383 +
2384 + '@vitest/snapshot@3.2.7':
2385 + dependencies:
2386 + '@vitest/pretty-format': 3.2.7
2387 + magic-string: 0.30.21
2388 + pathe: 2.0.3
2389 +
2390 + '@vitest/spy@3.2.7':
2391 + dependencies:
2392 + tinyspy: 4.0.6
2393 +
2394 + '@vitest/utils@3.2.7':
2395 + dependencies:
2396 + '@vitest/pretty-format': 3.2.7
2397 + loupe: 3.2.1
2398 + tinyrainbow: 2.0.0
2399 +
2400 + abstract-logging@2.0.1: {}
2401 +
2402 + ajv-formats@3.0.1(ajv@8.20.0):
2403 + optionalDependencies:
2404 + ajv: 8.20.0
2405 +
2406 + ajv@8.20.0:
2407 + dependencies:
2408 + fast-deep-equal: 3.1.3
2409 + fast-uri: 3.1.7
2410 + json-schema-traverse: 1.0.0
2411 + require-from-string: 2.0.2
2412 +
2413 + anynum@1.0.1: {}
2414 +
2415 + assertion-error@2.0.1: {}
2416 +
2417 + atomic-sleep@1.0.0: {}
2418 +
2419 + avvio@9.3.0:
2420 + dependencies:
2421 + '@fastify/error': 4.2.0
2422 + fastq: 1.20.3
2423 +
2424 + baseline-browser-mapping@2.11.22: {}
2425 +
2426 + cac@6.7.14: {}
2427 +
2428 + caniuse-lite@1.0.30001810: {}
2429 +
2430 + chai@5.3.3:
2431 + dependencies:
2432 + assertion-error: 2.0.1
2433 + check-error: 2.1.3
2434 + deep-eql: 5.0.2
2435 + loupe: 3.2.1
2436 + pathval: 2.0.1
2437 +
2438 + check-error@2.1.3: {}
2439 +
2440 + client-only@0.0.1: {}
2441 +
2442 + commander@2.20.3: {}
2443 +
2444 + cookie@1.1.1: {}
2445 +
2446 + csstype@3.2.3: {}
2447 +
2448 + d3-array@3.2.4:
2449 + dependencies:
2450 + internmap: 2.0.3
2451 +
2452 + d3-geo@3.1.1:
2453 + dependencies:
2454 + d3-array: 3.2.4
2455 +
2456 + debug@4.4.3:
2457 + dependencies:
2458 + ms: 2.1.3
2459 +
2460 + deep-eql@5.0.2: {}
2461 +
2462 + dequal@2.0.3: {}
2463 +
2464 + detect-libc@2.1.2: {}
2465 +
2466 + duplexify@4.1.3:
2467 + dependencies:
2468 + end-of-stream: 1.4.5
2469 + inherits: 2.0.4
2470 + readable-stream: 3.6.2
2471 + stream-shift: 1.0.3
2472 +
2473 + end-of-stream@1.4.5:
2474 + dependencies:
2475 + once: 1.4.0
2476 +
2477 + enhanced-resolve@5.24.5:
2478 + dependencies:
2479 + graceful-fs: 4.2.11
2480 + tapable: 2.3.3
2481 +
2482 + es-module-lexer@1.7.0: {}
2483 +
2484 + esbuild@0.25.12:
2485 + optionalDependencies:
2486 + '@esbuild/aix-ppc64': 0.25.12
2487 + '@esbuild/android-arm': 0.25.12
2488 + '@esbuild/android-arm64': 0.25.12
2489 + '@esbuild/android-x64': 0.25.12
2490 + '@esbuild/darwin-arm64': 0.25.12
2491 + '@esbuild/darwin-x64': 0.25.12
2492 + '@esbuild/freebsd-arm64': 0.25.12
2493 + '@esbuild/freebsd-x64': 0.25.12
2494 + '@esbuild/linux-arm': 0.25.12
2495 + '@esbuild/linux-arm64': 0.25.12
2496 + '@esbuild/linux-ia32': 0.25.12
2497 + '@esbuild/linux-loong64': 0.25.12
2498 + '@esbuild/linux-mips64el': 0.25.12
2499 + '@esbuild/linux-ppc64': 0.25.12
2500 + '@esbuild/linux-riscv64': 0.25.12
2501 + '@esbuild/linux-s390x': 0.25.12
2502 + '@esbuild/linux-x64': 0.25.12
2503 + '@esbuild/netbsd-arm64': 0.25.12
2504 + '@esbuild/netbsd-x64': 0.25.12
2505 + '@esbuild/openbsd-arm64': 0.25.12
2506 + '@esbuild/openbsd-x64': 0.25.12
2507 + '@esbuild/openharmony-arm64': 0.25.12
2508 + '@esbuild/sunos-x64': 0.25.12
2509 + '@esbuild/win32-arm64': 0.25.12
2510 + '@esbuild/win32-ia32': 0.25.12
2511 + '@esbuild/win32-x64': 0.25.12
2512 +
2513 + esbuild@0.28.2:
2514 + optionalDependencies:
2515 + '@esbuild/aix-ppc64': 0.28.2
2516 + '@esbuild/android-arm': 0.28.2
2517 + '@esbuild/android-arm64': 0.28.2
2518 + '@esbuild/android-x64': 0.28.2
2519 + '@esbuild/darwin-arm64': 0.28.2
2520 + '@esbuild/darwin-x64': 0.28.2
2521 + '@esbuild/freebsd-arm64': 0.28.2
2522 + '@esbuild/freebsd-x64': 0.28.2
2523 + '@esbuild/linux-arm': 0.28.2
2524 + '@esbuild/linux-arm64': 0.28.2
2525 + '@esbuild/linux-ia32': 0.28.2
2526 + '@esbuild/linux-loong64': 0.28.2
2527 + '@esbuild/linux-mips64el': 0.28.2
2528 + '@esbuild/linux-ppc64': 0.28.2
2529 + '@esbuild/linux-riscv64': 0.28.2
2530 + '@esbuild/linux-s390x': 0.28.2
2531 + '@esbuild/linux-x64': 0.28.2
2532 + '@esbuild/netbsd-arm64': 0.28.2
2533 + '@esbuild/netbsd-x64': 0.28.2
2534 + '@esbuild/openbsd-arm64': 0.28.2
2535 + '@esbuild/openbsd-x64': 0.28.2
2536 + '@esbuild/openharmony-arm64': 0.28.2
2537 + '@esbuild/sunos-x64': 0.28.2
2538 + '@esbuild/win32-arm64': 0.28.2
2539 + '@esbuild/win32-ia32': 0.28.2
2540 + '@esbuild/win32-x64': 0.28.2
2541 +
2542 + estree-walker@3.0.3:
2543 + dependencies:
2544 + '@types/estree': 1.0.9
2545 +
2546 + expect-type@1.4.0: {}
2547 +
2548 + fancy-canvas@2.1.0: {}
2549 +
2550 + fast-content-type-parse@3.0.0: {}
2551 +
2552 + fast-decode-uri-component@1.0.1: {}
2553 +
2554 + fast-deep-equal@3.1.3: {}
2555 +
2556 + fast-json-stringify@7.0.1:
2557 + dependencies:
2558 + '@fastify/merge-json-schemas': 0.2.1
2559 + ajv: 8.20.0
2560 + ajv-formats: 3.0.1(ajv@8.20.0)
2561 + fast-uri: 4.1.4
2562 + json-schema-ref-resolver: 3.0.0
2563 + rfdc: 1.4.1
2564 +
2565 + fast-querystring@1.1.2:
2566 + dependencies:
2567 + fast-decode-uri-component: 1.0.1
2568 +
2569 + fast-uri@3.1.7: {}
2570 +
2571 + fast-uri@4.1.4: {}
2572 +
2573 + fast-xml-builder@1.3.1:
2574 + dependencies:
2575 + path-expression-matcher: 1.6.2
2576 + xml-naming: 0.3.0
2577 +
2578 + fast-xml-parser@5.11.1:
2579 + dependencies:
2580 + '@nodable/entities': 3.0.0
2581 + fast-xml-builder: 1.3.1
2582 + is-unsafe: 2.0.2
2583 + path-expression-matcher: 1.6.2
2584 + strnum: 2.4.2
2585 + xml-naming: 0.3.0
2586 +
2587 + fastify-plugin@6.0.0: {}
2588 +
2589 + fastify@5.12.3:
2590 + dependencies:
2591 + '@fastify/ajv-compiler': 4.0.6
2592 + '@fastify/error': 4.2.0
2593 + '@fastify/fast-json-stringify-compiler': 5.1.0
2594 + '@fastify/proxy-addr': 5.1.0
2595 + abstract-logging: 2.0.1
2596 + avvio: 9.3.0
2597 + fast-json-stringify: 7.0.1
2598 + find-my-way: 9.9.0
2599 + light-my-request: 6.6.0
2600 + pino: 9.14.0
2601 + process-warning: 5.1.0
2602 + rfdc: 1.4.1
2603 + secure-json-parse: 4.1.0
2604 + semver: 7.8.5
2605 + toad-cache: 3.7.4
2606 +
2607 + fastq@1.20.3:
2608 + dependencies:
2609 + reusify: 1.1.0
2610 +
2611 + fdir@6.5.0(picomatch@4.0.7):
2612 + optionalDependencies:
2613 + picomatch: 4.0.7
2614 +
2615 + find-my-way@9.9.0:
2616 + dependencies:
2617 + fast-deep-equal: 3.1.3
2618 + fast-querystring: 1.1.2
2619 + safe-regex2: 5.1.1
2620 +
2621 + fsevents@2.3.3:
2622 + optional: true
2623 +
2624 + geist@1.7.2(next@16.3.4(@types/node@24.13.4)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)):
2625 + dependencies:
2626 + next: 16.3.4(@types/node@24.13.4)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
2627 +
2628 + graceful-fs@4.2.11: {}
2629 +
2630 + inherits@2.0.4: {}
2631 +
2632 + internmap@2.0.3: {}
2633 +
2634 + ipaddr.js@2.5.0: {}
2635 +
2636 + is-unsafe@2.0.2: {}
2637 +
2638 + jiti@2.7.0: {}
2639 +
2640 + js-tokens@9.0.1: {}
2641 +
2642 + json-schema-ref-resolver@3.0.0:
2643 + dependencies:
2644 + dequal: 2.0.3
2645 +
2646 + json-schema-traverse@1.0.0: {}
2647 +
2648 + light-my-request@6.6.0:
2649 + dependencies:
2650 + cookie: 1.1.1
2651 + process-warning: 4.0.1
2652 + set-cookie-parser: 2.7.2
2653 +
2654 + lightningcss-android-arm64@1.32.0:
2655 + optional: true
2656 +
2657 + lightningcss-darwin-arm64@1.32.0:
2658 + optional: true
2659 +
2660 + lightningcss-darwin-x64@1.32.0:
2661 + optional: true
2662 +
2663 + lightningcss-freebsd-x64@1.32.0:
2664 + optional: true
2665 +
2666 + lightningcss-linux-arm-gnueabihf@1.32.0:
2667 + optional: true
2668 +
2669 + lightningcss-linux-arm64-gnu@1.32.0:
2670 + optional: true
2671 +
2672 + lightningcss-linux-arm64-musl@1.32.0:
2673 + optional: true
2674 +
2675 + lightningcss-linux-x64-gnu@1.32.0:
2676 + optional: true
2677 +
2678 + lightningcss-linux-x64-musl@1.32.0:
2679 + optional: true
2680 +
2681 + lightningcss-win32-arm64-msvc@1.32.0:
2682 + optional: true
2683 +
2684 + lightningcss-win32-x64-msvc@1.32.0:
2685 + optional: true
2686 +
2687 + lightningcss@1.32.0:
2688 + dependencies:
2689 + detect-libc: 2.1.2
2690 + optionalDependencies:
2691 + lightningcss-android-arm64: 1.32.0
2692 + lightningcss-darwin-arm64: 1.32.0
2693 + lightningcss-darwin-x64: 1.32.0
2694 + lightningcss-freebsd-x64: 1.32.0
2695 + lightningcss-linux-arm-gnueabihf: 1.32.0
2696 + lightningcss-linux-arm64-gnu: 1.32.0
2697 + lightningcss-linux-arm64-musl: 1.32.0
2698 + lightningcss-linux-x64-gnu: 1.32.0
2699 + lightningcss-linux-x64-musl: 1.32.0
2700 + lightningcss-win32-arm64-msvc: 1.32.0
2701 + lightningcss-win32-x64-msvc: 1.32.0
2702 +
2703 + lightweight-charts@5.2.1:
2704 + dependencies:
2705 + fancy-canvas: 2.1.0
2706 +
2707 + loupe@3.2.1: {}
2708 +
2709 + lucide-react@1.44.0(react@19.2.8):
2710 + dependencies:
2711 + react: 19.2.8
2712 +
2713 + magic-string@0.30.21:
2714 + dependencies:
2715 + '@jridgewell/sourcemap-codec': 1.6.0
2716 +
2717 + ms@2.1.3: {}
2718 +
2719 + nanoid@3.3.19: {}
2720 +
2721 + next@16.3.4(@types/node@24.13.4)(react-dom@19.2.8(react@19.2.8))(react@19.2.8):
2722 + dependencies:
2723 + '@next/env': 16.3.4
2724 + '@swc/helpers': 0.5.23
2725 + baseline-browser-mapping: 2.11.22
2726 + caniuse-lite: 1.0.30001810
2727 + postcss: 8.5.23
2728 + react: 19.2.8
2729 + react-dom: 19.2.8(react@19.2.8)
2730 + styled-jsx: 5.1.6(react@19.2.8)
2731 + optionalDependencies:
2732 + '@next/swc-darwin-arm64': 16.3.4
2733 + '@next/swc-darwin-x64': 16.3.4
2734 + '@next/swc-linux-arm64-gnu': 16.3.4
2735 + '@next/swc-linux-arm64-musl': 16.3.4
2736 + '@next/swc-linux-x64-gnu': 16.3.4
2737 + '@next/swc-linux-x64-musl': 16.3.4
2738 + '@next/swc-win32-arm64-msvc': 16.3.4
2739 + '@next/swc-win32-x64-msvc': 16.3.4
2740 + sharp: 0.35.4(@types/node@24.13.4)
2741 + transitivePeerDependencies:
2742 + - '@babel/core'
2743 + - '@types/node'
2744 + - babel-plugin-macros
2745 +
2746 + on-exit-leak-free@2.1.2: {}
2747 +
2748 + once@1.4.0:
2749 + dependencies:
2750 + wrappy: 1.0.2
2751 +
2752 + path-expression-matcher@1.6.2: {}
2753 +
2754 + pathe@2.0.3: {}
2755 +
2756 + pathval@2.0.1: {}
2757 +
2758 + pg-cloudflare@1.4.0:
2759 + optional: true
2760 +
2761 + pg-connection-string@2.14.0: {}
2762 +
2763 + pg-int8@1.0.1: {}
2764 +
2765 + pg-pool@3.14.0(pg@8.23.0):
2766 + dependencies:
2767 + pg: 8.23.0
2768 +
2769 + pg-protocol@1.16.0: {}
2770 +
2771 + pg-types@2.2.0:
2772 + dependencies:
2773 + pg-int8: 1.0.1
2774 + postgres-array: 2.0.0
2775 + postgres-bytea: 1.0.1
2776 + postgres-date: 1.0.7
2777 + postgres-interval: 1.2.0
2778 +
2779 + pg@8.23.0:
2780 + dependencies:
2781 + pg-connection-string: 2.14.0
2782 + pg-pool: 3.14.0(pg@8.23.0)
2783 + pg-protocol: 1.16.0
2784 + pg-types: 2.2.0
2785 + pgpass: 1.0.5
2786 + optionalDependencies:
2787 + pg-cloudflare: 1.4.0
2788 +
2789 + pgpass@1.0.5:
2790 + dependencies:
2791 + split2: 4.2.0
2792 +
2793 + picocolors@1.1.1: {}
2794 +
2795 + picomatch@4.0.7: {}
2796 +
2797 + pino-abstract-transport@2.0.0:
2798 + dependencies:
2799 + split2: 4.2.0
2800 +
2801 + pino-std-serializers@7.1.0: {}
2802 +
2803 + pino@9.14.0:
2804 + dependencies:
2805 + '@pinojs/redact': 0.4.0
2806 + atomic-sleep: 1.0.0
2807 + on-exit-leak-free: 2.1.2
2808 + pino-abstract-transport: 2.0.0
2809 + pino-std-serializers: 7.1.0
2810 + process-warning: 5.1.0
2811 + quick-format-unescaped: 4.0.4
2812 + real-require: 0.2.0
2813 + safe-stable-stringify: 2.5.0
2814 + sonic-boom: 4.2.1
2815 + thread-stream: 3.2.0
2816 +
2817 + postcss@8.5.23:
2818 + dependencies:
2819 + nanoid: 3.3.19
2820 + picocolors: 1.1.1
2821 + source-map-js: 1.2.1
2822 +
2823 + postcss@8.5.28:
2824 + dependencies:
2825 + nanoid: 3.3.19
2826 + picocolors: 1.1.1
2827 + source-map-js: 1.2.1
2828 +
2829 + postgres-array@2.0.0: {}
2830 +
2831 + postgres-bytea@1.0.1: {}
2832 +
2833 + postgres-date@1.0.7: {}
2834 +
2835 + postgres-interval@1.2.0:
2836 + dependencies:
2837 + xtend: 4.0.2
2838 +
2839 + process-warning@4.0.1: {}
2840 +
2841 + process-warning@5.1.0: {}
2842 +
2843 + quick-format-unescaped@4.0.4: {}
2844 +
2845 + react-dom@19.2.8(react@19.2.8):
2846 + dependencies:
2847 + react: 19.2.8
2848 + scheduler: 0.27.0
2849 +
2850 + react@19.2.8: {}
2851 +
2852 + readable-stream@3.6.2:
2853 + dependencies:
2854 + inherits: 2.0.4
2855 + string_decoder: 1.3.0
2856 + util-deprecate: 1.0.2
2857 +
2858 + real-require@0.2.0: {}
2859 +
2860 + require-from-string@2.0.2: {}
2861 +
2862 + ret@0.5.0: {}
2863 +
2864 + reusify@1.1.0: {}
2865 +
2866 + rfdc@1.4.1: {}
2867 +
2868 + rollup@4.63.1:
2869 + dependencies:
2870 + '@types/estree': 1.0.9
2871 + optionalDependencies:
2872 + '@napi-rs/lzma-linux-x64-gnu': 1.5.1
2873 + '@rollup/rollup-android-arm-eabi': 4.63.1
2874 + '@rollup/rollup-android-arm64': 4.63.1
2875 + '@rollup/rollup-darwin-arm64': 4.63.1
2876 + '@rollup/rollup-darwin-x64': 4.63.1
2877 + '@rollup/rollup-freebsd-arm64': 4.63.1
2878 + '@rollup/rollup-freebsd-x64': 4.63.1
2879 + '@rollup/rollup-linux-arm-gnueabihf': 4.63.1
2880 + '@rollup/rollup-linux-arm-musleabihf': 4.63.1
2881 + '@rollup/rollup-linux-arm64-gnu': 4.63.1
2882 + '@rollup/rollup-linux-arm64-musl': 4.63.1
2883 + '@rollup/rollup-linux-loong64-gnu': 4.63.1
2884 + '@rollup/rollup-linux-loong64-musl': 4.63.1
2885 + '@rollup/rollup-linux-ppc64-gnu': 4.63.1
2886 + '@rollup/rollup-linux-ppc64-musl': 4.63.1
2887 + '@rollup/rollup-linux-riscv64-gnu': 4.63.1
2888 + '@rollup/rollup-linux-riscv64-musl': 4.63.1
2889 + '@rollup/rollup-linux-s390x-gnu': 4.63.1
2890 + '@rollup/rollup-linux-x64-gnu': 4.63.1
2891 + '@rollup/rollup-linux-x64-musl': 4.63.1
2892 + '@rollup/rollup-openbsd-x64': 4.63.1
2893 + '@rollup/rollup-openharmony-arm64': 4.63.1
2894 + '@rollup/rollup-win32-arm64-msvc': 4.63.1
2895 + '@rollup/rollup-win32-ia32-msvc': 4.63.1
2896 + '@rollup/rollup-win32-x64-gnu': 4.63.1
2897 + '@rollup/rollup-win32-x64-msvc': 4.63.1
2898 + fsevents: 2.3.3
2899 +
2900 + safe-buffer@5.2.1: {}
2901 +
2902 + safe-regex2@5.1.1:
2903 + dependencies:
2904 + ret: 0.5.0
2905 +
2906 + safe-stable-stringify@2.5.0: {}
2907 +
2908 + scheduler@0.27.0: {}
2909 +
2910 + secure-json-parse@4.1.0: {}
2911 +
2912 + semver@7.8.5: {}
2913 +
2914 + server-only@0.0.1: {}
2915 +
2916 + set-cookie-parser@2.7.2: {}
2917 +
2918 + sharp@0.35.4(@types/node@24.13.4):
2919 + dependencies:
2920 + '@img/colour': 1.1.0
2921 + detect-libc: 2.1.2
2922 + semver: 7.8.5
2923 + optionalDependencies:
2924 + '@img/sharp-darwin-arm64': 0.35.4
2925 + '@img/sharp-darwin-x64': 0.35.4
2926 + '@img/sharp-freebsd-wasm32': 0.35.4
2927 + '@img/sharp-libvips-darwin-arm64': 1.3.3
2928 + '@img/sharp-libvips-darwin-x64': 1.3.3
2929 + '@img/sharp-libvips-linux-arm': 1.3.3
2930 + '@img/sharp-libvips-linux-arm64': 1.3.3
2931 + '@img/sharp-libvips-linux-ppc64': 1.3.3
2932 + '@img/sharp-libvips-linux-riscv64': 1.3.3
2933 + '@img/sharp-libvips-linux-s390x': 1.3.3
2934 + '@img/sharp-libvips-linux-x64': 1.3.3
2935 + '@img/sharp-libvips-linuxmusl-arm64': 1.3.3
2936 + '@img/sharp-libvips-linuxmusl-x64': 1.3.3
2937 + '@img/sharp-linux-arm': 0.35.4
2938 + '@img/sharp-linux-arm64': 0.35.4
2939 + '@img/sharp-linux-ppc64': 0.35.4
2940 + '@img/sharp-linux-riscv64': 0.35.4
2941 + '@img/sharp-linux-s390x': 0.35.4
2942 + '@img/sharp-linux-x64': 0.35.4
2943 + '@img/sharp-linuxmusl-arm64': 0.35.4
2944 + '@img/sharp-linuxmusl-x64': 0.35.4
2945 + '@img/sharp-webcontainers-wasm32': 0.35.4
2946 + '@img/sharp-win32-arm64': 0.35.4
2947 + '@img/sharp-win32-ia32': 0.35.4
2948 + '@img/sharp-win32-x64': 0.35.4
2949 + '@types/node': 24.13.4
2950 + optional: true
2951 +
2952 + siginfo@2.0.0: {}
2953 +
2954 + sonic-boom@4.2.1:
2955 + dependencies:
2956 + atomic-sleep: 1.0.0
2957 +
2958 + source-map-js@1.2.1: {}
2959 +
2960 + split2@4.2.0: {}
2961 +
2962 + stackback@0.0.2: {}
2963 +
2964 + std-env@3.10.0: {}
2965 +
2966 + stream-shift@1.0.3: {}
2967 +
2968 + string_decoder@1.3.0:
2969 + dependencies:
2970 + safe-buffer: 5.2.1
2971 +
2972 + strip-literal@3.1.0:
2973 + dependencies:
2974 + js-tokens: 9.0.1
2975 +
2976 + strnum@2.4.2:
2977 + dependencies:
2978 + anynum: 1.0.1
2979 +
2980 + styled-jsx@5.1.6(react@19.2.8):
2981 + dependencies:
2982 + client-only: 0.0.1
2983 + react: 19.2.8
2984 +
2985 + tailwindcss@4.3.3: {}
2986 +
2987 + tapable@2.3.3: {}
2988 +
2989 + thread-stream@3.2.0:
2990 + dependencies:
2991 + real-require: 0.2.0
2992 +
2993 + tinybench@2.9.0: {}
2994 +
2995 + tinyexec@0.3.2: {}
2996 +
2997 + tinyglobby@0.2.17:
2998 + dependencies:
2999 + fdir: 6.5.0(picomatch@4.0.7)
3000 + picomatch: 4.0.7
3001 +
3002 + tinypool@1.1.1: {}
3003 +
3004 + tinyrainbow@2.0.0: {}
3005 +
3006 + tinyspy@4.0.6: {}
3007 +
3008 + toad-cache@3.7.4: {}
3009 +
3010 + topojson-client@3.1.0:
3011 + dependencies:
3012 + commander: 2.20.3
3013 +
3014 + tslib@2.8.1: {}
3015 +
3016 + tsx@4.23.13:
3017 + dependencies:
3018 + esbuild: 0.28.2
3019 + optionalDependencies:
3020 + fsevents: 2.3.3
3021 +
3022 + typescript@5.9.3: {}
3023 +
3024 + undici-types@7.18.2: {}
3025 +
3026 + undici@7.29.1: {}
3027 +
3028 + util-deprecate@1.0.2: {}
3029 +
3030 + vite-node@3.2.4(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13):
3031 + dependencies:
3032 + cac: 6.7.14
3033 + debug: 4.4.3
3034 + es-module-lexer: 1.7.0
3035 + pathe: 2.0.3
3036 + vite: 7.3.6(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
3037 + transitivePeerDependencies:
3038 + - '@types/node'
3039 + - jiti
3040 + - less
3041 + - lightningcss
3042 + - sass
3043 + - sass-embedded
3044 + - stylus
3045 + - sugarss
3046 + - supports-color
3047 + - terser
3048 + - tsx
3049 + - yaml
3050 +
3051 + vite@7.3.6(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13):
3052 + dependencies:
3053 + esbuild: 0.28.2
3054 + fdir: 6.5.0(picomatch@4.0.7)
3055 + picomatch: 4.0.7
3056 + postcss: 8.5.28
3057 + rollup: 4.63.1
3058 + tinyglobby: 0.2.17
3059 + optionalDependencies:
3060 + '@types/node': 24.13.4
3061 + fsevents: 2.3.3
3062 + jiti: 2.7.0
3063 + lightningcss: 1.32.0
3064 + tsx: 4.23.13
3065 +
3066 + vitest@3.2.7(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13):
3067 + dependencies:
3068 + '@types/chai': 5.2.3
3069 + '@vitest/expect': 3.2.7
3070 + '@vitest/mocker': 3.2.7(vite@7.3.6(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13))
3071 + '@vitest/pretty-format': 3.2.7
3072 + '@vitest/runner': 3.2.7
3073 + '@vitest/snapshot': 3.2.7
3074 + '@vitest/spy': 3.2.7
3075 + '@vitest/utils': 3.2.7
3076 + chai: 5.3.3
3077 + debug: 4.4.3
3078 + expect-type: 1.4.0
3079 + magic-string: 0.30.21
3080 + pathe: 2.0.3
3081 + picomatch: 4.0.7
3082 + std-env: 3.10.0
3083 + tinybench: 2.9.0
3084 + tinyexec: 0.3.2
3085 + tinyglobby: 0.2.17
3086 + tinypool: 1.1.1
3087 + tinyrainbow: 2.0.0
3088 + vite: 7.3.6(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
3089 + vite-node: 3.2.4(@types/node@24.13.4)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
3090 + why-is-node-running: 2.3.0
3091 + optionalDependencies:
3092 + '@types/node': 24.13.4
3093 + transitivePeerDependencies:
3094 + - jiti
3095 + - less
3096 + - lightningcss
3097 + - msw
3098 + - sass
3099 + - sass-embedded
3100 + - stylus
3101 + - sugarss
3102 + - supports-color
3103 + - terser
3104 + - tsx
3105 + - yaml
3106 +
3107 + why-is-node-running@2.3.0:
3108 + dependencies:
3109 + siginfo: 2.0.0
3110 + stackback: 0.0.2
3111 +
3112 + world-atlas@2.0.2: {}
3113 +
3114 + wrappy@1.0.2: {}
3115 +
3116 + ws@8.21.3: {}
3117 +
3118 + xml-naming@0.3.0: {}
3119 +
3120 + xtend@4.0.2: {}
3121 +
3122 + zod@3.25.76: {}
added pnpm-workspace.yaml +4 −0
@@ -0,0 +1,4 @@
1 +packages:
2 + - apps/*
3 + - packages/*
4 + - connectors
added tsconfig.base.json +20 −0
@@ -0,0 +1,20 @@
1 +{
2 + "compilerOptions": {
3 + "target": "ES2023",
4 + "lib": ["ES2023"],
5 + "module": "ESNext",
6 + "moduleResolution": "Bundler",
7 + "strict": true,
8 + "noUncheckedIndexedAccess": true,
9 + "noImplicitOverride": true,
10 + "exactOptionalPropertyTypes": false,
11 + "forceConsistentCasingInFileNames": true,
12 + "esModuleInterop": true,
13 + "resolveJsonModule": true,
14 + "skipLibCheck": true,
15 + "isolatedModules": true,
16 + "allowImportingTsExtensions": false,
17 + "noEmit": true,
18 + "types": ["node"]
19 + }
20 +}
added tsconfig.json +11 −0
@@ -0,0 +1,11 @@
1 +{
2 + "extends": "./tsconfig.base.json",
3 + "include": [
4 + "packages/*/src/**/*.ts",
5 + "connectors/src/**/*.ts",
6 + "apps/api/src/**/*.ts",
7 + "tests/**/*.ts",
8 + "scripts/**/*.ts"
9 + ],
10 + "exclude": ["node_modules", "apps/web", "**/dist"]
11 +}
added vitest.config.ts +24 −0
@@ -0,0 +1,24 @@
1 +import { defineConfig } from "vitest/config";
2 +import { fileURLToPath } from "node:url";
3 +
4 +const r = (p: string) => fileURLToPath(new URL(p, import.meta.url));
5 +
6 +export default defineConfig({
7 + resolve: {
8 + alias: {
9 + "@market-atlas/market-model": r("./packages/market-model/src/index.ts"),
10 + "@market-atlas/connector-sdk": r("./packages/connector-sdk/src/index.ts"),
11 + "@market-atlas/connectors": r("./connectors/src/index.ts"),
12 + },
13 + },
14 + test: {
15 + include: [
16 + "packages/**/*.test.ts",
17 + "connectors/**/*.test.ts",
18 + "apps/api/**/*.test.ts",
19 + "tests/**/*.test.ts",
20 + ],
21 + environment: "node",
22 + testTimeout: 20000,
23 + },
24 +});
25