spb/market-atlas

1# CLAUDE.md23# Market Atlas45**Production domain:** `https://www.market-atlas.co`6**Product name:** Market Atlas7**Project type:** Global financial market intelligence, market-data aggregation, live-feed reconstruction, source-observation, time-series, and market-event platform8**Primary interface:** Web application9**Secondary interfaces:** Public/internal APIs, WebSocket/SSE streaming endpoints, background ingestion workers, connector management console10**Deployment target:** Self-hosted production infrastructure / cluster (MacLustr, node M2U64 via `mld` — see `docs/DEPLOY.md`)11**Primary objective:** Build a global, continuously updating representation of financial markets using a heterogeneous network of market-data connectors, public web sources, official feeds where available, observable frontend data streams, filings, exchange publications, issuer pages, and additional legally usable sources.1213> Implementation state (v0.1.0, 2026-09-12): TypeScript monorepo — `packages/market-model`, `packages/connector-sdk`, `connectors/` (14 production connectors),14> `apps/api` (engines, REST/WS/SSE, admin, CLI), `apps/web` (Next.js). Read `docs/ARCHITECTURE.md`, `docs/CONNECTORS.md`, `docs/API.md`, `docs/FRONTEND.md`, `docs/DEPLOY.md` before changing anything structural.1516---1718# 1. CORE VISION1920Market Atlas must become a live map of global financial markets. This is NOT simply a stock screener, a quote website, a Yahoo Finance / TradingView / Bloomberg clone, a static company database, or an API wrapper. Market Atlas should behave as an independent **market observation infrastructure**: it continuously observes thousands of market-related sources, normalizes their outputs, identifies financial entities, compares independent observations, detects changes and anomalies, creates its own canonical event stream, stores historical observations, and redistributes normalized information through the Market Atlas application.2122```text23THE WEB / EXCHANGES / ISSUERS / DATA SOURCES → CONNECTOR NETWORK → RAW OBSERVATIONS → NORMALIZATION + RESOLUTION → CONSENSUS / VALIDATION → MARKET BUS24                                                                    ┌──────────────┼──────────────┐25                                                                 Live UI      Historical DB       APIs26                                                                 Alerts        Analytics        WebSocket27```2829Market Atlas becomes more valuable as time passes: every captured observation adds to the historical dataset. Two simultaneous products: **the live market state** and **the historical market memory**.3031# 2. PRIMARY PRODUCT PRINCIPLE3233Market Atlas must not depend structurally on one external financial-data API. External APIs may be optional sources, fallbacks, validation layers, or licensed sources — never architectural single points of failure. Build a flexible **connector infrastructure** able to observe many source types (exchange websites, issuer IR sites, regulatory filings, public quote pages and dashboards, exchange status pages, official downloadable files, RSS/Atom, public JSON endpoints, XHR/fetch responses used by frontends, GraphQL, SSE, public WebSocket streams, HTML, embedded JSON, JavaScript state, CSV, XML, PDF, official FTP/SFTP where permitted, public bulk downloads and object storage, press-release feeds, market-status endpoints). Always retain where an observation came from. Never detach market information from provenance.3435# 3. LEGAL AND DATA-USAGE PRINCIPLE3637Technical accessibility does NOT imply redistribution rights. Market Atlas has a first-class data-rights and provenance system. Never bypass authentication, paywalls, CAPTCHAs or access controls; never impersonate private users, use leaked credentials, scrape private pages, access data requiring unauthorized sessions, reverse-engineer protected feeds for unauthorized redistribution, or intentionally evade source restrictions.3839Every connector declares its data status: `PUBLIC_OPEN, PUBLIC_ATTRIBUTED, PUBLIC_RESTRICTED_REDISTRIBUTION, OFFICIAL_OPEN_DATA, LICENSED, DELAYED, INDICATIVE, RESEARCH_ONLY, INTERNAL_ONLY, UNKNOWN, DISABLED`. Every field supports `realtime_status: REALTIME, DELAYED, END_OF_DAY, INDICATIVE, STALE, UNKNOWN`. Without redistribution rights a source may be used internally where legally appropriate (validation, quality analysis) but its raw data must not be exposed publicly. This constraint is built into the architecture (`PUBLICLY_REDISTRIBUTABLE`, `publicQuote` withholding), not treated as documentation.4041# 4. PRODUCT PHILOSOPHY4243Make financial markets feel alive: Bloomberg infrastructure + FlightRadar24-like global visibility + GitHub activity feed + financial search engine + real-time event bus + market observatory — visually modern, approachable, fast, cleaner than a terminal. Avoid old-fashioned terminal aesthetics, excessive neon, fake "AI" gimmicks. Communicate scale, motion, data density, reliability, precision, geography, connectivity, provenance, temporal change.4445# 5. MAJOR PRODUCT SURFACES4647`/ /live /markets /stocks /etfs /indices /crypto /forex /commodities /bonds /rates /futures /options /exchanges /companies /countries /sectors /events /filings /earnings /dividends /splits /ipos /halts /insiders /news /sources /connectors /data-health /atlas /search /compare /api /developers /methodology /status`. Not every section ships in V1; the architecture must support them.4849# 6. HOMEPAGE5051"The world's markets, observed live." Hero: logo, universal search, live market status, instruments tracked, sources active, observations today, exchanges online, event throughput — **never hard-coded; metrics come from system telemetry** (`/v1/stats`). Then: Global Market Pulse (indices, crypto, FX, commodities, rates, top moves), Live Tape (real internal stream), Live Events (canonical events), World Market Map (open/closed, direction, activity), Source Health.5253# 7. THE LIVE FEED (`/live`)5455Global activity stream of market changes and events, filterable by asset class, country, exchange, sector, company, ticker, event type, magnitude, source confidence, time. Items come from canonical normalized events, never frontend hacks. Each item shows confidence and independent source count.5657# 8. CONNECTOR ARCHITECTURE5859The connector system is the core intellectual infrastructure — never a pile of scraping scripts. Standard interface (`packages/connector-sdk`): `metadata()`, optional `discover`, `start/stop` or `poll`, `normalize(raw)`, `health`. Metadata: id, name, organization, sourceType (WEBSOCKET, SSE, XHR, FETCH, GRAPHQL, HTML, RSS, CSV, XML, PDF, OFFICIAL_API, BULK_FILE), jurisdiction, rightsStatus, realtimeStatus, expectedLatency, supportsStreaming/Historical, symbols/exchanges/assetClasses, homepage, enabled, sourceFamily.6061# 9. CONNECTOR DISCOVERY ENGINE6263Automated (browser-based, Playwright/Chromium) discovery understands **how a page receives market information**: network requests, fetch/XHR, WebSocket, SSE, GraphQL, JSON/protobuf payloads, DOM mutations, embedded script state (Next/Nuxt/React hydration), downloadable files. It answers: does the page contain an instrument? how is the price obtained? which request changes with the price? symbol/price/volume/bid/ask/timestamp fields? delayed? multiplexed? subscription message? reconnect? public? authentication? restrictions? stability? Never auto-activate: `DISCOVERED → ANALYZED → VALIDATED → RIGHTS_REVIEW → STAGING → PRODUCTION`. (v0.1 ships a network-level prototype at `POST /v1/admin/discovery`.)6465# 10. NETWORK OBSERVATION6667Capture request URL/method/status/content type/headers/payload size/timing/initiator/resource type, WebSocket URL and frames, EventSource messages, JSON structure, field changes. Never store secrets; automatically redact cookies, Authorization, session identifiers, tokens, personal identifiers, CSRF secrets. Do not reproduce private sessions; discovery runs unauthenticated.6869# 11. AUTOMATIC PAYLOAD UNDERSTANDING7071Payloads vary enormously; normalize them to one canonical schema. Deterministic mappings first. LLMs may propose mappings, classify endpoints, generate connectors/docs/tests — production normalization must be reproducible. Never require an LLM per live tick.7273# 12. CANONICAL INSTRUMENT MODEL7475Global instrument master. Ticker alone never identifies an instrument: consider instrument_id, ticker, exchange, MIC, ISIN, FIGI (where legally usable), CUSIP (licensed), SEDOL, issuer, currency, country, asset class, security type. Example `eq_us_xnas_aapl`. Support symbol aliases (`BRK.B`, `BRK-B`, `BRK/B`, `BRK B`).7677# 13. ASSET CLASSES7879`EQUITY ETF ETN INDEX CRYPTO FOREX COMMODITY FUTURE OPTION BOND TREASURY INTEREST_RATE MUTUAL_FUND ADR GDR REIT WARRANT CFD OTHER`. Not all fields apply to all classes.8081# 14. OBSERVATION MODEL8283Atomic unit = observation: observation_id, instrument_id, field, value, currency, source_id, connector_id, source_timestamp, received_at, latency_ms, rights_status, realtime_status, confidence, raw_ref. Fields: `LAST_PRICE OPEN HIGH LOW PREVIOUS_CLOSE BID ASK BID_SIZE ASK_SIZE VOLUME VWAP MARKET_CAP CHANGE CHANGE_PERCENT OPEN_INTEREST YIELD RATE NAV 52_WEEK_HIGH 52_WEEK_LOW`.8485# 15. RAW DATA RETENTION8687Never discard raw input immediately. Layers: L0 RAW · L1 PARSED · L2 NORMALIZED · L3 CANONICAL · L4 DERIVED — debuggable, reprocessable.8889# 16. MARKET CONSENSUS ENGINE9091Sources disagree; never overwrite one price with the newest request. Weighted consensus using freshness, historical reliability, observed delay, source type, official status, error/disconnect rates, cross-source agreement, timestamp quality, coverage, known delay. Never conceal that a canonical value is an aggregation; expose methodology.9293# 17. SOURCE RELIABILITY SCORE9495Dynamic 0–100 operational score per source (availability 24h/30d, latency p50/p95, freshness, message rate, parse success, schema-change rate, reconnects, outlier rate, consensus agreement, timestamp integrity, coverage). It measures operational reliability, not financial truth.9697# 18. SOURCE DIVERGENCE9899Detect disagreement, mark stale sources, emit `SOURCE_DIVERGENCE`; useful internally and through data-health tools.100101# 19. MARKET EVENT ENGINE102103Transform noisy ticks into meaningful events: `PRICE_CHANGE SESSION_HIGH SESSION_LOW PRICE_BREAKOUT PRICE_GAP VOLUME_SPIKE VOLATILITY_SPIKE SPREAD_WIDENING SPREAD_COMPRESSION LIQUIDITY_CHANGE MARKET_OPEN MARKET_CLOSE TRADING_HALT TRADING_RESUME CORPORATE_ACTION DIVIDEND_DECLARED DIVIDEND_CHANGED STOCK_SPLIT EARNINGS_RELEASE GUIDANCE_CHANGE FILING_PUBLISHED INSIDER_TRANSACTION IPO_STATUS_CHANGE INDEX_CONSTITUENT_CHANGE RATING_CHANGE SOURCE_FAILURE SOURCE_RECOVERY SOURCE_DIVERGENCE`. Events carry id, type, instrument_ids, entity_ids, timestamp, severity, confidence, source_count, supporting observations, metadata.104105# 20. EVENT DEDUPLICATION106107Eleven sources publishing one announcement = one canonical event (entity, type, time window, semantic similarity, identifiers, references); confirmations increment source count.108109# 21–23. TEMPORAL DATABASE, AGGREGATION, COMPRESSION110111PostgreSQL for canonical metadata (instruments, exchanges, connectors, sources, events, configuration); time-series storage fit for purpose (Timescale/ClickHouse at scale — v0.1 uses daily-partitioned Postgres tables with gzip archive tiers); Redis for hot cache/pub-sub when needed; object storage for raw payloads, bulk files, PDFs, cold partitions. Canonical OHLCV bars 1s…1mo, reproducible from source observations. Columnar/delta/dictionary/ZSTD compression, partitioning, HOT (7 d) / WARM (90 d) / COLD tiers. Downsample strategically; never blindly delete history.112113# 24. INTERNAL MARKET BUS114115All normalized events enter one bus (NATS JetStream / Kafka-Redpanda / Redis Streams for small deployments; in-process today with spec topic names). `CONNECTORS → INGEST BUS → NORMALIZERS → CANONICAL BUS → { DB writers, WebSocket gateway, event engine, analytics, anomaly detection, monitoring }`. No connector → frontend coupling.116117# 25–27. STREAMING AND REST118119Own normalized stream (`wss://…/v1/stream`): subscribe/unsubscribe, heartbeat, reconnect, sequence ids, resume tokens, backpressure, rate limits, compression, batching; SSE for simple clients; REST `/v1/instruments, /v1/instruments/{id}, /v1/quotes/{symbol}, /v1/history/{symbol}, /v1/events, /v1/exchanges, /v1/markets, /v1/sources, /v1/status`. Never expose restricted observations without rights.120121# 28–31. COMPANY, EXCHANGE, COUNTRY PAGES, WORLD MAP122123Company/instrument page: live quote, market state, chart, volume, spread, market cap, performance, corporate actions, filings, earnings, dividends, related instruments, source confidence, events, profile, plus the unique **Data Provenance** panel (expandable, not overwhelming). Exchange page: status, local time, country, timezone, MIC, instruments tracked, advancers/decliners, volume, volatility, largest moves, events, halts, source health. Country page: exchanges, indices, largest companies, market cap, currency, bond yields, rates, commodity exposure, events, sector breakdown, hours — an atlas. World map: exchanges/countries/market centers with hours, volatility, activity; time-aware (New York activates at 09:30, Asia lights up as Europe closes). Informative, not decorative.124125# 32–34. SEARCH, COMPARE, MARKET MEMORY126127Universal hybrid search (exact identifiers, prefix, aliases, fuzzy, semantic) grouped by stocks, companies, indices, crypto, FX, commodities, bonds, exchanges, countries, events. `/compare` across asset classes with normalized performance, correlation, volatility, drawdown, event overlays. Market Replay: reconstruct prices, events, direction, volatility, filings, news, exchange status at time T.128129# 35–37. FILINGS, CORPORATE ACTIONS, MARKET HOURS130131Official regulators (SEC, SEDAR+, others, exchange announcements) → company, form type, timestamp, URLs, period, structured fields; never hallucinate contents; link every fact to its document. Corporate actions (dividends, splits, mergers, spin-offs, symbol changes, delistings, listings, rights, buybacks, special distributions) must adjust history correctly. Market-hours engine: time zones, DST, holidays, half days, special closures, overnight/pre/post/futures sessions; states `PRE OPEN HALTED CLOSED POST AUCTION UNKNOWN`; never hard-code 09:30–16:00.132133# 38–42. FRESHNESS, LATENCY, HEALTH, SCHEMA DRIFT, SELF-HEALING134135Every visible quote has freshness ("Updated 320 ms ago" / "Last update 4m 12s ago — data may be delayed"); never present stale as live. Track source/connector/frontend timestamps → source, ingestion, processing, delivery latency. Connector heartbeats and states `HEALTHY DEGRADED STALE RECONNECTING FAILED PAUSED DISABLED`; exponential backoff with jitter; no request storms. Schema fingerprints detect `SCHEMA_DRIFT`; pause unsafe normalization instead of producing wrong prices; notify admin. Self-healing workflow (replay → detect → discovery → proposed mapping → tests → compare → candidate → approval); an LLM never silently deploys production code.136137# 43–50. SDK, FIXTURES, ETHICS, EFFICIENCY, LIMITS, POLLING, CHANGE DETECTION138139Reusable SDK (`defineConnector`, HTTP, browser, WebSocket, SSE, retry, rate limiting, normalization, symbol resolution, timestamp parsing, logging, metrics, fixtures, tests). Every connector stores sample payloads; tests never touch production sources. Respectful crawling: host rate limits, connection limits, identifying user agent, caching, ETag / If-Modified-Since, backoff, robots awareness. Never amplify requests (one stream or bulk endpoint → many instruments; never one socket per symbol when multiplexing exists). Central rate limiter (token/leaky bucket, adaptive) keyed by source/host/connector/route/credential; connectors never sleep on their own. Adaptive polling (market state, activity, change frequency, constraints, importance, ETag). Content fingerprints (DOM/section/semantic/document/field hashes) trigger processing only on change.140141# 51–57. DOCUMENTS, AI, DERIVED METRICS, ANOMALIES, QUALITY, PROVENANCE GRAPH142143Document pipeline: download → hash → store raw → parse → metadata → classify → resolve company → extract facts → verify → index; keep the original. AI enhances (discovery, schema interpretation, classification, summarization, entity resolution, explanations, natural-language search) but never invents prices, trades, filings, market cap, exchange status. Derived metrics (returns, volatility, moving averages, drawdown, z-scores, relative volume, spread, breadth, correlation, beta, momentum, ATR, VWAP) are labelled derived. Statistically grounded anomaly detection with historical baselines. Every canonical datapoint may carry quality/confidence/source_count/freshness/dispersion/validation flags. Long-term provenance graph: Instrument → Observation → Connector → Source → Organization → URL; Company → Filing → Regulator → Document.144145# 58–61. ADMIN CONSOLE, CONNECTOR PAGES, DISCOVERY UI, STATUS146147`/admin` (strongly secured): overview, connectors, sources, instruments, entity resolution, observations, data quality, divergence, failed jobs, schema changes, events, storage, workers, streams, API, security, legal/data rights, logs. Per connector: status, type, host, rights, real-time status, last message/error, rates, latency, coverage, schema fingerprint, version, health history; actions pause/resume/restart/test/discover/fixtures/schema/sanitized payloads; never expose secrets. Discovery UI: submit URL → observed requests/WebSockets/SSE/JSON endpoints → likely quote source + field mapping → "Generate connector candidate" (not production). Public `/status` without infrastructure details.148149# 62–75. TECHNOLOGY, SERVICES, DEPLOYMENT, PROXY, FAILOVER, OBSERVABILITY, LOGGING, SECURITY, PERFORMANCE, MOBILE150151Frontend: Next.js, React, TypeScript, Tailwind, light state, WebSocket client, MapLibre/D3 for maps, ECharts/lightweight-charts/D3 by need; performance-driven library choices. Backend: TypeScript/Node for application services, Python for analytics/AI, Go/Rust only for proven high-throughput needs, PostgreSQL, ClickHouse/Timescale, Redis, NATS/Redpanda, Playwright; no new languages without clear reason. Services (web, api, stream-gateway, connector-manager/workers, discovery, normalization, entity-resolution, event-engine, analytics, database-writer, document-worker, scheduler, admin-api, health-monitor) start as a modular monolith and split when load justifies. Containerize where possible, health checks, safe restarts, no hand-started shells (MacLustr: PM2 via `mld`). TLS everywhere via reverse proxy (Caddy on the MacLustr Tunnel); no internal ports exposed. Workers support leases/locks/heartbeats/reassignment. Observability: Prometheus/Grafana/OpenTelemetry, structured JSON logs (never passwords, tokens, cookies, keys), metrics for observations/s, events/s, connector states, WebSocket connections, stream lag, latencies, queue depth, resources, storage growth. Security: HTTPS, secure headers, strict CORS, validation, parameterized SQL, rate limits, secret manager, admin MFA, least privilege, segmentation, dependency scanning, audit logs; isolated browser workers (no secrets, no DB credentials, temporary profiles); SSRF protection (localhost, private ranges, internal hostnames, metadata endpoints, redirects); API keys/quotas/analytics/revocation; service auth internally. Web performance: fast first paint, streaming updates without re-render storms, virtualized feeds, efficient charts, edge caching, SSR for public pages, granular subscriptions. Mobile-first surfaces (search, watch view, overview, instrument page, live feed, events, chart, world markets): bottom navigation, swipe-friendly cards, compact tape, sticky quote header, responsive charts, accessible touch targets, no horizontal overflow.152153# 76–84. DESIGN SYSTEM, MODES, COLOR, CHARTS, LIVE UI, SEO, SITEMAPS, LICENSING, DISCLAIMER154155Premium, dense but breathable, institutional, modern, high-information; not boxy, cartoonish or crypto-casino; hierarchy via spacing, typography, dividers, alignment, subtle surfaces — not isolated rounded cards. Dark and light modes first-class; avoid pure black; never red/green as the only signal. Color semantics: positive, negative, neutral, stale, warning, failure (+ icons/text). Charts: 1D 5D 1M 3M 6M YTD 1Y 5Y MAX, live intraday, line/area/candlestick/volume/comparison, no fake interpolation. Live UI: subtle directional flash, batch updates at 10–20 refresh cycles/s max. SEO: crawlable public pages with title/description/canonical/OpenGraph/structured data; no thin pages; segmented sitemaps (stocks, companies, exchanges, countries, indices, crypto, commodities). `/methodology`, `/data-sources`, `/licensing` explain methodology, delays, provenance, derived calculations, limitations, not-investment-advice, venue differences, licensing. Disclaimer: "Market Atlas is an informational market-data and analytics platform. Information may be delayed, indicative, incomplete, or sourced from third parties. Market Atlas is not an exchange, broker, investment adviser, or execution venue." (final legal review before commercial launch).156157# 85–94. TESTING, LOAD, REPLAY, MIGRATIONS, BACKUPS, RETENTION, REPROCESSING, VERSIONING158159Tests: unit, integration, connector (valid/missing/null/unexpected schema/stale timestamp/invalid symbol/duplicate/disconnect/reconnect/rate-limit/server error), schema, normalization, entity resolution, WebSocket, API, database, frontend, end-to-end, load, security. Consensus scenarios: all agree, one stale, one outlier, two clusters, single source, differing timestamps, market closed. Load: 100K obs/s, 10K clients, 100K subscriptions, 1M cached instruments, large historical queries — scale progressively. Internal replay feeds stored observations through normalizer/consensus/events. Versioned forward-only migrations; never edit production tables by hand. Automated backups (Postgres, metadata, connector config, object storage replication) — a backup never restored is not trusted. Configurable retention by source/asset class/field/resolution/rights (raw WS frames 7 d; normalized ticks permanent compressed; 1s/1m/1d bars permanent). Reprocess raw → canonical v2 with lineage; canonical datasets carry `normalizer_version`, `consensus_version`.160161# 95–105. IDENTIFIERS, SOURCES DIRECTORY, DATA HEALTH, "WHY THIS PRICE?", COVERAGE, PRIORITY, TIERS, MULTI-SOURCE, CORRELATION, LINEAGE, DELAY DETECTION162163Stable public ids (`eq_us_xnas_aapl`, `crypto_btc_usd`, `fx_eur_usd`, `index_us_spx`), never auto-increment ids. `/sources` directory by category with status/coverage/frequency/latency/rights/last observation — no endpoint details. `/data-health`: sources online, global freshness, markets covered, consensus, incidents — real metrics only. **"Why this price?"** on every instrument: canonical price, independent observations, dispersion, newest/oldest included, weights, reasons. Market Atlas coverage score per country/exchange (observation coverage, not market quality). Connector priority engine (market cap represented, demand, coverage gap, redundancy, legal accessibility, reliability, cost). Tier 1: major exchanges, regulators, large-cap IR, major public quote sources, crypto exchanges with public feeds, central banks, government debt, commodity references; Tier 2: regional exchanges, ETF issuers, index providers, economic data, specialized sites; Tier 3: small exchanges, local portals, specialized classes, regional announcements. Target ≥ 3 independent observations for major instruments (5–10+ for the most important). Source families: websites sharing an upstream are not independent; weight independence; infer lineage from timestamps/identical payloads/rounding/documentation/attribution — never assert hidden commercial relationships without evidence. Estimate delay against trusted references (0–2 s, 15 s, 1 min, 15 min, 20 min, EOD) and label estimates as such.164165# 106–121. MARKET CLOCK, TOPICS, IDEMPOTENCY, ORDERING, CLOCKS, TIMESTAMP TRUST, ENTITY & EXCHANGE GRAPHS, CURRENCY, LATER FEATURES, DEVELOPER PORTAL, API CONVENTIONS166167Global market clock (New York, London, Tokyo, Hong Kong, Sydney) with states. Topics: `raw.{source} normalized.quote normalized.trade normalized.event canonical.quote market.event connector.health source.divergence filing.detected corporate_action.detected`. Idempotent ingestion via fingerprints (source + instrument + timestamp + value + type); no duplicate rows on reconnect replay. Out-of-order messages: keep source sequence/source timestamp/receive timestamp. NTP-synchronized clocks. Timestamp trust: exchange / source / connector / unknown → influences confidence. Company ≠ security (common stock, ADR, preferred, bonds, subsidiaries). Exchange graph (operator, venue, MIC, country, timezone, asset classes, sessions). Store original currency; converted values record FX source/time. Later: watchlists, saved searches, alerts (price, volume, volatility, filing, earnings, dividend, halt, divergence, highs/lows, events), dashboards, API keys, exports, notebooks — the public atlas must be useful first. No brokerage in V1 (no execution, custody, deposits, leverage). `/developers`: REST, WebSocket, SSE, schemas, examples, SDKs (JS, Python, Go), status, rate limits. Versioned `/v1/`, no silent breaking changes; envelopes `{data, meta{request_id, timestamp, data_status}}`; structured errors `{error{code, message}}`, no stack traces.168169# 122. PROJECT STRUCTURE170171```text172market-atlas/173├── apps/          web · api (engines, stream gateway, admin, worker roles)174├── packages/      market-model · connector-sdk (+ future: schemas, database, streaming, logging, security, ui)175├── connectors/    one directory per connector (index.ts, fixtures/, README.md)176├── infra/         migrations (+ docker, monitoring, deployment as they come)177├── deploy/        mld manifest + render script (MacLustr)178├── docs/          ARCHITECTURE · CONNECTORS · API · FRONTEND · DEPLOY179├── scripts/ tests/180└── CLAUDE.md181```182183# 123. CODING RULES FOR CLAUDE1841851. Read this file before major architecture changes. 2. Inspect existing code before creating duplicate systems. 3. Prefer reusable infrastructure over one-off scripts. 4. Do not break existing connectors. 5. No hard-coded secrets. 6. No hard-coded production IPs. 7. No fake market data. 8. Never silently replace real data with mock values. 9. Strict TypeScript. 10. Validate external payloads. 11. Treat all external content as untrusted. 12. Add meaningful tests. 13. Keep the UI responsive. 14. Preserve data provenance. 15. Never present unknown freshness as real time. 16. Never remove data-rights metadata. 17. Prefer graceful degradation. 18. Add monitoring to new ingestion systems. 19. Document unusual source behaviour. 20. Keep connector code isolated from core business logic.186187# 124–127. WHEN ADDING A CONNECTOR / SCHEMA / FIELD / AI188189Connector = implementation + metadata + rights classification + normalization mapping + fixtures + unit tests + health check + documentation (no undocumented connectors). Schema change = inspect database, API, stream schemas, frontend consumers, tests, historical processing — no silent breaking changes. New market field = canonical name, type, unit, applicable asset classes, nullability, source-timestamp handling, display formatting. Adding AI: if deterministic code can solve it, use deterministic code; AI only where interpretation is needed; no LLM dependencies for basic parsing.190191# 128–137. PERFORMANCE, QUERIES, CACHE, CONTRACTS, EVENT SCHEMA, DISASTER MODES, CIRCUIT BREAKERS, SHUTDOWN, BACKPRESSURE, PRIORITY192193Never solve scale only with bigger servers — first inspect duplication, query structure, indexes, cacheability, batching, serialization, compression, memory copies, re-renders. Heavy historical queries: partition pruning, indexes, no `SELECT *`, limits, pagination, downsampling. Cache instrument/exchange metadata, latest quotes, market status, popular pages, aggregates — never user-specific or stale critical values incorrectly. Strongly versioned contracts (zod / JSON Schema / Protobuf / Avro); explicit bus contracts; event schema `{version, id, type, instrument_id, timestamp, confidence, severity, data, sources}`. Define degraded behaviour when Redis/ClickHouse/source/gateway/worker/browser/network/schema fail (e.g. historical DB down → live stream continues, charts unavailable). Circuit breakers on failing sources. Graceful shutdown: stop new jobs, flush, commit, release leases, close, exit. Bounded queues, batch writes, disk-backed queues where needed, priority tiers, drop policy only for expendable data — never silently drop canonical events. Priorities: P0 halts/critical events, P1 canonical quotes, P2 ticks, P3 secondary analytics, P4 reprocessing.194195# 138–151. ONBOARDING PIPELINE, EXPANSION, HISTORY, ARCHIVE, KNOWLEDGE GRAPH, CONSTITUENTS, DIFFS, EXPLORER, "MARKET CHANGES", BREADTH, ACTIVITY INDEX, PRECISION, PREDICTIONS196197Automated onboarding (candidate → metadata → discovery → rights → fixtures → mapping → validation → comparison → staging → observation → production). Long-term source discovery jobs feed a review queue; no unlimited crawling of arbitrary domains. Store source change history (versions, schema, endpoints, availability, delay class, rights class) and allow historical source-quality queries. Market observation archive: "what did the observable financial web say at time T?". Knowledge graph (Company, Instrument, Exchange, Country, Sector, Executive, ETF, Index, Filing, Corporate Action, Event, Source; LISTED_ON, HEADQUARTERED_IN, MEMBER_OF_INDEX, HOLDS, ISSUED_BY, FILED, ANNOUNCED, OBSERVED_BY). Index membership with effective dates. Diff log for slowly changing facts (old/new/detected/effective/source). Future `/explore` data laboratory. "Market changes" product (what changed in 1 min / 5 min / 1 h / 1 d: instruments > 1 %, filings, halts, new highs, source outages). Global breadth engine (advancers, decliners, highs/lows, median/weighted return, sector/country breadth, regional scopes). Internal activity score for ranking what is happening now — never a prediction. No fake precision (no 99.999 % confidence without justification; transparent methodology). Core product = observation, aggregation, historical intelligence, analytics — not guaranteed predictions or recommendations.198199# 152–158. V1 DEFINITION, CONNECTOR GOAL, PHASES, SUCCESS METRICS, NORTH STAR200201V1: homepage, universal search, market overview, live feed, equities, crypto, major FX, major indices, instrument pages, charts, basic map, sources directory, data health, connector framework, 5–20 strong production connectors, health monitoring, canonical normalization, basic consensus, PostgreSQL, time-series storage, cache, WebSocket gateway, admin console. Prove diversity: WebSocket, XHR/fetch, official source, HTML change detection, regulatory/filing source, bulk structured source. Phase 2: 50–100 connectors, multiple exchanges, corporate actions, filings, world map, consensus, country pages, advanced events. Phase 3: hundreds/thousands of connectors, discovery automation, global instrument graph, replay, developer APIs, advanced analytics, institutional monitoring. Phase 4: answer *what is happening, where, how fast, which sources confirm it, how reliable, what happened before, what else moved*. Metrics: active connectors, uptime, instruments covered, observations/day, median freshness, consensus coverage, quality incidents, historical depth, stream latency; product: searches, views, live-feed engagement, return users, API usage. **North star: high-quality, provenance-backed market observations successfully processed.**202203# 159–166. BRAND, STANDARD, THE MOST IMPORTANT RULE, MINDSET, IMPLEMENTATION ORDER, FIRST ENTITIES, NO 1,000 SCRAPERS, EXPECTATION204205Brand: **Market Atlas**, `www.market-atlas.co` — "The live map of global markets." Every feature reinforces GLOBAL, LIVE, HISTORICAL, VERIFIABLE, MULTI-SOURCE, CONNECTED, SEARCHABLE, FAST; otherwise reconsider it. **The proprietary asset is not the website**: it is connectors + normalization + entity graph + source reliability + consensus + historical observations + market events; the frontend is the best interface to that system. Build for 100,000+ instruments, 1,000+ sources, hundreds of millions of observations/day, years of history, thousands of stream consumers — incrementally, without premature distributed complexity. Implementation order: audit → structure → canonical schemas → DB models → connector SDK → rights metadata → representative connectors → ingestion bus → normalization → resolver → current-state cache → canonical quotes → WebSocket gateway → historical persistence → health metrics → consensus → REST → homepage → instrument page → live feed → admin dashboard → discovery prototype → world map → observability → load test/harden. First entities: Instrument, Company, Exchange, Country, Currency, Source, Connector, ConnectorVersion, Observation, CanonicalQuote, MarketEvent, CorporateAction, Filing, SourceHealth, ConnectorHealth, SymbolAlias, TradingSession. Strategy: excellent connector infrastructure → validate with 10 heterogeneous sources → improve → 50 → automate onboarding → hundreds → thousands; twenty excellent connectors beat a thousand fragile scripts. Claude behaves as a senior systems/data/quant-data/security/product engineer: inspect what exists, preserve working behaviour, smallest coherent architectural improvement, implement rather than describe, test, verify production compatibility, update documentation, leave the repository deployable. Never substitute placeholders for requested real functionality unless an upstream dependency makes it unavoidable; when a source cannot legally or technically provide real-time information, label it accurately.206207# END GOAL208209Market Atlas should function like a live nervous system for financial markets: thousands of independent sources emit observations; Market Atlas listens, resolves what they refer to, measures their reliability, compares them, records them, detects what changed, remembers what happened, and exposes the resulting market state through one unified global interface — **https://www.market-atlas.co**.210