spb/hfmarketdata

1# Fundamentals (SEC EDGAR) — architecture, schema, ingestion, point-in-time23Chantier 7 of the v2 upgrade. Module `hfmarketdata/api/fundamentals/` (router `/v1/fundamentals`), plus4`stream/` (WebSocket `/v1/stream`) and `bulk/` (`/v1/bulk/fundamentals/{year}.parquet`). Data source: the SEC5EDGAR XBRL APIs (`companyfacts`, `submissions`), never proxied live to users. Ratio formulas:6[fundamentals-ratios.md](fundamentals-ratios.md). Stream protocol: [asyncapi.yaml](asyncapi.yaml).78## 1. Architecture910```11        EDGAR (data.sec.gov / www.sec.gov)            ≤ 10 req/s token bucket, backoff 429/503, UA = settings.sec_user_agent12                 │ companyfacts + submissions (+ MetaLinks.json of the latest 10-K, Atom feed, daily index)13                 ▼14  data_root/edgar/raw/**.json.gz          raw cache (replayable offline, one file per document)15                 │ normalize.facts_frame16                 ▼17  data_root/edgar/facts/cik={cik}/facts.parquet     RAW FACTS LAKE (DuckDB) — every fact instance, versioned by (accn, filed)18                 │ normalize.normalize_company (mapping.py priority lists, fiscal calendar, versions, derivation)19                 ▼20  SQLite (core.db):  edgar_companies · edgar_filings · fund_statements (wide, versioned) · fund_mapping ·21                     fund_mapping_log · fund_coverage · fund_ingest_state · fund_latest (screener)22                 │                                   │23                 ▼                                   ▼24  /v1/fundamentals/* (service.py, DuckDB asof join with the price lake for ratios)      Redis `filings` → /v1/stream25  data_root/bulk/fundamentals_{year}.parquet (bulk/build.py, ETag/304, quota exempt)26```2728Why two stores:2930* **Parquet + DuckDB for facts** — ~25 000 facts per large filer, ~200 M for the universe; columnar, cheap to31  rescan when the mapping changes (re-normalisation never re-downloads), `view=as_reported` and32  `/facts/{concept}` read it directly with predicate pushdown. Never mutated: facts are appended by filing.33* **SQLite for standardized statements** — needs indexes for point-in-time lookups (`ticker, period_end,34  filed_date`), small (≈ 1–2 M rows for 7 600 companies × 15 years × 3 statements × ~1.2 versions), transactional35  replace per company, and the screener table lives next to it. `coverage` JSON per row explains every null.3637## 2. Schema (SQLite, `fundamentals/models.py`)3839| table | key | purpose |40|---|---|---|41| `edgar_companies` | `cik` | `ticker` (canonical = first SEC listing), `tickers` (all share classes: `["GOOGL","GOOG"]`), name, `sic`, `exchange`, `fiscal_year_end` (MMDD), `status` active/delisted, `ticker_history` JSON, `facts_updated_at`, `normalized_at` |42| `edgar_filings` | **(cik, accn)** | form, filed_date, period_of_report, primary_doc (full EDGAR URL), is_amendment, is_xbrl, parsed_at — indexes (cik, filed_date), accn. An accession is shared by co-registrants (Entergy 65984 + its utilities file one 10-Q/8-K): one row per registrant. Legacy key `accn` alone → `scripts/migrate_edgar_filings.py` (see §7) |43| `fund_coverage_blob` | id; unique sha1 | dictionary of distinct `coverage` JSON documents (`fundamentals/coverage_store.py`); `fund_statements.coverage_id` references it, the inline `coverage` column stays NULL for new rows (legacy inline JSON still read; `scripts/migrate_coverage.py` converts it) |44| `fund_statements` | id; unique (cik, statement, fiscal_year, fiscal_quarter, accn) | **wide, versioned**: cik, ticker, statement (income/balance/cashflow), fiscal_year, fiscal_quarter (1–4, **0 = annual**), period_start, period_end, calendar_quarter (`2024Q1`), form, accn, filed_date, derived, restated, currency, coverage JSON, mapping_version + one REAL column per account (49). Indexes: point-in-time `(ticker, period_end, filed_date)`, `(cik, statement, fiscal_year, fiscal_quarter, filed_date)`, `(statement, fiscal_quarter, calendar_quarter)` for frames |45| `fund_mapping` | id; unique (version, account, taxonomy, tag) | the prioritized mapping seeded from `mapping.py` — **183 rows** (49 accounts, of which 5 computed and 2 auxiliary), version `2026.09.1` (`GET /v1/fundamentals/_mapping`) |46| `fund_mapping_log` | (cik, taxonomy, tag) | tags seen but not mapped: standard-taxonomy tags outside the mapping (`is_extension=false`, occurrences, first/last seen, sample accn) and **company extensions** found in the statements of the latest 10-K via `MetaLinks.json` (`is_extension=true`, `hint_account`) |47| `fund_coverage` | ticker | first/last period, quarters, annuals, filings, completeness % (overall + per statement), `missing_accounts` {account: reason, periods}, `gaps` (missing fiscal quarters), derived/restated counts, extensions logged |48| `fund_ingest_state` | key (backfill/incremental/reconcile) | last run/success, last RSS check, `lag_seconds`, companies total/done, failures + samples, `mapping_failure_rate`, requests, events published — served by `GET /v1/fundamentals/_health` |49| `fund_latest` | ticker | precomputed screener row: identity, price/price_date, TTM flows + latest balances (public accounts) + every ratio, `reasons` JSON, `shares_source` |5051Raw facts lake columns: `cik, taxonomy, tag, unit, fy, fp, form, start, end, val, accn, filed, frame`52(de-duplicated on `(taxonomy, tag, unit, start, end, accn)` keeping the latest `filed`).5354## 3. Standard chart of accounts and mapping (`mapping.py`)5556Exactly the spec's accounts — income (16), balance (19), cash flow (12) — plus two **auxiliary** inputs that are57stored but not part of the public chart: `depreciation_amortization` (for `ebitda`) and `shares_outstanding`58(dei cover-page shares, all classes summed, for `market_cap`). Computed accounts (`*` in the spec): `ebitda`,59`total_debt`, `net_debt`, `working_capital`, `free_cash_flow` — never read from a tag, formula in `coverage`.6061Each account lists its tags **in priority order**; the first tag with a fact for the period wins and the chosen62tag + priority are written in `coverage[account]`. Examples (full table: `/v1/fundamentals/_mapping`):6364| account | priority list (us-gaap unless noted) | why the fallbacks |65|---|---|---|66| revenue | Revenues → RevenueFromContractWithCustomerExcludingAssessedTax → SalesRevenueNet → RevenueFromContractWithCustomerIncludingAssessedTax → SalesRevenueGoodsNet → SalesRevenueServicesNet → RevenuesNetOfInterestExpense → InterestAndDividendIncomeOperating → RegulatedAndUnregulatedOperatingRevenue → OperatingLeasesIncomeStatementLeaseRevenue → ifrs-full:Revenue | ASC 606 (2018) replaced SalesRevenueNet; banks/utilities/REITs use their own top line |67| net_income | NetIncomeLoss → NetIncomeLossAvailableToCommonStockholdersBasic → ProfitLoss → IncomeLossFromContinuingOperations → ifrs-full:ProfitLossAttributableToOwnersOfParent | ProfitLoss includes NCI, used when the parent figure is absent |68| cash_and_equivalents | CashAndCashEquivalentsAtCarryingValue → CashCashEquivalentsRestrictedCashAndRestrictedCashEquivalents → Cash → CashAndDueFromBanks → CashCashEquivalentsAndShortTermInvestments | ASU 2016-18 presentation incl. restricted cash |69| short_term_debt | DebtCurrent → LongTermDebtAndCapitalLeaseObligationsCurrent → LongTermDebtCurrent → ShortTermBorrowings → CommercialPaper → NotesPayableCurrent | when DebtCurrent is absent the disjoint components are **summed** (`components: true`); Apple = CommercialPaper + LongTermDebtCurrent |70| sga_expense | SellingGeneralAndAdministrativeExpense → G&A + Selling components | split SG&A summed, flagged |71| capex | PaymentsToAcquirePropertyPlantAndEquipment → PaymentsToAcquireProductiveAssets → PaymentsForCapitalImprovements → … | positive = outflow (EDGAR convention) |7273Accounting identities used as fallbacks (flagged `identity: true`): `gross_profit = revenue − cost_of_revenue`,74`total_liabilities = LiabilitiesAndStockholdersEquity − total_equity` (else `total_assets − total_equity`),75`operating_income = revenue − CostsAndExpenses`. Units: USD raw (not thousands), `shares`, `USD/share`; a76non-USD filer keeps its currency in `currency` and `coverage[account].currency` (20-F: ifrs-full tags are in77the lists). Extension tags (`aapl:`, `shak:`…) are **logged, never guessed**.7879## 4. Normalisation (`normalize.py`)80811. **Facts → frame**, de-dup by `(tag, unit, start, end, accn)` keeping the latest filed.822. **Fiscal calendar.** The `fy`/`fp` fields of a companyfacts fact describe the *filing*, not the fact83   (Apple's 10-Q for Q2 FY2025 re-reports Q2 FY2024 with `fy=2025, fp=Q2`). So the fiscal period of every fact84   is derived from its `end` against the fiscal year ends learnt from the 10-K report dates85   (`FiscalCalendar`, ±7-day snap for 52/53-week filers, MMDD extrapolation for years without a 10-K) and86   `fy`/`fp` are only used as a **sanity check on the filing's own period** (mismatches are counted in87   `stats.fiscal_mismatches`; 0 for the three prototypes). Apple Q2 FY2024: `end=2024-03-30`, FY end88   2024-09-28 → 182 days → quarter 4 − round(182/91.3) = **2**. `calendar_quarter` comes from `period_end`89   (an end in the first 7 days of a month belongs to the previous month: 2025-01-03 → 2024Q4).903. **Span classification** of duration facts: 75–105 days = quarter, 165–195 = 6-month YTD, 255–290 = 9-month91   YTD, 340–380 = fiscal year; anything else (stub periods) is ignored.924. **Resolution per filing** (accession) and period → `Resolved` values + coverage (section 3).935. **Versioning.** Filings are replayed in `filed` order. A period gets a new row only when a filing changes or94   completes what was known (`restated=true` when a previously served number changed; same-accession95   reported + derived pieces are one version). A later filing that re-reports a period with a *lower-priority*96   concept (cash incl. restricted cash in a comparative column) does **not** override the better concept.97   Unique key `(cik, statement, fiscal_year, fiscal_quarter, accn)`.986. **Derivation** (flagged `derived=true`, formula in `coverage[account].derived`): cash-flow statements in 10-Qs99   are year-to-date only, income statements often carry 9-month YTD facts — `Q2 = YTD6 − Q1`,100   `Q3 = YTD9 − YTD6` (or `YTD9 − Q1 − Q2`), `Q4 = FY − YTD9` or `FY − (Q1+Q2+Q3)`; only when every input101   belongs to the same fiscal year, same unit, and was known at that filing date. Weighted share counts use102   `4×FY − (Q1+Q2+Q3)` (`approx: true`); EPS Q4 = FY − ΣQ (`approx`). Nothing is derived when a quarter is missing.1037. **Computed accounts** (`ebitda`, `total_debt`, `net_debt`, `working_capital`, `free_cash_flow`) are attached104   to the latest version of the period with their formula; assumptions are flagged105   (`short_term_debt_assumed_zero`, `short_term_investments_assumed_zero`).1068. **TTM** (`ttm()`): sum of the last four *consecutive* fiscal quarters for flows, latest quarter for balances107   and share counts, EPS = sum of four quarterly EPS (`approx`). Windows with a hole are skipped108   (`missing_quarters` reason).109110### Point-in-time (anti look-ahead)111112Every row carries the `filed_date` of the filing that made it known. Serving "latest" = the version with the max113`filed_date` per (statement, fiscal_year, fiscal_quarter). `as_of=D` = the same selection restricted to114`filed_date ≤ D` (`normalize.select_as_of`). Consequences:115116* Apple's Q2 FY2024 income statement does not exist on 2024-05-02 and exists on 2024-05-03 (filing date).117* a 10-K/A or a later 10-K that restates a quarter creates a **new version**; `as_of` before the amendment returns118  the original numbers, after it the restated ones (`restated=true`).119* `/ratios?as_of=D` uses the fundamentals known at D **and** the last close at or before D;120  `/ratios/daily` builds one snapshot per `filed_date` and ASOF-joins it to the daily closes in DuckDB121  (`p.date >= s.valid_from`), so a 10-Q filed May 3 only affects May 3 onwards.122* `frames/{concept}?as_of=D` and the bulk files keep `accn`/`filed_date` for the same reason.123124## 5. Worked example — Apple, Q2 FY2024 (10-Q filed 2024-05-03, quarter ended 2024-03-30)125126`GET /v1/fundamentals/AAPL/statements?statement=income&period=quarterly&from=2024-03-30&to=2024-03-30`127128| account | value | provenance (`coverage`) |129|---|---|---|130| revenue | 90 753 000 000 | `us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax` (priority 2 — Apple does not tag `Revenues`) |131| cost_of_revenue | 48 482 000 000 | `CostOfGoodsAndServicesSold` |132| gross_profit | 42 271 000 000 | `GrossProfit` |133| operating_income | 27 900 000 000 | `OperatingIncomeLoss` |134| net_income | 23 636 000 000 | `NetIncomeLoss` |135| eps_diluted | 1.53 | `EarningsPerShareDiluted` (USD/shares) |136| shares_diluted | 15 464 709 000 | `WeightedAverageNumberOfDilutedSharesOutstanding` |137| ebitda | 30 736 000 000 | computed `operating_income + depreciation_amortization` (2 836 M from the cash-flow statement) |138| interest_expense, dividends_paid | null | `reason: no_mapped_tag` (Apple stopped tagging interest expense in FY2023) |139140Balance sheet (same period): total_assets **337 411 000 000**, cash 32 695 M, short-term investments 34 455 M,141short_term_debt 12 759 M (`CommercialPaper` + `LongTermDebtCurrent`, `components: true`), long_term_debt14291 831 M, total_debt 104 590 M (computed), total_equity 74 194 M, shares_outstanding 15 334 082 000 (cover page1432024-04-19).144145Cash flow Q2 FY2024 (`derived=true`): operating_cash_flow **22 690 000 000** = YTD6 62 585 M − Q1 39 895 M146(`derived: "YTD6-Q1"`), capex 1 996 M, free_cash_flow 20 694 M (computed).147148Q4 FY2024 (derived from the 10-K filed 2024-11-01): revenue 94 930 M = FY 391 035 M − YTD9 296 105 M.149TTM at Q2 FY2024: revenue 381 623 M, net income 100 389 M, diluted EPS 6.43 (sum of 4 quarters).150151`GET /v1/fundamentals/AAPL/ratios?as_of=2024-05-03` (real close 183.38 on 2024-05-03): market_cap =152183.38 × 15 334 082 000 = 2 812 T; pe = 183.38 / 6.43 = 28.5; pb = 37.9; gross_margin = 173 966 / 381 623 = 45.6 %;153net_margin 26.3 %; roe = 100 389 / 74 194 = 135 %; current_ratio = 128 416 / 123 822 = 1.04; debt_to_equity 1.41;154enterprise_value = market_cap + 104 590 M − 32 695 M − 34 455 M; forward_pe = null (`no_estimates`);155interest_coverage = null (`missing:interest_expense`); revenue_growth_yoy = 381 623 / 385 095 − 1 = −0.9 %.156(These numbers are asserted in `tests/test_fundamentals_unit.py::test_ratio_formulas_on_apple_q2_fy2024`157against the recorded fixture; in the test lake the *price* is synthetic, so the API test checks the formulas, not158the absolute valuation.)159160## 6. Prototype results (full companyfacts, 2026-09-04)161162| company | CIK | facts | filings | statement versions | derived rows | restated rows | completeness | extensions logged |163|---|---|---|---|---|---|---|---|---|164| Apple (FYE late Sept, 52/53 weeks) | 320193 | 25 135 | 70 | 412 | 113 | 76 | 87.2 % | 3 |165| Microsoft (FYE June 30; 10-K carries quarterly data) | 789019 | 32 671 | 68 | 458 | 60 | 120 | 85.9 % | 4 |166| Shake Shack (Russell 2000; FYE last Wednesday of Dec, 2025 → Dec 31) | 1620533 | 18 358 | 47 | 266 | 66 | 42 | 68.8 % | 9 |167168Shake Shack was chosen as the small cap because its income statement uses the company extension169`shak:OperatingMaterialsExpense` (food and paper costs) plus `us-gaap:LaborAndRelatedExpense` / `OccupancyNet`170instead of any cost-of-revenue concept: `cost_of_revenue` and `gross_profit` are therefore **null with171`no_mapped_tag`**, the extension is logged in `fund_mapping_log` with `hint_account=cost_of_revenue` and shown172in `/coverage.custom_extensions` — exactly the "log, don't guess" behaviour. Its 52/53-week calendar173(2023-12-27, 2024-12-25, then a change to 2025-12-31) and multi-class shares also exercise the calendar and174dei logic. Reconciliation against a fresh EDGAR fetch: 0 discrepancies for AAPL and SHAK.175176Latency on the test lake (TestClient, 25 runs, p95): statements 4–7 ms, ratios 7 ms, ratios/daily 1 year 30 ms177(2.5 years 51 ms), screener 3 ms, frames 3 ms, filings 3 ms, coverage 1 ms — far under the 300 ms / 1 s targets;178the screener reads a precomputed table so it stays O(rows in `fund_latest`) at universe scale.179180## 7. Ingestion181182* **Universe** (`ingest.sync_universe`): `company_tickers.json` + `company_tickers_exchange.json` restricted to183  the tickers present in `parquet/stock/1day/*` and `parquet/etf/1day/*`. One CIK with several tickers184  (GOOG/GOOGL, BRK-A/BRK-B) = one company, `tickers` lists the classes and any class resolves in the API. A185  ticker that leaves the SEC list is appended to `ticker_history` and the company becomes `delisted` (its data186  keeps being served). ETFs that are not SEC operating filers (SPY…) are simply absent →187  `FUNDAMENTALS_NOT_AVAILABLE`.188* **Backfill** (`scripts/edgar_backfill.py`): per CIK `companyfacts` + `submissions` (+ `MetaLinks.json` of the189  latest 10-K, 1 request) → lake → statements → coverage → screener row; manifest190  `data_root/edgar/backfill_manifest.json` makes it resumable (`ok` / `no_facts` are skipped, `error` retried;191  `--force` redoes). `--from-zip companyfacts.zip` reads the SEC bulk archive instead of the API. Ends with the192  bulk Parquet files.193* **Incremental** (`scripts/edgar_incremental.py`, every 2 min): Atom `getcurrent` feed per form194  (10-K, 10-Q, 8-K, 20-F — 4 requests) + the daily master index of the last two **business days** as a safety195  net (`Archives/edgar/daily-index/{year}/QTR{q}/master.YYYYMMDD.idx`; the SEC answers 403 for a day that has no196  index — weekend, holiday — which is treated as "not published", not as an error) → new **(cik, accession)197  pairs** of `TRACKED_FORMS` (10-K/10-Q/8-K/20-F/40-F/6-K, transition reports 10-KT/10-QT and every `/A`198  amendment) → refetch companyfacts (cache bypass) → re-normalise → statements rewritten **only if they changed**199  (fingerprint of the rows) → publish one `filing` event per registrant (Redis `filings` pub/sub +200  `filings:stream` capped at 1 000) → lag written to `fund_ingest_state`; the bulk extracts are rebuilt only when201  rows changed. Termination guards: Atom entries are filtered by form (the `type=8-K` feed also returns 8-K/A —202  an untracked form never landed in `edgar_filings` and re-ingested its company every cycle); a Redis SET203  `edgar:seen_accn` (TTL 7 d) remembers every pair handled, a HASH `edgar:failed_accn` counts transient errors204  (3 attempts, then the pair is left to the reconcile job); `companyfacts_404` (no XBRL facts) still records the205  filings and is not a failure. `failure_samples` accumulate (last 10, with the CIK) instead of being wiped by206  the next clean cycle. The efts full-text search needs a query term and is therefore not used as a "list207  everything" source.208* **Reconcile** (`scripts/edgar_reconcile.py`): 20 random companies refetched and re-normalised in memory,209  every public account of every latest period compared to the DB (`--fix` re-ingests the differing ones);210  result in `fund_ingest_state.reconcile` and `_health`. Exit code 1 on discrepancies for alerting. Ends with211  `PRAGMA incremental_vacuum` (effective once the database runs `auto_vacuum=INCREMENTAL`, which `init_db`212  switches on automatically for a database under 50 MB; an existing big file needs the one-off213  `scripts/migrate_coverage.py --vacuum`).214* **Schema maintenance** (`fundamentals/migrations.py`, run by `init_db()` at every start, idempotent):215  `fund_statements.coverage_id` column, `auto_vacuum` on a young DB, and the `edgar_filings` key216  `accn → (cik, accn)` — migrated inline only when the table has ≤ 300 000 rows; above that the API logs a217  warning and keeps running on the legacy key (co-registrant copies are skipped by `ON CONFLICT DO NOTHING`,218  the Redis guard stops the poller loop) until `scripts/migrate_edgar_filings.py` is run (one transaction,219  ~1.4 M rows ≈ 20–60 s; stop the ingestion jobs first, the API can keep serving).220* **Coverage dictionary** (`scripts/migrate_coverage.py`): converts the legacy inline `coverage` JSON of221  `fund_statements` (≈ 1.1 KB × 1.2 M rows) to `fund_coverage_blob` references, 2 000 rows per transaction,222  resumable (`--max-batches`, `--sleep`), API online. Space goes back to the OS only with `--vacuum` (offline).223* **Monitoring**: `GET /v1/fundamentals/_health` (hidden from OpenAPI) — companies, statement versions,224  screener rows, last filed date, per-job lag/failures/`mapping_failure_rate` (share of null public accounts in225  `fund_latest`).226227### Running on production (M3U96b)228229```bash230ssh M3U96b231cd ~/hfmarketdata232export HFMD_DATA_ROOT=~/firstratedata            # lake + edgar/ + bulk/ + state/hfmd.db ; Redis: redis://127.0.0.1:6379/0 (default)233venv/bin/pip install -r hfmarketdata/requirements.txt234235# 1) initial backfill (resumable; ~7 600 CIKs)236nohup venv/bin/python scripts/edgar_backfill.py --workers 4 > ~/edgar_backfill.log 2>&1 &237#    faster first pass: download the SEC bulk archive once (~1.3 GB) and use it for companyfacts238#    curl -A "$HFMD_SEC_USER_AGENT" -o /tmp/companyfacts.zip https://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip239#    venv/bin/python scripts/edgar_backfill.py --from-zip /tmp/companyfacts.zip --workers 6240241# 2) incremental poller under PM2 (2-minute cycle, publishes to Redis for /v1/stream)242pm2 start venv/bin/python --name edgar-incremental --cwd ~/hfmarketdata -- scripts/edgar_incremental.py243pm2 save244245# 3) nightly reconciliation (cron 03:15) — exit code 1 = discrepancies24615 3 * * * cd ~/hfmarketdata && HFMD_DATA_ROOT=~/firstratedata venv/bin/python scripts/edgar_reconcile.py --json ~/edgar_reconcile.json >> ~/edgar_reconcile.log 2>&1247248# health249curl -s https://www.hfmarketdata.io/v1/fundamentals/_health | jq .data.jobs250```251252The API process needs no restart: `init_db()` is idempotent and every request reads SQLite/Parquet.253254### Backfill duration estimate (~7 600 CIKs at 10 req/s)255256Requests: 2 per CIK (companyfacts + submissions) + 1 MetaLinks + ~0.3 for paginated older submissions ≈ 3.3 ×2577 600 ≈ **25 000 requests → ~42 min** at the 10 req/s ceiling. Volume: companyfacts average ≈ 2–3 MB258(Apple 10 MB, small caps 1 MB) → ~20 GB downloaded, gzip-cached at ~2 GB. CPU: normalisation measured at2590.7–1.2 s per company (pandas, one core) → ~2 h single-threaded, ~35 min with `--workers 4` overlapping network260and CPU. **Expected wall clock: 1 h – 1 h 30 with 4 workers** (network-bound at the bucket), ≈ 45 min with261`--from-zip` (no companyfacts requests; the zip download itself takes a few minutes). Resulting sizes: facts lake262≈ 2.5 GB Parquet, SQLite ≈ 1.5 GB, bulk files ≈ 20 × 30 MB.263264## 8. Endpoints (all `format=json|csv|parquet`, cursor pagination, `as_of`)265266| endpoint | notes |267|---|---|268| `GET /v1/fundamentals/{ticker}/statements` | `statement=income\|balance\|cashflow\|all`, `period=quarterly\|annual\|ttm`, `from`/`to` on period_end, `view=standardized\|as_reported` |269| `GET /v1/fundamentals/{ticker}/facts/{concept}` | standardized account → point-in-time series; raw `us-gaap:Tag` → fact instances from the lake (`CONCEPT_NOT_FOUND`) |270| `GET /v1/fundamentals/{ticker}/ratios` | 45 ratios in 7 groups + `inputs`, `meta.reasons` for nulls, `period=ttm\|annual` |271| `GET /v1/fundamentals/{ticker}/ratios/daily` | daily close × fundamentals known that day (DuckDB ASOF), `fields=` |272| `GET /v1/fundamentals/{ticker}/filings` | `form=` filter, EDGAR links |273| `GET /v1/fundamentals/{ticker}/coverage` | completeness, missing accounts + reasons, gaps, custom extensions |274| `GET /v1/fundamentals/screener` | `filters=pe<15,roe>15%,market_cap>1b,ev_ebitda=5..12,exchange=Nasdaq\|NYSE`, `sort=fcf_yield:desc`, `columns=`; `request_cost=2`, `requires_key=true` |275| `GET /v1/fundamentals/frames/{concept}` | `calendar_quarter=2024Q1` or `fiscal_year=&fiscal_quarter=`; `request_cost=2` |276| `GET /v1/fundamentals/_mapping` | chart of accounts + prioritized tags (public) · `GET /_health` internal |277| `GET /v1/bulk/fundamentals` · `GET /v1/bulk/fundamentals/{year}.parquet` | quota exempt, strong ETag, `If-None-Match` → 304 |278| `GET /v1/stream` (WebSocket) · `GET /v1/stream/info` | see below |279280Error codes added to `core/errors.py`: `FUNDAMENTALS_NOT_AVAILABLE` (404), `INVALID_FILTER` (400),281`CONCEPT_NOT_FOUND` (404), `STREAM_CONNECTION_LIMIT` (429).282283## 9. Stream (`stream/`)284285`GET /v1/stream` (WebSocket). Auth by `?api_key=` or `Authorization: Bearer` (keyless → JSON error then close2864001). Client `{"action":"subscribe","channel":"filings","tickers":[…]|"all","forms":[…],"resume_token":<seq>}`;287server `hello`, `subscribed`, `filing` (`ticker, cik, form, period, filed_date, url, accn, seq, summary:{revenue,288net_income, eps_diluted, total_assets, operating_cash_flow, fiscal_year, fiscal_quarter, yoy:{…}}`), `heartbeat`289every 20 s, `pong`, `error`. Max 5 concurrent sockets per key: Redis ZSET `stream:conns:{principal}` (member =290connection id, score = last heartbeat, refreshed every 30 s; members silent for 90 s are pruned at every count,291so a worker that dies without cleanup never leaves phantom slots; 6th live socket → `STREAM_CONNECTION_LIMIT`,292close 4029). Buffer = Redis stream `filings:stream` (MAXLEN 1 000, ids `<seq>-0`, `seq` from `INCR filings:seq`);293`resume_token` replays everything after it. Delivery = one blocking `XREAD` (1 s) per subscribed socket on its294own async Redis connection — no polling; the `filings` pub/sub channel is published for other consumers.295Full spec: `docs/asyncapi.yaml`.296297**Authentication**: the rate-limit middleware authenticates every `/v1/*` socket before the handler and stores298`principal` (`key:<id>`), `principal_kind`, `tier` in `scope["state"]`; `stream/routes.py` reads those (never a299hash of the key — the billed principal is exactly the one the HTTP limiter uses). Keyless → JSON `AUTH_REQUIRED`300then close 4001; no middleware context while `HFMD_RATELIMIT` is on → close 4401. With the limiter disabled,301`stream/auth.py::authenticate(key)` looks the key up in `api_keys` (`sha256(salt + key)`); unknown keys are302refused whatever their shape.303304**Stream accounting interface** (for the ratelimit module):305306```python307from stream import accounting308accounting.set_charger(fn)            # fn(principal: str, rows: int) -> None ; called once per delivered `filing` message309accounting.charge(principal, rows=1)  # what the socket calls (REDUCED_RATE_ROWS_PER_MESSAGE = 1)310accounting.rows_charged(principal)    # default fallback counter: Redis `stream:rows:{principal}:{YYYY-MM-DD}` (24 h TTL)311```312313`principal` is the same `key:<id>` string the HTTP limiter uses, so stream rows can be folded into the rows314quota; heartbeats, hello/subscribed and errors are free.315316## 10. Tests and fixtures317318`tests/fixtures/edgar/` holds **real, trimmed** EDGAR documents (≈ 80 KB gzipped: companyfacts + submissions of319Apple, Microsoft, Shake Shack limited to mapped tags and filings since 2022, SHAK's MetaLinks tag list, one Atom320page, one daily index) recorded by `tests/fixtures/edgar/record.py`. `tests/conftest.py::edgar_mock` serves them321through `respx` — no test touches the network. The synthetic price lake gained the SHAK, GOOG and GOOGL tickers.322Suites: `test_fundamentals_unit.py` (mapping, calendar incl. Apple Q2 FY2024, resolution priority/units,323Q4 derivation, restatements/as_of, TTM, ratio formulas on Apple's real numbers, filter grammar, client backoff),324`test_fundamentals_api.py` (every endpoint in json/csv/parquet, envelopes, as_of, share classes),325`test_bulk.py` (ETag/304), `test_stream.py` (auth via the middleware principal, subscribe, blocking delivery,326resume, heartbeat, ZSET connection limit + phantom pruning, accounting hook), `test_fundamentals_ingest.py`327(co-registrant accessions, poller termination and retries, failure samples, 403 daily index, conditional328statement rewrite, coverage dictionary, `edgar_filings` key migration). `./.venv/bin/python -m pytest` must stay green.329330## 11. Limitations and next steps331332* Companyfacts only carries standard taxonomies: values reported **exclusively** through company extensions333  (SHAK cost lines, many bank/insurer line items) stay null with a reason; the `fund_mapping_log` hints are the334  input for the next mapping version (bump `MAPPING_VERSION`, re-run normalisation from the cached raw JSON).335* Dimensional facts (segments, share classes) are not in companyfacts; `shares_outstanding` sums the dei cover336  facts, which is right for market cap but not per class.337* `forward_pe` is always null (no consensus estimates). Growth/CAGR need enough history (null otherwise).338* Fiscal-year labels of Jan–Mar year ends follow the filer's convention learnt from its 10-Ks; a filer with no339  10-K in the window falls back to the MMDD extrapolation.340* Restatement detection is value-based (tolerance 1e-6 relative / 0.5 absolute); a filer re-tagging the same341  number under a lower-priority concept does not create a version, a re-tag under a *higher*-priority concept does.342* 20-F/40-F filers are mapped through ifrs-full tags but were not part of the prototype; currency is kept, not343  converted.344* The ratelimit module must call `stream.accounting.set_charger` and honour `request.state.requires_key` /345  `request_cost` / `quota_exempt` set by these routes; the WebSocket itself is not rate-limited beyond the346  5-connection cap.347