docs: fundamentals — clé (cik, accn), garde-fous du poller, dictionnaire de couverture, stream ZSET/XREAD et principal du middleware
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +45 −15
modified
docs/fundamentals.md
+45 −15
@@ -39,7 +39,8 @@ Why two stores: | ||
| 39 | 39 | | table | key | purpose | |
| 40 | 40 | |---|---|---| |
| 41 | 41 | | `edgar_companies` | `cik` | `ticker` (canonical = first SEC listing), `tickers` (all share classes: `["GOOGL","GOOG"]`), name, `sic`, `exchange`, `fiscal_year_end` (MMDD), `status` active/delisted, `ticker_history` JSON, `facts_updated_at`, `normalized_at` | |
| 42 | −| `edgar_filings` | `accn` | cik, form, filed_date, period_of_report, primary_doc (full EDGAR URL), is_amendment, is_xbrl, parsed_at — index (cik, filed_date) | | |
| 42 | +| `edgar_filings` | **(cik, accn)** | form, filed_date, period_of_report, primary_doc (full EDGAR URL), is_amendment, is_xbrl, parsed_at — indexes (cik, filed_date), accn. An accession is shared by co-registrants (Entergy 65984 + its utilities file one 10-Q/8-K): one row per registrant. Legacy key `accn` alone → `scripts/migrate_edgar_filings.py` (see §7) | | |
| 43 | +| `fund_coverage_blob` | id; unique sha1 | dictionary of distinct `coverage` JSON documents (`fundamentals/coverage_store.py`); `fund_statements.coverage_id` references it, the inline `coverage` column stays NULL for new rows (legacy inline JSON still read; `scripts/migrate_coverage.py` converts it) | | |
| 43 | 44 | | `fund_statements` | id; unique (cik, statement, fiscal_year, fiscal_quarter, accn) | **wide, versioned**: cik, ticker, statement (income/balance/cashflow), fiscal_year, fiscal_quarter (1–4, **0 = annual**), period_start, period_end, calendar_quarter (`2024Q1`), form, accn, filed_date, derived, restated, currency, coverage JSON, mapping_version + one REAL column per account (49). Indexes: point-in-time `(ticker, period_end, filed_date)`, `(cik, statement, fiscal_year, fiscal_quarter, filed_date)`, `(statement, fiscal_quarter, calendar_quarter)` for frames | |
| 44 | 45 | | `fund_mapping` | id; unique (version, account, taxonomy, tag) | the prioritized mapping seeded from `mapping.py` — **183 rows** (49 accounts, of which 5 computed and 2 auxiliary), version `2026.09.1` (`GET /v1/fundamentals/_mapping`) | |
| 45 | 46 | | `fund_mapping_log` | (cik, taxonomy, tag) | tags seen but not mapped: standard-taxonomy tags outside the mapping (`is_extension=false`, occurrences, first/last seen, sample accn) and **company extensions** found in the statements of the latest 10-K via `MetaLinks.json` (`is_extension=true`, `hint_account`) | |
@@ -190,13 +191,35 @@ the screener reads a precomputed table so it stays O(rows in `fund_latest`) at u | ||
| 190 | 191 | `--force` redoes). `--from-zip companyfacts.zip` reads the SEC bulk archive instead of the API. Ends with the |
| 191 | 192 | bulk Parquet files. |
| 192 | 193 | * **Incremental** (`scripts/edgar_incremental.py`, every 2 min): Atom `getcurrent` feed per form |
| 193 | − (10-K, 10-Q, 8-K, 20-F — 4 requests) + today's/yesterday's daily master index as a safety net → new accessions | |
| 194 | − of tracked CIKs → refetch companyfacts (cache bypass) → re-normalise → publish a `filing` event | |
| 195 | − (Redis `filings` pub/sub + `filings:stream` capped at 1 000) → lag written to `fund_ingest_state`. The efts | |
| 196 | − full-text search needs a query term and is therefore not used as a "list everything" source. | |
| 194 | + (10-K, 10-Q, 8-K, 20-F — 4 requests) + the daily master index of the last two **business days** as a safety | |
| 195 | + net (`Archives/edgar/daily-index/{year}/QTR{q}/master.YYYYMMDD.idx`; the SEC answers 403 for a day that has no | |
| 196 | + index — weekend, holiday — which is treated as "not published", not as an error) → new **(cik, accession) | |
| 197 | + pairs** of `TRACKED_FORMS` (10-K/10-Q/8-K/20-F/40-F/6-K, transition reports 10-KT/10-QT and every `/A` | |
| 198 | + amendment) → refetch companyfacts (cache bypass) → re-normalise → statements rewritten **only if they changed** | |
| 199 | + (fingerprint of the rows) → publish one `filing` event per registrant (Redis `filings` pub/sub + | |
| 200 | + `filings:stream` capped at 1 000) → lag written to `fund_ingest_state`; the bulk extracts are rebuilt only when | |
| 201 | + rows changed. Termination guards: Atom entries are filtered by form (the `type=8-K` feed also returns 8-K/A — | |
| 202 | + an untracked form never landed in `edgar_filings` and re-ingested its company every cycle); a Redis SET | |
| 203 | + `edgar:seen_accn` (TTL 7 d) remembers every pair handled, a HASH `edgar:failed_accn` counts transient errors | |
| 204 | + (3 attempts, then the pair is left to the reconcile job); `companyfacts_404` (no XBRL facts) still records the | |
| 205 | + filings and is not a failure. `failure_samples` accumulate (last 10, with the CIK) instead of being wiped by | |
| 206 | + the next clean cycle. The efts full-text search needs a query term and is therefore not used as a "list | |
| 207 | + everything" source. | |
| 197 | 208 | * **Reconcile** (`scripts/edgar_reconcile.py`): 20 random companies refetched and re-normalised in memory, |
| 198 | 209 | every public account of every latest period compared to the DB (`--fix` re-ingests the differing ones); |
| 199 | − result in `fund_ingest_state.reconcile` and `_health`. Exit code 1 on discrepancies for alerting. | |
| 210 | + result in `fund_ingest_state.reconcile` and `_health`. Exit code 1 on discrepancies for alerting. Ends with | |
| 211 | + `PRAGMA incremental_vacuum` (effective once the database runs `auto_vacuum=INCREMENTAL`, which `init_db` | |
| 212 | + switches on automatically for a database under 50 MB; an existing big file needs the one-off | |
| 213 | + `scripts/migrate_coverage.py --vacuum`). | |
| 214 | +* **Schema maintenance** (`fundamentals/migrations.py`, run by `init_db()` at every start, idempotent): | |
| 215 | + `fund_statements.coverage_id` column, `auto_vacuum` on a young DB, and the `edgar_filings` key | |
| 216 | + `accn → (cik, accn)` — migrated inline only when the table has ≤ 300 000 rows; above that the API logs a | |
| 217 | + warning and keeps running on the legacy key (co-registrant copies are skipped by `ON CONFLICT DO NOTHING`, | |
| 218 | + the Redis guard stops the poller loop) until `scripts/migrate_edgar_filings.py` is run (one transaction, | |
| 219 | + ~1.4 M rows ≈ 20–60 s; stop the ingestion jobs first, the API can keep serving). | |
| 220 | +* **Coverage dictionary** (`scripts/migrate_coverage.py`): converts the legacy inline `coverage` JSON of | |
| 221 | + `fund_statements` (≈ 1.1 KB × 1.2 M rows) to `fund_coverage_blob` references, 2 000 rows per transaction, | |
| 222 | + resumable (`--max-batches`, `--sleep`), API online. Space goes back to the OS only with `--vacuum` (offline). | |
| 200 | 223 | * **Monitoring**: `GET /v1/fundamentals/_health` (hidden from OpenAPI) — companies, statement versions, |
| 201 | 224 | screener rows, last filed date, per-job lag/failures/`mapping_failure_rate` (share of null public accounts in |
| 202 | 225 | `fund_latest`). |
@@ -263,15 +286,20 @@ Error codes added to `core/errors.py`: `FUNDAMENTALS_NOT_AVAILABLE` (404), `INVA | ||
| 263 | 286 | 4001). Client `{"action":"subscribe","channel":"filings","tickers":[…]|"all","forms":[…],"resume_token":<seq>}`; |
| 264 | 287 | server `hello`, `subscribed`, `filing` (`ticker, cik, form, period, filed_date, url, accn, seq, summary:{revenue, |
| 265 | 288 | net_income, eps_diluted, total_assets, operating_cash_flow, fiscal_year, fiscal_quarter, yoy:{…}}`), `heartbeat` |
| 266 | −every 20 s, `pong`, `error`. Max 5 concurrent sockets per key (Redis counter `stream:conns:{principal}`, 6th → | |
| 267 | −`STREAM_CONNECTION_LIMIT`, close 4029). Buffer = Redis stream `filings:stream` (MAXLEN 1 000, ids `<seq>-0`, | |
| 268 | −`seq` from `INCR filings:seq`); `resume_token` replays everything after it. Delivery polls the stream every | |
| 269 | −250 ms per socket (also gives resume for free); the `filings` pub/sub channel is published for other consumers. | |
| 289 | +every 20 s, `pong`, `error`. Max 5 concurrent sockets per key: Redis ZSET `stream:conns:{principal}` (member = | |
| 290 | +connection id, score = last heartbeat, refreshed every 30 s; members silent for 90 s are pruned at every count, | |
| 291 | +so a worker that dies without cleanup never leaves phantom slots; 6th live socket → `STREAM_CONNECTION_LIMIT`, | |
| 292 | +close 4029). Buffer = Redis stream `filings:stream` (MAXLEN 1 000, ids `<seq>-0`, `seq` from `INCR filings:seq`); | |
| 293 | +`resume_token` replays everything after it. Delivery = one blocking `XREAD` (1 s) per subscribed socket on its | |
| 294 | +own async Redis connection — no polling; the `filings` pub/sub channel is published for other consumers. | |
| 270 | 295 | Full spec: `docs/asyncapi.yaml`. |
| 271 | 296 | |
| 272 | −**Authentication hook**: `stream/auth.py` delegates to `accounts.security.{verify_api_key|resolve_api_key| | |
| 273 | −authenticate_key}(key)` when the accounts module is present; otherwise a key with the documented shape | |
| 274 | −`hfmd_live_<32 base62>` is accepted and the principal is `key:<sha256(salt+key)[:16]>`. | |
| 297 | +**Authentication**: the rate-limit middleware authenticates every `/v1/*` socket before the handler and stores | |
| 298 | +`principal` (`key:<id>`), `principal_kind`, `tier` in `scope["state"]`; `stream/routes.py` reads those (never a | |
| 299 | +hash of the key — the billed principal is exactly the one the HTTP limiter uses). Keyless → JSON `AUTH_REQUIRED` | |
| 300 | +then close 4001; no middleware context while `HFMD_RATELIMIT` is on → close 4401. With the limiter disabled, | |
| 301 | +`stream/auth.py::authenticate(key)` looks the key up in `api_keys` (`sha256(salt + key)`); unknown keys are | |
| 302 | +refused whatever their shape. | |
| 275 | 303 | |
| 276 | 304 | **Stream accounting interface** (for the ratelimit module): |
| 277 | 305 | |
@@ -294,8 +322,10 @@ through `respx` — no test touches the network. The synthetic price lake gained | ||
| 294 | 322 | Suites: `test_fundamentals_unit.py` (mapping, calendar incl. Apple Q2 FY2024, resolution priority/units, |
| 295 | 323 | Q4 derivation, restatements/as_of, TTM, ratio formulas on Apple's real numbers, filter grammar, client backoff), |
| 296 | 324 | `test_fundamentals_api.py` (every endpoint in json/csv/parquet, envelopes, as_of, share classes), |
| 297 | −`test_bulk.py` (ETag/304), `test_stream.py` (auth, subscribe, delivery, resume, heartbeat, connection limit, | |
| 298 | −accounting hook). `./.venv/bin/python -m pytest` → 53 passed. | |
| 325 | +`test_bulk.py` (ETag/304), `test_stream.py` (auth via the middleware principal, subscribe, blocking delivery, | |
| 326 | +resume, heartbeat, ZSET connection limit + phantom pruning, accounting hook), `test_fundamentals_ingest.py` | |
| 327 | +(co-registrant accessions, poller termination and retries, failure samples, 403 daily index, conditional | |
| 328 | +statement rewrite, coverage dictionary, `edgar_filings` key migration). `./.venv/bin/python -m pytest` must stay green. | |
| 299 | 329 | |
| 300 | 330 | ## 11. Limitations and next steps |
| 301 | 331 | |
| 302 | 332 | |