docs: STATUS final checks, acceptance tests, next steps

72

| catalog (Postgres) | 100 / 16 | 101 ms | 209 ms | 135 |

73

Interpretation: file selection is by `lake_files` date range, so a 31-day 1m request scans the symbol's yearly base file (≈ 100 k rows) then paginates — acceptable; compaction of the 5m deltas (6 748 session files from the Z backfill) into base files was queued to fix the derived dataset's file count.

74

75

−**Compaction / backup #2 / restore drill / quality report**: see "Final checks" below.

75

+## Final checks (2026-11-01 evening, real outputs)

76

+

77

+- **Derived 5m compaction**: the invariant check caught a real defect — the first backfill-driven derivation had written a Z-only delta for 2026-10-30 next to the full-market delta (94 duplicate (instrument, ts) for ticker Z). The compaction job refused to publish (`compaction invariant violated`), the duplicated file was retired by a versioned publish (v3, note recorded), `derive` was fixed to write per-symbol base files for backfills, and the compaction then succeeded: 6 747 delta files → 6 935 base files, 7 470 593 rows, 198 MB, 29 s (v4). No duplicates reached readers at any version.

78

+- **Backup #2** (after the backfill): pg_dump 5.9 MB zstd; rsync 14 627 files / 567.7 MB to `M4M64a:~/backups/findata`; remote curated file count 7 236 = local 7 236 → `verified: true`; 6.8 s.

79

+- **Restore drill** (`findata restore-drill`): latest dump restored into a scratch database (472 live `lake_files` rows for the 1m dataset), dataset `market_bars.stock.1m.raw` copied back from M4M64a, **472/472 Parquet files SHA-256-verified against manifest v58, 0 missing**, scratch database dropped. Measured RTO for metadata + one dataset: < 2 min on LAN.

80

+- **Quality report** (1m): 26 152 471 bars, 6 521 symbols, 111 956 symbol-sessions, 2000-01-04 → 2026-10-30; anomalies (30 d): price_jump 1 480 (warning), outside_session 130 (warning); mean regular-hours completeness 0.58 (honest: many micro-caps trade a fraction of the 391 regular minutes; vendor omits zero-volume bars).

81

+- **Storage**: curated 653 MB (1m 331.5 MB in 472 files; 5m 198 MB in 6 935 files), raw 17 MB (incremental archives kept; the 258 MB full Z archive deleted after publication, sha256 kept), `_retired` 209 MB (GC after 7 days), manifests 7.4 MB. Free 506.5 GB.

82

+- **Security scan on the node**: vendor id absent from PM2 logs, PM2 dump, app directory (except `.env` 600), lake manifests/raw names, and Postgres (`raw_archives`, jobs, audit, checkpoints); API listens `*:8600` behind the source guard, Postgres loopback only; session cookie `HttpOnly; Secure; SameSite=strict; Max-Age=43200`. A restore-drill traceback had echoed the DB password to the operator terminal → fixed (credentials via environment, redacted errors) and **the Postgres password was rotated** (`.env` on node + laptop secrets copy).

83

+- **Dependency audit**: `pip-audit` on the exported lock → no known vulnerabilities (2026-11-01).

84

+

85

+## Acceptance tests

86

+| # | Test | Result |

87

+|---|---|---|

88

+| 1 | Private login at https://findata.spboucher.ai on desktop and mobile | ✅ headless Chromium 375 px / 1280 px, 0 console errors; owner login verified (TOTP to enrol by owner) |

89

+| 2 | Licensed symbol/date range fetched, validated, stored, queried, re-fetched without duplication | ✅ stock 1m session 2026-10-30 (6 515 symbols): re-run skipped by vendor date, forced re-fetch recognised by sha256; supersede path tested; Z backfill excluded the session already live (10 570 rows) |

90

+| 3 | 1m and aggregated 5m match fixture expectations incl. session/DST | ✅ unit tests (open/high/low/close/volume/trade_count, wall-clock buckets across 2026-03-08 and 2026-11-01 DST, volume conservation on production AAPL) |

91

+| 4 | Fundamentals as_of hides later filings/restatements | ✅ test + production: AAPL as_of 2025-01-01 → latest period 2024-09-28 (filed 2024-11-01), 536 facts hidden |

92

+| 5 | SDK retrieves a bounded sample and a larger Parquet export | ✅ 900 bars in 0.30 s; 5-symbol export 4 508 rows Parquet in 1.15 s |

93

+| 6 | Interrupted ingestion resumes from durable checkpoints | ✅ test (backfill checkpoint `done_symbols`); production: killed compaction requeued, orphan GC added; jobs/checkpoints in Postgres survive restarts |

94

+| 7 | Unauthorized access fails; vendor secrets never in bundles or logs | ✅ 401/403 paths tested and verified in production; web bundle, logs, DB, manifests scanned |

95

+| 8 | Backup restoration tested; monitoring reports SSD health and freshness | ✅ restore drill 472/472 hashes; `/metrics` + health: SMART Verified, vendor lag 0 d, backup age |

96

+| 9 | Licensing and unsupported datasets surfaced in UI and docs | ✅ license banner, dataset statuses `unsupported` (ticks) / `pending_configuration` (SEC EDGAR) / `planned` |

97

+| 10 | Final operator report | ✅ this file + `docs/DISCOVERY.md`, `docs/OPERATIONS.md` |

98

+

99

+## Known limitations / exact next steps

100

+1. **TOTP**: enrol the owner's authenticator in Settings → Security (one click + code). Until then login is password-only.

101

+2. **Coverage**: production holds one full-market session (2026-10-30) + full history for the 57 Z tickers. Extend with `findata ingest backfill <letters>` (≈ 258 MB zip and 67 s per small letter; large letters such as A/S take several GB and tens of minutes; budget guard stops below 60 GB free). Daily incremental runs automatically from 02:00 ET.

102

+3. **Fundamentals universe**: 5 tickers loaded; set tracked tickers in Settings (or `findata ingest fundamentals T1,T2,…`); the 05:00 ET job currently ingests the first 50 active stocks unless tracked tickers are configured (wire `fundamentals_tracked_tickers` into the scheduler — small follow-up).

103

+4. **SEC EDGAR**: set `SEC_EDGAR_USER_AGENT="Name email"` in `.env` to activate the independent source (parser + PIT already implemented and tested).

104

+5. **AI assistant**: set `ANTHROPIC_API_KEY` (server-side) then run a capability/cost test (`POST /api/v1/assistant`, 5–10 questions) and record tokens/latency here before enabling for research use; OpenAI not implemented.

105

+6. **Bars endpoint latency** on multi-year base files (≈1.5 s for a 31-day 1m window): acceptable now; if needed, add month-level partitioning or a symbol/year row-group index.

106

+7. **Encryption at rest**: FileVault is off by cluster policy (unattended reboots); enable on M3U96a + M4M64a if required.

107

+8. **Fundamentals revision history** before first ingestion is unknowable from the vendor; EDGAR filing-level history fixes this once configured.

108

+9. Daily scheduler, weekly compaction and backups are in place; verify the first automatic cycle on 2026-11-03 (Monday) in Ingestion → jobs.

76

109

spb/findata

Theme

		`@@ -72,4 +72,37 @@ Lake: 506.9 GB free, curated 0.54 GB, raw 0.02 GB (full Z archive deleted after`
72	72	\| catalog (Postgres) \| 100 / 16 \| 101 ms \| 209 ms \| 135 \|
73	73	Interpretation: file selection is by `lake_files` date range, so a 31-day 1m request scans the symbol's yearly base file (≈ 100 k rows) then paginates — acceptable; compaction of the 5m deltas (6 748 session files from the Z backfill) into base files was queued to fix the derived dataset's file count.
74	74
75		−Compaction / backup #2 / restore drill / quality report: see "Final checks" below.
	75	+## Final checks (2026-11-01 evening, real outputs)
	76	+
	77	+- Derived 5m compaction: the invariant check caught a real defect — the first backfill-driven derivation had written a Z-only delta for 2026-10-30 next to the full-market delta (94 duplicate (instrument, ts) for ticker Z). The compaction job refused to publish (`compaction invariant violated`), the duplicated file was retired by a versioned publish (v3, note recorded), `derive` was fixed to write per-symbol base files for backfills, and the compaction then succeeded: 6 747 delta files → 6 935 base files, 7 470 593 rows, 198 MB, 29 s (v4). No duplicates reached readers at any version.
	78	+- Backup #2 (after the backfill): pg_dump 5.9 MB zstd; rsync 14 627 files / 567.7 MB to `M4M64a:~/backups/findata`; remote curated file count 7 236 = local 7 236 → `verified: true`; 6.8 s.
	79	+- Restore drill (`findata restore-drill`): latest dump restored into a scratch database (472 live `lake_files` rows for the 1m dataset), dataset `market_bars.stock.1m.raw` copied back from M4M64a, 472/472 Parquet files SHA-256-verified against manifest v58, 0 missing, scratch database dropped. Measured RTO for metadata + one dataset: < 2 min on LAN.
	80	+- Quality report (1m): 26 152 471 bars, 6 521 symbols, 111 956 symbol-sessions, 2000-01-04 → 2026-10-30; anomalies (30 d): price_jump 1 480 (warning), outside_session 130 (warning); mean regular-hours completeness 0.58 (honest: many micro-caps trade a fraction of the 391 regular minutes; vendor omits zero-volume bars).
	81	+- Storage: curated 653 MB (1m 331.5 MB in 472 files; 5m 198 MB in 6 935 files), raw 17 MB (incremental archives kept; the 258 MB full Z archive deleted after publication, sha256 kept), `_retired` 209 MB (GC after 7 days), manifests 7.4 MB. Free 506.5 GB.
	82	+- Security scan on the node: vendor id absent from PM2 logs, PM2 dump, app directory (except `.env` 600), lake manifests/raw names, and Postgres (`raw_archives`, jobs, audit, checkpoints); API listens `:8600` behind the source guard, Postgres loopback only; session cookie `HttpOnly; Secure; SameSite=strict; Max-Age=43200`. A restore-drill traceback had echoed the DB password to the operator terminal → fixed (credentials via environment, redacted errors) and the Postgres password was rotated* (`.env` on node + laptop secrets copy).
	83	+- Dependency audit: `pip-audit` on the exported lock → no known vulnerabilities (2026-11-01).
	84	+
	85	+## Acceptance tests
	86	+\| # \| Test \| Result \|
	87	+\|---\|---\|---\|
	88	+\| 1 \| Private login at https://findata.spboucher.ai on desktop and mobile \| ✅ headless Chromium 375 px / 1280 px, 0 console errors; owner login verified (TOTP to enrol by owner) \|
	89	+\| 2 \| Licensed symbol/date range fetched, validated, stored, queried, re-fetched without duplication \| ✅ stock 1m session 2026-10-30 (6 515 symbols): re-run skipped by vendor date, forced re-fetch recognised by sha256; supersede path tested; Z backfill excluded the session already live (10 570 rows) \|
	90	+\| 3 \| 1m and aggregated 5m match fixture expectations incl. session/DST \| ✅ unit tests (open/high/low/close/volume/trade_count, wall-clock buckets across 2026-03-08 and 2026-11-01 DST, volume conservation on production AAPL) \|
	91	+\| 4 \| Fundamentals as_of hides later filings/restatements \| ✅ test + production: AAPL as_of 2025-01-01 → latest period 2024-09-28 (filed 2024-11-01), 536 facts hidden \|
	92	+\| 5 \| SDK retrieves a bounded sample and a larger Parquet export \| ✅ 900 bars in 0.30 s; 5-symbol export 4 508 rows Parquet in 1.15 s \|
	93	+\| 6 \| Interrupted ingestion resumes from durable checkpoints \| ✅ test (backfill checkpoint `done_symbols`); production: killed compaction requeued, orphan GC added; jobs/checkpoints in Postgres survive restarts \|
	94	+\| 7 \| Unauthorized access fails; vendor secrets never in bundles or logs \| ✅ 401/403 paths tested and verified in production; web bundle, logs, DB, manifests scanned \|
	95	+\| 8 \| Backup restoration tested; monitoring reports SSD health and freshness \| ✅ restore drill 472/472 hashes; `/metrics` + health: SMART Verified, vendor lag 0 d, backup age \|
	96	+\| 9 \| Licensing and unsupported datasets surfaced in UI and docs \| ✅ license banner, dataset statuses `unsupported` (ticks) / `pending_configuration` (SEC EDGAR) / `planned` \|
	97	+\| 10 \| Final operator report \| ✅ this file + `docs/DISCOVERY.md`, `docs/OPERATIONS.md` \|
	98	+
	99	+## Known limitations / exact next steps
	100	+1. TOTP: enrol the owner's authenticator in Settings → Security (one click + code). Until then login is password-only.
	101	+2. Coverage: production holds one full-market session (2026-10-30) + full history for the 57 Z tickers. Extend with `findata ingest backfill <letters>` (≈ 258 MB zip and 67 s per small letter; large letters such as A/S take several GB and tens of minutes; budget guard stops below 60 GB free). Daily incremental runs automatically from 02:00 ET.
	102	+3. Fundamentals universe: 5 tickers loaded; set tracked tickers in Settings (or `findata ingest fundamentals T1,T2,…`); the 05:00 ET job currently ingests the first 50 active stocks unless tracked tickers are configured (wire `fundamentals_tracked_tickers` into the scheduler — small follow-up).
	103	+4. SEC EDGAR: set `SEC_EDGAR_USER_AGENT="Name email"` in `.env` to activate the independent source (parser + PIT already implemented and tested).
	104	+5. AI assistant: set `ANTHROPIC_API_KEY` (server-side) then run a capability/cost test (`POST /api/v1/assistant`, 5–10 questions) and record tokens/latency here before enabling for research use; OpenAI not implemented.
	105	+6. Bars endpoint latency on multi-year base files (≈1.5 s for a 31-day 1m window): acceptable now; if needed, add month-level partitioning or a symbol/year row-group index.
	106	+7. Encryption at rest: FileVault is off by cluster policy (unattended reboots); enable on M3U96a + M4M64a if required.
	107	+8. Fundamentals revision history before first ingestion is unknowable from the vendor; EDGAR filing-level history fixes this once configured.
	108	+9. Daily scheduler, weekly compaction and backups are in place; verify the first automatic cycle on 2026-11-03 (Monday) in Ingestion → jobs.
76	109