TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# Connector health monitoring23## 1. Statuses (SPEC §12)45| internal (`connector_health.status`) | operator label | meaning |6|---|---|---|7| `healthy` | **UP** | ≥ 85 % page success in 24 h, no anomalies |8| `degraded` | **DEGRADED** | 50–85 % success, or anomalies (schema drift, result-count collapse, duplicate explosion, price shift, challenges) |9| `failing` | **BROKEN** | < 50 % success or the last 3 runs failed |10| `paused` / `disabled` | **DISABLED** | paused by an operator, disabled in the registry, or gated (missing `requires` env vars) |11| `maintenance` | **MAINTENANCE** | operator flag; scheduler skips it, health keeps computing |12| `unknown` | UNKNOWN | never ran |1314Health is computed every 30 min (`health.compute` job) by `healthFromRuns()` over the last 7 days of15`connector_runs`, plus a data-freshness query. `pnpm ri health [--probe]` forces a recompute (with a live probe).1617## 2. Metrics per connector (`ConnectorHealth`)1819success_rate_24h · pages_attempted/success · requests_24h · items (`records_24h`, `duplicates_24h`) ·20latest successful ingest (`last_success`) · `last_error` · **HTTP error distribution** (`http_errors {code: n}`) ·21`parse_failure_rate` · `schema_drift[]` · `latency_ms_avg` · `rate_limited_24h` (429s) ·22**captcha/challenge events** (`challenges_24h`) · `circuit_refusals_24h` · `data_freshness` (newest source-side23date ingested) · `records_7d_daily_avg` (volume baseline) · Firecrawl success rate / Scrapfly fallback rate ·24`missing_requirements`. Queue backlog comes from pg-boss (`/admin` overview). Costs per connector from `costs`.2526## 3. Schema-drift detection (SPEC §13)2728Detectors, all producing anomaly strings that mark the run and degrade health:2930| signal | where | anomaly |31|---|---|---|32| expected selectors/fields missing | connector `parse()` → router quality < 0.6 → escalation; connector `ctx.anomaly('selector_missing' | 'parse_failure_page')` | `parse_failure_page`, `selector_missing` |33| field-null increase | normalizer: `connector_field_stats` daily presence of price/currency/date/images/title/identifiers/grade/number/set/year/description vs trailing 7 days; +30 pts null rate on ≥ 50 records | `schema_drift:<field>` |34| price parsing failure / currency problem | normalizer `priceDistributionAnomaly` (median shift ×3 vs last 500 sales), Zod schema failures | `price_distribution_shift`, `parse_failure: schema …` |35| pagination failure | connector `ctx.anomaly('pagination_failure')`, `all_pages_failed` | `pagination_failure`, `all_pages_failed` |36| result-count collapse | health: `records_24h < 20 %` of the 7-day daily average (≥ 50/day) | `result_count_collapse` |37| layout change / JSON schema change | Zod payload schemas throw in `normalize()` → recorded on the raw row (`process_error`) and counted as parse failures | `parse_failure` |38| duplicate explosion | crawler: ≥ 90 % duplicates on ≥ 50 records | `duplicate_explosion` |39| anti-bot | router `isChallengePage()` | `challenge_events: n` |4041The connector page shows the field-presence monitor (today vs 7 days) so a drift is visible before the42connector silently produces garbage.4344## 4. Alerts4546`connector_run_finished` events carry status and anomalies; the account/alert worker and the MacLustr47monitoring apps read `/api/admin/connectors` (health labels) — a BROKEN or DEGRADED-with-drift connector is48an alert condition. Circuit-breaker openings are logged at `warn` with the retry time.4950## 5. Admin explorer (SPEC §29) — `/admin/connectors`5152Columns: logo · domain · category · country · health label · status · acquisition method (+ engines, refresh53class) · last sync / last ok / data freshness · items 24 h (normalised, duplicates, pages/min) · errors54(HTTP errors, anomalies, challenges) · Firecrawl/Scrapfly credits 24 h and rates · latency / 429 · priority ·55trust · backfill progress. Filters: text, health, category, country.5657Buttons: **Test** (25-record probe) · **Run now** · **Backfill** (start/continue a resumable campaign) ·58**Pause / Resume**; on the connector page also **Retry from cursor**, **Pause backfill**, **Reset backfill**,59**Maintenance**, **View logs** (runs, anomalies, costs), **View payload** (`/admin/connectors/<id>/raw/<rawId>`),60**View parsed record** (`/admin/connectors/<id>/record/<recId>`), config/schedule editor, access notes and the61full registry entry.6263JSON: `GET /api/admin/connectors` (health per connector) · `GET /api/admin/coverage` (SPEC §30 coverage stats) ·64`GET /api/admin/status` (pipeline counts + queues). All require the admin cookie.6566## 6. Coverage dashboard (SPEC §30) — `/admin/coverage`6768Connectors by health / category / country / source type / engine / capability; live listings indexed;69historical sales indexed (and how many older than a year, oldest sale); auction houses covered; grading providers70(certificates, population reports); canonical assets and variants; guide observations; daily ingested71observations (14-day bars); sales by family with 30-day counts and oldest dates; sales by year (historical depth).72