# Ingestion and maintenance pipeline

## Stages
```
discover → fetch → extract → normalize → entity-resolve → evidence-check → score → editorial-review → publish
```
| Stage | Implementation | Durable state |
|---|---|---|
| discover | Editorial research (batch briefs in `docs/_agent-brief.md` style), admin "New record", bulk import (JSON/CSV) | `content/`, `tools.editorial_status=draft/reviewed` |
| fetch | `safeFetch` (`src/lib/url.ts`): SSRF guard, manual redirects, caps | `sources.http_status/final_url/fetched_at` |
| extract | `extractHtmlMeta` (title, canonical, description) ; optional AI suggestion with evidence quotes (`/api/admin/ai/suggest`, human-reviewed) | not persisted until the editor saves |
| normalize | `normalizeUrl`, controlled vocabularies (zod), `keywords`, `embedding_text` | `tools` |
| entity-resolve | slug as identity; `aliases`; duplicate candidates by trigram/homepage/alias (`/admin/duplicates`), merge keeps target and redirects relations | `tools.duplicate_of` |
| evidence-check | `link_check` job per record (every URL field); `source_rescan` hashes the official page; validator rules (sources per depth, policy sources for claims) | `link_checks`, `sources.content_hash`, flags in `settings` |
| score | `qualityScore` 0–100 (completeness × evidence × freshness) | `tools.quality_score` |
| editorial-review | Admin → Review queue: reviewed records, automated flags, public error reports | `audit_log`, `tool_revisions` |
| publish | `publish` action (quality ≥ 40, no broken links) or `content:import --publish` | `published_at`, public view |

## Jobs (PostgreSQL queue, `src/lib/jobs`)
`jobs(kind, payload, dedupe_key, status queued|running|succeeded|failed|dead|cancelled, attempts/max_attempts, run_at, locked_by)`. Claim with `FOR UPDATE SKIP LOCKED`; failures requeue with exponential backoff (30 s · 2^attempt, cap 6 h); after `max_attempts` → `dead` (retry from admin); stale `running` jobs (lease 20 min) are requeued.

| Kind | Trigger | Effect |
|---|---|---|
| link_check {toolId} | save, publish, sweep, admin | checks homepage/docs/repo/pricing/license/privacy/security; 401/403/405/429 = unverifiable (ok, soft); off-site redirect → `redirected`; 404/5xx/timeout → `broken` |
| link_sweep | daily 03:10 | fan-out link_check for records not checked in 6 days, 3 s apart |
| source_rescan {toolId} | admin | fetch official page, compare content hash, flag `review:<slug>` / canonical move |
| freshness_sweep | daily 03:40 | flags `stale:<slug>` when pricing > 90 d, review > 365 d, oldest source > 180 d |
| embed_tool / embed_sweep | save / daily 04:00 | embeddings when `AI_PROVIDER` configured (budget-guarded) |
| backup | daily 04:20 | encrypted pg_dump, retention 14, optional rsync |

Schedule is wall-clock America/Toronto inside the worker (`src/worker/index.ts`), deduped per day. Manual: Admin → Jobs, or `pnpm links:check [--sample N]`.

## Freshness policy by field
| Field | Re-check | Where surfaced |
|---|---|---|
| URLs | weekly | link_state badge, Admin → Links |
| pricing_model | 90 days | `stale:` flag |
| license | 365 days (and on repo change) | `stale:`/`review:` flag |
| description / editorial review | 365 days | `stale:` flag |
| official page content | on demand (rescan) | `review:` flag with new title/canonical |

## Failure recovery
- Worker crash: PM2 restarts; running jobs are reaped after 20 min.
- Network outage: jobs fail → backoff → dead after 5 attempts; nothing is unpublished automatically (an outage is not evidence of discontinuation).
- Bad import: validator blocks malformed records; `--partial` writes valid rows only; every change has a revision snapshot to restore from (`tool_revisions.snapshot` can be re-imported through the admin editor's Raw JSON tab).

## Verification rules enforced by code
`scripts/content-validate.ts`: schema, unique slug/name, file name = slug, leaf categories, one primary, description word counts per depth, minimum sources per depth, official site present, price figures and certification claims rejected, privacy claims require a policy source, marketing words warned. `pnpm eval:ranking` guards search quality (MRR ≥ 0.7).