spb/company-atlas

1# Event taxonomy — deterministic rules, wording policy, importance & confidence23Owner: intelligence layer (`services/events.py`, `services/clustering.py`, rules version `rules-v1`, schema `event-v1`).4Vocabulary: `taxonomy.EVENT_SUBTYPES` (subtype → type + default importance). The LLM classifier may only pick from that list.56## Pipeline78```9changes (status='pending', kind ≥ meaningful) ─▶ derive_events(change, company, sensor, baseline)   pure rules, no I/O10                                             ─▶ persist_change_events()   events + event_sources + clusters + review_queue + llm_jobs11                                             ─▶ changes.status = 'processed' · sensors.event_count · companies.last_event_at12                                             ─▶ alerts.evaluate_alerts(new_event_ids)13```1415- `catlas process-changes [--limit] [--loop]` · periodic `process-changes` every 20 s (SKIP LOCKED, batch 200).16- `catlas reprocess-events --since 7d [--company]` re-runs the rules on processed changes **without refetching**; dedupe keys make it idempotent,17  new rules simply add the events they now produce.18- **Noise / minor changes never produce events** (they are archived if still pending). Meaningful (≥ 0.40), major (≥ 0.65), critical (≥ 0.85) do.1920## Rules → subtypes2122| Family | Input (`structured_delta`) | Subtype(s) | Title pattern |23|---|---|---|---|24| HIRING | `jobs.added/removed/open_before/open_after` | `JOB_COUNT_INCREASE` when net > 0 (entities.jobs ≤ 50) | *12 new positions detected on careers page* |25| | | `JOB_COUNT_DECREASE` when net < 0 | *7 monitored job listings no longer visible on careers page* |26| | AI keyword in added titles (`taxonomy.AI_KEYWORDS`) or `is_ai` | `AI_HIRING` | *2 AI-related positions detected on careers page* |27| | ≤ 5 jobs added | `NEW_JOB` per job | *New position listed: Senior ML Engineer (Toronto, CA)* |28| | added ≥ mean + 2σ of `baselines.jobs_new_weekly` (≥ 4 samples) — fallback ≥ 10 and ≥ 50 % of open_before | `HIRING_SURGE` | *Hiring surge signal: 25 new positions detected in one observation (baseline ≈ 3.0 new/week)* |29| | removed ≥ 10, ≥ 50 % of open_before and open_after ≤ 50 % | `HIRING_FREEZE_SIGNAL` (+ review `unexpected_activity`) | *Hiring slowdown signal: 30 of 40 monitored listings no longer visible* |30| PRICING | `plans.price_changed` | `PRICE_INCREASE` / `PRICE_DECREASE` (old/new values, `payload.pct`) | *Pro plan price observed at $59 (was $49)* |31| | `plans.added` | `NEW_PRICING_TIER` (tag `enterprise` when contact-sales) | *New pricing tier listed: Enterprise (contact sales)* |32| | `plans.removed` | `PRICING_TIER_REMOVED` | *Pricing tier no longer listed: Starter* |33| | pricing surface, no typed delta | `PRICING_CHANGE` (text-diff) | *Pricing page materially updated (3 blocks changed)* |34| LEADERSHIP | `people.added` (executive) | `NEW_EXECUTIVE` | *Jane Doe listed as Chief Financial Officer on leadership page* |35| | `people.removed` (executive) | `EXECUTIVE_NO_LONGER_LISTED` | *Jane Doe no longer listed on leadership page* |36| | `people.title_changed` | `EXECUTIVE_TITLE_CHANGE` | *Ann Lee now listed as COO (was VP Operations)* |37| | ≥ 3 people changes, or non-executive changes only | `LEADERSHIP_CHANGE` aggregate | *Leadership page updated: 2 added, 1 no longer listed* |38| PRODUCT | `products.added` / `removed` | `NEW_PRODUCT` / `PRODUCT_REMOVED` | *New product listed: Atlas Pro* · *Product no longer listed: Atlas Lite* |39| LOCATION | `locations.added` | `NEW_LOCATION` (kind-aware label) | *New office listed: Toronto, CA* |40| | `locations.new_countries` | `COUNTRY_EXPANSION` | *New country presence listed: Japan (Tokyo)* |41| | `locations.removed` | `OFFICE_REMOVED` | *Office no longer listed: Berlin, DE* |42| COMMUNICATION / IR / DEVELOPER | `news.added` (≤ 20 per change) | `NEWS_RELEASE` · `BLOG_POST` · `CHANGELOG_ENTRY` · `INVESTOR_UPDATE` · `EARNINGS_RELEASE` (title heuristics: earnings/quarter/fiscal → earnings; investor/annual report/dividend → IR; category/surface otherwise) | *News release: <title>* |43| DEVELOPER (text-diff) | docs/developer · api · changelog | `DOC_CHANGE` · `API_CHANGE` · `CHANGELOG_ENTRY` | *Documentation updated (5 sections changed)* |44| LEGAL (text-diff) | legal_terms · legal_privacy · security | `TERMS_CHANGE` · `PRIVACY_POLICY_CHANGE` · `SECURITY_UPDATE` (+ review `legal_sensitive`) | *Terms of service page materially updated (3 sections changed)* — sections from block `path` |45| WEBSITE | homepage meaningful · major+ · `meta.title_changed` | `WEBSITE_CHANGE` · `HOMEPAGE_REDESIGN` · `MESSAGING_CHANGE` | *Homepage materially redesigned (9 blocks changed, 62% of text)* |46| other surfaces (text-diff) | products/services/solutions · investor_relations · sustainability · status · careers · locations · about | `PRODUCT_UPDATE` · `INVESTOR_UPDATE` · `SUSTAINABILITY_UPDATE` · `OPERATIONS_UPDATE` · `WEBSITE_CHANGE` | *<Surface> updated (n sections changed)* |4748Surfaces with no rule (`other`, `partners`, `customers`, `support`, …) and meaningful+ significance produce no deterministic event; the change is49queued for the LLM classifier instead (`llm_jobs.kind = classify_change`), together with ambiguous surfaces (homepage/about/products text-only).5051## Wording policy (spec §167–168)5253- Verbs: **detected, observed, listed, now listed, no longer listed, no longer visible, appears, signal**.54- Never: *fired, laid off, layoffs, shut down, bankrupt, collapsed* (`taxonomy.FORBIDDEN_WORDING`). `safe_wording()` rewrites them defensively;55  the LLM schemas reject them at validation time.56- Disappearance ≠ departure: `EXECUTIVE_NO_LONGER_LISTED` summaries say so explicitly; `HIRING_FREEZE_SIGNAL` / `hiring_freeze` are labelled *signal*.57- Titles ≤ 200 chars, summaries ≤ 600; entity lists bounded to 50; per-entity events capped (5 jobs, 10 people, 20 news items).5859## Importance & confidence6061- `importance = default(subtype) × (0.7 + 0.6·significance) × (1 + 0.3·magnitude)` clamped to [0.05, 1]. `magnitude` is rule-specific and 0–1:62  relative job delta (`n / max(open_before, 5)`), |price pct| / 50, headquarters vs office, country expansion 0.6, launch-tagged news 0.5, text63  ratio × 2 for page changes.64- `confidence` = evidence quality (`taxonomy.EVIDENCE_CONFIDENCE`): structured ATS JSON **0.95** (jobs_board surface, Greenhouse/Lever/Ashby/65  SmartRecruiters/Workday/JSON connectors), JSON-LD / feed **0.9**, HTML extraction **0.8**, text-diff-only **0.7**. `confidence_label` via66  `taxonomy.confidence_label` (VERIFIED ≥ 0.95, HIGH_CONFIDENCE ≥ 0.85, LIKELY ≥ 0.7, INFERRED ≥ 0.5, else LOW_CONFIDENCE).67- Corroboration: a second surface in the same cluster adds **+0.03 per extra surface**, capped at 0.99 (clustering below).68- Review queue: `major_event` for critical changes, `legal_sensitive` for legal inferences, `low_confidence` (< 0.5), `unexpected_activity` for freeze signals.6970## Idempotency & clustering7172- `events.dedupe_key = sha(company, subtype, normalised entity key, sensor, detection day)` → inserting the same change twice is a no-op73  (`on conflict do nothing`). Entity keys: job label, plan name, person name, product name, place, news title, `jobs:<before>><after>` for aggregates,74  `page:<sensor>` for page-level events.75- `event_clusters.cluster_key = sha(company, subtype, normalised entity key, 7-day bucket)` — surface-independent, so the same press release seen76  on the newsroom **and** the feed, or the same executive on leadership **and** about, fold into one cluster. The first event is canonical; later77  ones are stored with `status='duplicate'` + `cluster_id`, the canonical event gains an `event_sources` row (`kind='corroboration'`),78  `payload.sources[]`, `payload.corroborations` and the confidence bump. Clusters track `source_count`, `surfaces[]`, `confidence`.7980## LLM enrichment hooks8182- `classify_change` (small model) when no deterministic event or ambiguous surface, significance ≥ `CA_LLM_MIN_SIGNIFICANCE`, within83  `CA_LLM_DAILY_BUDGET`, only when `settings.llm_configured`. Material classifications create `origin='llm'` events (dedupe84  `sha(company, 'llm', change, subtype)`); when a deterministic event of the same subtype already exists for the change, the classification enriches85  it instead (`origin='hybrid'`, `payload.llm_classification`).86- `summarize_event` (medium model) for legal, homepage/messaging and news/IR events that carry diff content → `events.summary`, `origin='hybrid'`,87  `payload.llm` (key points or legal sections + materiality), model/prompt/schema versions recorded. See docs/LLM.md.88