docs: condensed product brief
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +66 −0
added
docs/PRODUCT-BRIEF.md
+66 −0
@@ -0,0 +1,66 @@ | |||||
| 1 | +# WebSensor.io — product brief (condensed from the founding CLAUDE.md, 2026-09-08) | ||||
| 2 | + | ||||
| 3 | +**Mission.** A production-grade real-time web intelligence platform: *a global sensor network for the changing | ||||
| 4 | +Web*. Not a generic crawler. Sources → sensors/connectors → fetch → change detection → AI interpretation → | ||||
| 5 | +entity resolution → event clustering → importance scoring → historical event store → real-time WebSocket feed. | ||||
| 6 | +Feel: Bloomberg Terminal × GitHub activity × Cloudflare Radar × security monitoring × an Internet-wide changelog. | ||||
| 7 | + | ||||
| 8 | +**Philosophy.** Never just "HTML changed". Answer: what / when / where changed, is it meaningful, which entity, | ||||
| 9 | +how important, part of a larger event, publicly announced or silent, has it happened before. Preserve both the | ||||
| 10 | +RAW OBSERVATION and the INTERPRETED EVENT. | ||||
| 11 | + | ||||
| 12 | +**Engineering principles.** Read docs and test integrations; prefer webhooks → WebSocket → SSE → official API → | ||||
| 13 | +RSS/Atom → structured feeds → JSON/XML → sitemaps → conditional GET → static HTML → rendered HTML → Firecrawl → | ||||
| 14 | +Scrapfly (fallback only, where permitted). Never defeat CAPTCHAs or explicit access controls; honour robots, | ||||
| 15 | +terms, rate limits. Cache and deduplicate aggressively, retain history, keep connectors independently testable, | ||||
| 16 | +never discard raw evidence, idempotent processing, UTC everywhere. Build a connector framework, not one crawler. | ||||
| 17 | + | ||||
| 18 | +**Sensors and tiers.** Every endpoint is a sensor (type, tier, interval, importance weight). Tiers S (15–60 s), | ||||
| 19 | +A (1–5 min), B (5–30 min), C (30 min–6 h), D (6–24 h). Adaptive polling from change frequency/recency, patterns, | ||||
| 20 | +importance history, error rate and cache behaviour. Conditional requests (ETag / Last-Modified) preferred. | ||||
| 21 | + | ||||
| 22 | +**Snapshots, canonicalization, diffs.** Immutable snapshots with raw/canonical/semantic hashes, blobs in object | ||||
| 23 | +storage (content-addressed, compressed). Strip render noise before diffing. Diff engines: text, DOM, JSON, | ||||
| 24 | +table, sitemap, PDF. Staged semantic detection: raw diff → heuristics → structured extraction → semantic | ||||
| 25 | +classification → LLM interpretation → event candidate. | ||||
| 26 | + | ||||
| 27 | +**Events.** Canonical schema (ids, sensor/entities, detected/observed timestamps, type, title, summary, | ||||
| 28 | +importance/confidence/novelty, snapshots, diff, categories, silent_change, processing version). Open taxonomy | ||||
| 29 | +(announcement, product_launch, model_release, pricing_change, policy/terms/API changes, security_advisory, | ||||
| 30 | +vulnerability, breach, incident, outage, recall, drug_approval, filings, earnings, leadership, acquisition, | ||||
| 31 | +funding, layoffs, new_region, DNS/certificate changes, releases, standards, government/economic releases, | ||||
| 32 | +page_created/removed, unknown…). Importance 0–100 = 25 % severity + 20 % source + 15 % entity + 15 % novelty + | ||||
| 33 | +10 % magnitude + 5 % confirmation + 5 % user impact + 5 % unusualness (components stored). Confidence is | ||||
| 34 | +separate. Novelty against recent events (no 50 events for one syndicated press release). Clustering by entity | ||||
| 35 | +overlap, similarity, time proximity. **Silent change** = important change without announcement (flagship). | ||||
| 36 | + | ||||
| 37 | +**Entities & graph.** Everything resolves to entities (organization, product, AI_model, API, drug, person, | ||||
| 38 | +standard, vulnerability…), aliases, relations (owns, released, approved, acquired…), PostgreSQL first. | ||||
| 39 | +Timelines per entity and per URL ("git history for the public Web"), compare any two versions. Latency fields | ||||
| 40 | +(published / first seen / detected / processed / published to feed) — compete on detecting first. | ||||
| 41 | + | ||||
| 42 | +**Real-time & product.** Redis Streams → WebSocket gateway with channels (global, breaking, categories, | ||||
| 43 | +entity, source, watchlist). Frontend: the live feed is the hero; nav Live · Breaking · Explore · Sources · | ||||
| 44 | +Entities · Timelines · Silent Changes · Watchlists · Alerts · API; event detail with importance/confidence, | ||||
| 45 | +evidence, semantic + raw diffs, related events, cluster, timeline, reliability; Explore, Trending, activity | ||||
| 46 | +anomaly per source, early signals (never speculation as fact), watchlists, alerts, search, dark + light, | ||||
| 47 | +performance targets, stable canonical URLs, machine-readable (API, RSS, JSON-LD, sitemaps). | ||||
| 48 | + | ||||
| 49 | +**Platform.** TypeScript/Node/Fastify, PostgreSQL, Redis, object store, Playwright later; Python micro-services | ||||
| 50 | +where useful; Next.js frontend. Deployed on the private Apple Silicon cluster as separable processes, exposed | ||||
| 51 | +through ngrok at https://www.websensor.io (apex redirects to www). Never expose datastores publicly. Strict | ||||
| 52 | +security: rate limiting, validation, SSRF protection (private ranges, metadata endpoints, cluster hosts blocked), | ||||
| 53 | +redirect validation, size/time limits. Observability: Prometheus metrics, structured logs, request ids. | ||||
| 54 | + | ||||
| 55 | +**Registry.** 200 launch organizations across AI, cloud, developer platforms, cybersecurity, consumer tech, | ||||
| 56 | +finance/regulators, government/statistics, healthcare/pharma, science/space, automotive, commerce/payments, | ||||
| 57 | +internet platforms/standards. Seed format in `config/sources.yaml`; discovery validates every candidate. | ||||
| 58 | +Expected 1 000–4 000 sensors at phase 1, then 10 k, 100 k, millions of monitored endpoints. | ||||
| 59 | + | ||||
| 60 | +**MVP definition.** Register sources, discover endpoints, schedule, fetch efficiently, preserve snapshots, | ||||
| 61 | +detect raw changes, eliminate noise, classify, create events, resolve entities, compute importance, cluster, | ||||
| 62 | +publish over WebSocket, live feed, event detail, raw + semantic diffs, source history, watchlists, connector | ||||
| 63 | +health, continuous operation on the cluster. No fake connectors, no placeholder production data. | ||||
| 64 | + | ||||
| 65 | +**Build order.** Foundation → structured sources → intelligence → product → real-time → advanced acquisition. | ||||
| 66 | +**Tagline.** *WebSensor — Detect What Changed. Know Why It Matters.* | ||||
| 67 | |||||