Source and evidence policy
Trust ladder
- Official site / official documentation / repository of the product (primary). Required for every record (
official_site). - Official license page, pricing page, policy page, release notes, package registry — required to assert
license_spdx,pricing_model(commercial),privacy_posture,product_statuschanges. - Primary announcements (vendor blog, press release) — accepted for renames and acquisitions, cited as
news. - Academic papers for method claims.
- Third-party directories, forums, reviews — discovery only; never cited as evidence for a published claim.
What a record may claim
| Claim | Evidence required |
|---|---|
| Name, type, summary, capabilities | official site/docs |
| License | license file or page (license_page/repository), SPDX identifier |
| Pricing model | pricing page (pricing_page) with pricing_checked_at; prices themselves are never recorded |
| Privacy posture, data residency, certifications | vendor policy/security page (policy_page); otherwise omitted |
| Status (deprecated, discontinued, renamed) | explicit vendor statement or archived repository; a down site alone → unknown + note |
| Comparisons | documented differences only; no unmeasured performance claims |
Everything else is uncertainty_note + confidence: medium|low. Low-confidence records are not published.
Crawling and copyright
- Fetches use a declared User-Agent (
OUTBOUND_USER_AGENT), GET with 20 s timeout, 1 MiB cap, ≤ 5 redirects, 250 ms pacing, daily/weekly schedules — a handful of requests per site per week. robots.txt: the checker only fetches URLs that editors recorded as official pages (homepage, docs, pricing, license, policy). It does not crawl, follow links or index content; it stores a title, a canonical URL and a content hash, never the page body.- Sites that block automated clients (401/403/405/429) are marked unverifiable, not broken, and are re-checked manually.
- Descriptions are original editorial prose; vendor text is never copied. Logos are not redistributed (monograms are generated). Product names are used nominatively.
- Official APIs/feeds are preferred when they exist (e.g. GitHub API for license metadata) — see
docs/ingestion.md.
Security of ingestion
Fetched content is untrusted: SSRF guard (private/loopback/link-local/CGNAT/metadata ranges, .local/.internal/cluster hostnames, redirect re-validation, https→http downgrade rejected); HTML is never rendered; AI suggestions wrap page text as inert data, demand JSON, and keep only evidence quotes that exist verbatim in the page.