SPB Git forge
spboucher.ai ↗

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
743 B
View source Raw Blame History

# Fallback: Scrapfly (scrapfly)

Acquisition hierarchy step 14. Used only when the direct fetch returned 403/429/503 and the source has fallback: { scrapfly: true } and the daily budget (WS_SCRAPFLY_DAILY_BUDGET, default 300 calls) is not exhausted. Request: GET https://api.scrapfly.io/scrape?key&url&asp=true&render_js=false&country=us&retry=true; response result.content is fed to the normal connector normalization; fetch_method = API, x-websensor-via: scrapfly. Cost observed: ~30 credits per call with anti-scraping protection (residential proxy

  • auto browser). Sources enabled at launch: OpenAI (pricing page), xAI, Tesla, NHTSA, NIH — all tier C (hourly). Tested live 2026-09-08 (tesla.com/blog: 200, 1.25 MB).