SPB Git forge

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
1.5 KB

# Connector: CSV / TSV files (csv)

Purpose: open-data time series and registers published as delimited files — FRED fredgraph.csv?id=… (no API key), ECB Data Portal format=csvdata, NOAA Mauna Loa CO₂, NASA GISTEMP, Cboe VIX history, statistics offices, sanctions/registers exported as CSV. A new observation, a revised value or a removed row is a list event. Sensor types: FILE. Config: keyColumn (name or list of names; "@row" = hash of the row), compareColumns (default: all non-key columns), titleColumn, dateColumn, delimiter (auto: , \t ; |), skipRows (leading title lines, e.g. GISTEMP), commentPrefix (# lines, e.g. NOAA), maxRows (2000), tail: true (keep the last rows of a growing series — always set it on time series), encoding, headers. Normalization: RFC 4180 parser (quotes, escaped quotes, CRLF, BOM) → keyed rows; compareFields = the compared columns; publishedAt = newest dateColumn value. extra: rows, columns, header, truncated. Failure handling: HTML instead of CSV → html_not_csv; unknown key/compare column → bad_config with the header shown; < 2 rows → bad_shape. Limits 64 MB / 60 s. Known quirks: FRED returns application/csv; values may be . (missing) — kept as text so a revision from . to a number is a change; Bank of Canada Valet CSV has a metadata preamble → use its JSON endpoint with jsonlist. Tests: connectors2.test.ts. Verified: 2026-09-08 (16 FRED series, ECB EXR/FM/ICP, NOAA, GISTEMP, VIX).