SPB Git forge

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
1.5 KB · 16 lines markdown
Rendered Raw Blame History
1# Connector: CSV / TSV files (`csv`)2**Purpose**: open-data time series and registers published as delimited files — FRED `fredgraph.csv?id=…` (no API3key), ECB Data Portal `format=csvdata`, NOAA Mauna Loa CO₂, NASA GISTEMP, Cboe VIX history, statistics offices,4sanctions/registers exported as CSV. A new observation, a revised value or a removed row is a list event.5**Sensor types**: FILE. **Config**: `keyColumn` (name or list of names; `"@row"` = hash of the row), `compareColumns`6(default: all non-key columns), `titleColumn`, `dateColumn`, `delimiter` (auto: `,` `\t` `;` `|`), `skipRows`7(leading title lines, e.g. GISTEMP), `commentPrefix` (`#` lines, e.g. NOAA), `maxRows` (2000), `tail: true` (keep the8**last** rows of a growing series — always set it on time series), `encoding`, `headers`.9**Normalization**: RFC 4180 parser (quotes, escaped quotes, CRLF, BOM) → keyed rows; `compareFields` = the compared10columns; `publishedAt` = newest `dateColumn` value. `extra`: rows, columns, header, truncated.11**Failure handling**: HTML instead of CSV → `html_not_csv`; unknown key/compare column → `bad_config` with the header12shown; < 2 rows → `bad_shape`. Limits 64 MB / 60 s.13**Known quirks**: FRED returns `application/csv`; values may be `.` (missing) — kept as text so a revision from `.`14to a number is a change; Bank of Canada Valet CSV has a metadata preamble → use its JSON endpoint with `jsonlist`.15**Tests**: `connectors2.test.ts`. **Verified**: 2026-09-08 (16 FRED series, ECB EXR/FM/ICP, NOAA, GISTEMP, VIX).16