# FinData OS — Data dictionary and conventions

## Bars (`market_bars.*`) — Arrow schema v1 (`GET /api/v1/catalog/schema/bars`)
| Column | Type | Meaning |
|---|---|---|
| instrument_id | string | `firstrate:<asset_class>:<SYMBOL>` (vendor-keyed). Delisted: `…:<SYMBOL>-DELISTED` |
| symbol_as_reported | string | vendor file ticker (uppercase) |
| venue | string? | unknown from vendor (US consolidated); NULL |
| asset_class | string | stock, etf, futures, index, fx, crypto |
| interval | string | 1m, 5m, 30m, 1h, 1d |
| **timestamp_utc** | timestamp[us, UTC] | **bar OPEN** instant. The 09:30 bar covers 09:30:00–09:30:59 ET |
| session_date | date | trading session date in the exchange timezone (America/New_York; crypto: UTC date) |
| open/high/low/close | float64 | prices in `currency` |
| volume | float64 | shares; integral for raw data (fractional only in vendor-adjusted series) |
| currency | string | USD |
| trade_count | int32? | number of 1-minute bars aggregated (derived datasets only) |
| vwap | float64? | volume-weighted close of the aggregated bars (derived datasets only; NULL when volume = 0) |
| adjustment_type | string | raw (stored) · split · split_div (query-time) · continuous (futures) |
| source_id | string | firstrate |
| ingested_at | timestamp[us, UTC] | when FinData normalized the row |
| quality_flags | list<string> | see below; never alter prices |
| revision_id | int32 | increments when a session is re-published by the vendor |

### Sessions, DST, calendars
- Vendor timestamps are US Eastern wall-clock (EST/EDT); conversion to UTC uses `zoneinfo America/New_York`. Non-existent local times (spring-forward hour) are dropped and counted (`rows_dropped_unparseable` in the job result); extended hours never overlap the ambiguous fall-back hour.
- Extended session 04:00–20:00 ET (last 1-minute bar opens 19:59). Regular session 09:30–16:00; the 16:00 bar is the closing print. Expected regular bars per session = minutes(open→close) + 1 (391 on a normal day, 211 on a 13:00 early close), from `exchange_calendars` XNYS.
- **Zero-volume minutes are absent at the source** — FinData never forward-fills or manufactures bars. Coverage = bars present / expected regular bars.
- Overnight futures sessions: daily futures bars follow the contract trading day (≈18:00 ET previous day → 17:00); `session_date` is the vendor's date (dataset planned).
- Crypto: UTC timestamps, 7 days a week (dataset planned).

### Quality flags (row level) and anomaly codes
`ohlc_inconsistent`, `nonpositive_price`, `negative_volume`, `zero_volume`, `fractional_volume`, `off_interval`, `duplicate`, `outside_session`, `non_trading_day`, `price_jump` (> 25 % between consecutive bars within a session), `gap` (coverage level). Severity: error (ohlc/nonpositive/negative/duplicate/off_interval/non_trading_day) · warning (price_jump, outside_session) · info (zero_volume, fractional_volume, gap).

### Adjustments
- Stored data is **raw** (vendor `UNADJUSTED`). `adjustment=split`: prices × Π(1/ratio) and volume × Π(ratio) for splits with ex_date after the bar's session. `adjustment=split_div`: additionally × Π(1 − dividend / last close before ex_date) using the closes present in the requested range (if no prior close is in range the dividend factor is skipped and reported in `meta.adjustment_notes`). Splits and cash dividends only (vendor meta files); no mergers/spin-offs.
- Vendor-adjusted history is rewritten after each corporate action; FinData avoids that drift by storing raw bars and applying actions explicitly.

### Symbols, delistings, survivorship
- Vendor merges ticker changes under the latest symbol (fully since 2022); earlier changes mapped by the vendor. Delisted instruments keep a `-DELISTED` suffix; `instrument_aliases` records the base ticker. The instrument master includes 8 069 delisted tickers (listing of 2026-11-01), so universes can be built without survivorship bias when delisted archives are ingested (planned).
- Futures roll policy is the vendor's (fixed dates ≈ 7–10 days before expiry for ES/NQ); three continuous series (unadjusted, ratio, absolute); absolute-adjusted series may be negative (dataset planned).

## Fundamentals (`fundamentals_facts`)
| Column | Meaning |
|---|---|
| entity_id | `frd:<TICKER>` (vendor) or `sec:CIK##########` (EDGAR) |
| security_id | `firstrate:stock:<TICKER>` when known |
| metric | vendor field name (rev, net_inc, eps_d, assets_tot, cash_op, shares, roe, …) — 76 fields, units in `unit` |
| value / unit / currency | numeric; USD, USD/share, shares, ratio |
| fiscal_period_start/end, fiscal_period_type (q/a), fiscal_year, fiscal_period | vendor period; start is approximated (vendor gives end only) |
| filing_date | SEC filing date (vendor) |
| published_at | unknown from vendor → NULL |
| **available_at** | PIT policy: `filing_date + 1 day 00:00 America/New_York` (fallback period_end + 91 d when filing_date missing, flagged `pit_fallback`) |
| observed_at | when FinData ingested this value |
| revision_id / restatement_flag / superseded_at | a later ingestion seeing a different value creates revision n+1 with `restatement_flag=true` and supersedes revision n |
| source_document / source_id | `SEC 10-Q via FirstRate (filing_date=…)` / `firstrate_fundamentals` |

**As-of semantics** (`GET /api/v1/fundamentals/{entity}?as_of=`): visible iff `available_at <= as_of` and (`revision_id = 1` or `observed_at <= as_of`). Revision 1 is the proxy for "as originally filed"; restatements observed after `as_of` are never shown. Limitation: restatements that happened before FinData's first ingestion are not reconstructible from this vendor (SEC EDGAR connector, once configured, provides filing-level history).

## Provenance recorded per row / file
vendor + product (`sources`), endpoint + sanitized request (`raw_archives.request`), retrieval timestamp (`raw_archives.fetched_at`, `ingested_at`), vendor update date (`raw_archives.vendor_last_update`), archive sha256, Parquet sha256 (`lake_files`), dataset version + snapshot id (`dataset_versions`, `X-Dataset-Version` header), transformation code version (`ingest_runs.code_version` = git short SHA), completeness (`coverage`), license class (`datasets.license_class`).