Data dictionary — WP38
Generated by src/reports/data_dictionary.py from the parquet schemas. Raw snapshot fields are described in reports/00_source_audit.md.
data/interim/listings_clean.parquet — 390 108 rows × 56 columns
| column | dtype | non-null | description |
|---|---|---|---|
uid |
object | 1.000 | Fabri-Ka listing id (hash of store + platform product id); stable across syncs |
store_id |
object | 1.000 | canonical store domain (merchant identifier) |
external_id |
object | 1.000 | platform-local product id (not unique across stores) |
url |
object | 1.000 | canonical product URL on the merchant site |
title |
object | 1.000 | listing title as published |
description_head |
object | 1.000 | first 600 characters of the merchant description |
description_len |
int64 | 1.000 | length of the full description (chars) |
price |
float64 | 0.962 | listed price = MINIMUM variant price at the snapshot (merchant currency) |
price_max |
float64 | 0.962 | maximum variant price |
compare_at_price |
float64 | 0.146 | merchant strike-through price (on-sale indicator) |
currency |
object | 1.000 | merchant currency code |
category |
object | 1.000 | Fabri-Ka canonical category (20 keys, rule-based inference; audited in reports/phase2_data_audit.md) |
product_type |
object | 1.000 | merchant's own product type |
vendor |
object | 1.000 | merchant's brand/vendor field (often the store's own name) |
available |
Int64 | 0.987 | merchant availability flag (1 in stock, 0 sold out, NULL unknown) |
content_hash |
object | 1.000 | hash of listing content used by the platform's upsert |
first_seen |
float64 | 1.000 | epoch seconds of first observation |
last_seen |
float64 | 1.000 | epoch seconds of last observation |
active |
int64 | 1.000 | 1 if seen in the last two syncs |
miss_count |
int64 | 1.000 | consecutive misses (0–2) |
listing_status |
object | 1.000 | published |
price_on_request |
int64 | 1.000 | 1 if the merchant shows no price |
n_variants |
float64 | 0.756 | number of variants (Shopify/Woo/Wix details) |
has_sku |
bool | 1.000 | a merchant SKU is present |
sku |
object | 0.769 | first merchant SKU (64 chars max) |
grams |
float64 | 0.427 | variant weight in grams (Shopify grams or Woo kg×1000) — first variant |
n_options |
float64 | 0.472 | number of option dimensions |
variant_prices_min |
float64 | 0.697 | min of variant prices in details |
variant_prices_max |
float64 | 0.697 | max of variant prices in details |
n_variant_prices |
float64 | 0.697 | number of variant prices |
on_sale_flag |
object | 0.085 | Shopify on_sale flag |
merchant_categories |
object | 0.184 | merchant's own category path (Woo/Wix) |
published_at |
object | 0.701 | Shopify published_at |
created_at |
object | 0.701 | Shopify created_at |
n_images |
int64 | 1.000 | number of images |
tags_list |
object | 1.000 | merchant tags (JSON list) |
n_tags |
int64 | 1.000 | number of tags |
first_seen_utc |
datetime64[ns, UTC] | 1.000 | first observation (UTC) |
last_seen_utc |
datetime64[ns, UTC] | 1.000 | last observation (UTC) |
f_published |
bool | 1.000 | active = 1 and listing_status = 'published' |
f_live_store |
bool | 1.000 | store enabled with >= 1 published listing |
store_kind |
object | 1.000 | standard |
f_hors_mission |
bool | 1.000 | store flagged out of mission by the platform |
title_norm |
object | 1.000 | ASCII-folded, lower-cased, punctuation-stripped title |
dup_group_size |
int64 | 1.000 | number of listings with the same normalised title in the store |
f_dup_in_store |
bool | 1.000 | duplicated title within store |
f_dup_rep |
bool | 1.000 | representative (first-seen) listing of its (store, title) group |
f_sibling_dup |
bool | 1.000 | store is a non-representative mirror domain of another store |
f_analysis_assort |
bool | 1.000 | ASSORTMENT SAMPLE membership |
store_cad_share |
float64 | 1.000 | share of the store's listings priced in CAD |
f_cad |
bool | 1.000 | CAD listing in a CAD store (>= 95 %) |
f_price_ok |
bool | 1.000 | 0 < price <= 50 000, not on request |
f_analysis_price |
bool | 1.000 | PRICE SAMPLE membership |
log_price |
float64 | 0.962 | natural log of price (price sample) |
on_sale |
bool | 1.000 | compare_at_price > price |
has_variants |
bool | 1.000 | price_max > price or n_variants > 1 |
data/interim/stores_master.parquet — 1 569 rows × 43 columns
| column | dtype | non-null | description |
|---|---|---|---|
id |
object | 1.000 | store domain |
name |
object | 1.000 | store name |
platform |
object | 1.000 | e-commerce platform (connector) |
city |
object | 1.000 | city/sector label from the FSA centroid or directory |
region_db |
object | 0.779 | Fabri-Ka region (directory / site / FSA map) |
region_pip |
object | 0.797 | region containing the FSA centroid (official MERN layer, point-in-polygon) |
region_city |
object | 0.250 | |
region_pipfirst |
object | 1.000 | |
mrc_pip |
object | 0.797 | |
region_final |
object | 1.000 | region_db, else region_pip, else 'Inconnue' |
region_source |
object | 1.000 | db |
region_final_vs_polygon |
bool | 1.000 | |
region_conflict |
bool | 1.000 | region_db ≠ region_pip (both present) |
has_geo |
bool | 1.000 | FSA centroid available |
lat |
float64 | 0.797 | FSA centroid latitude |
lng |
float64 | 0.797 | FSA centroid longitude |
postal_prefix |
object | 0.824 | forward sortation area (first 3 characters) |
phone_shared |
bool | 1.000 | phone shared by >= 3 stores (directory contamination; never used) |
origin_class |
object | 1.000 | store-level origin class A/C/E assigned by the platform from discovery-source priors + site signals (NOT product provenance) |
origin_confidence |
float64 | 1.000 | platform confidence in origin_class |
origin_evidence |
object | 1.000 | retained textual evidence for origin_class |
certified |
bool | 1.000 | store carries >= 1 structured third-party certification |
cert_adq |
bool | 1.000 | Aliments du Québec member |
cert_lpq |
bool | 1.000 | Les Produits du Québec certified |
cert_bio |
bool | 1.000 | CARTV organic |
cert_labels |
object | 1.000 | certification labels |
store_kind |
object | 1.000 | standard |
language |
object | 1.000 | site language |
n_sources |
int64 | 1.000 | number of discovery sources |
first_source |
object | 1.000 | first discovery source |
reg_status |
object | 1.000 | registry verification status (verified/probable) |
product_count |
int64 | 1.000 | platform's published count |
n_listings |
int64 | 1.000 | assortment-sample listings |
n_categories |
float64 | 0.992 | distinct categories in the assortment sample |
share_autre |
float64 | 0.992 | share of 'autre' |
median_price |
float64 | 0.953 | median listed price (CAD) |
share_available |
float64 | 0.985 | share available = 1 |
share_on_sale |
float64 | 0.992 | share on sale |
main_category |
object | 0.992 | modal category |
first_listing_utc |
datetime64[ns, UTC] | 0.992 | earliest first_seen |
last_listing_utc |
datetime64[ns, UTC] | 0.992 | latest first_seen |
last_sync_utc |
datetime64[ns, UTC] | 1.000 | last sync |
fail_streak |
int64 | 1.000 | consecutive failed syncs |
data/interim/sibling_stores.parquet — 22 rows × 5 columns
| column | dtype | non-null | description |
|---|---|---|---|
sibling_group |
int64 | 1.000 | mirror-domain group id |
id |
object | 1.000 | store domain |
sibling_rep |
object | 1.000 | representative store of the group |
is_rep |
bool | 1.000 | is the representative |
n_titles |
int64 | 1.000 | distinct titles |
data/interim/match_pairs.parquet — 5 453 rows × 16 columns
| column | dtype | non-null | description |
|---|---|---|---|
uid_a |
object | 1.000 | listing a |
uid_b |
object | 1.000 | listing b |
tier |
object | 1.000 | matching tier A/B/C/D |
score |
float64 | 1.000 | rapidfuzz score |
store_a |
object | 1.000 | |
price_a |
float64 | 1.000 | |
nv_a |
int64 | 1.000 | |
grams_a |
float64 | 0.469 | |
cat_a |
object | 1.000 | |
store_b |
object | 1.000 | |
price_b |
float64 | 1.000 | |
nv_b |
int64 | 1.000 | |
grams_b |
float64 | 0.484 | |
cat_b |
object | 1.000 | |
comparable |
bool | 1.000 | variant-structure comparability flag |
log_gap |
float64 | 1.000 |
data/interim/match_groups.parquet — 4 832 rows × 11 columns
| column | dtype | non-null | description |
|---|---|---|---|
group_id |
int64 | 1.000 | match group (connected component over tiers A–C) |
uid |
object | 1.000 | Fabri-Ka listing id (hash of store + platform product id); stable across syncs |
store_id |
object | 1.000 | canonical store domain (merchant identifier) |
price |
float64 | 1.000 | listed price = MINIMUM variant price at the snapshot (merchant currency) |
category |
object | 1.000 | Fabri-Ka canonical category (20 keys, rule-based inference; audited in reports/phase2_data_audit.md) |
nv |
int64 | 1.000 | variant count |
grams |
float64 | 0.542 | variant weight in grams (Shopify grams or Woo kg×1000) — first variant |
vendor_norm |
object | 1.000 | normalised brand |
n |
int64 | 1.000 | group size |
n_stores |
int64 | 1.000 | stores in group |
n_cat |
int64 | 1.000 | categories in group |
data/interim/sync_log.parquet — 577 773 rows × 8 columns
| column | dtype | non-null | description |
|---|---|---|---|
ts |
float64 | 1.000 | epoch |
store_id |
object | 1.000 | canonical store domain (merchant identifier) |
found |
int64 | 1.000 | listings found in the sync |
added |
int64 | 1.000 | added |
updated |
int64 | 1.000 | updated |
removed |
int64 | 1.000 | removed |
status |
object | 1.000 | ok |
ts_utc |
datetime64[ns, UTC] | 1.000 | UTC time |