SPB Git

spb/wp38_uqo

Public
13.8 KB

# Phase 0 — Environment, model and source audit (WP38)

Written 2026-11-01. All figures below were measured on the authorised snapshot or on the production node with read-only queries; nothing was modified in production.

# 0.1 Environment

Item Value
Working directory ~/Desktop/UQO/Recherche/Recherche-UQO/economie/economic-geography-digital-production (git, local main; remote to be created on spbgit as wp38_uqo.git)
Laptop Apple Silicon, 18 cores, 48 GB RAM, 1.0 TB free; macOS 27
Cluster MacLustr (29 Macs, 401 cores) reachable over SSH; only M4M64a (Fabri-Ka host) was used in Phase 0
Python uv venv .venv, Python 3.12, pinned via pyproject.toml (pandas, pyfixest, statsmodels, networkx, geopandas, duckdb, rapidfuzz, …)
LaTeX TeX Live (latexmk, pdflatex), pandoc
Network Outbound HTTPS allowed. Crossref, Statistics Canada WDS, ISQ and Données Québec are reachable
API keys .env (git-ignored) holds one OpenAI key and one Anthropic key inherited from WP37; never printed

# 0.1.1 Model inventory (official provider endpoints, tiny test calls)

See reports/model_inventory.md / logs/model_inventory.json (checked 2026-11-01T17:42Z). OpenAI GET /v1/models lists 140 IDs; gpt-5.5, gpt-5 and gpt-5-mini answered 200 (8-token budget fully consumed by reasoning, so empty text — expected for reasoning models with a tiny budget); gpt-5.1 returned 400 ("could not finish the message" under the 8-token budget). Anthropic lists 14 IDs; claude-fable-5-1, claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5 all answered "OK" in about 1 s. Total cost of the inventory: below 0.002 USD. LLMs will be used only for (a) second annotation of a product-matching validation sample and (b) optional category-reclassification audit; every call will be logged in logs/run_ledger.jsonl with its cost. No empirical result depends on an LLM.

# 0.2 Source identification: Fabri-Ka

Fabri-Ka (www.fabri-ka.com) is an aggregator built and operated by the author (Groupe Ka). It does not sell: it discovers Quebec online stores from certification registries, sector associations, regional agri-food brands and marketplaces (deliverables/research_report.md), verifies each domain, classifies the store by origin class A–E with retained textual evidence, and then syncs each store's public catalogue endpoint (Shopify /products.json, WooCommerce Store API, Wix GraphQL, Squarespace ?format=json, Square, Votresite, generic JSON-LD) roughly every 6 hours (PM2 process fabri-ka-sync). Compliance per source is documented in docs/CONFORMITE.md (reviewed 2026-08-19/21): public endpoints only, nothing behind authentication, polite rates, removal on request.

# 0.2.1 Access and authorisation

  • Authorised access: the author owns the platform and the database. The research copy is a first-party export; no scraping of fabri-ka.com and no use of its /api/ (which robots.txt disallows for crawlers) beyond one control request to /api/stats on 2026-11-01T17:37Z to date-stamp the public counters (products 390 108, stores_live 1 569, stores_registry 4 082, regions 17). The mandate's "approximately 390 456 products, 1 572 stores" was therefore an interface figure from a slightly earlier moment; the counts drift by a few hundred per 6-hour cycle.
  • Snapshot: sqlite3.Connection.backup() of M4M64a:~/fabri-ka/data/fabrika.db on 2026-11-01T17:38:30Z, application commit 689c7bb1…; tables users (2 rows) and inscriptions (1 row) — the only personal data — and the derived FTS index were dropped on the node before transfer. SHA-256 8d963ba9…98fb, 1.24 GB, stored in data/raw/ (git-ignored). The discovery registry stores.json and collections_taxonomy.json were copied with checksums (data/raw/README.md).
  • Production untouched: all queries used mode=ro; the WAL file was not modified.

# 0.2.2 Schema (three tables kept)

table rows key fields
stores 4 082 (1 569 enabled with ≥ 1 published product; 132 enabled with 0; 2 381 disabled = directory-only or dead) id (canonical domain), name, platform, city, region (17 canonical names or blank), origin_class A/C/E, origin_confidence, origin_evidence (text), categories (JSON), discovery_sources (JSON), language, postal_prefix (RTA/FSA), lat,lng (FSA centroid via zippopotam/GeoNames, precision "fsa"), store_kind (revendeur/collectif/blank), certifications (JSON: Aliments du Québec, Les Produits du Québec, CARTV bio…), product_count, fail_streak, presumed_closed, shipping_info
products 465 388 (390 108 active=1 ∧ listing_status='published') uid (20-hex hash of store+external id), store_id, external_id, url, title, description, price (min variant), price_max, compare_at_price, currency, images (JSON), category (20 canonical keys, inferred by keyword rules fabrika/schema.py::infer_category from product type, tags, title and option names), product_type (merchant's own), tags (JSON), vendor (merchant's brand field), available, content_hash, first_seen, last_seen, active, miss_count, details (JSON: variants with price/sku/grams/available, options, attributes, categories), listing_status (published / excluded:{carte_cadeau, atelier_cours, abonnement, billet, test} / quarantine:{prix_hors_bornes, titre_vide}), price_on_request
sync_log 577 773 (ok 572 749, error 4 972, presumed_closed 52) one row per store per sync: ts, store_id, found, added, updated, removed, status, message

There is no price-history table. The database upserts by content_hash and keeps only the current price per listing; sync_log records counts, not prices. Consequently every price analysis in WP38 is cross-sectional at the snapshot date; temporal analysis is limited to listing entry/exit (first_seen/last_seen) and store-level sync counts.

# 0.2.3 Temporal coverage

  • first_seen runs 2026-08-12 → 2026-11-01 (81 days). 291 125 listings carry first_seen = 2026-08-12 (initial load) and a further 48 820 + 30 173 on 2026-08-19/23 (expansion waves); 1 266 stores were onboarded in ISO week 32 and 328 in week 33; only 15 stores were first observed later. Hence store onboarding ended by 2026-08-24, and product-level entry after that date reflects merchant catalogue changes (new listings, connector fixes) rather than platform expansion — this is the only usable "dynamic" margin and it must be audited connector by connector before any H4 statement.
  • Sync cadence: median 6.3 h between successive syncs of a store (p10 1.2 h, p90 8.8 h; 200-store sample, last 30 days). A listing becomes active=0 after 2 consecutive misses (MISS_GRACE = 2, ≈ 12 h). 73 618 published listings are inactive at the snapshot (last seen Aug 6 970, Sep 31 642, Oct 35 222). A disappearance is not a sale (sold-out products usually stay listed with available=0: 48 997 active listings).

# 0.2.4 Field-level quality and missingness (active published listings, N = 390 108)

issue measurement
Price missing 14 786 (3.8 %) price IS NULL (all flagged price_on_request); 31 quarantined prix_hors_bornes (> 100 000 CAD) excluded upstream
Currency CAD 389 378; USD 578 (8 stores); EUR 112; GBP 40 — non-CAD stores to be excluded from price analyses (12 stores)
Price distribution p1 1.32, p25 12.95, median 25.00, p75 69.00, p95 766, p99 3 745 CAD; max 95 999.99
Variants price_max > price for 57 728 listings (14.8 %) — the listing price is the minimum variant price; details.variants carries per-variant prices for Shopify/WooCommerce
Compare-at (strike-through) price 33 311 listings (8.5 %) have compare_at_price > price (on-sale indicator, merchant-declared)
Availability available = 1: 336 038; = 0: 48 997; NULL: 5 073
Category 20 keys; autre = 46 094 (11.8 %); rule-based inference, no manual validation yet → WP38 will audit a stratified sample and report category precision
Brand / vendor vendor non-empty for 290 827 (74.5 %); it is the merchant's own brand field (often the store name)
Product type / tags / description / images non-empty for 305 860 / 261 421 / 339 265 / 360 508
SKU / GTIN SKU present inside details for Shopify/Woo variants; no GTIN/UPC field → cross-merchant matching must rely on vendor + normalised title (+ SKU where present)
Within-store duplicate titles 7 411 (store, title) pairs with > 1 listing (variants listed as separate products, colour/format)
Cross-store identical titles (len > 15) 2 595 distinct titles covering 5 961 listings — first candidate pool for H5
external_id collisions across stores 15 033 — platform-local integer IDs, not product identity (ignored)

# 0.2.5 Merchant-level quality

issue measurement
Region 346 of 1 569 live stores (22 %) have a blank region, carrying 159 096 listings (41 %), including the four largest catalogues (qublivre.ca 24 946 books; ici-la.co 21 412 souvenirs; remorquetrailer.com 19 482; beteaunaturel.com 9 910). Of these 346, 117 have an FSA centroid (lat/lng) → region recoverable by point-in-polygon against the official administrative-region layer; 229 have no geographic field at all → to be resolved from stores.json (qc_postal, city mentions) or left as "unknown region" and excluded from regional analyses (sensitivity: included as an 18th pseudo-region)
Region provenance assigned by (1) regional directory source (certain), (2) regions mentioned on the site, (3) FSA → region map; coordinates = FSA centroid (neighbourhood precision), never the civic address
Origin class A 1 526 stores (322 007 listings), C 12 (63 885), E 31 (4 216); B and D unused. Class is a store-level prior from the discovery source (e.g. PPAQ maple producer = A 0.95; Les Produits du Québec retailer = C 0.75) adjusted by site signals; origin_evidence text retained. Many A entries read "aucun signal de localisation québécoise détecté sur le site (à vérifier)". It is not product-level manufacturing provenance
Certifications 718 live stores carry at least one structured certification (Aliments du Québec, Les Produits du Québec, CARTV organic, …) — third-party store/enterprise certification, usable as a store-level provenance indicator with its own caveats
Platform shopify 696 (273 504 listings), woocommerce 498 (91 100), wix 244 (8 798), votresite 42, square 36, squarespace 23, generic 20, lightspeed 5, others ≤ 2
Catalogue size median 36 listings per store, p90 324, p99 3 997; top-10 stores hold 139 735 listings (36 %)
Store kind revendeur 23, collectif 3 (marketplaces such as epipresto.ca), hors_mission 1, blank 1 542
Sync health 26 stores with an error status in the last 7 days; stores.product_count equals the live published count for all 1 569 stores (consistency check passed)

# 0.2.6 Known selection mechanisms (from the platform's own documentation)

Discovery favours certified, association-member and search-indexed businesses; under-represents CMAQ artisans, industrial manufacturers selling direct, Instagram/Facebook-only micro-shops, Nord-du-Québec, Etsy sellers (excluded by Etsy ToS), ASC farms; Wix stores were only connected in a later wave; Ecwid storefronts cannot be connected; Montréal was deliberately topped-up via a dedicated source (acheterquebecois_mtl, modemtl). Connector caps (max_pages 200) truncate the very largest catalogues. These mechanisms are region- and category-correlated and are central to the representativeness audit (Phase 5).

# 0.3 Feasibility verdict by hypothesis

hypothesis feasible with this source? conditions
H1 regional specialization (RAA) Yes after region recovery; merchant-weighted and largest-merchant-excluded variants mandatory
H2 diversity vs regional scale Yes (n = 17 regions → low power; report as descriptive association with bootstrap CIs) external population/business counts from ISQ/StatCan
H3 concentration / long tail Yes rank-size of merchant catalogues, listing shares
H4 related diversification (dynamic) Partially / exploratory only no store onboarding after week 33; only within-store listing entry over ~10 weeks; no cross-snapshot category entry at region level beyond a handful of events → mark as exploratory, likely inconclusive
H5 price dispersion of matched products Yes, cross-sectional only no price history; matched products via vendor + title + variant; merchant/region collinearity: one merchant = one region → region effect identified only across merchants
H6 local-origin premium Not testable at product level (no product provenance) store-level certification contrast reported as a descriptive, clearly-labelled proxy, not as H6
H7 digital-assortment complexity Yes (17 × 20 matrix; small) null models and merchant-preserving permutations required; eigenvector interpretation guarded

# 0.4 Blockers and decisions carried forward

  1. No price history → temporal price analysis blocked (recorded in reports/limitations.md). A prospective collection plan is written but not executed.
  2. Region unknown for 22 % of live stores → Phase 2 region-recovery procedure with quality flags; all regional results reported with and without recovered stores.
  3. Origin class is store-level → wording throughout: "online assortment of Quebec-based merchants", never "Quebec production".
  4. Category labels are rule-based → stratified audit (manual + LLM second annotation, logged cost) to estimate confusion rates and propagate uncertainty.
  5. Public counters vs cleaned sample: 390 108 published active listings at the snapshot; the research sample will be smaller after exclusions (non-CAD, missing price, unknown region, duplicates) and the full attrition is reported in reports/sample_flow.md.
Files · 70dce80b4301256e2be67a8ee6de33a9a67212cb
Menu

Theme