SPB Git
spboucher.ai

spb/resto-ka

Public

Resto·Ka — tous les restaurants du Québec, menus complets et prix réels (famille ·Ka)

Major upgrade connecteurs : 7 nouvelles sources (mtl-alim, tastet, chownow, gloriafood, square, doordash, skip) + site-resto hub de découverte multi-plateformes

- mtl-alim : découverte massive Montréal (CSV open data, ~7 500 fiches food-service)
- tastet : découverte éditoriale (1 516 critiques, coordonnées + lien source)
- chownow/gloriafood/square : Palier 1 takeout (prix réels), dormants jusqu'à
  découverte de leurs intégrations par site-resto (*-discovered.json avec uid porteur)
- doordash/skip : Palier 2 delivery, enrichissement conservateur (jamais de fiche,
  croisement téléphone/postal+civique/GPS+nom, menus étiquetés delivery)
- siteresto : détection des widgets GloriaFood/ChowNow/Square/Clover sur les sites
- dedup : priorités mtl-alim/tastet ; registre sources.json complété (17 sources)
- 7 fichiers de tests (110 tests verts), smoke-runs live validés

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Aug 25, 2026) parent 19425cb Browse files

18 changed files +2,320 −10

18 changed files
Load all 18 files Unified Split
modified .gitignore +2 −0
@@ -15,3 +15,5 @@ frontend/dist/
15 frontend/tsconfig.tsbuildinfo 15 frontend/tsconfig.tsbuildinfo
16 .DS_Store 16 .DS_Store
17 data/ubereats-stores.json 17 data/ubereats-stores.json
18+data/doordash-stores.json
19+data/skip-stores.json
modified data/sources.json +91 −0
@@ -334,6 +334,97 @@
334 "acces_legal": "Données ouvertes du gouvernement du Québec (RACJ), licence CC-BY 4.0 — attribution RACJ/Données Québec affichée avec l'information de permis.", 334 "acces_legal": "Données ouvertes du gouvernement du Québec (RACJ), licence CC-BY 4.0 — attribution RACJ/Données Québec affichée avec l'information de permis.",
335 "cadence": "hebdomadaire (guard 6 jours dans ingest.enrich)", 335 "cadence": "hebdomadaire (guard 6 jours dans ingest.enrich)",
336 "status": "actif" 336 "status": "actif"
337+ },
338+ {
339+ "id": "mtl-alim",
340+ "name": "Montréal — Établissements alimentaires (données ouvertes)",
341+ "url": "https://donnees.montreal.ca/dataset/etablissements-alimentaires",
342+ "platform": "donnees-montreal",
343+ "connector": "mtl-alim",
344+ "tier": 4,
345+ "price_context": null,
346+ "extraction": "CSV ouvert businesses.csv (donnees.montreal.ca, ~54 000 lignes) : whitelist de 12 types d'établissements où l'on mange (Restaurant, service rapide, casse-croûte, bar, boulangerie-pâtisserie, traiteur, camion-cuisine…), statut Ouvert seulement, GPS bornés à l'île de Montréal. Nom, adresse, ville, GPS, type -> fiches de découverte SANS menu (les connecteurs de menus s'y greffent par déduplication). ~7 500 fiches.",
347+ "acces_legal": "Données ouvertes de la Ville de Montréal, licence CC-BY 4.0 — attribution affichée (page Sources). Un seul téléchargement CSV par cycle, UA identifiable.",
348+ "cadence": "mensuel",
349+ "status": "actif"
350+ },
351+ {
352+ "id": "tastet",
353+ "name": "Tastet (média food — découverte éditoriale)",
354+ "url": "https://tastet.ca",
355+ "platform": "tastet",
356+ "connector": "tastet",
357+ "tier": 4,
358+ "price_context": null,
359+ "extraction": "Sitemaps Yoast reviews-sitemap*.xml -> pages critiques ; chaque page embarque un lien Google Maps (nom + adresse complète) et un lien tel:. Fiches de découverte SANS menu (nom, adresse, ville, postal, téléphone, photo og:image, lien critique) ; la déduplication fusionne avec OSM/UEAT et la fiche canonique gagne le lien éditorial. Cache BD par page (clé lastmod), budget TASTET_BUDGET (80 pages/cycle), corpus incrémental.",
360+ "acces_legal": "Média public québécois, découverte minimale (coordonnées + lien sortant vers la critique = attribution). Politesse 1,2 s, UA identifiable, pas d'anti-bot à contourner (CLAUDE.md §15).",
361+ "cadence": "hebdomadaire par lots (80 pages), re-crawl si lastmod change",
362+ "status": "actif"
363+ },
364+ {
365+ "id": "chownow",
366+ "name": "ChowNow (commande en ligne)",
367+ "url": "https://www.chownow.com",
368+ "platform": "chownow",
369+ "connector": "chownow",
370+ "tier": 1,
371+ "price_context": "takeout",
372+ "extraction": "API JSON publique non authentifiée : api.chownow.com/api/restaurant/<id> (fiche complète : nom, adresse, téléphone, GPS, site) + /<id>/menu (menu_categories + graphe complet des modificateurs -> options §5.2). 2 requêtes/resto, politesse 0,8 s. Émet des fiches complètes avec menu takeout ; filtre state == QC. Intégrations découvertes automatiquement par site-resto (liens direct.chownow.com sur les sites des restos -> data/chownow-discovered.json).",
373+ "acces_legal": "Source de première partie (plateforme de commande du resto lui-même, prix réels non majorés — CLAUDE.md §9 Palier 1, §15). API ouverte, extraction minimale, lien de commande conservé.",
374+ "cadence": "hebdomadaire",
375+ "status": "dormant (aucune intégration découverte encore — SkipSource)"
376+ },
377+ {
378+ "id": "gloriafood",
379+ "name": "GloriaFood / Oracle (widget de commande)",
380+ "url": "https://www.gloriafood.com",
381+ "platform": "gloriafood",
382+ "connector": "gloriafood",
383+ "tier": 1,
384+ "price_context": "takeout",
385+ "extraction": "API publique du widget : GET pos.globalfoodsoft.com/pos/menu (Authorization: <ruid>, Glf-Api-Version: 2) -> categories[].items[] {name, description, price, sizes}. ENRICHISSEMENT PUR : le menu est posé sur le resto porteur (uid consigné par site-resto avec les data-glf-cuid/ruid détectés dans le HTML -> data/gloriafood-discovered.json). Re-visite 14 j, refus 403 défensifs.",
386+ "acces_legal": "API du widget de commande du resto lui-même (première partie, prix réels — CLAUDE.md §9 Palier 1, §15). Extraction minimale, lien conservé.",
387+ "cadence": "hebdomadaire, re-visite 14 jours",
388+ "status": "dormant (aucune intégration découverte encore — SkipSource)"
389+ },
390+ {
391+ "id": "square",
392+ "name": "Square Online (boutiques de commande)",
393+ "url": "https://squareup.com/ca/fr/online-store",
394+ "platform": "square-online",
395+ "connector": "square",
396+ "tier": 1,
397+ "price_context": "takeout",
398+ "extraction": "Boutiques <sous-domaine>.square.site détectées par site-resto (lien de commande sur le site du resto -> data/square-discovered.json avec l'uid porteur). SPA sans JSON-LD ni prix inline : page /s/order rendue en JS via Scrapfly puis texte structuré par Claude Haiku (menullm, validation stricte des prix). ENRICHISSEMENT PUR : menu takeout posé sur le resto porteur + details.square. Budget SQUARE_BUDGET (15 rendus/cycle), re-visite 30 j.",
399+ "acces_legal": "Boutique de commande du resto lui-même (première partie, prix réels — CLAUDE.md §9 Palier 1, §15). Extraction minimale, lien source conservé.",
400+ "cadence": "hebdomadaire par petits lots (15 rendus), re-visite 30 jours",
401+ "status": "dormant (aucune boutique découverte encore — SkipSource)"
402+ },
403+ {
404+ "id": "doordash",
405+ "name": "DoorDash (menus livraison + notes)",
406+ "url": "https://www.doordash.com",
407+ "platform": "doordash",
408+ "connector": "doordash",
409+ "tier": 2,
410+ "price_context": "delivery",
411+ "extraction": "Découverte par le sitemap public QC (sitemap-doordash-qc-stores.xml, ~11 700 URLs /en-CA/store/, index local data/doordash-stores.json rafraîchi 30 j). Chaque page resto embarque du JSON-LD schema.org (Restaurant + Menu -> sections/items/prix, note, adresse, GPS). Croisement conservateur avec un resto EXISTANT (GPS<120 m + nom similaire OU civique+ville+nom) — n'émet aucune fiche. Menu -> table menus en price_context delivery (prix majorés ~25-30 %), note -> details.doordash. Budget DOORDASH_BUDGET (40 pages/cycle), verdicts en cache, re-visite 30 j.",
412+ "acces_legal": "Palier 2 (CLAUDE.md §9, §15) : CGU restrictives, anti-bot — Scrapfly ASP. Usage d'appoint minimal : menus pour restos sans source dine-in/takeout, prix TOUJOURS étiquetés delivery, lien vers la fiche source conservé. Préférer UEAT/site du resto dès que disponible.",
413+ "cadence": "quotidien par petits lots (40 pages), re-vérification 30 jours",
414+ "status": "actif"
415+ },
416+ {
417+ "id": "skip",
418+ "name": "SkipTheDishes (menus livraison)",
419+ "url": "https://www.skipthedishes.com",
420+ "platform": "skip",
421+ "connector": "skip",
422+ "tier": 2,
423+ "price_context": "delivery",
424+ "extraction": "Découverte par sitemaps publics (sitemap2..7.xml, URLs partenaires), index local data/skip-partners.json rafraîchi 30 j. Chaque page partenaire embarque __NEXT_DATA__ (menu complet en cents, téléphone dans les disclaimers, ville/province/postal). Croisement conservateur avec un resto EXISTANT (téléphone identique OU postal+civique), rejet hors-QC — n'émet aucune fiche. Menu -> table menus en price_context delivery, note -> details.skip. Budget SKIP_BUDGET (40 pages/cycle), verdicts en cache, re-visite 30 j.",
425+ "acces_legal": "Palier 2 (CLAUDE.md §9, §15) : CGU restrictives, anti-bot — Scrapfly ASP. Usage d'appoint minimal : menus pour restos sans source dine-in/takeout, prix TOUJOURS étiquetés delivery, lien vers la fiche source conservé.",
426+ "cadence": "quotidien par petits lots (40 pages), re-vérification 30 jours",
427+ "status": "actif"
337 } 428 }
338 ] 429 ]
339 } 430 }
\ No newline at end of file
added restoka/connectors/chownow.py +204 −0
@@ -0,0 +1,204 @@
1+# ==============================================================================
2+# Author: Simon-Pierre Boucher <contact@spboucher.ai>
3+# File: restoka/connectors/chownow.py
4+# Desc: Connecteur templatisé ChowNow (Palier 1 — plateforme de commande en
5+# ligne des restos eux-mêmes -> PRIX RÉELS, contexte `takeout`).
6+# API JSON publique et non authentifiée :
7+# GET api.chownow.com/api/restaurant/<id> (fiche : nom,
8+# adresse, téléphone, GPS « lat,lng », cuisines, site web)
9+# GET api.chownow.com/api/restaurant/<id>/menu (menu_categories[]
10+# .items[] {name, description, price, size} + graphe complet
11+# des modificateurs -> options §5.2)
12+# Une intégration = un id de resto ChowNow, consigné dans
13+# data/sources.json (source « chownow » -> integrations[]) et/ou
14+# découvert automatiquement par le connecteur site-resto
15+# (data/chownow-discovered.json). CONNECTEUR DORMANT tant qu'aucune
16+# intégration n'est connue (SkipSource — pas un échec).
17+# Seuls les restos au Québec (state == QC) sont émis.
18+#
19+# Mode d'extraction : API directe (2 requêtes/resto), politesse 0.6 s.
20+# Base légale §15 : API ouverte de la plateforme du resto (source de
21+# première partie), extraction minimale, lien source conservé.
22+# ==============================================================================
23+from __future__ import annotations
24+
25+import json
26+import sys
27+from pathlib import Path
28+
29+from ..schema import Restaurant
30+from .base import BaseConnector, SkipSource
31+
32+API = "https://api.chownow.com/api/restaurant"
33+SOURCES_PATH = Path(__file__).resolve().parents[2] / "data" / "sources.json"
34+
35+
36+def parse_geo(geo: str) -> tuple[float | None, float | None]:
37+ """« 45.5017,-73.5673 » -> (lat, lng)."""
38+ try:
39+ lat_s, lng_s = (geo or "").split(",", 1)
40+ return float(lat_s), float(lng_s)
41+ except (ValueError, AttributeError):
42+ return None, None
43+
44+
45+def build_options(item: dict, mod_cats: dict[str, dict],
46+ mods: dict[str, dict]) -> list[dict]:
47+ """Options §5.2 depuis le graphe de modificateurs ChowNow (item ->
48+ modifier_categories -> modifiers {name, price})."""
49+ options: list[dict] = []
50+ for cat_id in item.get("modifier_categories") or []:
51+ cat = mod_cats.get(str(cat_id))
52+ if not cat:
53+ continue
54+ choices = []
55+ for mod_id in cat.get("modifiers") or []:
56+ mod = mods.get(str(mod_id))
57+ if not mod or not (mod.get("name") or "").strip():
58+ continue
59+ price = mod.get("price")
60+ choices.append({
61+ "name": mod["name"].strip(),
62+ "price_delta": round(float(price), 2)
63+ if isinstance(price, (int, float)) else 0.0,
64+ })
65+ if choices:
66+ options.append({
67+ "group": (cat.get("name") or "").strip() or "Options",
68+ "required": bool(cat.get("min_qty")),
69+ "choices": choices,
70+ })
71+ return options
72+
73+
74+def build_menu(menu_payload: dict, captured_at: str) -> dict | None:
75+ """Menu standard Resto·Ka (CLAUDE.md §5.2) depuis menu_categories."""
76+ mod_cats = {str(c.get("id")): c
77+ for c in menu_payload.get("modifier_categories") or []}
78+ mods = {str(m.get("id")): m for m in menu_payload.get("modifiers") or []}
79+ sections_out: list[dict] = []
80+ seen: set[str] = set()
81+ for cat in menu_payload.get("menu_categories") or []:
82+ name = (cat.get("name") or "").strip()
83+ if not name or name in seen:
84+ continue
85+ items = []
86+ for it in cat.get("items") or []:
87+ title = (it.get("name") or "").strip()
88+ if not title:
89+ continue
90+ size = (it.get("size") or "").strip()
91+ if size and size.lower() not in ("regular", "régulier"):
92+ title = f"{title} ({size})"
93+ price = it.get("price")
94+ entry = {
95+ "name": title,
96+ "description": (it.get("description") or "").strip(),
97+ "price": round(float(price), 2)
98+ if isinstance(price, (int, float)) and price > 0 else None,
99+ }
100+ options = build_options(it, mod_cats, mods)
101+ if options:
102+ entry["options"] = options
103+ items.append(entry)
104+ if items:
105+ seen.add(name)
106+ sections_out.append({"name": name, "items": items})
107+ if not sections_out:
108+ return None
109+ return {
110+ "price_context": "takeout",
111+ "price_source": "chownow",
112+ "currency": "CAD",
113+ "captured_at": captured_at,
114+ "sections": sections_out,
115+ }
116+
117+
118+class ChowNowConnector(BaseConnector):
119+ source_id = "chownow"
120+ request_delay = 0.8
121+ timeout = 30
122+ use_detail_cache = False # 2 requêtes API légères par resto
123+
124+ def _integrations(self) -> list[dict]:
125+ registry = json.loads(SOURCES_PATH.read_text(encoding="utf-8"))
126+ out: list[dict] = []
127+ for src in registry.get("sources", []):
128+ if src.get("id") == self.source_id:
129+ out.extend(src.get("integrations") or [])
130+ # ids découverts automatiquement par site-resto (widget ChowNow
131+ # détecté sur le site d'un resto) — voir siteresto.py
132+ discovered = SOURCES_PATH.parent / "chownow-discovered.json"
133+ if discovered.exists():
134+ try:
135+ extra = json.loads(discovered.read_text(encoding="utf-8"))
136+ known = {str(i.get("key")) for i in out}
137+ out.extend(i for i in extra.get("integrations", [])
138+ if str(i.get("key")) not in known)
139+ except ValueError:
140+ pass
141+ return out
142+
143+ def _now(self) -> str:
144+ import datetime
145+ return datetime.datetime.now(datetime.timezone.utc) \
146+ .strftime("%Y-%m-%dT%H:%M:%SZ")
147+
148+ def _fetch_restaurant(self, cn_id: str) -> Restaurant | None:
149+ info = self.get(f"{API}/{cn_id}",
150+ headers={"Accept": "application/json"}).json()
151+ addr = info.get("address") or {}
152+ state = (addr.get("state") or "").strip().upper()
153+ if state and state != "QC":
154+ print(f"[resto-ka] chownow: {cn_id} hors Québec ({state}) — ignoré",
155+ file=sys.stderr)
156+ return None
157+ name = (info.get("name") or "").strip()
158+ if not name:
159+ return None
160+ lat, lng = parse_geo(addr.get("geo") or "")
161+ menu_payload = self.get(f"{API}/{cn_id}/menu",
162+ headers={"Accept": "application/json"}).json()
163+ menu = build_menu(menu_payload or {}, self._now())
164+ services = ["takeout"]
165+ if (info.get("fulfillment") or {}).get("delivery") \
166+ or info.get("has_low_cost_delivery"):
167+ services.append("delivery")
168+ return Restaurant(
169+ source=self.source_id,
170+ external_id=str(cn_id),
171+ name=name,
172+ url=f"https://direct.chownow.com/order/{cn_id}/locations",
173+ address=(addr.get("street_address1") or "").strip(),
174+ city=(addr.get("city") or "").strip(),
175+ postal_code=(addr.get("zip") or "").strip(),
176+ lat=lat, lng=lng,
177+ phone=(info.get("phone") or "").strip(),
178+ website=(info.get("website_url") or "").strip(),
179+ services=services,
180+ menu=menu,
181+ )
182+
183+ def fetch(self) -> list[Restaurant]:
184+ integrations = [i for i in self._integrations()
185+ if i.get("key") and i.get("status") != "inactif"]
186+ if not integrations:
187+ raise SkipSource(
188+ "aucune intégration ChowNow connue (data/sources.json et "
189+ "data/chownow-discovered.json vides) — connecteur dormant, "
190+ "en attente des découvertes du connecteur site-resto")
191+ out: list[Restaurant] = []
192+ for integ in integrations:
193+ cn_id = str(integ["key"])
194+ try:
195+ resto = self._fetch_restaurant(cn_id)
196+ except Exception as exc: # un resto ne bloque pas les autres
197+ print(f"[resto-ka] chownow: intégration {cn_id} ignorée : "
198+ f"{exc}", file=sys.stderr)
199+ continue
200+ if resto is not None:
201+ out.append(resto)
202+ print(f"[resto-ka] chownow: {len(integrations)} intégration(s) -> "
203+ f"{len(out)} resto(s) QC avec menu takeout")
204+ return out
added restoka/connectors/doordash.py +411 −0
@@ -0,0 +1,411 @@
1+# ==============================================================================
2+# Author: Simon-Pierre Boucher <contact@spboucher.ai>
3+# File: restoka/connectors/doordash.py
4+# Desc: Connecteur d'ENRICHISSEMENT DoorDash (menus livraison + notes) —
5+# même patron conservateur qu'ubereats.py. Découverte par le sitemap
6+# public QC (cdn.doordash.com/sitemaps/…/sitemap-doordash-qc-stores.xml,
7+# ~11 700 magasins, index local data/doordash-stores.json, 30 j).
8+# Chaque page magasin embarque du JSON-LD schema.org : un bloc
9+# `Restaurant` (adresse, GPS, cuisines, priceRange, aggregateRating —
10+# PAS de téléphone) et un bloc `Menu` (hasMenuSection imbriqué ->
11+# hasMenuItem {name, description, offers.price "$X.XX"}).
12+# Croisement CONSERVATEUR avec un resto EXISTANT seulement (GPS <120 m
13+# + nom similaire, OU numéro civique + ville + nom similaire) : n'émet
14+# AUCUNE fiche. Menu -> table menus en price_context `delivery` (prix
15+# majorés ~25-30 %, CLAUDE.md §6.2), note -> details.doordash.
16+#
17+# Mode d'extraction : Scrapfly ASP (anti-bot ; les gros contenus
18+# reviennent en « large_object » à re-télécharger), budget
19+# DOORDASH_BUDGET (défaut 40) pages/cycle, verdict par magasin en
20+# cache, re-visite 30 j. Conformité §15 : extraction minimale, usage
21+# d'appoint, prix toujours étiquetés `delivery`, lien source conservé.
22+# ==============================================================================
23+from __future__ import annotations
24+
25+import datetime
26+import json
27+import os
28+import re
29+import sys
30+import time
31+import urllib.parse
32+from pathlib import Path
33+
34+import requests
35+
36+from ..inspections import _CIVIC_RE, _name_similar, norm_name
37+from ..regions import strip_accents
38+from ..schema import Restaurant
39+from .base import BaseConnector, SkipSource
40+
41+SITEMAP_QC = ("https://cdn.doordash.com/sitemaps/sitemaps/"
42+ "sitemap-doordash-qc-stores.xml")
43+INDEX_PATH = Path(__file__).resolve().parents[2] / "data" / "doordash-stores.json"
44+INDEX_REFRESH_DAYS = 30
45+MAX_BUDGET = int(os.environ.get("DOORDASH_BUDGET", "40")) # pages/cycle
46+REFRESH_DAYS = 30
47+MAX_CANDIDATES = 3
48+MAX_CONSECUTIVE_FAILURES = 3
49+MAX_GPS_M = 120.0
50+
51+# /en-CA/store/<slug>-<id>/ — les /convenience/store/ (épiceries) sont exclus
52+_STORE_URL_RE = re.compile(r"doordash\.com/en-CA/store/([^/]+?)-(\d+)/?$")
53+
54+_LDJSON_RE = re.compile(
55+ r'<script type="application/ld\+json"[^>]*>(.*?)</script>', re.S)
56+_PRICE_RE = re.compile(r"(\d+(?:[.,]\d{1,2})?)")
57+
58+
59+def parse_ldjson(page_html: str) -> tuple[dict | None, dict | None]:
60+ """Extrait les blocs JSON-LD `Restaurant` et `Menu` d'une page magasin."""
61+ resto = menu = None
62+ for raw in _LDJSON_RE.findall(page_html):
63+ try:
64+ data = json.loads(raw)
65+ except ValueError:
66+ continue
67+ for obj in (data if isinstance(data, list) else [data]):
68+ if not isinstance(obj, dict):
69+ continue
70+ if obj.get("@type") == "Restaurant" and resto is None:
71+ resto = obj
72+ elif obj.get("@type") == "Menu" and menu is None:
73+ menu = obj
74+ return resto, menu
75+
76+
77+def _parse_price(offers) -> float | None:
78+ """« $13.20 » -> 13.20 ; None si absent/illisible."""
79+ raw = str((offers or {}).get("price") or "")
80+ m = _PRICE_RE.search(raw.replace(",", "."))
81+ if not m:
82+ return None
83+ price = float(m.group(1))
84+ return price if 0 < price <= 10000 else None
85+
86+
87+def _flatten_sections(node, out: list[dict]) -> None:
88+ """hasMenuSection est parfois une liste de listes — aplatir récursivement."""
89+ if isinstance(node, list):
90+ for x in node:
91+ _flatten_sections(x, out)
92+ elif isinstance(node, dict):
93+ out.append(node)
94+
95+
96+def build_menu(ld_menu: dict, captured_at: str) -> dict | None:
97+ """Menu standard Resto·Ka (CLAUDE.md §5.2) depuis le JSON-LD `Menu`."""
98+ raw_sections: list[dict] = []
99+ _flatten_sections((ld_menu or {}).get("hasMenuSection"), raw_sections)
100+ sections_out: list[dict] = []
101+ seen: set[str] = set()
102+ for sec in raw_sections:
103+ name = (sec.get("name") or "").strip()
104+ if not name or name in seen:
105+ continue
106+ items = []
107+ for it in sec.get("hasMenuItem") or []:
108+ title = (it.get("name") or "").strip()
109+ if not title:
110+ continue
111+ items.append({
112+ "name": title,
113+ "description": (it.get("description") or "").strip(),
114+ "price": _parse_price(it.get("offers")),
115+ })
116+ if items:
117+ seen.add(name)
118+ sections_out.append({"name": name, "items": items})
119+ if not sections_out:
120+ return None
121+ return {
122+ "price_context": "delivery",
123+ "price_source": "doordash",
124+ "currency": "CAD",
125+ "captured_at": captured_at,
126+ "sections": sections_out,
127+ }
128+
129+
130+def _haversine_m(lat1, lng1, lat2, lng2) -> float:
131+ import math
132+ r = 6371000.0
133+ p1, p2 = math.radians(lat1), math.radians(lat2)
134+ dp, dl = math.radians(lat2 - lat1), math.radians(lng2 - lng1)
135+ a = (math.sin(dp / 2) ** 2
136+ + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2)
137+ return 2 * r * math.asin(math.sqrt(a))
138+
139+
140+def _norm_city(city: str) -> str:
141+ return re.sub(r"[^a-z0-9]+", "", strip_accents((city or "").lower()))
142+
143+
144+def verify_store(row, ld: dict) -> str:
145+ """Vérifie qu'une page magasin DoorDash correspond bien au resto de la
146+ base — CONSERVATEUR (le JSON-LD n'a PAS de téléphone) : GPS <120 m + nom
147+ similaire, OU numéro civique + ville + nom similaire. Retourne le mode de
148+ croisement ('' = pas le même établissement)."""
149+ addr = ld.get("address") or {}
150+ geo = ld.get("geo") or {}
151+ street = (addr.get("streetAddress") or "").strip()
152+ addr_tokens = set(re.sub(r"[^a-z0-9]+", " ",
153+ strip_accents(street.lower())).split())
154+ similar = _name_similar(norm_name(row["name"]),
155+ norm_name(ld.get("name") or ""), addr_tokens)
156+ try:
157+ lat, lng = float(geo.get("latitude")), float(geo.get("longitude"))
158+ except (TypeError, ValueError):
159+ lat = lng = None
160+ if (lat is not None and row["lat"] is not None and row["lng"] is not None
161+ and _haversine_m(row["lat"], row["lng"], lat, lng) <= MAX_GPS_M
162+ and similar):
163+ return "gps+nom"
164+ civic_m = _CIVIC_RE.match(street)
165+ rcivic_m = _CIVIC_RE.match(row["address"] or "")
166+ if (civic_m and rcivic_m and civic_m.group(1) == rcivic_m.group(1)
167+ and _norm_city(addr.get("addressLocality") or "")
168+ == _norm_city(row["city"] or "") and similar):
169+ return "civique+ville+nom"
170+ return ""
171+
172+
173+def slugify(name: str) -> str:
174+ """Slug façon DoorDash : minuscules sans accents, tirets."""
175+ s = strip_accents((name or "").lower())
176+ s = re.sub(r"['’´`.]", "", s)
177+ s = re.sub(r"&", " ", s)
178+ s = re.sub(r"[^a-z0-9]+", "-", s).strip("-")
179+ return s
180+
181+
182+class DoorDashConnector(BaseConnector):
183+ source_id = "doordash"
184+ request_delay = 1.0
185+ timeout = 60
186+ use_detail_cache = False # verdicts gérés à la main (detail_cache)
187+ enrichment_only = True # n'émet aucune fiche (ingest.run)
188+
189+ # -- Scrapfly / large_object -------------------------------------------------
190+ def _content(self, url: str, **kw) -> tuple[int, str]:
191+ """Page via Scrapfly ASP ; résout les réponses « large_object »
192+ (contenu volumineux renvoyé comme URL à re-télécharger avec la clé)."""
193+ result = self.scrapfly(url, render_js=False, **kw)
194+ status = result.get("status_code") or 0
195+ content = result.get("content") or ""
196+ if content.startswith("https://api.scrapfly.io/scrape/large_object/"):
197+ resp = requests.get(content,
198+ params={"key": os.environ["SCRAPFLY_KEY"]},
199+ timeout=120)
200+ content = resp.text if resp.ok else ""
201+ return status, content
202+
203+ # -- index sitemap -----------------------------------------------------------
204+ def load_index(self) -> dict[str, str]:
205+ """Index slug -> URL magasin depuis le sitemap QC public, en cache
206+ local 30 jours (data/doordash-stores.json). Épiceries/dépanneurs
207+ (/convenience/store/) exclus."""
208+ if INDEX_PATH.exists():
209+ try:
210+ data = json.loads(INDEX_PATH.read_text(encoding="utf-8"))
211+ if time.time() - float(data.get("fetched_at") or 0) \
212+ < INDEX_REFRESH_DAYS * 86400:
213+ return data.get("stores") or {}
214+ except (ValueError, OSError):
215+ pass
216+ # le CDN sitemap répond en direct ; Scrapfly en secours (large_object)
217+ try:
218+ xml = self.get(SITEMAP_QC).text
219+ except Exception:
220+ _, xml = self._content(SITEMAP_QC)
221+ stores: dict[str, str] = {}
222+ for loc in re.findall(r"<loc>([^<]+)</loc>", xml):
223+ if "/convenience/" in loc:
224+ continue
225+ mm = _STORE_URL_RE.search(urllib.parse.unquote(loc))
226+ if mm:
227+ stores[f"{mm.group(1)}-{mm.group(2)}".lower()] = loc
228+ if not stores:
229+ raise RuntimeError("sitemap QC DoorDash vide (blocage ?)")
230+ INDEX_PATH.parent.mkdir(parents=True, exist_ok=True)
231+ INDEX_PATH.write_text(json.dumps(
232+ {"fetched_at": time.time(), "stores": stores},
233+ ensure_ascii=False), encoding="utf-8")
234+ print(f"[resto-ka] doordash: index sitemap rafraîchi — "
235+ f"{len(stores)} magasin(s) QC")
236+ return stores
237+
238+ @staticmethod
239+ def candidates_for(name: str, slugs: list[str]) -> list[str]:
240+ """Slugs DoorDash candidats pour un nom de resto (préfixe strict —
241+ le slug DoorDash embarque la ville et l'id : nom-ville-123456)."""
242+ base = slugify(name)
243+ if len(base) < 5:
244+ return []
245+ import bisect
246+ i = bisect.bisect_left(slugs, base)
247+ out = []
248+ while i < len(slugs) and len(out) < MAX_CANDIDATES:
249+ s = slugs[i]
250+ if s == base or s.startswith(base + "-"):
251+ out.append(s)
252+ i += 1
253+ else:
254+ break
255+ return out
256+
257+ # -- cycle -------------------------------------------------------------------
258+ def _now(self) -> str:
259+ return datetime.datetime.now(datetime.timezone.utc) \
260+ .strftime("%Y-%m-%dT%H:%M:%SZ")
261+
262+ def _fresh(self, stamp: str, now: float, stale_s: float) -> bool:
263+ try:
264+ ts = datetime.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ") \
265+ .replace(tzinfo=datetime.timezone.utc).timestamp()
266+ return ts > now - stale_s
267+ except (ValueError, TypeError):
268+ return False
269+
270+ def _details_payload(self, ld: dict, url: str, how: str) -> dict:
271+ rating = ld.get("aggregateRating") or {}
272+ return {
273+ "rating": rating.get("ratingValue"),
274+ "review_count": rating.get("reviewCount"),
275+ "price_range": ld.get("priceRange"),
276+ "cuisines": ld.get("servesCuisine"),
277+ "url": url.split("?")[0],
278+ "matched_by": how,
279+ "fetched_at": self._now(),
280+ }
281+
282+ def _probe_store(self, con, row, url: str) -> tuple[bool, bool]:
283+ """Visite une page magasin et l'attache au resto si c'est le même
284+ établissement. Retourne (matched, menu_added)."""
285+ from .. import db
286+ mm = _STORE_URL_RE.search(urllib.parse.unquote(url))
287+ store_id = mm.group(2) if mm else url
288+ cached = db.get_cached_detail(con, self.source_id, store_id,
289+ "verdict-v1")
290+ if cached is not None and cached.get("matched_uid") != row["uid"]:
291+ return False, False # déjà identifié comme un autre resto
292+ status, content = self._content(url)
293+ if status in (400, 404, 410, 451): # magasin retiré de DoorDash
294+ db.put_cached_detail(con, self.source_id, store_id,
295+ "verdict-v1", {"matched_uid": None,
296+ "gone": status})
297+ return False, False
298+ if status != 200:
299+ raise RuntimeError(f"HTTP {status}")
300+ ld_resto, ld_menu = parse_ldjson(content)
301+ if not ld_resto:
302+ if ld_menu or "application/ld+json" in content:
303+ # variante de page sans bloc Restaurant (constat de terrain) :
304+ # identité invérifiable -> miss prudent, PAS un blocage
305+ db.put_cached_detail(con, self.source_id, store_id,
306+ "verdict-v1", {"matched_uid": None,
307+ "no_restaurant_ld": True})
308+ return False, False
309+ raise RuntimeError("JSON-LD absent (page non rendue ?)")
310+ how = verify_store(row, ld_resto)
311+ if not how:
312+ db.put_cached_detail(con, self.source_id, store_id,
313+ "verdict-v1", {"matched_uid": None})
314+ return False, False
315+ db.put_cached_detail(con, self.source_id, store_id, "verdict-v1",
316+ {"matched_uid": row["uid"]})
317+ menu = build_menu(ld_menu or {}, self._now())
318+ menu_added = False
319+ if menu:
320+ # validation stricte du schéma menu (prix implausibles, etc.)
321+ Restaurant(source=self.source_id, external_id=store_id,
322+ name=row["name"], menu=menu)._validate_menu()
323+ db.upsert_menu(con, row["uid"], menu, time.time())
324+ menu_added = True
325+ db.merge_details(con, row["uid"],
326+ {"doordash": self._details_payload(ld_resto, url, how)})
327+ con.commit()
328+ return True, menu_added
329+
330+ def fetch(self) -> list[Restaurant]:
331+ if not os.environ.get("SCRAPFLY_KEY"):
332+ raise SkipSource("SCRAPFLY_KEY manquant (.env) — DoorDash est "
333+ "derrière un anti-bot, scraping direct impossible")
334+ from .. import db
335+ index = self.load_index()
336+ slugs = sorted(index.keys())
337+ con = db.connect()
338+ now = time.time()
339+ stale_s = REFRESH_DAYS * 86400.0
340+ budget = MAX_BUDGET
341+ matched = menus = misses = failures_row = 0
342+ rows = con.execute(
343+ "SELECT uid, name, address, city, postal_code, phone, lat, lng,"
344+ " details,"
345+ " EXISTS (SELECT 1 FROM menus m WHERE m.uid=restaurants.uid)"
346+ " AS has_menu"
347+ " FROM restaurants WHERE active=1 AND dup_of IS NULL AND name<>''"
348+ " AND ((lat IS NOT NULL AND lng IS NOT NULL)"
349+ " OR (address<>'' AND city<>''))"
350+ " ORDER BY has_menu ASC, lat IS NULL ASC, updated_at DESC"
351+ ).fetchall()
352+ for row in rows:
353+ if budget <= 0:
354+ break
355+ if failures_row >= MAX_CONSECUTIVE_FAILURES:
356+ print("[resto-ka] doordash: Scrapfly bloqué "
357+ f"{failures_row} fois de suite — arrêt du cycle",
358+ file=sys.stderr)
359+ break
360+ try:
361+ details = json.loads(row["details"] or "{}")
362+ except ValueError:
363+ details = {}
364+ dd = details.get("doordash") or {}
365+ if self._fresh(dd.get("fetched_at", ""), now, stale_s):
366+ continue # déjà frais (<30 j)
367+ probe = details.get("doordash_probe") or {}
368+ if self._fresh(probe.get("fetched_at", ""), now, stale_s):
369+ continue # échec récent : re-visite dans 30 j
370+ if dd.get("url"): # déjà croisé : rafraîchir directement
371+ urls = [dd["url"]]
372+ else:
373+ cand = self.candidates_for(row["name"], slugs)
374+ urls = [index[s] for s in cand if s in index]
375+ if not urls:
376+ continue # aucun candidat : pas de marqueur,
377+ # l'index du mois prochain peut changer
378+ found = False
379+ for url in urls[:MAX_CANDIDATES]:
380+ if budget <= 0:
381+ break
382+ budget -= 1
383+ try:
384+ ok, menu_added = self._probe_store(con, row, url)
385+ except Exception as exc:
386+ failures_row += 1
387+ print(f"[resto-ka] doordash: {row['uid']} erreur: {exc}",
388+ file=sys.stderr)
389+ continue
390+ failures_row = 0
391+ if ok:
392+ matched += 1
393+ menus += 1 if menu_added else 0
394+ found = True
395+ break
396+ if not found and not dd.get("url"):
397+ db.merge_details(con, row["uid"],
398+ {"doordash_probe":
399+ {"miss": "aucun magasin correspondant",
400+ "fetched_at": self._now()}})
401+ con.commit()
402+ misses += 1
403+ con.commit()
404+ con.close()
405+ self.enriched_count = matched
406+ self.enrich_message = (f"{matched} resto(s) croisés DoorDash "
407+ f"({menus} menu(s) livraison), {misses} sans "
408+ f"correspondance, budget restant "
409+ f"{max(budget, 0)} page(s)")
410+ print(f"[resto-ka] doordash: {self.enrich_message}")
411+ return []
added restoka/connectors/gloriafood.py +192 −0
@@ -0,0 +1,192 @@
1+# ==============================================================================
2+# Author: Simon-Pierre Boucher <contact@spboucher.ai>
3+# File: restoka/connectors/gloriafood.py
4+# Desc: Connecteur templatisé GloriaFood/Oracle (Palier 1 — widget de
5+# commande embarqué sur le site du resto -> PRIX RÉELS, contexte
6+# `takeout`). Le widget s'identifie par data-glf-cuid/data-glf-ruid
7+# dans le HTML du site ; le connecteur site-resto consigne ces ids
8+# dans data/gloriafood-discovered.json AVEC l'uid du resto porteur.
9+# Menu via l'API publique du widget :
10+# GET pos.globalfoodsoft.com/pos/menu
11+# (Authorization: <ruid>, Glf-Api-Version: 2, Accept: JSON)
12+# -> categories[].items[] {name, description, price, sizes[]}.
13+# ENRICHISSEMENT PUR : n'émet aucune fiche — le menu est posé sur le
14+# resto porteur (db.upsert_menu, price_context takeout) + traçabilité
15+# details.gloriafood. CONNECTEUR DORMANT tant qu'aucune intégration
16+# n'est découverte (SkipSource — pas un échec). L'endpoint refuse les
17+# requêtes sans Authorization (403) : chaque intégration est tentée
18+# défensivement, un refus n'interrompt pas le cycle.
19+#
20+# Base légale §15 : API du widget de commande du resto lui-même
21+# (source de première partie), extraction minimale, lien conservé.
22+# ==============================================================================
23+from __future__ import annotations
24+
25+import datetime
26+import json
27+import sys
28+import time
29+from pathlib import Path
30+
31+from ..schema import Restaurant
32+from .base import BaseConnector, SkipSource
33+
34+MENU_API = "https://pos.globalfoodsoft.com/pos/menu"
35+SOURCES_PATH = Path(__file__).resolve().parents[2] / "data" / "sources.json"
36+DISCOVERED = SOURCES_PATH.parent / "gloriafood-discovered.json"
37+REFRESH_DAYS = 14 # menus takeout : rafraîchissement §14
38+
39+
40+def build_menu(payload: dict, captured_at: str) -> dict | None:
41+ """Menu standard Resto·Ka (§5.2) depuis le payload GloriaFood — parseur
42+ TOLÉRANT (structure observée : categories[].items[], variantes `sizes`)."""
43+ if isinstance(payload, list): # payload directement en liste
44+ cats = payload
45+ else:
46+ cats = (payload or {}).get("categories")
47+ sections_out: list[dict] = []
48+ seen: set[str] = set()
49+ for cat in cats or []:
50+ if not isinstance(cat, dict):
51+ continue
52+ name = (cat.get("name") or "").strip()
53+ if not name or name in seen:
54+ continue
55+ items = []
56+ for it in cat.get("items") or []:
57+ if not isinstance(it, dict):
58+ continue
59+ title = (it.get("name") or "").strip()
60+ if not title:
61+ continue
62+ price = it.get("price")
63+ entry = {
64+ "name": title,
65+ "description": (it.get("description") or "").strip(),
66+ "price": round(float(price), 2)
67+ if isinstance(price, (int, float)) and price > 0 else None,
68+ }
69+ sizes = [s for s in it.get("sizes") or [] if isinstance(s, dict)
70+ and (s.get("name") or "").strip()]
71+ if sizes:
72+ base = entry["price"] or 0.0
73+ choices = []
74+ for s in sizes:
75+ sp = s.get("price")
76+ delta = (round(float(sp) - base, 2)
77+ if isinstance(sp, (int, float)) else 0.0)
78+ choices.append({"name": s["name"].strip(),
79+ "price_delta": delta})
80+ entry["options"] = [{"group": "Format", "required": True,
81+ "choices": choices}]
82+ items.append(entry)
83+ if items:
84+ seen.add(name)
85+ sections_out.append({"name": name, "items": items})
86+ if not sections_out:
87+ return None
88+ return {
89+ "price_context": "takeout",
90+ "price_source": "gloriafood",
91+ "currency": "CAD",
92+ "captured_at": captured_at,
93+ "sections": sections_out,
94+ }
95+
96+
97+class GloriaFoodConnector(BaseConnector):
98+ source_id = "gloriafood"
99+ request_delay = 1.0
100+ timeout = 30
101+ use_detail_cache = False # fraîcheur gérée via details.gloriafood
102+ enrichment_only = True # n'émet aucune fiche (ingest.run)
103+
104+ def _integrations(self) -> list[dict]:
105+ registry = json.loads(SOURCES_PATH.read_text(encoding="utf-8"))
106+ out: list[dict] = []
107+ for src in registry.get("sources", []):
108+ if src.get("id") == self.source_id:
109+ out.extend(src.get("integrations") or [])
110+ if DISCOVERED.exists():
111+ try:
112+ extra = json.loads(DISCOVERED.read_text(encoding="utf-8"))
113+ known = {i.get("key") for i in out}
114+ out.extend(i for i in extra.get("integrations", [])
115+ if i.get("key") not in known)
116+ except ValueError:
117+ pass
118+ return out
119+
120+ def _now(self) -> str:
121+ return datetime.datetime.now(datetime.timezone.utc) \
122+ .strftime("%Y-%m-%dT%H:%M:%SZ")
123+
124+ def _fetch_menu(self, ruid: str) -> dict | None:
125+ resp = self.get(MENU_API, headers={
126+ "Authorization": ruid,
127+ "Accept": "application/json",
128+ "Glf-Api-Version": "2",
129+ })
130+ return build_menu(resp.json(), self._now())
131+
132+ def fetch(self) -> list[Restaurant]:
133+ integrations = [i for i in self._integrations()
134+ if i.get("key") and i.get("status") != "inactif"]
135+ if not integrations:
136+ raise SkipSource(
137+ "aucune intégration GloriaFood découverte "
138+ "(data/gloriafood-discovered.json vide) — connecteur dormant, "
139+ "en attente des découvertes du connecteur site-resto")
140+ from .. import db
141+ con = db.connect()
142+ now = time.time()
143+ menus = errors = 0
144+ for integ in integrations:
145+ ruid, uid = str(integ["key"]), integ.get("uid") or ""
146+ if not uid:
147+ print(f"[resto-ka] gloriafood: {ruid} sans uid porteur — "
148+ "ignoré", file=sys.stderr)
149+ continue
150+ row = con.execute(
151+ "SELECT details FROM restaurants WHERE uid=? AND active=1",
152+ (uid,)).fetchone()
153+ if row is None:
154+ continue # resto porteur disparu
155+ try:
156+ details = json.loads(row["details"] or "{}")
157+ except ValueError:
158+ details = {}
159+ gf = details.get("gloriafood") or {}
160+ try:
161+ ts = datetime.datetime.strptime(
162+ gf.get("fetched_at", ""), "%Y-%m-%dT%H:%M:%SZ") \
163+ .replace(tzinfo=datetime.timezone.utc).timestamp()
164+ if ts > now - REFRESH_DAYS * 86400:
165+ continue # déjà frais
166+ except (ValueError, TypeError):
167+ pass
168+ try:
169+ menu = self._fetch_menu(ruid)
170+ except Exception as exc: # 403/refus : défensif, on continue
171+ errors += 1
172+ print(f"[resto-ka] gloriafood: {uid} ({ruid}) erreur: {exc}",
173+ file=sys.stderr)
174+ continue
175+ if not menu:
176+ continue
177+ # validation stricte du schéma menu avant stockage
178+ Restaurant(source=self.source_id, external_id=ruid,
179+ name=uid, menu=menu)._validate_menu()
180+ db.upsert_menu(con, uid, menu, time.time())
181+ db.merge_details(con, uid, {"gloriafood": {
182+ "ruid": ruid, "cuid": integ.get("cuid"),
183+ "fetched_at": self._now()}})
184+ con.commit()
185+ menus += 1
186+ con.close()
187+ self.enriched_count = menus
188+ self.enrich_message = (f"{menus} menu(s) takeout GloriaFood posés, "
189+ f"{errors} refus/erreur(s) sur "
190+ f"{len(integrations)} intégration(s)")
191+ print(f"[resto-ka] gloriafood: {self.enrich_message}")
192+ return []
added restoka/connectors/mtlalim.py +129 −0
@@ -0,0 +1,129 @@
1+# ==============================================================================
2+# Author: Simon-Pierre Boucher <contact@spboucher.ai>
3+# File: restoka/connectors/mtlalim.py
4+# Desc: Connecteur de DÉCOUVERTE massif « mtl-alim » (Palier 4) — registre
5+# officiel des établissements alimentaires inspectés par la Ville de
6+# Montréal (données ouvertes, ~17 000 établissements OUVERTS dont
7+# ~7 500 de restauration). Complète OSM sur l'île de Montréal : chaque
8+# resto détenteur d'un permis y figure, y compris ceux absents des
9+# cartes communautaires.
10+#
11+# Mode d'extraction : téléchargement direct du CSV
12+# (donnees.montreal.ca, licence CC-BY 4.0 — attribution « Ville de
13+# Montréal » sur la page Sources). Une seule requête par run.
14+# Base légale : données ouvertes, réutilisation permise avec
15+# attribution. Filtres : statut == « Ouvert » + whitelist des types
16+# food-service (restaurants, casse-croûtes, bars, boulangeries,
17+# traiteurs, camions-cuisine…) — épiceries, garderies, cafétérias
18+# institutionnelles et distributrices EXCLUES (hors périmètre §2).
19+#
20+# Contexte de prix produit : AUCUN (pas de menu — fiche de découverte).
21+# ==============================================================================
22+from __future__ import annotations
23+
24+import csv
25+import io
26+import sys
27+
28+from ..schema import Restaurant
29+from .base import BaseConnector
30+
31+CSV_URL = ("https://donnees.montreal.ca/dataset/"
32+ "c1d65779-d3cb-44e8-af0a-b9f2c5f7766d/resource/"
33+ "28a4957d-732e-48f9-8adb-0624867d9bb0/download/businesses.csv")
34+
35+# type d'établissement (registre Ville) -> establishment_type Resto·Ka.
36+# Whitelist STRICTE : tout type absent d'ici est ignoré (épiceries,
37+# distributrices, écoles, hôpitaux, entrepôts… — on ne mange pas « au resto »).
38+TYPE_MAP = {
39+ "Restaurant": "restaurant",
40+ "Restaurant service rapide": "fast-food",
41+ "Restaurant mets pour emporter": "fast-food",
42+ "Casse-croûte": "casse-croute",
43+ "Bar salon, taverne": "bar",
44+ "Brasserie": "bar",
45+ "Pâtisserie/Boulangerie": "boulangerie-patisserie",
46+ "Boulangerie": "boulangerie-patisserie",
47+ "Traiteur": "traiteur",
48+ "Camion-cuisine": "food-truck",
49+ "Bar laitier": "casse-croute",
50+ "Bar laitier saisonnier": "casse-croute",
51+}
52+
53+# ces types n'offrent pas de salle : takeout seulement
54+_TAKEOUT_ONLY = {"fast-food", "traiteur", "food-truck", "casse-croute",
55+ "boulangerie-patisserie"}
56+
57+# borne île de Montréal (lat/lng du CSV parfois 0/0 ou aberrants)
58+_LAT = (45.35, 45.75)
59+_LNG = (-74.05, -73.35)
60+
61+
62+class MtlAlimConnector(BaseConnector):
63+ source_id = "mtl-alim"
64+ request_delay = 1.0
65+ timeout = 120
66+ use_detail_cache = False # un seul CSV, pas de pages détail
67+
68+ def _rows(self) -> list[dict]:
69+ resp = self.get(CSV_URL)
70+ resp.encoding = "utf-8"
71+ return list(csv.DictReader(io.StringIO(resp.text)))
72+
73+ def _to_restaurant(self, row: dict) -> Restaurant | None:
74+ etype = TYPE_MAP.get((row.get("type") or "").strip())
75+ if not etype:
76+ return None
77+ name = (row.get("name") or "").strip()
78+ ext_id = (row.get("business_id") or "").strip()
79+ if not name or not ext_id:
80+ return None
81+ lat = lng = None
82+ try:
83+ lat, lng = float(row.get("latitude") or ""), float(row.get("longitude") or "")
84+ except ValueError:
85+ pass
86+ if lat is not None and not (_LAT[0] <= lat <= _LAT[1]
87+ and _LNG[0] <= lng <= _LNG[1]):
88+ lat = lng = None
89+ services = (["takeout"] if etype in _TAKEOUT_ONLY
90+ else ["salle", "takeout"])
91+ return Restaurant(
92+ source=self.source_id,
93+ external_id=ext_id,
94+ name=name,
95+ url=("https://donnees.montreal.ca/dataset/"
96+ "inspection-aliments-contrevenants"),
97+ establishment_type=etype,
98+ address=(row.get("address") or "").strip(),
99+ city=(row.get("city") or "").strip() or "Montréal",
100+ region="Montréal", # registre = île de Montréal
101+ lat=lat, lng=lng,
102+ services=services,
103+ languages=["fr"],
104+ menu=None, # découverte : pas de menu ici
105+ )
106+
107+ def fetch(self) -> list[Restaurant]:
108+ rows = self._rows()
109+ out: dict[str, Restaurant] = {}
110+ skipped_types: dict[str, int] = {}
111+ for row in rows:
112+ if (row.get("statut") or "").strip() != "Ouvert":
113+ continue
114+ resto = self._to_restaurant(row)
115+ if resto is None:
116+ t = (row.get("type") or "?").strip()
117+ skipped_types[t] = skipped_types.get(t, 0) + 1
118+ continue
119+ out[resto.uid] = resto # business_id unique -> dédup interne
120+ # garde-fou : un CSV tronqué/vidé ne doit pas déclencher des retraits
121+ # de masse — on signale le run comme partiel sous un plancher plausible
122+ if len(out) < 3000:
123+ self.partial_run = True
124+ print(f"[resto-ka] mtl-alim: seulement {len(out)} fiches "
125+ "(plancher 3000) — run marqué partiel, retraits suspendus",
126+ file=sys.stderr)
127+ print(f"[resto-ka] mtl-alim: {len(rows)} lignes CSV -> "
128+ f"{len(out)} établissements food-service ouverts")
129+ return list(out.values())
modified restoka/connectors/siteresto.py +96 −9
@@ -62,6 +62,21 @@ _UEAT_RE = re.compile(
62 r"(?:order\.ueat\.io/integration/|order-online\.ai.{0,200}?integration/)" 62 r"(?:order\.ueat\.io/integration/|order-online\.ai.{0,200}?integration/)"
63 r"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})", re.I) 63 r"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})", re.I)
64 _ORDER_ONLINE_RE = re.compile(r"https?://([a-z0-9-]+)\.order-online\.ai", re.I) 64 _ORDER_ONLINE_RE = re.compile(r"https?://([a-z0-9-]+)\.order-online\.ai", re.I)
65+
66+# autres plateformes de commande (CLAUDE.md §9 P1) détectables sur le site du
67+# resto — consignées dans data/<plateforme>-discovered.json, couvertes par
68+# leurs connecteurs dédiés (gloriafood.py, chownow.py, squareonline.py ;
69+# Clover : consigné comme candidat, pas de connecteur — SPA opaque)
70+_GLF_RUID_RE = re.compile(r'data-glf-ruid="([0-9a-f-]{30,40})"', re.I)
71+_GLF_CUID_RE = re.compile(r'data-glf-cuid="([0-9a-f-]{30,40})"', re.I)
72+_CHOWNOW_RE = re.compile(
73+ r"(?:direct\.chownow\.com/order/|ordering\.chownow\.com/order/"
74+ r"|api\.chownow\.com/api/restaurant/|chownow\.com/order/)(\d+)", re.I)
75+_SQUARE_SITE_RE = re.compile(r"https?://([a-z0-9-]+\.square\.site)", re.I)
76+_CLOVER_RE = re.compile(r"clover\.com/online-ordering/([a-z0-9-]+)", re.I)
77+# plateformes dont le connecteur dédié capte un MENU structuré : quand l'une
78+# d'elles est détectée, site-resto ne crawle pas le menu maison (comme UEAT)
79+_MENU_PLATFORMS = ("gloriafood", "chownow", "square")
65 _MENU_WORD_RE = re.compile(r"menu|carte|nos-plats|nos_plats|food", re.I) 80 _MENU_WORD_RE = re.compile(r"menu|carte|nos-plats|nos_plats|food", re.I)
66 _PRICE_HINT_RE = re.compile(r"\d{1,3}[.,]\d{2}\s*\$|\$\s*\d{1,3}[.,]\d{2}|\d{1,3}\s*\$") 81 _PRICE_HINT_RE = re.compile(r"\d{1,3}[.,]\d{2}\s*\$|\$\s*\d{1,3}[.,]\d{2}|\d{1,3}\s*\$")
67 82
@@ -309,6 +324,53 @@ class SiteRestoConnector(BaseConnector):
309 with _UEAT_FILE_LOCK: 324 with _UEAT_FILE_LOCK:
310 self._record_ueat_locked(guid, name, domain) 325 self._record_ueat_locked(guid, name, domain)
311 326
327+ def _discover_platforms(self, html: str) -> list[dict]:
328+ """Plateformes de commande tierces (hors UEAT) embarquées sur le site."""
329+ out: list[dict] = []
330+ m = _GLF_RUID_RE.search(html)
331+ if m:
332+ cuid = _GLF_CUID_RE.search(html)
333+ entry = {"name": "gloriafood", "key": m.group(1)}
334+ if cuid:
335+ entry["cuid"] = cuid.group(1)
336+ out.append(entry)
337+ m = _CHOWNOW_RE.search(html)
338+ if m:
339+ out.append({"name": "chownow", "key": m.group(1)})
340+ m = _SQUARE_SITE_RE.search(html)
341+ if m:
342+ out.append({"name": "square", "key": m.group(1).lower()})
343+ m = _CLOVER_RE.search(html)
344+ if m:
345+ out.append({"name": "clover", "key": m.group(1)})
346+ return out
347+
348+ def _record_platform(self, platform: dict, name: str, domain: str,
349+ uid: str) -> None:
350+ """Consigne une intégration découverte dans
351+ data/<plateforme>-discovered.json (avec l'uid du resto porteur pour
352+ les connecteurs d'enrichissement gloriafood/square)."""
353+ path = DATA_DIR / f"{platform['name']}-discovered.json"
354+ with _UEAT_FILE_LOCK:
355+ data = {"integrations": []}
356+ if path.exists():
357+ try:
358+ data = json.loads(path.read_text(encoding="utf-8"))
359+ except ValueError:
360+ pass
361+ if any(i.get("key") == platform["key"]
362+ for i in data.get("integrations", [])):
363+ return
364+ entry = {"key": platform["key"], "uid": uid,
365+ "label": f"{name} (découvert via {domain})"}
366+ if platform.get("cuid"):
367+ entry["cuid"] = platform["cuid"]
368+ data.setdefault("integrations", []).append(entry)
369+ path.write_text(json.dumps(data, ensure_ascii=False, indent=1),
370+ encoding="utf-8")
371+ print(f"[resto-ka] site-resto: plateforme {platform['name']} "
372+ f"découverte chez {name} ({platform['key']}) -> {path.name}")
373+
312 def _record_ueat_locked(self, guid: str, name: str, domain: str) -> None: 374 def _record_ueat_locked(self, guid: str, name: str, domain: str) -> None:
313 data = {"integrations": []} 375 data = {"integrations": []}
314 if UEAT_DISCOVERED.exists(): 376 if UEAT_DISCOVERED.exists():
@@ -335,9 +397,16 @@ class SiteRestoConnector(BaseConnector):
335 return None 397 return None
336 contact = self._extract_contact(home, website) 398 contact = self._extract_contact(home, website)
337 images = self._extract_images(home, website) 399 images = self._extract_images(home, website)
400+ platforms = self._discover_platforms(home)
338 guid = self._discover_ueat(home, website) 401 guid = self._discover_ueat(home, website)
339 if guid: 402 if guid:
340− return {"ueat": guid, "contact": contact, "images": images} 403+ return {"ueat": guid, "contact": contact, "images": images,
404+ "platforms": platforms}
405+ if any(p["name"] in _MENU_PLATFORMS for p in platforms):
406+ # une plateforme à menu structuré couvre ce resto : son connecteur
407+ # dédié prendra le relais (comme UEAT) — pas de crawl maison
408+ return {"contact": contact, "images": images,
409+ "platforms": platforms}
341 pages, pdfs = self._menu_links(home, website) 410 pages, pdfs = self._menu_links(home, website)
342 # 1) pages HTML de menu 411 # 1) pages HTML de menu
343 for page_url in pages: 412 for page_url in pages:
@@ -355,7 +424,8 @@ class SiteRestoConnector(BaseConnector):
355 menu = menullm.menu_from_text(text) 424 menu = menullm.menu_from_text(text)
356 if menu: 425 if menu:
357 return {"sections": menu["sections"], "url": page_url, 426 return {"sections": menu["sections"], "url": page_url,
358− "contact": contact, "images": images} 427+ "contact": contact, "images": images,
428+ "platforms": platforms}
359 # 2) PDF de menu 429 # 2) PDF de menu
360 for pdf_url in pdfs: 430 for pdf_url in pdfs:
361 pdf = self._get_pdf(pdf_url) 431 pdf = self._get_pdf(pdf_url)
@@ -363,17 +433,20 @@ class SiteRestoConnector(BaseConnector):
363 menu = menullm.menu_from_pdf(pdf) 433 menu = menullm.menu_from_pdf(pdf)
364 if menu: 434 if menu:
365 return {"sections": menu["sections"], "url": pdf_url, 435 return {"sections": menu["sections"], "url": pdf_url,
366− "contact": contact, "images": images} 436+ "contact": contact, "images": images,
437+ "platforms": platforms}
367 # 3) la page d'accueil elle-même contient parfois le menu 438 # 3) la page d'accueil elle-même contient parfois le menu
368 text = self._html_text(home) 439 text = self._html_text(home)
369 if len(_PRICE_HINT_RE.findall(text)) >= 8: 440 if len(_PRICE_HINT_RE.findall(text)) >= 8:
370 menu = menullm.menu_from_text(text) 441 menu = menullm.menu_from_text(text)
371 if menu: 442 if menu:
372 return {"sections": menu["sections"], "url": website, 443 return {"sections": menu["sections"], "url": website,
373− "contact": contact, "images": images} 444+ "contact": contact, "images": images,
374− if any(v for v in contact.values()) or images: 445+ "platforms": platforms}
375− # pas de menu, mais du contact et/ou des photos captés 446+ if any(v for v in contact.values()) or images or platforms:
376− return {"contact": contact, "images": images} 447+ # pas de menu, mais du contact/photos/plateformes captés
448+ return {"contact": contact, "images": images,
449+ "platforms": platforms}
377 return None 450 return None
378 451
379 # -- reconstruction des fiches déjà captées ------------------------------------------ 452 # -- reconstruction des fiches déjà captées ------------------------------------------
@@ -427,8 +500,8 @@ class SiteRestoConnector(BaseConnector):
427 500
428 firecrawl_budget = [MAX_FIRECRAWL_PER_RUN 501 firecrawl_budget = [MAX_FIRECRAWL_PER_RUN
429 if os.environ.get("FIRECRAWL_API_KEY") else 0] 502 if os.environ.get("FIRECRAWL_API_KEY") else 0]
430− stats = {"crawled": 0, "menus": 0, "ueat": 0, "vides": 0, 503+ stats = {"crawled": 0, "menus": 0, "ueat": 0, "plateformes": 0,
431− "contacts": 0, "photos": 0} 504+ "vides": 0, "contacts": 0, "photos": 0}
432 now = time.time() 505 now = time.time()
433 # enrichissements à appliquer APRÈS sync_source (la fiche site-resto 506 # enrichissements à appliquer APRÈS sync_source (la fiche site-resto
434 # n'existe en base qu'après l'upsert) — consommé par ingest.run 507 # n'existe en base qu'après l'upsert) — consommé par ingest.run
@@ -514,12 +587,26 @@ class SiteRestoConnector(BaseConnector):
514 (json.dumps(images), row["uid"])) 587 (json.dumps(images), row["uid"]))
515 con.commit() 588 con.commit()
516 stats["photos"] += 1 589 stats["photos"] += 1
590+ # plateformes de commande découvertes : consignées ici (fil
591+ # principal seulement — les workers n'écrivent ni BD ni fichiers)
592+ for p in (result or {}).get("platforms") or []:
593+ self._record_platform(p, row["name"], domain, row["uid"])
517 if result and result.get("ueat"): 594 if result and result.get("ueat"):
518 self._record_ueat(result["ueat"], row["name"], domain) 595 self._record_ueat(result["ueat"], row["name"], domain)
519 self._set_status(domain, {"status": "ueat", 596 self._set_status(domain, {"status": "ueat",
520 "guid": result["ueat"]}) 597 "guid": result["ueat"]})
521 stats["ueat"] += 1 598 stats["ueat"] += 1
522 continue 599 continue
600+ blocking = [p for p in (result or {}).get("platforms") or []
601+ if p["name"] in _MENU_PLATFORMS]
602+ if blocking and not (result and result.get("sections")):
603+ # menu servi par une plateforme connue : le connecteur
604+ # dédié (gloriafood/chownow/square) prendra le relais
605+ self._set_status(domain, {"status": "platform",
606+ "platforms":
607+ [p["name"] for p in blocking]})
608+ stats["plateformes"] += 1
609+ continue
523 if not result or not result.get("sections"): 610 if not result or not result.get("sections"):
524 self._set_status(domain, {"status": "no_menu"}) 611 self._set_status(domain, {"status": "no_menu"})
525 stats["vides"] += 1 612 stats["vides"] += 1
added restoka/connectors/skip.py +383 −0
@@ -0,0 +1,383 @@
1+# ==============================================================================
2+# Author: Simon-Pierre Boucher <contact@spboucher.ai>
3+# File: restoka/connectors/skip.py
4+# Desc: Connecteur d'ENRICHISSEMENT SkipTheDishes (menus livraison + score) —
5+# même patron conservateur qu'ubereats.py/doordash.py. Découverte par
6+# les sitemaps publics (sitemap2..7.xml, pages resto à la racine
7+# /<slug>, doublons /fr/ exclus ; index local data/skip-stores.json,
8+# 30 j). Chaque page resto embarque un état Next.js (__NEXT_DATA__)
9+# avec props.pageProps.partner (nom, rue, « Ville, PROV, POSTAL, CAN »,
10+# score Skip, téléphone dans les textes du disclaimerModal) et
11+# menuV2.categories[].menuItems[] (name, description, subtotal en
12+# cents). Croisement CONSERVATEUR avec un resto EXISTANT seulement
13+# (téléphone identique OU code postal + numéro civique identiques ;
14+# province ≠ QC rejetée d'emblée) : n'émet AUCUNE fiche. Menu -> table
15+# menus en price_context `delivery` (prix majorés ~25-30 %, §6.2),
16+# score -> details.skip.
17+#
18+# Mode d'extraction : Scrapfly ASP (anti-bot ; gros contenus résolus
19+# via « large_object »), budget SKIP_BUDGET (défaut 40) pages/cycle,
20+# verdict par resto en cache, re-visite 30 j. Conformité §15 :
21+# extraction minimale, usage d'appoint, prix étiquetés `delivery`.
22+# ==============================================================================
23+from __future__ import annotations
24+
25+import datetime
26+import json
27+import os
28+import re
29+import sys
30+import time
31+from pathlib import Path
32+
33+import requests
34+
35+from ..inspections import _CIVIC_RE
36+from ..normalize import normalize_phone
37+from ..regions import strip_accents
38+from ..schema import Restaurant
39+from .base import BaseConnector, SkipSource
40+
41+BASE = "https://www.skipthedishes.com"
42+SITEMAPS = [f"{BASE}/sitemap{i}.xml" for i in range(2, 8)]
43+INDEX_PATH = Path(__file__).resolve().parents[2] / "data" / "skip-stores.json"
44+INDEX_REFRESH_DAYS = 30
45+MAX_BUDGET = int(os.environ.get("SKIP_BUDGET", "40")) # pages/cycle
46+REFRESH_DAYS = 30
47+MAX_CANDIDATES = 3
48+MAX_CONSECUTIVE_FAILURES = 3
49+
50+_NEXT_DATA_RE = re.compile(
51+ r'<script id="__NEXT_DATA__" type="application/json"[^>]*>(.*?)</script>',
52+ re.S)
53+# pages resto = un seul segment à la racine ; /fr/... = doublon francophone
54+_STORE_URL_RE = re.compile(r"skipthedishes\.com/([a-z0-9-]+)/?$")
55+# « Windsor, ON, N9E 1S3, CAN »
56+_LOCDETAILS_RE = re.compile(
57+ r"^\s*(.+?),\s*([A-Z]{2}),\s*([A-Z]\d[A-Z]\s?\d[A-Z]\d),", re.I)
58+
59+
60+def decode_next_data(page_html: str) -> dict | None:
61+ """Décode le bloc __NEXT_DATA__ (JSON brut) d'une page Skip."""
62+ m = _NEXT_DATA_RE.search(page_html)
63+ if not m:
64+ return None
65+ try:
66+ return json.loads(m.group(1))
67+ except ValueError:
68+ return None
69+
70+
71+def partner_payload(state: dict) -> tuple[dict | None, dict | None]:
72+ """(partner, menuV2) depuis props.pageProps."""
73+ pp = ((state or {}).get("props") or {}).get("pageProps") or {}
74+ partner = pp.get("partner")
75+ menu = pp.get("menuV2") or (partner or {}).get("menuV2")
76+ return (partner if isinstance(partner, dict) else None,
77+ menu if isinstance(menu, dict) else None)
78+
79+
80+def partner_phone(partner: dict) -> str:
81+ """Téléphone du resto, enfoui dans les textes du disclaimerModal."""
82+ dm = (partner or {}).get("disclaimerModal") or {}
83+ for d in dm.get("disclaimers") or []:
84+ phone = normalize_phone(d.get("text") or "")
85+ if phone:
86+ return phone
87+ return ""
88+
89+
90+def parse_location_details(details: str) -> tuple[str, str, str]:
91+ """« Windsor, ON, N9E 1S3, CAN » -> (ville, province, code postal)."""
92+ m = _LOCDETAILS_RE.match(details or "")
93+ if not m:
94+ return "", "", ""
95+ return (m.group(1).strip(), m.group(2).upper(),
96+ m.group(3).replace(" ", "").upper())
97+
98+
99+def build_menu(menu_v2: dict, captured_at: str) -> dict | None:
100+ """Menu standard Resto·Ka (CLAUDE.md §5.2) depuis menuV2.categories.
101+ Prix Skip en cents (`subtotal`) -> dollars, contexte `delivery`."""
102+ sections_out: list[dict] = []
103+ seen: set[str] = set()
104+ for cat in (menu_v2 or {}).get("categories") or []:
105+ name = (cat.get("name") or "").strip()
106+ if not name or name in seen:
107+ continue
108+ items = []
109+ for it in cat.get("menuItems") or []:
110+ title = (it.get("name") or "").strip()
111+ if not title:
112+ continue
113+ sub = it.get("subtotal")
114+ items.append({
115+ "name": title,
116+ "description": (it.get("description") or "").strip(),
117+ "price": round(sub / 100.0, 2)
118+ if isinstance(sub, (int, float)) and sub > 0 else None,
119+ })
120+ if items:
121+ seen.add(name)
122+ sections_out.append({"name": name, "items": items})
123+ if not sections_out:
124+ return None
125+ return {
126+ "price_context": "delivery",
127+ "price_source": "skip",
128+ "currency": "CAD",
129+ "captured_at": captured_at,
130+ "sections": sections_out,
131+ }
132+
133+
134+def verify_partner(row, partner: dict) -> str:
135+ """Vérifie qu'une page Skip correspond bien au resto de la base —
136+ CONSERVATEUR : province QC obligatoire, puis téléphone identique OU
137+ code postal + numéro civique identiques. Retourne le mode de croisement
138+ ('' = pas le même établissement)."""
139+ _, prov, postal = parse_location_details(
140+ partner.get("locationDetails") or "")
141+ if prov and prov != "QC":
142+ return ""
143+ phone = partner_phone(partner)
144+ if phone and normalize_phone(row["phone"]) == phone:
145+ return "telephone"
146+ rpostal = (row["postal_code"] or "").replace(" ", "").upper()
147+ civic_m = _CIVIC_RE.match(partner.get("location") or "")
148+ rcivic_m = _CIVIC_RE.match(row["address"] or "")
149+ if postal and rpostal == postal and civic_m and rcivic_m \
150+ and civic_m.group(1) == rcivic_m.group(1):
151+ return "postal+civique"
152+ return ""
153+
154+
155+def slugify(name: str) -> str:
156+ """Slug façon Skip (cleanUrl) : minuscules sans accents, tirets."""
157+ s = strip_accents((name or "").lower())
158+ s = re.sub(r"['’´`.]", "", s)
159+ s = re.sub(r"&", " ", s)
160+ s = re.sub(r"[^a-z0-9]+", "-", s).strip("-")
161+ return s
162+
163+
164+class SkipConnector(BaseConnector):
165+ source_id = "skip"
166+ request_delay = 1.0
167+ timeout = 60
168+ use_detail_cache = False # verdicts gérés à la main (detail_cache)
169+ enrichment_only = True # n'émet aucune fiche (ingest.run)
170+
171+ # -- Scrapfly / large_object -------------------------------------------------
172+ def _content(self, url: str, **kw) -> tuple[int, str]:
173+ result = self.scrapfly(url, render_js=False, **kw)
174+ status = result.get("status_code") or 0
175+ content = result.get("content") or ""
176+ if content.startswith("https://api.scrapfly.io/scrape/large_object/"):
177+ resp = requests.get(content,
178+ params={"key": os.environ["SCRAPFLY_KEY"]},
179+ timeout=120)
180+ content = resp.text if resp.ok else ""
181+ return status, content
182+
183+ # -- index sitemap -----------------------------------------------------------
184+ def load_index(self) -> dict[str, str]:
185+ """Index slug -> URL resto depuis sitemap2..7 (pages à la racine),
186+ en cache local 30 jours (data/skip-stores.json). Les /fr/… (doublons)
187+ et sitemap1 (pages villes/cuisines) sont exclus."""
188+ if INDEX_PATH.exists():
189+ try:
190+ data = json.loads(INDEX_PATH.read_text(encoding="utf-8"))
191+ if time.time() - float(data.get("fetched_at") or 0) \
192+ < INDEX_REFRESH_DAYS * 86400:
193+ return data.get("stores") or {}
194+ except (ValueError, OSError):
195+ pass
196+ stores: dict[str, str] = {}
197+ for sm_url in SITEMAPS:
198+ _, xml = self._content(sm_url)
199+ for loc in re.findall(r"<loc>([^<]+)</loc>", xml):
200+ mm = _STORE_URL_RE.search(loc)
201+ if mm:
202+ stores.setdefault(mm.group(1).lower(), loc)
203+ if not stores:
204+ raise RuntimeError("sitemaps Skip vides (blocage ?)")
205+ INDEX_PATH.parent.mkdir(parents=True, exist_ok=True)
206+ INDEX_PATH.write_text(json.dumps(
207+ {"fetched_at": time.time(), "stores": stores},
208+ ensure_ascii=False), encoding="utf-8")
209+ print(f"[resto-ka] skip: index sitemap rafraîchi — "
210+ f"{len(stores)} resto(s) Canada")
211+ return stores
212+
213+ @staticmethod
214+ def candidates_for(name: str, slugs: list[str]) -> list[str]:
215+ """Slugs Skip candidats pour un nom de resto (préfixe strict — le
216+ slug Skip est le nom seul, parfois suffixé de la rue/ville)."""
217+ base = slugify(name)
218+ if len(base) < 5:
219+ return []
220+ import bisect
221+ i = bisect.bisect_left(slugs, base)
222+ out = []
223+ while i < len(slugs) and len(out) < MAX_CANDIDATES:
224+ s = slugs[i]
225+ if s == base or s.startswith(base + "-"):
226+ out.append(s)
227+ i += 1
228+ else:
229+ break
230+ return out
231+
232+ # -- cycle -------------------------------------------------------------------
233+ def _now(self) -> str:
234+ return datetime.datetime.now(datetime.timezone.utc) \
235+ .strftime("%Y-%m-%dT%H:%M:%SZ")
236+
237+ def _fresh(self, stamp: str, now: float, stale_s: float) -> bool:
238+ try:
239+ ts = datetime.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ") \
240+ .replace(tzinfo=datetime.timezone.utc).timestamp()
241+ return ts > now - stale_s
242+ except (ValueError, TypeError):
243+ return False
244+
245+ def _details_payload(self, partner: dict, url: str, how: str) -> dict:
246+ score = partner.get("score") or {}
247+ return {
248+ "score": score.get("formatted") or score.get("value"),
249+ "url": url.split("?")[0],
250+ "partner_id": partner.get("id"),
251+ "matched_by": how,
252+ "fetched_at": self._now(),
253+ }
254+
255+ def _probe_store(self, con, row, url: str) -> tuple[bool, bool]:
256+ """Visite une page resto Skip et l'attache au resto si c'est le même
257+ établissement. Retourne (matched, menu_added)."""
258+ from .. import db
259+ mm = _STORE_URL_RE.search(url)
260+ slug = mm.group(1) if mm else url
261+ cached = db.get_cached_detail(con, self.source_id, slug, "verdict-v1")
262+ if cached is not None and cached.get("matched_uid") != row["uid"]:
263+ return False, False # déjà identifié comme un autre resto
264+ status, content = self._content(url)
265+ if status in (400, 404, 410, 451): # resto retiré de Skip
266+ db.put_cached_detail(con, self.source_id, slug, "verdict-v1",
267+ {"matched_uid": None, "gone": status})
268+ return False, False
269+ if status != 200:
270+ raise RuntimeError(f"HTTP {status}")
271+ state = decode_next_data(content)
272+ if not state:
273+ raise RuntimeError("__NEXT_DATA__ absent (page non rendue ?)")
274+ partner, menu_v2 = partner_payload(state)
275+ if not partner:
276+ # page rendue mais sans fiche partner (page de groupe/chaîne) :
277+ # identité invérifiable -> miss prudent, PAS un blocage
278+ db.put_cached_detail(con, self.source_id, slug, "verdict-v1",
279+ {"matched_uid": None, "no_partner": True})
280+ return False, False
281+ how = verify_partner(row, partner)
282+ if not how:
283+ db.put_cached_detail(con, self.source_id, slug, "verdict-v1",
284+ {"matched_uid": None})
285+ return False, False
286+ db.put_cached_detail(con, self.source_id, slug, "verdict-v1",
287+ {"matched_uid": row["uid"]})
288+ menu = build_menu(menu_v2 or {}, self._now())
289+ menu_added = False
290+ if menu:
291+ # validation stricte du schéma menu (prix implausibles, etc.)
292+ Restaurant(source=self.source_id, external_id=slug,
293+ name=row["name"], menu=menu)._validate_menu()
294+ db.upsert_menu(con, row["uid"], menu, time.time())
295+ menu_added = True
296+ db.merge_details(con, row["uid"],
297+ {"skip": self._details_payload(partner, url, how)})
298+ con.commit()
299+ return True, menu_added
300+
301+ def fetch(self) -> list[Restaurant]:
302+ if not os.environ.get("SCRAPFLY_KEY"):
303+ raise SkipSource("SCRAPFLY_KEY manquant (.env) — SkipTheDishes "
304+ "est derrière un anti-bot, scraping direct "
305+ "impossible")
306+ from .. import db
307+ index = self.load_index()
308+ slugs = sorted(index.keys())
309+ con = db.connect()
310+ now = time.time()
311+ stale_s = REFRESH_DAYS * 86400.0
312+ budget = MAX_BUDGET
313+ matched = menus = misses = failures_row = 0
314+ # Skip n'expose ni GPS ni téléphone en clair : le croisement passe par
315+ # le téléphone (disclaimer) ou postal+civique -> exiger ces champs
316+ rows = con.execute(
317+ "SELECT uid, name, address, city, postal_code, phone, lat, lng,"
318+ " details,"
319+ " EXISTS (SELECT 1 FROM menus m WHERE m.uid=restaurants.uid)"
320+ " AS has_menu"
321+ " FROM restaurants WHERE active=1 AND dup_of IS NULL AND name<>''"
322+ " AND (phone<>'' OR (postal_code<>'' AND address<>''))"
323+ " ORDER BY has_menu ASC, phone<>'' DESC, updated_at DESC"
324+ ).fetchall()
325+ for row in rows:
326+ if budget <= 0:
327+ break
328+ if failures_row >= MAX_CONSECUTIVE_FAILURES:
329+ print("[resto-ka] skip: Scrapfly bloqué "
330+ f"{failures_row} fois de suite — arrêt du cycle",
331+ file=sys.stderr)
332+ break
333+ try:
334+ details = json.loads(row["details"] or "{}")
335+ except ValueError:
336+ details = {}
337+ sk = details.get("skip") or {}
338+ if self._fresh(sk.get("fetched_at", ""), now, stale_s):
339+ continue # déjà frais (<30 j)
340+ probe = details.get("skip_probe") or {}
341+ if self._fresh(probe.get("fetched_at", ""), now, stale_s):
342+ continue # échec récent : re-visite dans 30 j
343+ if sk.get("url"): # déjà croisé : rafraîchir directement
344+ urls = [sk["url"]]
345+ else:
346+ cand = self.candidates_for(row["name"], slugs)
347+ urls = [index[s] for s in cand if s in index]
348+ if not urls:
349+ continue # aucun candidat : pas de marqueur
350+ found = False
351+ for url in urls[:MAX_CANDIDATES]:
352+ if budget <= 0:
353+ break
354+ budget -= 1
355+ try:
356+ ok, menu_added = self._probe_store(con, row, url)
357+ except Exception as exc:
358+ failures_row += 1
359+ print(f"[resto-ka] skip: {row['uid']} erreur: {exc}",
360+ file=sys.stderr)
361+ continue
362+ failures_row = 0
363+ if ok:
364+ matched += 1
365+ menus += 1 if menu_added else 0
366+ found = True
367+ break
368+ if not found and not sk.get("url"):
369+ db.merge_details(con, row["uid"],
370+ {"skip_probe":
371+ {"miss": "aucun resto correspondant",
372+ "fetched_at": self._now()}})
373+ con.commit()
374+ misses += 1
375+ con.commit()
376+ con.close()
377+ self.enriched_count = matched
378+ self.enrich_message = (f"{matched} resto(s) croisés Skip "
379+ f"({menus} menu(s) livraison), {misses} sans "
380+ f"correspondance, budget restant "
381+ f"{max(budget, 0)} page(s)")
382+ print(f"[resto-ka] skip: {self.enrich_message}")
383+ return []
added restoka/connectors/squareonline.py +165 −0
Large diff (165 lines) is not loaded by default to keep the page fast.Load diff
added restoka/connectors/tastet.py +173 −0
Large diff (173 lines) is not loaded by default to keep the page fast.Load diff
modified restoka/dedup.py +2 −1
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_chownow.py +58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_doordash.py +99 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_gloriafood.py +46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_mtlalim.py +72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_siteresto_platforms.py +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_skip.py +86 −0
Large diff (86 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_tastet.py +56 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere