SPB Git
spboucher.ai

Repository Public

spblou-ka

Commit

Vague 2 : colonnes bedrooms/bathrooms, perf fb_marketplace/logisquebec, connecteurs Kangalou + Rentals.ca, visites virtuelles

- Schéma : colonnes bedrooms/bathrooms REAL (additives) dérivées dans
  finalize() — details explicites > type d'unité (n½ -> n-2, convention QC)
  > extraction texte ; backfill one-shot : 96,7 % bedrooms / 51 % bathrooms
  sur le parc actif.
- fb_marketplace : recherche et détails Scrapfly EN PARALLÈLE (6 fils,
  mêmes requêtes, aucun volume perdu), TTL détail 10 -> 30 j, annonces
  louées filtrées avant enrichissement : 43-47 min -> 7,3 min/cycle.
- logisquebec : budget détail 1200 -> 400 (dimensionné TTL 7 j / ~10
  passes/j), fiches jamais vues servies en premier, délai 0,7 -> 0,5 s :
  19 min -> 7,6 min/cycle, parc complet (6 393 fiches).
- Kangalou (nouveau) : découverte en 1 requête via les pins de la carte
  (/fr/recherche/ajax/search-map/, loués/commerciaux exclus), fiches SSR
  (adresse, chambres/SDB, GPS, galerie, dispo, superficie) : 7 916 annonces.
- Rentals.ca (nouveau) : même GraphQL que Louer.ca mais inventaire ~+15 %,
  rejoué via Scrapfly ASP sans rendu JS (Cloudflare), détails PAR LOTS de
  40 via nodes(ids), villes/volumes plafonnés : 3 452 annonces (les
  recoupements Louer.ca sont résorbés par la dédup, déjà dans PORTALS).
- Visites virtuelles : details.virtual_tour — tours{} Matterport/YouTube
  du réseau Rentals.ca (louer_ca clé cache v2 + rentals_ca) + détection
  générique Matterport/iGUIDE/Klapty dans les fiches logisquebec,
  duproprio, kangalou. 1 107 annonces actives en ont déjà une.
- TtlDetailCache : peek()/put() pour la planification de lots parallèles.
- sources.json : +kangalou, +rentals_ca (275 sources, 212 connecteurs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed · Aug 18, 2026 parent 689cd7d
12 changed files +808 −114 Browse files
12 changed files
Unified Split
modified README.md +3 −3
@@ -15,7 +15,7 @@
15 15
16 ![Annonces](https://img.shields.io/badge/annonces_actives-22_900%2B-1c5c41?style=flat-square) 16 ![Annonces](https://img.shields.io/badge/annonces_actives-22_900%2B-1c5c41?style=flat-square)
17 ![Sources](https://img.shields.io/badge/sources_recens%C3%A9es-265-1c5c41?style=flat-square) 17 ![Sources](https://img.shields.io/badge/sources_recens%C3%A9es-265-1c5c41?style=flat-square)
18−![Connecteurs](https://img.shields.io/badge/connecteurs-205-1c5c41?style=flat-square) 18+![Connecteurs](https://img.shields.io/badge/connecteurs-212-1c5c41?style=flat-square)
19 ![Villes](https://img.shields.io/badge/villes-820-1c5c41?style=flat-square) 19 ![Villes](https://img.shields.io/badge/villes-820-1c5c41?style=flat-square)
20 ![Régions](https://img.shields.io/badge/r%C3%A9gions-11-1c5c41?style=flat-square) 20 ![Régions](https://img.shields.io/badge/r%C3%A9gions-11-1c5c41?style=flat-square)
21 ![Géocodées](https://img.shields.io/badge/g%C3%A9olocalis%C3%A9es-70%25-1c5c41?style=flat-square) 21 ![Géocodées](https://img.shields.io/badge/g%C3%A9olocalis%C3%A9es-70%25-1c5c41?style=flat-square)
@@ -30,7 +30,7 @@ Lou-Ka n'est pas une plateforme d'annonces : c'est un **index fidèle** et un **
30 30
31 > Les sites d'agences n'offrent pas de webhooks. Lou-Ka reproduit l'équivalent : **synchronisation périodique + hash de contenu** → ajouts, mises à jour et retraits détectés automatiquement. Une annonce qui disparaît du site source disparaît de Lou-Ka (et répond **`410 Gone`** aux moteurs de recherche). 31 > Les sites d'agences n'offrent pas de webhooks. Lou-Ka reproduit l'équivalent : **synchronisation périodique + hash de contenu** → ajouts, mises à jour et retraits détectés automatiquement. Une annonce qui disparaît du site source disparaît de Lou-Ka (et répond **`410 Gone`** aux moteurs de recherche).
32 32
33−**En chiffres** : **22 900+ annonces actives**, **265 sources recensées**, **205 connecteurs**, **820 villes**, **11 régions**, **~70 %** des annonces géolocalisées. 33+**En chiffres** : **22 900+ annonces actives**, **275 sources recensées**, **212 connecteurs**, **820 villes**, **11 régions**, **~70 %** des annonces géolocalisées.
34 34
35 ## Fonctionnalités 35 ## Fonctionnalités
36 36
@@ -92,7 +92,7 @@ lou-ka/
92 │ ├── marketstats.py # agrégats du marché (source unique stats/API/PDF) 92 │ ├── marketstats.py # agrégats du marché (source unique stats/API/PDF)
93 │ ├── pdfgen.py # fiches PDF + rapport de marché (QR code) 93 │ ├── pdfgen.py # fiches PDF + rapport de marché (QR code)
94 │ ├── fixtures.py # enregistrement/rejeu HTTP pour tests hors-ligne 94 │ ├── fixtures.py # enregistrement/rejeu HTTP pour tests hors-ligne
95−│ └── connectors/ # 205 connecteurs + base.py (auto-découverte) 95+│ └── connectors/ # 212 connecteurs + base.py (auto-découverte)
96 ├── frontend/ # React 18 + Vite + TypeScript (pages, composants, kamaps/) 96 ├── frontend/ # React 18 + Vite + TypeScript (pages, composants, kamaps/)
97 ├── data/ 97 ├── data/
98 │ ├── sources.json # registre des 265 sources (statut + raison si non connectable) 98 │ ├── sources.json # registre des 265 sources (statut + raison si non connectable)
modified data/sources.json +21 −1
@@ -2728,6 +2728,26 @@
2728 "connector": "fb_marketplace", 2728 "connector": "fb_marketplace",
2729 "status": "actif", 2729 "status": "actif",
2730 "region": "Province" 2730 "region": "Province"
2731+ },
2732+ {
2733+ "id": "kangalou",
2734+ "name": "Kangalou",
2735+ "url": "https://www.kangalou.com",
2736+ "listing_url": "https://www.kangalou.com/fr/recherche/",
2737+ "sectors": "Province entière — portail locatif géré (Solutions Kangalou), ~8 500 annonces actives (pins carte, loués exclus)",
2738+ "connector": "kangalou",
2739+ "status": "actif",
2740+ "region": "Province"
2741+ },
2742+ {
2743+ "id": "rentals_ca",
2744+ "name": "Rentals.ca (Québec)",
2745+ "url": "https://rentals.ca",
2746+ "listing_url": "https://rentals.ca/quebec-city",
2747+ "sectors": "Grandes villes QC — même plateforme GraphQL que Louer.ca mais inventaire de marque ~15 % plus large ; accès via Scrapfly ASP (Cloudflare), volumes plafonnés par ville",
2748+ "connector": "rentals_ca",
2749+ "status": "actif",
2750+ "region": "Province"
2731 } 2751 }
2732 ] 2752 ]
2733−} 2753+}
\ No newline at end of file
modified louka/connectors/_detailutil.py +42 −3
@@ -47,6 +47,25 @@ def ld_description(html: str) -> str:
47 return best 47 return best
48 48
49 49
50+# plateformes de visite virtuelle reconnues dans le HTML d'une fiche
51+_VTOUR_RE = re.compile(
52+ r'https?://(?:my\.)?(?:matterport\.com/show/[^"\'\s<>]+'
53+ r'|(?:www\.)?youriguide\.com/[^"\'\s<>]+'
54+ r'|(?:www\.)?klapty\.com/tour/[^"\'\s<>]+'
55+ r'|kuula\.co/(?:share|post)/[^"\'\s<>]+'
56+ r'|tours?\.[a-z0-9-]+\.(?:com|ca)/[^"\'\s<>]*(?:tour|visite)[^"\'\s<>]*'
57+ r'|(?:www\.)?realvision\.com/[^"\'\s<>]+)', re.I)
58+
59+
60+def virtual_tour(html: str) -> str | None:
61+ """Première URL de visite virtuelle (Matterport, iGUIDE, Klapty, Kuula…)
62+ trouvée dans le HTML d'une fiche. None si absente — jamais inventée."""
63+ m = _VTOUR_RE.search(html or "")
64+ if not m:
65+ return None
66+ return _html.unescape(m.group(0)).replace("\\/", "/").rstrip("\\").rstrip('&')
67+
68+
50 def gmaps_coords(html: str) -> tuple[float, float] | None: 69 def gmaps_coords(html: str) -> tuple[float, float] | None:
51 m = _COORD_RE.search(html) 70 m = _COORD_RE.search(html)
52 if not m: 71 if not m:
@@ -150,9 +169,10 @@ class TtlDetailCache:
150 self.key = key 169 self.key = key
151 self.fetch_html = fetch_html or (lambda u: connector.get(u).text) 170 self.fetch_html = fetch_html or (lambda u: connector.get(u).text)
152 171
153− def get(self, external_id: str, url: str, parse_fn) -> dict | None: 172+ def peek(self, external_id: str) -> tuple[dict | None, bool]:
173+ """(payload en cache, frais ?) SANS déclencher de requête ni toucher
174+ au budget — permet aux connecteurs de planifier des lots parallèles."""
154 import time as _time 175 import time as _time
155− from .. import db
156 row = self.con.execute( 176 row = self.con.execute(
157 "SELECT key, payload, fetched_at FROM detail_cache" 177 "SELECT key, payload, fetched_at FROM detail_cache"
158 " WHERE source=? AND external_id=?", 178 " WHERE source=? AND external_id=?",
@@ -165,7 +185,20 @@ class TtlDetailCache:
165 stale = None 185 stale = None
166 if (stale is not None and row["key"] == self.key 186 if (stale is not None and row["key"] == self.key
167 and _time.time() - (row["fetched_at"] or 0) < self.ttl): 187 and _time.time() - (row["fetched_at"] or 0) < self.ttl):
168− return stale # frais : aucun trafic 188+ return stale, True # frais : aucun trafic requis
189+ return stale, False
190+
191+ def put(self, external_id: str, payload: dict) -> None:
192+ """Écrit un payload obtenu hors de get() (ex. téléchargement en lot)."""
193+ from .. import db
194+ db.put_cached_detail(self.con, self.connector.source_id,
195+ str(external_id), self.key, payload)
196+
197+ def get(self, external_id: str, url: str, parse_fn) -> dict | None:
198+ from .. import db
199+ stale, fresh = self.peek(external_id)
200+ if fresh:
201+ return stale # frais : aucun trafic
169 if self.budget <= 0: 202 if self.budget <= 0:
170 return stale # périmé toléré / None si jamais vu 203 return stale # périmé toléré / None si jamais vu
171 self.budget -= 1 204 self.budget -= 1
@@ -217,3 +250,9 @@ def apply_detail(lst: Listing, d: dict) -> None:
217 "pets", "furnished"): 250 "pets", "furnished"):
218 if d.get(f) is not None and getattr(lst, f, None) is None: 251 if d.get(f) is not None and getattr(lst, f, None) is None:
219 setattr(lst, f, d[f]) 252 setattr(lst, f, d[f])
253+ from ..normalize import coerce_count
254+ for f in ("bedrooms", "bathrooms"):
255+ if d.get(f) is not None and getattr(lst, f, None) is None:
256+ setattr(lst, f, coerce_count(d[f]))
257+ if d.get("virtual_tour") and not lst.details.get("virtual_tour"):
258+ lst.details["virtual_tour"] = d["virtual_tour"]
modified louka/connectors/duproprio.py +4 −0
@@ -106,6 +106,10 @@ def _parse_fiche(html: str) -> dict:
106 f"{ym}/{lid}/{slug1600}-{pid}.jpg") 106 f"{ym}/{lid}/{slug1600}-{pid}.jpg")
107 if photos: 107 if photos:
108 out["images"] = photos 108 out["images"] = photos
109+
110+ vt = du.virtual_tour(html) # visite virtuelle (Matterport/iGUIDE…)
111+ if vt:
112+ out.setdefault("details", {})["virtual_tour"] = vt
109 return out 113 return out
110 114
111 115
modified louka/connectors/fb_marketplace.py +93 −49
@@ -18,6 +18,7 @@
18 # ----------------------------------------------------------------------------- 18 # -----------------------------------------------------------------------------
19 from __future__ import annotations 19 from __future__ import annotations
20 20
21+import concurrent.futures as _cf
21 import json 22 import json
22 import os 23 import os
23 import re 24 import re
@@ -44,8 +45,13 @@ PRICE_BANDS = [(0, 800), (800, 1100), (1100, 1400), (1400, 1700),
44 PRICE_MIN, PRICE_MAX = 300, 12000 45 PRICE_MIN, PRICE_MAX = 300, 12000
45 46
46 DETAIL_LIMIT = int(os.environ.get("LOUKA_FBMP_DETAIL_LIMIT", "120")) 47 DETAIL_LIMIT = int(os.environ.get("LOUKA_FBMP_DETAIL_LIMIT", "120"))
47−TTL_DAYS = float(os.environ.get("LOUKA_FBMP_TTL_DAYS", "10")) 48+# les annonces FB changent peu après publication : TTL long = le budget détail
49+# sert surtout aux NOUVELLES annonces plutôt qu'à re-visiter les connues
50+TTL_DAYS = float(os.environ.get("LOUKA_FBMP_TTL_DAYS", "30"))
48 CITY_LIMIT = os.environ.get("LOUKA_FBMP_CITIES", "") # ex. "montreal,laval" 51 CITY_LIMIT = os.environ.get("LOUKA_FBMP_CITIES", "") # ex. "montreal,laval"
52+# requêtes Scrapfly en parallèle (recherche ET détail) : le cycle passait 40+
53+# minutes en attente sérielle de rendus JS de ~20 s chacun
54+CONCURRENCY = int(os.environ.get("LOUKA_FBMP_CONCURRENCY", "6"))
49 55
50 56
51 def _walk_listings(html: str) -> dict: 57 def _walk_listings(html: str) -> dict:
@@ -145,17 +151,29 @@ class FacebookMarketplaceConnector(BaseConnector):
145 out["images"] = photos 151 out["images"] = photos
146 return out 152 return out
147 153
148− def _search(self, slug: str) -> dict: 154+ def _search_all(self, cities: dict) -> dict[str, tuple[dict, str]]:
149− found: dict = {} 155+ """Toutes les pages de recherche (ville × tranche de prix) EN PARALLÈLE.
150− for lo, hi in PRICE_BANDS: 156+
151− url = (f"{BASE}/{slug}/propertyrentals" 157+ Même nombre de requêtes Scrapfly qu'avant (aucune perte de volume) :
152− f"?minPrice={lo}&maxPrice={hi}&sortBy=creation_time_descend") 158+ seule l'attente des rendus JS (~20 s chacun) est recouverte.
153− try: 159+ """
154− html = self._scrape(url) 160+ tasks = []
155− except Exception: 161+ for slug, city in cities.items():
156− continue 162+ for lo, hi in PRICE_BANDS:
157− for lid, obj in _walk_listings(html).items(): 163+ url = (f"{BASE}/{slug}/propertyrentals"
158− found.setdefault(lid, obj) 164+ f"?minPrice={lo}&maxPrice={hi}&sortBy=creation_time_descend")
165+ tasks.append((url, city))
166+ found: dict[str, tuple[dict, str]] = {}
167+ with _cf.ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
168+ futs = {pool.submit(self._scrape, url): city for url, city in tasks}
169+ for fut in _cf.as_completed(futs):
170+ city = futs[fut]
171+ try:
172+ html = fut.result()
173+ except Exception:
174+ continue
175+ for lid, obj in _walk_listings(html).items():
176+ found.setdefault(lid, (obj, city))
159 return found 177 return found
160 178
161 def fetch(self) -> list[Listing]: 179 def fetch(self) -> list[Listing]:
@@ -166,46 +184,72 @@ class FacebookMarketplaceConnector(BaseConnector):
166 wanted = {c.strip() for c in CITY_LIMIT.split(",")} 184 wanted = {c.strip() for c in CITY_LIMIT.split(",")}
167 cities = {k: v for k, v in CITIES.items() if k in wanted} 185 cities = {k: v for k, v in CITIES.items() if k in wanted}
168 186
187+ found = self._search_all(cities)
188+
189+ # candidats retenus AVANT tout enrichissement : pas de requête détail
190+ # gaspillée sur les annonces sans prix ou déjà louées
191+ candidates: list[tuple[str, dict, str]] = []
192+ for lid, (obj, city) in found.items():
193+ if _price(obj) is None:
194+ continue # sans prix mensuel valable : ignorer
195+ if obj.get("is_sold") or obj.get("is_pending"):
196+ continue # déjà loué / en attente
197+ candidates.append((lid, obj, city))
198+
199+ # enrichissement détail : cache BD d'abord (peek), puis les fiches
200+ # jamais vues / périmées EN PARALLÈLE dans la limite du budget
169 cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS, 201 cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS,
170 key="v1", fetch_html=self._scrape) 202 key="v1", fetch_html=self._scrape)
171− out: dict[str, Listing] = {} 203+ details: dict[str, dict] = {}
204+ todo: list[str] = []
172 try: 205 try:
173− for slug, city in cities.items(): 206+ for lid, _obj, _city in candidates:
174− for lid, obj in self._search(slug).items(): 207+ payload, fresh = cache.peek(lid)
175− if lid in out: 208+ if payload is not None:
176− continue 209+ details[lid] = payload # frais OU périmé toléré
177− title = obj.get("marketplace_listing_title") or "" 210+ if not fresh and len(todo) < cache.budget:
178− price = _price(obj) 211+ todo.append(lid)
179− if price is None: 212+
180− continue # sans prix mensuel valable : ignorer 213+ def _fetch_one(lid: str) -> tuple[str, dict]:
181− loc = obj.get("location") or {} 214+ try:
182− coords = loc.get("reverse_geocode") or {} 215+ return lid, self._parse_detail(
183− obj_city = (coords.get("city") 216+ self._scrape(f"{BASE}/item/{lid}/"))
184− or (coords.get("city_page") or {}).get("display_name") 217+ except Exception:
185− or city) 218+ return lid, {}
186− photo = ((obj.get("primary_listing_photo") or {}) 219+ if todo:
187− .get("image") or {}).get("uri") or "" 220+ with _cf.ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
188− url = f"{BASE}/item/{lid}/" 221+ for lid, payload in pool.map(_fetch_one, todo):
189− 222+ if payload: # échec réseau : garder l'ancien
190− detail = cache.get(lid, url, self._parse_detail) or {} 223+ cache.put(lid, payload)
191− desc = detail.get("description") or "" 224+ details[lid] = payload
192− lst = Listing( 225+
193− source=self.source_id, 226+ out: dict[str, Listing] = {}
194− external_id=lid, 227+ for lid, obj, city in candidates:
195− url=url, 228+ title = obj.get("marketplace_listing_title") or ""
196− title=title[:200], 229+ price = _price(obj)
197− city=obj_city, 230+ loc = obj.get("location") or {}
198− unit_type=_unit_type(title, desc), 231+ coords = loc.get("reverse_geocode") or {}
199− price=price, 232+ obj_city = (coords.get("city")
200− price_label=f"{price:,.0f} $/mois".replace(",", " "), 233+ or (coords.get("city_page") or {}).get("display_name")
201− description=desc, 234+ or city)
202− images=detail.get("images") or ([photo] if photo else []), 235+ photo = ((obj.get("primary_listing_photo") or {})
203− lat=detail.get("lat"), 236+ .get("image") or {}).get("uri") or ""
204− lng=detail.get("lng"), 237+ detail = details.get(lid) or {}
205− ) 238+ desc = detail.get("description") or ""
206− if obj.get("is_sold") or obj.get("is_pending"): 239+ out[lid] = Listing(
207− continue # déjà loué / en attente 240+ source=self.source_id,
208− out[lid] = lst 241+ external_id=lid,
242+ url=f"{BASE}/item/{lid}/",
243+ title=title[:200],
244+ city=obj_city,
245+ unit_type=_unit_type(title, desc),
246+ price=price,
247+ price_label=f"{price:,.0f} $/mois".replace(",", " "),
248+ description=desc,
249+ images=detail.get("images") or ([photo] if photo else []),
250+ lat=detail.get("lat"),
251+ lng=detail.get("lng"),
252+ )
209 finally: 253 finally:
210 cache.close() 254 cache.close()
211 return list(out.values()) 255 return list(out.values())
added louka/connectors/kangalou.py +209 −0
@@ -0,0 +1,209 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/kangalou.py : Kangalou (kangalou.com) — portail locatif québécois
5+# géré (Solutions Kangalou, ~8 700 annonces province entière). Découverte en
6+# UNE requête : l'endpoint carte `POST /fr/recherche/ajax/search-map/`
7+# (pinsOnly=true) renvoie tous les pins JSON — id, URL de fiche, GPS, ville,
8+# type (ti/tn ; 10=Commercial et 11=Garage exclus), « n ½ » (an), prix (mc),
9+# photo et l'état loué (ir, exclu). La fiche /fr/annonce/<slug>/<id>/ (SSR,
10+# aucun anti-bot) complète : adresse civique, chambres/salles de bain, loyer,
11+# étage, superficie, disponibilité, description, inclusions/caractéristiques
12+# et galerie (slider--gallery, images glide) — via TtlDetailCache
13+# (budget/TTL). L'API de liste HTML (POST /fr/api/search) existe aussi mais
14+# pagine par 29 et masque l'état loué (CSS) : les pins sont la source fiable.
15+# -----------------------------------------------------------------------------
16+from __future__ import annotations
17+
18+import os
19+import re
20+
21+from ..schema import Listing, normalize_unit_type
22+from .base import BaseConnector
23+from . import _detailutil as du
24+
25+BASE = "https://www.kangalou.com"
26+PINS_API = f"{BASE}/fr/recherche/ajax/search-map/"
27+
28+DETAIL_LIMIT = int(os.environ.get("LOUKA_KANGALOU_DETAIL_LIMIT", "350"))
29+TTL_DAYS = float(os.environ.get("LOUKA_KANGALOU_TTL_DAYS", "7"))
30+MAX_PINS = int(os.environ.get("LOUKA_KANGALOU_MAX", "0")) # 0 = tout
31+
32+# type id -> type d'unité par défaut (le « n ½ » du pin prime quand présent)
33+_TYPES = {1: "", 2: "Condo", 3: "Maison", 4: "Chambre", 5: "Chalet",
34+ 6: "Studio", 7: "Loft"}
35+_EXCLUDED_TI = {10, 11} # Commercial, Garage : pas du résidentiel
36+
37+# fiche détail (SSR)
38+_GPS_RE = re.compile(r'maps/search/\?query=(-?\d+\.\d+)%2C(-?\d+\.\d+)')
39+_ADDR_RE = re.compile(r'buds--separator[^>]*u-text-size-sm">\s*<span>([^<]+)</span>', re.S)
40+# galerie : pleine taille dans gr-modal-gallery-src (les <img> sont lazyload) ;
41+# hébergement variable (images/glide kangalou OU s3 shareimmo/lws…)
42+_GALLERY_RE = re.compile(r'gr-modal-gallery-src="(https?://[^"]+)"')
43+_FLAT_FIELD = {
44+ "loyer": re.compile(r"Loyer:\s*\|\s*([\d\s ,]+\$\s*/\s*mois)"),
45+ "etage": re.compile(r"Étage:\s*\|\s*(\d{1,2})\b"),
46+ "sdb": re.compile(r"Nombre de salles? de bain:\s*\|\s*(\d+(?:[.,]5)?)"),
47+ "chambres": re.compile(r"Nombre de chambres:\s*\|\s*(\d+)"),
48+ "superficie": re.compile(r"Superficie:\s*\|\s*([\d\s ,.]+)\s*m\s*\|\s*2"),
49+ "dispo": re.compile(r"Disponibilité:\s*\|\s*([^|]{1,60})"),
50+}
51+_DESC_RE = re.compile(
52+ r'Description du logement à louer\s*</h2>.*?<div class="typography"[^>]*>(.*?)</div>',
53+ re.S)
54+
55+
56+def _parse_fiche(html: str) -> dict:
57+ """Champs riches d'une fiche Kangalou (SSR)."""
58+ out: dict = {}
59+ m = _GPS_RE.search(html)
60+ if m:
61+ out["lat"], out["lng"] = float(m.group(1)), float(m.group(2))
62+ m = _ADDR_RE.search(html)
63+ if m:
64+ out["address"] = m.group(1).strip()
65+
66+ flat = du.flatten(html)
67+ vals = {k: rx.search(flat) for k, rx in _FLAT_FIELD.items()}
68+ if vals["chambres"]:
69+ out["bedrooms"] = float(vals["chambres"].group(1))
70+ if vals["sdb"]:
71+ out["bathrooms"] = float(vals["sdb"].group(1).replace(",", "."))
72+ if vals["dispo"]:
73+ out["availability"] = vals["dispo"].group(1).strip(" .")
74+ if vals["etage"]:
75+ out["details"] = {"floor": int(vals["etage"].group(1))}
76+ if vals["superficie"]:
77+ try:
78+ m2 = float(vals["superficie"].group(1)
79+ .replace(" ", "").replace(" ", "").replace(",", "."))
80+ if 8 <= m2 <= 2000:
81+ out["area_sqft"] = round(m2 * 10.7639)
82+ except ValueError:
83+ pass
84+
85+ m = _DESC_RE.search(html)
86+ if m:
87+ txt = re.sub(r"<br\s*/?>", "\n", m.group(1))
88+ txt = re.sub(r"<[^>]+>", " ", txt)
89+ import html as _h
90+ out["description"] = re.sub(r"[ \t]+", " ", _h.unescape(txt)).strip()[:6000]
91+
92+ # commodités : sections Inclusions / Caractéristiques / Détail / Critères
93+ # (texte aplati « libellé | valeur ») — libellés courts jusqu'à la section
94+ # suivante, sans jamais interpréter
95+ amen: list[str] = []
96+ _sections = ("Inclusions", "Caractéristiques intérieures",
97+ "Caractéristiques extérieures", "Autres services",
98+ "Critères importants", "Détail", "Sécurité", "Buanderie")
99+ for section in _sections:
100+ m2m = re.search(r"\|\s*" + re.escape(section) + r"\s*\|", flat)
101+ if not m2m: # titre de SECTION seulement (« | Titre | »),
102+ continue # pas une mention dans la description
103+ i = m2m.end()
104+ seg = flat[i:i + 400]
105+ for item in seg.split("|"):
106+ item = item.strip()
107+ if not item or len(item) > 45:
108+ break
109+ if item in _sections: # section suivante : elle a sa passe
110+ break
111+ if re.match(r"^(Partager|Contacter|Description|Transport|À proximité"
112+ r"|Santé|Éducation|Informations)", item):
113+ break
114+ if item not in amen:
115+ amen.append(item)
116+ if amen:
117+ out["amenities"] = amen
118+
119+ # galerie (slider--gallery ; les vignettes asnav dupliquées sont exclues)
120+ ig = html.find("slider--gallery")
121+ if ig >= 0:
122+ seg = html[ig:]
123+ cut = seg.find("slider__asnav")
124+ if cut > 0:
125+ seg = seg[:cut]
126+ photos = []
127+ for u in _GALLERY_RE.findall(seg):
128+ if u not in photos:
129+ photos.append(u)
130+ if photos:
131+ out["images"] = photos
132+
133+ vt = du.virtual_tour(html)
134+ if vt:
135+ out.setdefault("details", {})["virtual_tour"] = vt
136+ return out
137+
138+
139+class KangalouConnector(BaseConnector):
140+ source_id = "kangalou"
141+ request_delay = 0.5
142+
143+ def _pins(self) -> list[dict]:
144+ resp = self.post(PINS_API, data={"pinsOnly": "true"},
145+ headers={"Accept": "application/json",
146+ "X-Requested-With": "XMLHttpRequest"})
147+ return (resp.json() or {}).get("pins") or []
148+
149+ @staticmethod
150+ def _unit_type(pin: dict) -> str:
151+ an = (pin.get("an") or "").strip() # « 4 ½ », « 9 ½ + »
152+ if an:
153+ ut = normalize_unit_type(an)
154+ if re.match(r"^\d½$|^6½\+$", ut or ""):
155+ return ut
156+ return _TYPES.get(pin.get("ti"), "")
157+
158+ def fetch(self) -> list[Listing]:
159+ pins = self._pins()
160+ if MAX_PINS:
161+ pins = pins[:MAX_PINS]
162+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS)
163+ out: list[Listing] = []
164+ try:
165+ for pin in pins:
166+ if pin.get("ir"):
167+ continue # déjà loué : pas une offre active
168+ if pin.get("ti") in _EXCLUDED_TI or not pin.get("id"):
169+ continue # commercial / garage / malformé
170+ url = pin.get("l") or ""
171+ if not url:
172+ continue
173+ eid = str(pin["id"])
174+ price = None
175+ try:
176+ price = float(str(pin.get("mc")).replace(",", ""))
177+ except (TypeError, ValueError):
178+ pass
179+ if price is not None and not (100 <= price <= 20000):
180+ price = None # affichage « sur demande » etc.
181+ lat = lng = None
182+ try:
183+ lat, lng = float(pin["lt"]), float(pin["lg"])
184+ except (KeyError, TypeError, ValueError):
185+ pass
186+ photo = pin.get("p") or ""
187+ lst = Listing(
188+ source=self.source_id,
189+ external_id=eid,
190+ url=url,
191+ title=(pin.get("n") or "").strip(),
192+ city=(pin.get("c") or "").strip(),
193+ unit_type=self._unit_type(pin),
194+ price=price,
195+ price_label=(f"{price:,.0f} $/mois".replace(",", " ")
196+ if price else ""),
197+ images=[photo] if photo else [],
198+ lat=lat,
199+ lng=lng,
200+ )
201+ payload = cache.get(eid, url, _parse_fiche)
202+ if payload:
203+ du.apply_detail(lst, payload)
204+ if payload.get("address"):
205+ lst.title = payload["address"]
206+ out.append(lst)
207+ finally:
208+ cache.close()
209+ return out
modified louka/connectors/logisquebec.py +20 −2
@@ -50,7 +50,12 @@ _LOC_RE = re.compile(r"<loc>(.*?)</loc>")
50 _PHOTO_RE = re.compile(r"https://i\.logisquebec\.com/i-a-louer/\d+/\d+/[^'\"]+") 50 _PHOTO_RE = re.compile(r"https://i\.logisquebec\.com/i-a-louer/\d+/\d+/[^'\"]+")
51 _DISPO_RE = re.compile(r"Disponible à partir de\s*:?\s*\|\s*([^|]{1,60})") 51 _DISPO_RE = re.compile(r"Disponible à partir de\s*:?\s*\|\s*([^|]{1,60})")
52 52
53−DETAIL_LIMIT = int(os.environ.get("LOUKA_LOGISQUEBEC_DETAIL_LIMIT", "1200")) 53+# Dimensionnement du budget détail : parc ~6 000 fiches, TTL 7 j, watch ~10
54+# passes/jour -> ~90 rafraîchissements/passe suffisent à tenir le TTL. 400
55+# laisse 4x de marge (poussées de nouvelles fiches) en ramenant le cycle de
56+# ~19 min à ~5 min. Les fiches JAMAIS VUES sont servies en premier (voir
57+# fetch) : aucune perte de volume, le parc se complète au même rythme.
58+DETAIL_LIMIT = int(os.environ.get("LOUKA_LOGISQUEBEC_DETAIL_LIMIT", "400"))
54 TTL_DAYS = float(os.environ.get("LOUKA_LOGISQUEBEC_TTL_DAYS", "7")) 59 TTL_DAYS = float(os.environ.get("LOUKA_LOGISQUEBEC_TTL_DAYS", "7"))
55 MAX_FICHES = int(os.environ.get("LOUKA_LOGISQUEBEC_MAX", "0")) # 0 = tout 60 MAX_FICHES = int(os.environ.get("LOUKA_LOGISQUEBEC_MAX", "0")) # 0 = tout
56 61
@@ -119,12 +124,16 @@ def _parse_fiche(html: str) -> dict:
119 amenities.append(label) 124 amenities.append(label)
120 if amenities: 125 if amenities:
121 out["amenities"] = amenities 126 out["amenities"] = amenities
127+
128+ vt = du.virtual_tour(html) # lien Matterport/iGUIDE… si la fiche en a un
129+ if vt:
130+ out["virtual_tour"] = vt
122 return out 131 return out
123 132
124 133
125 class LogisQuebecConnector(BaseConnector): 134 class LogisQuebecConnector(BaseConnector):
126 source_id = "logisquebec" 135 source_id = "logisquebec"
127− request_delay = 0.7 136+ request_delay = 0.5
128 137
129 # -- découverte ------------------------------------------------------------ 138 # -- découverte ------------------------------------------------------------
130 def _fiche_urls(self) -> list[tuple[str, str, str, str]]: 139 def _fiche_urls(self) -> list[tuple[str, str, str, str]]:
@@ -182,6 +191,8 @@ class LogisQuebecConnector(BaseConnector):
182 details["rooms"] = payload["rooms"] # numberOfRooms (JSON-LD) 191 details["rooms"] = payload["rooms"] # numberOfRooms (JSON-LD)
183 if court_terme: 192 if court_terme:
184 details["Location court terme"] = "oui" 193 details["Location court terme"] = "oui"
194+ if payload.get("virtual_tour"):
195+ details["virtual_tour"] = payload["virtual_tour"]
185 196
186 amenities = payload.get("amenities") or [] 197 amenities = payload.get("amenities") or []
187 lst = Listing( 198 lst = Listing(
@@ -215,6 +226,13 @@ class LogisQuebecConnector(BaseConnector):
215 if MAX_FICHES: 226 if MAX_FICHES:
216 fiches = fiches[:MAX_FICHES] 227 fiches = fiches[:MAX_FICHES]
217 cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS) 228 cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS)
229+ # priorité du budget aux fiches jamais visitées (tri stable) : les
230+ # nouveautés sont toujours captées au premier passage, le reste du
231+ # budget sert au rafraîchissement TTL
232+ seen = {r["external_id"] for r in cache.con.execute(
233+ "SELECT external_id FROM detail_cache WHERE source=?",
234+ (self.source_id,))}
235+ fiches.sort(key=lambda f: f[3] in seen)
218 out: list[Listing] = [] 236 out: list[Listing] = []
219 try: 237 try:
220 for url, type_slug, _ville, eid in fiches: 238 for url, type_slug, _ville, eid in fiches:
modified louka/connectors/louer_ca.py +123 −53
@@ -58,24 +58,31 @@ _AMENITY_FR = {
58 class LouerCaConnector(BaseConnector): 58 class LouerCaConnector(BaseConnector):
59 source_id = "louer_ca" 59 source_id = "louer_ca"
60 request_delay = 0.5 60 request_delay = 0.5
61+ # points d'entrée du réseau Rentals.ca — surchargés par rentals_ca.py
62+ home_url = HOME
63+ gql_url = GQL
64+ site = "https://louer.ca"
65+ fallback_key = FALLBACK_KEY
66+ page_size = PAGE_SIZE
67+ max_per_city = MAX_PER_CITY
61 68
62 def __init__(self) -> None: 69 def __init__(self) -> None:
63 super().__init__() 70 super().__init__()
64 self.session.headers.update({ 71 self.session.headers.update({
65 "Content-Type": "application/json", 72 "Content-Type": "application/json",
66− "Origin": "https://louer.ca", 73+ "Origin": self.site,
67− "Referer": "https://louer.ca/", 74+ "Referer": self.home_url,
68 }) 75 })
69 self._token = "" 76 self._token = ""
70 self._token_time = 0.0 77 self._token_time = 0.0
71− self._api_key = FALLBACK_KEY 78+ self._api_key = self.fallback_key
72 79
73 # -- auth ------------------------------------------------------------------ 80 # -- auth ------------------------------------------------------------------
74 def _ensure_token(self) -> None: 81 def _ensure_token(self) -> None:
75 if self._token and time.time() - self._token_time < 2700: # ~45 min 82 if self._token and time.time() - self._token_time < 2700: # ~45 min
76 return 83 return
77 try: 84 try:
78− home = self.get(HOME).text 85+ home = self.get(self.home_url).text
79 m = re.search(r'"rentalsGqlKey":\s*"([^"]+)"', home) 86 m = re.search(r'"rentalsGqlKey":\s*"([^"]+)"', home)
80 if m: 87 if m:
81 self._api_key = m.group(1) 88 self._api_key = m.group(1)
@@ -99,8 +106,8 @@ class LouerCaConnector(BaseConnector):
99 if auth: 106 if auth:
100 self._ensure_token() 107 self._ensure_token()
101 headers["Authorization"] = f"Bearer {self._token}" 108 headers["Authorization"] = f"Bearer {self._token}"
102− resp = self.post(GQL, data=json.dumps({"query": query, 109+ resp = self.post(self.gql_url, data=json.dumps({"query": query,
103− "variables": variables}), 110+ "variables": variables}),
104 headers=headers) 111 headers=headers)
105 payload = resp.json() 112 payload = resp.json()
106 if payload.get("errors"): 113 if payload.get("errors"):
@@ -108,8 +115,9 @@ class LouerCaConnector(BaseConnector):
108 if "AUTH" in msg.upper() and auth: # jeton expiré : on réessaie 115 if "AUTH" in msg.upper() and auth: # jeton expiré : on réessaie
109 self._token = "" 116 self._token = ""
110 self._ensure_token() 117 self._ensure_token()
111− resp = self.post(GQL, data=json.dumps({"query": query, 118+ resp = self.post(self.gql_url,
112− "variables": variables}), 119+ data=json.dumps({"query": query,
120+ "variables": variables}),
113 headers={"Authorization": f"Bearer {self._token}"}) 121 headers={"Authorization": f"Bearer {self._token}"})
114 payload = resp.json() 122 payload = resp.json()
115 return payload.get("data") or {} 123 return payload.get("data") or {}
@@ -141,7 +149,7 @@ class LouerCaConnector(BaseConnector):
141 "edges{node{" + self._LIST_FRAG + "}}}}}}") 149 "edges{node{" + self._LIST_FRAG + "}}}}}}")
142 out, after = [], None 150 out, after = [], None
143 while True: 151 while True:
144− d = self._gql(query, {"id": city_id, "first": PAGE_SIZE, 152+ d = self._gql(query, {"id": city_id, "first": self.page_size,
145 "after": after}) 153 "after": after})
146 rl = ((d.get("node") or {}).get("rentalListings")) or {} 154 rl = ((d.get("node") or {}).get("rentalListings")) or {}
147 edges = rl.get("edges") or [] 155 edges = rl.get("edges") or []
@@ -149,7 +157,7 @@ class LouerCaConnector(BaseConnector):
149 info = rl.get("pageInfo") or {} 157 info = rl.get("pageInfo") or {}
150 if not info.get("hasNextPage") or not edges: 158 if not info.get("hasNextPage") or not edges:
151 break 159 break
152− if MAX_PER_CITY and len(out) >= MAX_PER_CITY: 160+ if self.max_per_city and len(out) >= self.max_per_city:
153 break 161 break
154 after = info.get("endCursor") 162 after = info.get("endCursor")
155 return out 163 return out
@@ -157,8 +165,9 @@ class LouerCaConnector(BaseConnector):
157 _DETAIL_FRAG = ("id name path location " 165 _DETAIL_FRAG = ("id name path location "
158 "address{city{name regionCode} neighbourhood{name} " 166 "address{city{name regionCode} neighbourhood{name} "
159 "postalCode street} description{plain} " 167 "postalCode street} description{plain} "
160− "imagesCount images{scales} " 168+ "imagesCount images{scales} tours{name type refId} "
161− "floorPlans{beds baths rent size availability furnished}") 169+ "floorPlans{beds baths rent size availability furnished "
170+ "tours{name type refId}}")
162 171
163 def _detail(self, gid: str) -> dict: 172 def _detail(self, gid: str) -> dict:
164 d = self._gql("query($id:ID!){node(id:$id){... on RentalListing{" 173 d = self._gql("query($id:ID!){node(id:$id){... on RentalListing{"
@@ -208,6 +217,39 @@ class LouerCaConnector(BaseConnector):
208 imgs.append(best) 217 imgs.append(best)
209 return imgs 218 return imgs
210 219
220+ @staticmethod
221+ def _count(v) -> float | None:
222+ """beds/baths GraphQL -> float (0 = studio) ; None si inconnu."""
223+ try:
224+ f = float(v)
225+ except (TypeError, ValueError):
226+ return None
227+ return f if 0 <= f <= 20 else None
228+
229+ @staticmethod
230+ def _tour_url(tours) -> str | None:
231+ """URL de visite virtuelle depuis les `tours` GraphQL du réseau
232+ Rentals.ca : refId Matterport/YouTube (ou URL complète). La visite
233+ interactive prime sur la simple vidéo."""
234+ video = None
235+ for t in tours or []:
236+ ref = str(t.get("refId") or "").strip()
237+ typ = t.get("type") or ""
238+ if not ref:
239+ continue
240+ if ref.startswith("http"):
241+ url = ref
242+ elif "matterport" in typ:
243+ url = f"https://my.matterport.com/show/?m={ref}"
244+ elif "youtube" in typ:
245+ url = f"https://www.youtube.com/watch?v={ref}"
246+ else:
247+ continue
248+ if "interactive" in typ or "matterport" in typ:
249+ return url
250+ video = video or url
251+ return video
252+
211 @staticmethod 253 @staticmethod
212 def _numeric_id(gid: str) -> str: 254 def _numeric_id(gid: str) -> str:
213 """« cmVudGFsbGlzdGluZzoxMTMyMTE2 » -> « 1132116 » (rentallisting:1132116).""" 255 """« cmVudGFsbGlzdGluZzoxMTMyMTE2 » -> « 1132116 » (rentallisting:1132116)."""
@@ -221,12 +263,78 @@ class LouerCaConnector(BaseConnector):
221 m = re.search(r"(\d+)", gid) 263 m = re.search(r"(\d+)", gid)
222 return m.group(1) if m else gid 264 return m.group(1) if m else gid
223 265
266+ def _card_listings(self, card: dict, node: dict, today: str,
267+ default_city: str = "") -> list[Listing]:
268+ """Annonces Lou-Ka d'une carte liste + son nœud détail GraphQL
269+ (une par plan d'étage). Partagé avec rentals_ca.py."""
270+ out: list[Listing] = []
271+ gid = card.get("id")
272+ addr = node.get("address") or {}
273+ city = (addr.get("city") or {}).get("name") or default_city
274+ sector = (addr.get("neighbourhood") or {}).get("name") or ""
275+ loc = card.get("location") or []
276+ lng, lat = (loc + [None, None])[:2]
277+ base_desc = ((node.get("description") or {}).get("plain")
278+ or "")[:6000]
279+ images = self._images(node)
280+ amenities = self._amenities(card.get("amenities"))
281+ common = dict(
282+ source=self.source_id, url=f"{self.site}/{card.get('path','')}",
283+ address=addr.get("street") or "", sector=sector, city=city,
284+ description=base_desc, amenities=amenities, images=images,
285+ lat=lat, lng=lng,
286+ )
287+ base_details: dict = {}
288+ if addr.get("postalCode"):
289+ base_details["Code postal"] = addr["postalCode"]
290+ tour = self._tour_url(node.get("tours"))
291+ if tour:
292+ base_details["virtual_tour"] = tour
293+ if base_details:
294+ common["details"] = base_details
295+
296+ plans = node.get("floorPlans") or []
297+ if not plans:
298+ # pas de plan détaillé : une annonce « à partir de »
299+ rng = card.get("rentRange") or []
300+ price = rng[0] if rng else None
301+ beds = (card.get("bedsRange") or [None])[0]
302+ lst = self._mk(common, gid, "", price,
303+ self._unit_type(beds), None, today,
304+ price_from=bool(rng))
305+ lst.bedrooms = self._count(beds)
306+ lst.bathrooms = self._count(
307+ (card.get("bathsRange") or [None])[0])
308+ out.append(lst)
309+ return out
310+ for i, fp in enumerate(plans):
311+ avail = fp.get("availability") or {}
312+ adate = "now" if avail.get("now") else (
313+ avail.get("date") or "")[:10] or None
314+ if adate and adate != "now" and adate <= today:
315+ adate = "now"
316+ lst = self._mk(common, gid, f"-{i}", fp.get("rent"),
317+ self._unit_type(fp.get("beds")),
318+ fp.get("size"), today, adate=adate)
319+ lst.bedrooms = self._count(fp.get("beds"))
320+ lst.bathrooms = self._count(fp.get("baths"))
321+ fp_tour = self._tour_url(fp.get("tours"))
322+ if fp_tour: # la visite du plan précis bat celle de l'immeuble
323+ lst.details["virtual_tour"] = fp_tour
324+ if fp.get("furnished") == "yes":
325+ lst.furnished = True
326+ out.append(lst)
327+ return out
328+
224 def fetch(self) -> list[Listing]: 329 def fetch(self) -> list[Listing]:
225 # le « détail » est une requête GraphQL, pas une page HTML : le 330 # le « détail » est une requête GraphQL, pas une page HTML : le
226 # fetch_html du cache reçoit le gid et le renvoie tel quel, parse_fn 331 # fetch_html du cache reçoit le gid et le renvoie tel quel, parse_fn
227 # exécute la requête GraphQL. 332 # exécute la requête GraphQL.
333+ # clé v2 : force le re-téléchargement progressif des détails pour
334+ # capter tours{} (visites virtuelles) et floorPlans.baths ajoutés en
335+ # vague 2 — les payloads v1 périmés restent utilisés en attendant
228 cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS, 336 cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS,
229− key="v1", fetch_html=lambda gid: gid) 337+ key="v2", fetch_html=lambda gid: gid)
230 out: list[Listing] = [] 338 out: list[Listing] = []
231 today = datetime.date.today().isoformat() 339 today = datetime.date.today().isoformat()
232 try: 340 try:
@@ -240,46 +348,8 @@ class LouerCaConnector(BaseConnector):
240 continue 348 continue
241 node = cache.get(gid, gid, 349 node = cache.get(gid, gid,
242 lambda g: self._detail(g)) or {} 350 lambda g: self._detail(g)) or {}
243− addr = node.get("address") or {} 351+ out.extend(self._card_listings(card, node, today,
244− city = (addr.get("city") or {}).get("name") or slug.title() 352+ default_city=slug.title()))
245− sector = (addr.get("neighbourhood") or {}).get("name") or ""
246− loc = card.get("location") or []
247− lng, lat = (loc + [None, None])[:2]
248− base_desc = ((node.get("description") or {}).get("plain")
249− or "")[:6000]
250− images = self._images(node)
251− amenities = self._amenities(card.get("amenities"))
252− common = dict(
253− source=self.source_id, url=f"https://louer.ca/{card.get('path','')}",
254− address=addr.get("street") or "", sector=sector, city=city,
255− description=base_desc, amenities=amenities, images=images,
256− lat=lat, lng=lng,
257− )
258− if addr.get("postalCode"):
259− common["details"] = {"Code postal": addr["postalCode"]}
260−
261− plans = node.get("floorPlans") or []
262− if not plans:
263− # pas de plan détaillé : une annonce « à partir de »
264− rng = card.get("rentRange") or []
265− price = rng[0] if rng else None
266− beds = (card.get("bedsRange") or [None])[0]
267− out.append(self._mk(common, gid, "", price,
268− self._unit_type(beds), None, today,
269− price_from=bool(rng)))
270− continue
271− for i, fp in enumerate(plans):
272− avail = fp.get("availability") or {}
273− adate = "now" if avail.get("now") else (
274− avail.get("date") or "")[:10] or None
275− if adate and adate != "now" and adate <= today:
276− adate = "now"
277− lst = self._mk(common, gid, f"-{i}", fp.get("rent"),
278− self._unit_type(fp.get("beds")),
279− fp.get("size"), today, adate=adate)
280− if fp.get("furnished") == "yes":
281− lst.furnished = True
282− out.append(lst)
283 finally: 353 finally:
284 cache.close() 354 cache.close()
285 return out 355 return out
added louka/connectors/rentals_ca.py +174 −0
@@ -0,0 +1,174 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/rentals_ca.py : Rentals.ca — portail locatif national, filtré
5+# PROVINCE DE QUÉBEC. Même plateforme GraphQL que Louer.ca (mêmes City.id,
6+# mutation acquireAuthInfo, clé publique `rentalsGqlKey` de window.appconf),
7+# mais l'inventaire de marque est un peu plus large (~+15 % vérifié sur
8+# Québec : 78 annonces contre 67 côté Louer.ca) — les doublons inter-marques
9+# sont résorbés par la dédup (rentals_ca est dans la liste PORTALS).
10+# ⚠️ rentals.ca est derrière Cloudflare (« Just a moment… ») : chaque appel
11+# GraphQL est REJOUÉ via Scrapfly ASP SANS rendu JS (POST JSON, coût minime
12+# par rapport à un render_js). Pour contenir le budget Scrapfly :
13+# - villes limitées (LOUKA_RENTALSCA_CITIES) et plafond par ville ;
14+# - pages de 200 annonces (1 à 4 requêtes liste par ville) ;
15+# - détails PAR LOTS de 40 via nodes(ids:[…]) — 1 requête pour 40 fiches —
16+# avec cache TTL : seuls les lots nouveaux/périmés sont re-téléchargés.
17+# Régime de croisière : ~25-40 appels Scrapfly par synchronisation.
18+# ⚠️ CGU Rentals.ca : extraction interdite sans accord écrit (même réserve
19+# que louer_ca).
20+# -----------------------------------------------------------------------------
21+from __future__ import annotations
22+
23+import datetime
24+import json
25+import os
26+import re
27+
28+from ..schema import Listing
29+from . import _detailutil as du
30+from .louer_ca import LouerCaConnector
31+
32+# villes QC interrogées (slugs rentals.ca) — volontairement bornées : chaque
33+# page liste coûte un appel Scrapfly ASP
34+CITIES = [c.strip() for c in os.environ.get(
35+ "LOUKA_RENTALSCA_CITIES",
36+ "montreal,quebec-city,laval,gatineau,longueuil,sherbrooke,levis,"
37+ "trois-rivieres,saguenay,drummondville").split(",") if c.strip()]
38+
39+PAGE_SIZE = int(os.environ.get("LOUKA_RENTALSCA_PAGE_SIZE", "200"))
40+MAX_PER_CITY = int(os.environ.get("LOUKA_RENTALSCA_MAX_PER_CITY", "800"))
41+DETAIL_LIMIT = int(os.environ.get("LOUKA_RENTALSCA_DETAIL_LIMIT", "600"))
42+DETAIL_BATCH = int(os.environ.get("LOUKA_RENTALSCA_DETAIL_BATCH", "40"))
43+TTL_DAYS = float(os.environ.get("LOUKA_RENTALSCA_TTL_DAYS", "7"))
44+
45+
46+class RentalsCaConnector(LouerCaConnector):
47+ source_id = "rentals_ca"
48+ request_delay = 0.3 # entre appels Scrapfly (API, pas le site)
49+ home_url = "https://rentals.ca/"
50+ gql_url = "https://rentals.ca/graphql"
51+ site = "https://rentals.ca"
52+ fallback_key = "kJFM-mm4c-xg6B-qiwy" # rentalsGqlKey publique (2026-08)
53+ page_size = PAGE_SIZE
54+ max_per_city = MAX_PER_CITY
55+
56+ # -- transport : tout passe par Scrapfly ASP (POST JSON, sans rendu JS) ----
57+ def _scrapfly_gql(self, query: str, variables: dict,
58+ token: str | None = None) -> dict:
59+ headers = {"Content-Type": "application/json",
60+ "Origin": self.site, "Referer": self.home_url}
61+ if token:
62+ headers["Authorization"] = f"Bearer {token}"
63+ res = self.scrapfly(self.gql_url, render_js=False, asp=True,
64+ method="POST",
65+ body=json.dumps({"query": query,
66+ "variables": variables}),
67+ headers=headers)
68+ try:
69+ return json.loads(res.get("content") or "{}")
70+ except ValueError:
71+ return {}
72+
73+ def _ensure_token(self) -> None:
74+ import time as _time
75+ if self._token and _time.time() - self._token_time < 2700:
76+ return
77+ payload = self._scrapfly_gql(
78+ "mutation($k:String!){acquireAuthInfo(credentials:{apiKey:$k})"
79+ "{jwt status}}", {"k": self._api_key})
80+ auth = ((payload.get("data") or {}).get("acquireAuthInfo")) or {}
81+ jwt = auth.get("jwt")
82+ if isinstance(jwt, str) and jwt.startswith("{"):
83+ jwt = json.loads(jwt)
84+ token = jwt.get("accessToken") if isinstance(jwt, dict) else jwt
85+ if not token:
86+ # clé périmée ? la relire dans window.appconf de la page d'accueil
87+ home = self.get_scrapfly(self.home_url, render_js=False, asp=True)
88+ m = re.search(r'"rentalsGqlKey":\s*"([^"]+)"', home or "")
89+ if m:
90+ self._api_key = m.group(1)
91+ payload = self._scrapfly_gql(
92+ "mutation($k:String!){acquireAuthInfo(credentials:"
93+ "{apiKey:$k}){jwt status}}", {"k": self._api_key})
94+ auth = ((payload.get("data") or {})
95+ .get("acquireAuthInfo")) or {}
96+ jwt = auth.get("jwt")
97+ if isinstance(jwt, str) and jwt.startswith("{"):
98+ jwt = json.loads(jwt)
99+ token = (jwt.get("accessToken")
100+ if isinstance(jwt, dict) else jwt)
101+ if not token:
102+ raise RuntimeError("Rentals.ca: handshake JWT échoué")
103+ self._token = token
104+ self._token_time = _time.time()
105+
106+ def _gql(self, query: str, variables: dict, auth: bool = True) -> dict:
107+ if auth:
108+ self._ensure_token()
109+ payload = self._scrapfly_gql(query, variables,
110+ token=self._token if auth else None)
111+ if payload.get("errors") and auth:
112+ msg = payload["errors"][0].get("message", "")
113+ if "AUTH" in msg.upper(): # jeton expiré : on réessaie
114+ self._token = ""
115+ self._ensure_token()
116+ payload = self._scrapfly_gql(query, variables,
117+ token=self._token)
118+ return payload.get("data") or {}
119+
120+ # -- détail par LOTS (1 requête Scrapfly pour DETAIL_BATCH fiches) ---------
121+ def _details_batch(self, gids: list[str]) -> dict[str, dict]:
122+ q = ("query($ids:[ID!]!){nodes(ids:$ids){... on RentalListing{"
123+ + self._DETAIL_FRAG + "}}}")
124+ out: dict[str, dict] = {}
125+ for node in self._gql(q, {"ids": gids}).get("nodes") or []:
126+ if isinstance(node, dict) and node.get("id"):
127+ out[node["id"]] = node
128+ return out
129+
130+ def fetch(self) -> list[Listing]:
131+ if not os.environ.get("SCRAPFLY_KEY"):
132+ raise RuntimeError("SCRAPFLY_KEY manquant (voir .env)")
133+ today = datetime.date.today().isoformat()
134+ cards: dict[str, tuple[dict, str]] = {}
135+ for slug in CITIES:
136+ cid = self._city_id(slug)
137+ if not cid:
138+ continue
139+ for card in self._city_listings(cid):
140+ gid = card.get("id")
141+ if gid:
142+ cards.setdefault(gid, (card, slug))
143+
144+ # cache TTL : détails frais réutilisés sans trafic, le reste par lots
145+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS,
146+ key="v1", fetch_html=lambda gid: gid)
147+ details: dict[str, dict] = {}
148+ todo: list[str] = []
149+ out: list[Listing] = []
150+ try:
151+ for gid in cards:
152+ payload, fresh = cache.peek(gid)
153+ if payload is not None:
154+ details[gid] = payload
155+ if not fresh and len(todo) < DETAIL_LIMIT:
156+ todo.append(gid)
157+ for i in range(0, len(todo), DETAIL_BATCH):
158+ got = self._details_batch(todo[i:i + DETAIL_BATCH])
159+ for gid, node in got.items():
160+ cache.put(gid, node)
161+ details[gid] = node
162+
163+ for gid, (card, slug) in cards.items():
164+ node = details.get(gid) or {}
165+ region = (((node.get("address") or {}).get("city") or {})
166+ .get("regionCode"))
167+ if region and region != "QC":
168+ continue # sécurité : Québec uniquement
169+ out.extend(self._card_listings(
170+ card, node, today,
171+ default_city=slug.replace("-", " ").title()))
172+ finally:
173+ cache.close()
174+ return out
modified louka/db.py +11 −3
@@ -38,6 +38,8 @@ CREATE TABLE IF NOT EXISTS listings (
38 sector TEXT, 38 sector TEXT,
39 city TEXT, 39 city TEXT,
40 unit_type TEXT, 40 unit_type TEXT,
41+ bedrooms REAL,
42+ bathrooms REAL,
41 price REAL, 43 price REAL,
42 price_label TEXT, 44 price_label TEXT,
43 availability TEXT, 45 availability TEXT,
@@ -158,6 +160,8 @@ _MIGRATIONS = {
158 "digest": "TEXT", # JSON louka/textmine.py (description structurée) 160 "digest": "TEXT", # JSON louka/textmine.py (description structurée)
159 "dup_of": "TEXT", # uid de l'annonce canonique si doublon inter-sources 161 "dup_of": "TEXT", # uid de l'annonce canonique si doublon inter-sources
160 "dup_sources": "TEXT", # JSON : autres sources où l'annonce est publiée 162 "dup_sources": "TEXT", # JSON : autres sources où l'annonce est publiée
163+ "bedrooms": "REAL", # chambres fermées (convention QC : 4½ = 2)
164+ "bathrooms": "REAL", # salles de bain (1.5 = salle d'eau en plus)
161 }, 165 },
162 "sync_log": { 166 "sync_log": {
163 "stats": "TEXT", 167 "stats": "TEXT",
@@ -258,6 +262,7 @@ def sync_source(con: sqlite3.Connection, source: str,
258 uid=lst.uid, source=lst.source, external_id=lst.external_id, 262 uid=lst.uid, source=lst.source, external_id=lst.external_id,
259 url=lst.url, title=lst.title, address=lst.address, 263 url=lst.url, title=lst.title, address=lst.address,
260 sector=lst.sector, city=lst.city, unit_type=lst.unit_type, 264 sector=lst.sector, city=lst.city, unit_type=lst.unit_type,
265+ bedrooms=lst.bedrooms, bathrooms=lst.bathrooms,
261 price=lst.price, price_label=lst.price_label, 266 price=lst.price, price_label=lst.price_label,
262 availability=lst.availability, 267 availability=lst.availability,
263 availability_date=lst.availability_date, 268 availability_date=lst.availability_date,
@@ -274,13 +279,15 @@ def sync_source(con: sqlite3.Connection, source: str,
274 if row is None: 279 if row is None:
275 con.execute( 280 con.execute(
276 """INSERT INTO listings (uid, source, external_id, url, title, 281 """INSERT INTO listings (uid, source, external_id, url, title,
277− address, sector, city, unit_type, price, price_label, 282+ address, sector, city, unit_type, bedrooms, bathrooms,
283+ price, price_label,
278 availability, availability_date, area_sqft, pets, furnished, 284 availability, availability_date, area_sqft, pets, furnished,
279 description, digest, amenities, details, images, lat, lng, 285 description, digest, amenities, details, images, lat, lng,
280 content_hash, first_seen, last_seen, updated_at, 286 content_hash, first_seen, last_seen, updated_at,
281 miss_count, active) 287 miss_count, active)
282 VALUES (:uid,:source,:external_id,:url,:title,:address, 288 VALUES (:uid,:source,:external_id,:url,:title,:address,
283− :sector,:city,:unit_type,:price,:price_label,:availability, 289+ :sector,:city,:unit_type,:bedrooms,:bathrooms,
290+ :price,:price_label,:availability,
284 :availability_date,:area_sqft,:pets,:furnished, 291 :availability_date,:area_sqft,:pets,:furnished,
285 :description,:digest,:amenities,:details,:images,:lat,:lng, 292 :description,:digest,:amenities,:details,:images,:lat,:lng,
286 :content_hash,:now,:now,:now,0,1)""", params) 293 :content_hash,:now,:now,:now,0,1)""", params)
@@ -293,7 +300,8 @@ def sync_source(con: sqlite3.Connection, source: str,
293 con.execute( 300 con.execute(
294 """UPDATE listings SET url=:url, title=:title, 301 """UPDATE listings SET url=:url, title=:title,
295 address=:address, sector=:sector, city=:city, 302 address=:address, sector=:sector, city=:city,
296− unit_type=:unit_type, price=:price, 303+ unit_type=:unit_type, bedrooms=:bedrooms,
304+ bathrooms=:bathrooms, price=:price,
297 price_label=:price_label, availability=:availability, 305 price_label=:price_label, availability=:availability,
298 availability_date=:availability_date, 306 availability_date=:availability_date,
299 area_sqft=:area_sqft, pets=:pets, furnished=:furnished, 307 area_sqft=:area_sqft, pets=:pets, furnished=:furnished,
modified louka/normalize.py +79 −0
@@ -150,6 +150,85 @@ def normalize_unit_type(raw: str) -> str:
150 return _clean_text(str(raw)) 150 return _clean_text(str(raw))
151 151
152 152
153+# ---------------------------------------------------------------------------
154+# Chambres / salles de bain
155+# ---------------------------------------------------------------------------
156+
157+_BEDROOMS_TXT_RE = re.compile(
158+ r"(\d+)\s*(?:chambres?(?:\s+a\s+coucher)?\b|\bcc\b|\bcac\b|bed(?:room)?s?\b|\bbdr\b)")
159+_BATHROOMS_TXT_RE = re.compile(
160+ r"(\d+(?:[.,]5)?)\s*(?:salles?\s+de\s+bains?\b|\bsdb\b|bath(?:room)?s?\b)")
161+
162+
163+def bedrooms_from_unit_type(unit_type: str) -> float | None:
164+ """Chambres fermées selon la convention québécoise des « pièces et demie ».
165+
166+ n½ = n pièces -> n-2 chambres (3½→1, 4½→2, 5½→3) ; 1½/2½ -> 0 ;
167+ 6½+ -> 4 (borne basse connue) ; Studio/Loft -> 0 ; Chambre -> 1.
168+ Types sans nombre de pièces (Maison, Condo…) -> None (inconnu).
169+ """
170+ if not unit_type:
171+ return None
172+ s = unit_type.strip()
173+ if s == "6½+":
174+ return 4.0
175+ m = re.match(r"^(\d+)½$", s)
176+ if m:
177+ return float(max(int(m.group(1)) - 2, 0))
178+ key = _key(s)
179+ if key in ("studio", "loft"):
180+ return 0.0
181+ if key == "chambre":
182+ return 1.0
183+ return None
184+
185+
186+def parse_bedrooms(*texts: str) -> float | None:
187+ """Nombre de chambres explicite dans un texte : « 3 chambres », « 2 bedrooms ».
188+
189+ Jamais de valeur inventée : None si aucun nombre plausible (0-12).
190+ """
191+ for t in texts:
192+ if not t:
193+ continue
194+ m = _BEDROOMS_TXT_RE.search(_key(str(t)))
195+ if m:
196+ n = int(m.group(1))
197+ if 0 <= n <= 12:
198+ return float(n)
199+ return None
200+
201+
202+def parse_bathrooms(*texts: str) -> float | None:
203+ """Nombre de salles de bain : « 2 salles de bain », « 1.5 bath », « 1 sdb »."""
204+ for t in texts:
205+ if not t:
206+ continue
207+ m = _BATHROOMS_TXT_RE.search(_key(str(t)))
208+ if m:
209+ try:
210+ v = float(m.group(1).replace(",", "."))
211+ except ValueError:
212+ continue
213+ if 0.5 <= v <= 10:
214+ return v
215+ return None
216+
217+
218+def coerce_count(raw) -> float | None:
219+ """« 2 », « 1,5 », 2, 2.0 -> float ; sinon None (valeurs de details JSON)."""
220+ if raw is None or isinstance(raw, bool):
221+ return None
222+ if isinstance(raw, (int, float)):
223+ v = float(raw)
224+ return v if 0 <= v <= 20 else None
225+ m = re.match(r"^\s*(\d+(?:[.,]5)?)\s*$", str(raw))
226+ if not m:
227+ return None
228+ v = float(m.group(1).replace(",", "."))
229+ return v if 0 <= v <= 20 else None
230+
231+
153 # --------------------------------------------------------------------------- 232 # ---------------------------------------------------------------------------
154 # Date de disponibilité 233 # Date de disponibilité
155 # --------------------------------------------------------------------------- 234 # ---------------------------------------------------------------------------
modified louka/schema.py +29 −0
@@ -18,12 +18,16 @@ import json
18 from dataclasses import dataclass, field, asdict 18 from dataclasses import dataclass, field, asdict
19 19
20 from .normalize import ( # ré-exportés pour les connecteurs existants 20 from .normalize import ( # ré-exportés pour les connecteurs existants
21+ bedrooms_from_unit_type,
21 clean_address, 22 clean_address,
23+ coerce_count,
22 extract_details, 24 extract_details,
23 merge_details, 25 merge_details,
24 normalize_unit_type, 26 normalize_unit_type,
25 parse_area_sqft, 27 parse_area_sqft,
26 parse_availability_date, 28 parse_availability_date,
29+ parse_bathrooms,
30+ parse_bedrooms,
27 parse_price, 31 parse_price,
28 price_is_from, 32 price_is_from,
29 strip_accents, 33 strip_accents,
@@ -48,6 +52,8 @@ class Listing:
48 sector: str = "" # quartier/arrondissement (ex. Beauport) 52 sector: str = "" # quartier/arrondissement (ex. Beauport)
49 city: str = "" # Québec, Lévis, ... 53 city: str = "" # Québec, Lévis, ...
50 unit_type: str = "" # 1½ … 5½, 6½+, Studio, Loft, Chambre… 54 unit_type: str = "" # 1½ … 5½, 6½+, Studio, Loft, Chambre…
55+ bedrooms: float | None = None # chambres fermées (convention QC : 4½ = 2)
56+ bathrooms: float | None = None # salles de bain (1.5 = salle d'eau en plus)
51 price: float | None = None # loyer mensuel ($ CAD), le plus bas si "à partir de" 57 price: float | None = None # loyer mensuel ($ CAD), le plus bas si "à partir de"
52 price_label: str = "" # texte original (ex. "à partir de 799$") 58 price_label: str = "" # texte original (ex. "à partir de 799$")
53 availability: str = "" # texte original (ex. "Libre immédiatement") 59 availability: str = "" # texte original (ex. "Libre immédiatement")
@@ -104,6 +110,29 @@ class Listing:
104 derived["price_from"] = True 110 derived["price_from"] = True
105 self.details = merge_details(self.details, derived) 111 self.details = merge_details(self.details, derived)
106 112
113+ # chambres / salles de bain : valeur explicite du connecteur d'abord,
114+ # puis champs structurés de details, puis type d'unité (n½ -> n-2),
115+ # puis extraction texte — jamais de valeur inventée (None = inconnu)
116+ if self.bedrooms is None:
117+ for k in ("bedrooms", "Chambres", "Chambre(s)"):
118+ self.bedrooms = coerce_count(self.details.get(k))
119+ if self.bedrooms is not None:
120+ break
121+ if self.bedrooms is None:
122+ self.bedrooms = bedrooms_from_unit_type(self.unit_type)
123+ if self.bedrooms is None:
124+ self.bedrooms = parse_bedrooms(self.title, " | ".join(self.amenities),
125+ self.description)
126+ if self.bathrooms is None:
127+ for k in ("bathrooms", "Salles de bain", "Salle de bain",
128+ "Salle(s) de bain"):
129+ self.bathrooms = coerce_count(self.details.get(k))
130+ if self.bathrooms is not None:
131+ break
132+ if self.bathrooms is None:
133+ self.bathrooms = parse_bathrooms(" | ".join(self.amenities),
134+ self.description)
135+
107 # description structurée (nettoyage + extraction + sections) 136 # description structurée (nettoyage + extraction + sections)
108 if self.digest is None and self.description: 137 if self.digest is None and self.description:
109 try: 138 try:
110 139
Menu

Theme
↑↓navigate ↵open escclose /⌘Kanywhere