SPB Git
spboucher.ai

spb/lou-ka

Public

Lou·Ka — tous les logements à louer du Québec, un seul endroit.

www.lou-ka.com
webappreal-estatequebecrental

Nouvelles sources + déduplication inter-sources

8 connecteurs (208 → 212), testés en live :
- logisquebec — portail (~9 800 fiches) : sitemaps + JSON-LD, expiration HTTP 410
- duproprio — location par particuliers (~4 200, 0 % chevauchement) : sitemaps gz + JSON-LD, retrait = 302
- louer_ca — réseau Rentals.ca (~1 700) : API GraphQL interne (JWT), 1 annonce par plan d'étage
- roomies — chambres/colocation : JSON-LD Room
- groupe_maurice — 37 résidences pour aînés (RPA) : API REST + grilles de prix
- bons_locataires — agence Montréal : carte MapListPro + fiches WordPress
- omhm — logements abordables ENHARMONIE
- fb_marketplace — Scrapfly ASP, pagination par tranches de prix (scroll bloqué hors login)

Déduplication inter-sources (louka/dedup.py) :
- colonnes dup_of / dup_sources (migration db.py), recalcul idempotent dans ingest.watch() après géocodage
- blocage par clé d'adresse (civique + rue normalisée FR/EN + ville), confirmation type + prix ±4 %
- contrainte « ≤1 annonce par source par groupe » (évite de fusionner des unités distinctes d'un immeuble)
- canonique = source la plus autoritaire (gestionnaire direct > portail > petites annonces)
- web.py n'affiche que les annonces canoniques (dup_of IS NULL) ; dup_sources exposé pour le badge « aussi publiée sur »
- résultat : 147 groupes, 154 doublons masqués sur 18 536 annonces

Utilitaire _detailutil.TtlDetailCache : cache de pages détail avec TTL + budget par sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Aug 17, 2026) parent 3c42232 Browse files

14 changed files +1,936 −8

14 changed files
Load all 14 files Unified Split
modified data/sources.json +81 −1
@@ -2648,6 +2648,86 @@
2648 "connector": "sutton", 2648 "connector": "sutton",
2649 "status": "actif", 2649 "status": "actif",
2650 "region": "Province" 2650 "region": "Province"
2651+ },
2652+ {
2653+ "id": "logisquebec",
2654+ "name": "LogisQuébec",
2655+ "url": "https://www.logisquebec.com",
2656+ "listing_url": "https://www.logisquebec.com/appartement-a-louer-montreal",
2657+ "sectors": "Province entière — portail locatif (Réglisse Média), ~9 800 annonces",
2658+ "connector": "logisquebec",
2659+ "status": "actif",
2660+ "region": "Province"
2661+ },
2662+ {
2663+ "id": "duproprio",
2664+ "name": "DuProprio (location)",
2665+ "url": "https://duproprio.com",
2666+ "listing_url": "https://duproprio.com/fr/location/montreal",
2667+ "sectors": "Province entière — location par des particuliers (sans courtier), ~4 200 annonces",
2668+ "connector": "duproprio",
2669+ "status": "actif",
2670+ "region": "Province"
2671+ },
2672+ {
2673+ "id": "louer_ca",
2674+ "name": "Louer.ca",
2675+ "url": "https://louer.ca",
2676+ "listing_url": "https://louer.ca/montreal",
2677+ "sectors": "Province entière — portail locatif (réseau Rentals.ca), API GraphQL",
2678+ "connector": "louer_ca",
2679+ "status": "actif",
2680+ "region": "Province"
2681+ },
2682+ {
2683+ "id": "roomies",
2684+ "name": "Roomies.ca",
2685+ "url": "https://www.roomies.ca",
2686+ "listing_url": "https://www.roomies.ca/montreal-qc",
2687+ "sectors": "Grandes villes QC — chambres et colocation",
2688+ "connector": "roomies",
2689+ "status": "actif",
2690+ "region": "Province"
2691+ },
2692+ {
2693+ "id": "groupe_maurice",
2694+ "name": "Le Groupe Maurice",
2695+ "url": "https://www.legroupemaurice.com",
2696+ "listing_url": "https://www.legroupemaurice.com/residences/",
2697+ "sectors": "37 résidences pour aînés (RPA) — Montréal, Laval, Rive-Sud/Nord, Québec, Gatineau",
2698+ "connector": "groupe_maurice",
2699+ "status": "actif",
2700+ "region": "Province"
2701+ },
2702+ {
2703+ "id": "bons_locataires",
2704+ "name": "Bons Locataires",
2705+ "url": "https://www.bonslocataires.com",
2706+ "listing_url": "https://www.bonslocataires.com/logement-appartement-a-louer-enquete-proprio-credit-locataire-visite-virtuelle-360/",
2707+ "sectors": "Montréal, Laval, Longueuil — agence de location",
2708+ "connector": "bons_locataires",
2709+ "status": "actif",
2710+ "region": "Grand Montréal"
2711+ },
2712+ {
2713+ "id": "omhm",
2714+ "name": "OMHM — logements abordables",
2715+ "url": "https://www.omhm.qc.ca",
2716+ "listing_url": "https://www.omhm.qc.ca/fr/soumettre-une-demande/logements-disponibles",
2717+ "sectors": "Montréal — résidences abordables ENHARMONIE (aînés autonomes)",
2718+ "connector": "omhm",
2719+ "status": "actif",
2720+ "region": "Montréal"
2721+ },
2722+ {
2723+ "id": "fb_marketplace",
2724+ "name": "Facebook Marketplace (location)",
2725+ "url": "https://www.facebook.com/marketplace",
2726+ "listing_url": "https://www.facebook.com/marketplace/montreal/propertyrentals",
2727+ "sectors": "Grandes villes QC — annonces de particuliers (Scrapfly)",
2728+ "connector": "fb_marketplace",
2729+ "status": "actif",
2730+ "region": "Province"
2651 } 2731 }
2652 ] 2732 ]
2653−}
\ No newline at end of file
2733+}
modified louka/connectors/_detailutil.py +60 −0
@@ -128,6 +128,66 @@ def enrich(connector, listings, limit, parse_fn, key="v1", fetch_html=None):
128 con.close() 128 con.close()
129 129
130 130
131+class TtlDetailCache:
132+ """Cache BD des pages détail AVEC durée de vie, pour les portails où la
133+ fiche est la seule source de données (LogisQuébec, DuProprio…).
134+
135+ Contrairement à enrich() (cache permanent tant que la clé ne change pas),
136+ chaque fiche est re-visitée après `ttl_days` pour capter les changements
137+ de prix/disponibilité — dans la limite de `budget` requêtes par
138+ synchronisation. Budget épuisé : le payload périmé est réutilisé tel quel
139+ (jamais de trou de données) ; une fiche jamais visitée retourne None et
140+ sera captée à une synchronisation suivante.
141+ """
142+
143+ def __init__(self, connector, budget: int, ttl_days: float = 7.0,
144+ key: str = "v1", fetch_html=None) -> None:
145+ from .. import db
146+ self.connector = connector
147+ self.con = db.connect()
148+ self.budget = budget
149+ self.ttl = ttl_days * 86400
150+ self.key = key
151+ self.fetch_html = fetch_html or (lambda u: connector.get(u).text)
152+
153+ def get(self, external_id: str, url: str, parse_fn) -> dict | None:
154+ import time as _time
155+ from .. import db
156+ row = self.con.execute(
157+ "SELECT key, payload, fetched_at FROM detail_cache"
158+ " WHERE source=? AND external_id=?",
159+ (self.connector.source_id, str(external_id))).fetchone()
160+ stale = None
161+ if row and row["payload"]:
162+ try:
163+ stale = json.loads(row["payload"])
164+ except ValueError:
165+ stale = None
166+ if (stale is not None and row["key"] == self.key
167+ and _time.time() - (row["fetched_at"] or 0) < self.ttl):
168+ return stale # frais : aucun trafic
169+ if self.budget <= 0:
170+ return stale # périmé toléré / None si jamais vu
171+ self.budget -= 1
172+ try:
173+ payload = parse_fn(self.fetch_html(url)) or {}
174+ except Exception as exc: # 404/410 = fiche retirée
175+ code = getattr(getattr(exc, "response", None), "status_code", None)
176+ if code in (404, 410):
177+ payload = {"gone": True}
178+ else:
179+ return stale # erreur réseau : on garde l'ancien
180+ db.put_cached_detail(self.con, self.connector.source_id,
181+ str(external_id), self.key, payload)
182+ return payload
183+
184+ def close(self) -> None:
185+ try:
186+ self.con.close()
187+ except Exception:
188+ pass
189+
190+
131 def apply_detail(lst: Listing, d: dict) -> None: 191 def apply_detail(lst: Listing, d: dict) -> None:
132 """Applique un payload détail au Listing sans écraser les valeurs déjà 192 """Applique un payload détail au Listing sans écraser les valeurs déjà
133 présentes (sauf images : on garde la plus grande galerie).""" 193 présentes (sauf images : on garde la plus grande galerie)."""
added louka/connectors/bons_locataires.py +160 −0
@@ -0,0 +1,160 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/bons_locataires.py : Bons Locataires (bonslocataires.com) — agence
5+# de location Montréal/Laval/Longueuil. WordPress, aucun anti-bot (seul
6+# mod_security bloque les sitemaps XML — on ne les utilise pas).
7+# Source de vérité : la page carte MapListPro (~8 Mo) dont le tableau JSON
8+# inline contient les 1 600+ emplacements avec lat/lng, taxonomies
9+# (région/quartier/taille) et description complète. Le site garde les
10+# logements loués sur la carte (« DÉJÀ LOUÉ », pin PouceVert) : on ne
11+# conserve que les annonces encore offertes, et on valide que leur page
12+# (locationUrl) répond toujours 200 (certains vieux liens sont morts).
13+# Prix et disponibilité en texte libre -> regex. ~20-40 annonces actives.
14+# -----------------------------------------------------------------------------
15+from __future__ import annotations
16+
17+import html as H
18+import json
19+import re
20+
21+from ..schema import Listing, normalize_unit_type
22+from .base import BaseConnector
23+
24+MAP_PAGE = ("https://www.bonslocataires.com/logement-appartement-a-louer-"
25+ "enquete-proprio-credit-locataire-visite-virtuelle-360/")
26+
27+_PRICE_RE = re.compile(r"(\d{3,4})\s*\$\s*(?:/|par)?\s*mois", re.I)
28+_PRICE_ALT_RE = re.compile(r"(\d{3,4})\s*\$")
29+_IMG_RE = re.compile(r'src="(https?://www\.bonslocataires\.com/wp-content/'
30+ r'uploads/[^"]+)"')
31+_LOC_ID_RE = re.compile(r"loc-(\d+)")
32+_TAG_RE = re.compile(r"<[^>]+>")
33+
34+
35+def _texte(fragment: str) -> str:
36+ return re.sub(r"\s+\n", "\n", H.unescape(_TAG_RE.sub("", fragment or ""))).strip()
37+
38+
39+class BonsLocatairesConnector(BaseConnector):
40+ source_id = "bons_locataires"
41+ request_delay = 1.0
42+ timeout = 120 # la page carte fait ~8 Mo
43+
44+ def _map_objects(self) -> list[dict]:
45+ import time
46+ html = ""
47+ for attempt in range(3): # mod_security coupe parfois la page 8 Mo
48+ try:
49+ html = self.get(MAP_PAGE).text
50+ break
51+ except Exception:
52+ if attempt == 2:
53+ raise
54+ time.sleep(20 * (attempt + 1))
55+ i = html.find('[{"title":')
56+ if i < 0:
57+ return []
58+ depth, end = 0, None
59+ for j in range(i, len(html)):
60+ if html[j] == "[":
61+ depth += 1
62+ elif html[j] == "]":
63+ depth -= 1
64+ if depth == 0:
65+ end = j + 1
66+ break
67+ try:
68+ return json.loads(html[i:end])
69+ except ValueError:
70+ return []
71+
72+ @staticmethod
73+ def _available(obj: dict) -> bool:
74+ blob = (obj.get("title", "") + (obj.get("description") or "")
75+ + (obj.get("icon") or "") + (obj.get("pinImageUrl") or ""))
76+ return "LOU" not in blob.upper() or "PouceVert" not in blob
77+
78+ def fetch(self) -> list[Listing]:
79+ out: list[Listing] = []
80+ for obj in self._map_objects():
81+ desc_html = obj.get("description") or ""
82+ blob = (obj.get("title", "") + desc_html).upper()
83+ pin = (obj.get("pinImageUrl") or "") + (obj.get("icon") or "")
84+ if "DÉJÀ LOU" in blob or "DEJA LOU" in blob or "LOUÉ!" in blob \
85+ or "PouceVert" in pin:
86+ continue
87+ url = (obj.get("locationUrl") or "").replace("http://", "https://")
88+ if not url:
89+ continue
90+ try: # vieux liens morts -> annonce écartée
91+ page_html = self.get(url).text
92+ except Exception:
93+ continue
94+ m = _LOC_ID_RE.search(obj.get("cssClass") or "")
95+ eid = m.group(1) if m else url.rstrip("/").rsplit("/", 1)[-1]
96+
97+ cats = obj.get("customCategories") or {}
98+ def _first(tax: str) -> str:
99+ terms = cats.get(tax) or []
100+ return terms[0].get("name", "") if terms else ""
101+ city = _first("map_location_categories_region") or "Montréal"
102+ sector = _first("map_location_categories_city")
103+ taille = _first("map_location_categories_taille")
104+
105+ title = _texte(obj.get("title", "")).strip("* ")
106+ unit_type = normalize_unit_type(taille or title)
107+ if not re.match(r"^\d½$|^6½\+$|^Studio$|^Loft$|^Chambre", unit_type or ""):
108+ unit_type = ""
109+
110+ # la fiche WordPress (déjà téléchargée pour la validation) est plus
111+ # riche que le popup carte : prix, galerie, disponibilité
112+ page_main = ""
113+ m = re.search(r'<div class="entry-content[^"]*">(.*?)</div>\s*'
114+ r'(?:<footer|<div class="entry-footer)', page_html, re.S)
115+ if not m:
116+ m = re.search(r"<article\b.*?</article>", page_html, re.S)
117+ if m:
118+ page_main = m.group(0)
119+ page_text = _texte(page_main)
120+
121+ description = _texte(desc_html)
122+ if 100 < len(page_text) and len(page_text) > len(description):
123+ description = page_text
124+ price = None # le popup carte prime : texte ciblé
125+ for texte in (_texte(desc_html), page_text):
126+ m = _PRICE_RE.search(texte) or _PRICE_ALT_RE.search(texte)
127+ if m and 400 <= float(m.group(1)) <= 5000:
128+ price = float(m.group(1))
129+ break
130+ availability = ""
131+ m = re.search(r"[Dd]isponible[^.\n<]{0,70}", description)
132+ if m:
133+ availability = m.group(0).strip()
134+ images = []
135+ for u in _IMG_RE.findall(page_main) + _IMG_RE.findall(desc_html):
136+ u = u.replace("http://", "https://")
137+ if u not in images:
138+ images.append(u)
139+ try:
140+ lat, lng = float(obj.get("latitude")), float(obj.get("longitude"))
141+ except (TypeError, ValueError):
142+ lat = lng = None
143+ out.append(Listing(
144+ source=self.source_id,
145+ external_id=str(eid),
146+ url=url,
147+ title=title,
148+ sector=sector,
149+ city=city,
150+ unit_type=unit_type,
151+ price=price,
152+ price_label=(f"{price:,.0f} $/mois".replace(",", " ")
153+ if price else ""),
154+ availability=availability,
155+ description=description[:6000],
156+ images=images,
157+ lat=lat,
158+ lng=lng,
159+ ))
160+ return out
added louka/connectors/duproprio.py +225 −0
@@ -0,0 +1,225 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/duproprio.py : DuProprio (duproprio.com) — section LOCATION
5+# ~4 200 annonces de particuliers (sans courtier : zéro chevauchement MLS).
6+# Découverte par les 21 sitemaps régionaux `sitemaps/fr/<region>-listings.xml.gz`
7+# filtrés sur `/fr/location/`. Fiches SSR (Laravel/CloudFront, aucun anti-bot) :
8+# JSON-LD `Apartment` (adresse + code postal + chambres/SDB + amenityFeature)
9+# et `RealEstateListing` (offers.price + availabilityStarts), lat/lng inline,
10+# superficie « 970 pi² », galerie dans un JSON échappé ("photos":[…]).
11+# Fiche retirée -> 302 vers la page parente (détectée par l'URL finale sans
12+# `/hab-`). Cache détail avec TTL (_detailutil.TtlDetailCache).
13+# Respect robots.txt : ne jamais toucher /webservice/, /fr-ca/api, /api-proxy.
14+# -----------------------------------------------------------------------------
15+from __future__ import annotations
16+
17+import gzip
18+import html as H
19+import os
20+import re
21+
22+from ..schema import Listing
23+from .base import BaseConnector
24+from . import _detailutil as du
25+
26+BASE = "https://duproprio.com"
27+SITEMAP_INDEX = f"{BASE}/sitemaps/fr/index.xml.gz"
28+
29+_LOC_RE = re.compile(r"<loc>(.*?)</loc>")
30+# .../fr/location/<region>/<ville>/<type>-a-louer/hab-<slug>-<ID>
31+_FICHE_RE = re.compile(
32+ r"^https://duproprio\.com/fr/location/([a-z0-9-]+)/([a-z0-9-]+)/"
33+ r"([a-z0-9-]+)-a-louer/hab-[a-z0-9-]+-(\d+)$")
34+_GEO_RE = re.compile(r'"latitude":([\d.-]+),"longitude":([\d.-]+)')
35+_AREA_RE = re.compile(r"([\d\s,.]{1,10})\s*pi²")
36+_PHOTO_RE = re.compile(
37+ r"photos\\?/public\\?/for_rent\\?/[0-9]+\\?/[0-9]+\\?/[a-z0-9-]+-(\d+)\.jpg")
38+_PHOTO_FULL_RE = re.compile(
39+ r"photos[\\/]+public[\\/]+for_rent[\\/]+(\d+)[\\/]+(\d+)[\\/]+([a-z0-9-]+)-(\d+)\.jpg")
40+_TYPE_HALF_RE = re.compile(r"^(\d+)-1-2$")
41+
42+# types non résidentiels (premier mot du slug type) exclus du parc
43+_EXCLUDED_TYPES = {"commerce", "bureau", "local", "entrepot", "espace",
44+ "terrain", "garage", "stationnement", "industriel"}
45+
46+DETAIL_LIMIT = int(os.environ.get("LOUKA_DUPROPRIO_DETAIL_LIMIT", "800"))
47+TTL_DAYS = float(os.environ.get("LOUKA_DUPROPRIO_TTL_DAYS", "7"))
48+MAX_FICHES = int(os.environ.get("LOUKA_DUPROPRIO_MAX", "0")) # 0 = tout
49+
50+
51+def _parse_fiche(html: str) -> dict:
52+ out: dict = {}
53+ for node in du.ld_nodes(html):
54+ t = node.get("@type") or ""
55+ if t in ("Apartment", "House", "SingleFamilyResidence", "Accommodation"):
56+ out["title"] = (node.get("name") or "").strip()
57+ addr = node.get("address") or {}
58+ out["address"] = (addr.get("streetAddress") or "").strip()
59+ out["city"] = (addr.get("addressLocality") or "").strip()
60+ out["postal_code"] = (addr.get("postalCode") or "").strip()
61+ for src, lab in (("numberOfBedrooms", "Chambres"),
62+ ("numberOfFullBathrooms", "Salles de bain"),
63+ ("floorLevel", "Étage")):
64+ if node.get(src):
65+ out.setdefault("details", {})[lab] = str(node[src])
66+ if node.get("numberOfBedrooms"):
67+ out["bedrooms"] = str(node["numberOfBedrooms"])
68+ feats = node.get("amenityFeature") or []
69+ names = [f.get("name", "").strip() for f in feats
70+ if isinstance(f, dict) and f.get("name")]
71+ if names:
72+ out["amenities"] = names
73+ if node.get("description"):
74+ out["description"] = H.unescape(str(node["description"])).strip()[:6000]
75+ elif t == "RealEstateListing":
76+ offers = node.get("offers") or {}
77+ try:
78+ out["price"] = float(str(offers.get("price")).replace(",", "."))
79+ except (TypeError, ValueError):
80+ pass
81+ if offers.get("availabilityStarts"):
82+ out["availability_date"] = str(offers["availabilityStarts"])[:10]
83+
84+ m = _GEO_RE.search(html)
85+ if m:
86+ out["lat"], out["lng"] = float(m.group(1)), float(m.group(2))
87+ m = _AREA_RE.search(html)
88+ if m:
89+ try:
90+ area = float(m.group(1).replace(" ", "").replace(",", ""))
91+ if 80 <= area <= 20000:
92+ out["area_sqft"] = area
93+ except ValueError:
94+ pass
95+
96+ # galerie : URLs (souvent échappées \/) dédupliquées par id photo, en 1600 px
97+ photos, seen = [], set()
98+ for m in _PHOTO_FULL_RE.finditer(html):
99+ ym, lid, slug, pid = m.groups()
100+ if pid in seen:
101+ continue
102+ seen.add(pid)
103+ # le slug embarque la taille en avant-dernier segment : la re-forcer à 1600
104+ slug1600 = re.sub(r"-\d+$", "-1600", slug)
105+ photos.append(f"https://photos.duproprio.com/photos/public/for_rent/"
106+ f"{ym}/{lid}/{slug1600}-{pid}.jpg")
107+ if photos:
108+ out["images"] = photos
109+ return out
110+
111+
112+class DuProprioConnector(BaseConnector):
113+ source_id = "duproprio"
114+ request_delay = 0.8
115+
116+ def _gz(self, url: str) -> str:
117+ resp = self.get(url)
118+ try:
119+ return gzip.decompress(resp.content).decode("utf-8", "replace")
120+ except (OSError, EOFError):
121+ return resp.text
122+
123+ def _fiche_urls(self) -> list[tuple[str, str, str, str]]:
124+ """(url, ville, type_slug, id) des fiches location, tous sitemaps."""
125+ index = self._gz(SITEMAP_INDEX)
126+ subs = [u for u in _LOC_RE.findall(index) if u.endswith("-listings.xml.gz")]
127+ out, seen = [], set()
128+ for sub in subs:
129+ try:
130+ xml = self._gz(sub)
131+ except Exception:
132+ continue
133+ for u in _LOC_RE.findall(xml):
134+ m = _FICHE_RE.match(u)
135+ if m and m.group(3).split("-")[0] in _EXCLUDED_TYPES:
136+ continue
137+ if m and m.group(4) not in seen:
138+ seen.add(m.group(4))
139+ out.append((u, m.group(2), m.group(3), m.group(4)))
140+ return out
141+
142+ def _fetch_fiche(self, url: str) -> str:
143+ """GET d'une fiche ; une redirection vers la page parente = retirée."""
144+ resp = self.get(url, allow_redirects=True)
145+ if "/hab-" not in resp.url:
146+ return "" # 302 « annonce disparue »
147+ return resp.text
148+
149+ @staticmethod
150+ def _unit_type(type_slug: str, bedrooms: str | None) -> str:
151+ m = _TYPE_HALF_RE.match(type_slug)
152+ if m:
153+ n = int(m.group(1))
154+ return "6½+" if n >= 6 else f"{n}½"
155+ base = {"studio": "Studio", "loft": "Loft", "chambre": "Chambre",
156+ "maison": "Maison", "condo": "", "appartement": ""}
157+ label = base.get(type_slug.split("-")[0], "")
158+ if not label and bedrooms and bedrooms.isdigit():
159+ n = int(bedrooms) + 2
160+ return "6½+" if n >= 6 else f"{n}½"
161+ return label or type_slug.replace("-", " ").capitalize()
162+
163+ def fetch(self) -> list[Listing]:
164+ fiches = self._fiche_urls()
165+ if MAX_FICHES:
166+ fiches = fiches[:MAX_FICHES]
167+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS,
168+ fetch_html=self._fetch_fiche)
169+ out: list[Listing] = []
170+ try:
171+ for url, _ville, type_slug, eid in fiches:
172+ payload = cache.get(eid, url, _parse_fiche)
173+ if payload is None: # jamais visitée + budget épuisé
174+ continue
175+ if not payload or payload.get("gone"):
176+ continue # fiche retirée (302) ou page vide
177+ price = payload.get("price")
178+ # availabilityStarts dans le passé = déjà libre aujourd'hui
179+ avail = payload.get("availability_date")
180+ if avail:
181+ import datetime as _dt
182+ if avail <= _dt.date.today().isoformat():
183+ avail = "now"
184+ # « Ahuntsic (Montréal) » -> secteur + ville ; « X (X) » -> X
185+ locality = payload.get("city") or ""
186+ sector = ""
187+ m = re.match(r"^(.*?)\s*\((.+)\)\s*$", locality)
188+ if m:
189+ inner, outer = m.group(1).strip(), m.group(2).strip()
190+ if inner.lower() == outer.lower():
191+ locality, sector = inner, ""
192+ else:
193+ locality, sector = outer, inner
194+ lst = Listing(
195+ source=self.source_id,
196+ external_id=str(eid),
197+ url=url,
198+ title=payload.get("title") or "",
199+ address=payload.get("address") or "",
200+ sector=sector,
201+ city=locality,
202+ unit_type=self._unit_type(type_slug, payload.get("bedrooms")),
203+ price=price,
204+ price_label=(f"{price:,.0f} $/mois".replace(",", " ")
205+ if price else ""),
206+ availability_date=avail,
207+ availability=("Libre immédiatement" if avail == "now"
208+ else f"Libre le {avail}" if avail else ""),
209+ area_sqft=payload.get("area_sqft"),
210+ description=payload.get("description") or "",
211+ amenities=payload.get("amenities") or [],
212+ details=payload.get("details") or {},
213+ images=payload.get("images") or [],
214+ lat=payload.get("lat"),
215+ lng=payload.get("lng"),
216+ )
217+ low = [a.lower() for a in lst.amenities]
218+ if any("animaux" in a and "permis" in a for a in low):
219+ lst.pets = "oui"
220+ if any(a.startswith("meublé") for a in low):
221+ lst.furnished = True
222+ out.append(lst)
223+ finally:
224+ cache.close()
225+ return out
added louka/connectors/fb_marketplace.py +211 −0
@@ -0,0 +1,211 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/fb_marketplace.py : Facebook Marketplace — catégorie location
5+# (propertyrentals) par ville du Québec. Annonces de particuliers, fort
6+# complément aux gestionnaires. Accès via Scrapfly ASP (render_js + proxy
7+# résidentiel CA), SANS compte ni cookie (le scraping public déconnecté est
8+# couvert par Meta v. Bright Data 2024). Les données vivent dans des balises
9+# <script type="application/json"> (objets marketplace_listing_title).
10+# Le scroll infini est bloqué hors connexion (~25 annonces/URL, mur de
11+# login) : on pagine par TRANCHES DE PRIX (minPrice/maxPrice), qui renvoient
12+# des lots quasi disjoints, et on accumule sur plusieurs synchronisations.
13+# Détail (/marketplace/item/<id>) : lat/lng + description (Scrapfly, coûteux)
14+# uniquement pour les nouvelles annonces via le cache. Type d'unité et
15+# disponibilité déduits du titre/description (pas d'attributs structurés
16+# hors connexion). ⚠️ Coût Scrapfly élevé + données personnelles (Loi 25) :
17+# ne jamais republier nom/téléphone du vendeur, garder le lien sortant.
18+# -----------------------------------------------------------------------------
19+from __future__ import annotations
20+
21+import json
22+import os
23+import re
24+
25+from ..schema import Listing, normalize_unit_type
26+from .base import BaseConnector
27+from . import _detailutil as du
28+
29+BASE = "https://www.facebook.com/marketplace"
30+
31+# villes québécoises (slug Marketplace) -> nom d'affichage
32+CITIES = {
33+ "montreal": "Montréal", "laval": "Laval", "longueuil": "Longueuil",
34+ "quebec": "Québec", "gatineau": "Gatineau", "sherbrooke": "Sherbrooke",
35+ "trois-rivieres": "Trois-Rivières", "levis": "Lévis",
36+ "saguenay": "Saguenay", "drummondville": "Drummondville",
37+}
38+# tranches de prix (bornes en $) : le flux déconnecté sert un sous-ensemble
39+# tournant, chaque tranche renvoie un lot quasi disjoint
40+PRICE_BANDS = [(0, 800), (800, 1100), (1100, 1400), (1400, 1700),
41+ (1700, 2100), (2100, 2800), (2800, 6000)]
42+
43+# garde-fous de prix mensuel (rejette « $90 » = /nuit, et les valeurs à vendre)
44+PRICE_MIN, PRICE_MAX = 300, 12000
45+
46+DETAIL_LIMIT = int(os.environ.get("LOUKA_FBMP_DETAIL_LIMIT", "120"))
47+TTL_DAYS = float(os.environ.get("LOUKA_FBMP_TTL_DAYS", "10"))
48+CITY_LIMIT = os.environ.get("LOUKA_FBMP_CITIES", "") # ex. "montreal,laval"
49+
50+
51+def _walk_listings(html: str) -> dict:
52+ """Objets d'annonce (marketplace_listing_title) des <script> JSON."""
53+ out: dict = {}
54+
55+ def walk(o):
56+ if isinstance(o, dict):
57+ if "marketplace_listing_title" in o and o.get("id"):
58+ out[str(o["id"])] = o
59+ for v in o.values():
60+ walk(v)
61+ elif isinstance(o, list):
62+ for v in o:
63+ walk(v)
64+
65+ for m in re.finditer(r'<script type="application/json"[^>]*>(.*?)</script>',
66+ html, re.S):
67+ try:
68+ walk(json.loads(m.group(1)))
69+ except ValueError:
70+ continue
71+ return out
72+
73+
74+def _price(obj: dict) -> float | None:
75+ pr = obj.get("listing_price") or {}
76+ for k in ("amount", "amount_with_offset_in_currency"):
77+ v = pr.get(k)
78+ if v is None:
79+ continue
80+ try:
81+ val = float(v)
82+ except (TypeError, ValueError):
83+ continue
84+ if k == "amount_with_offset_in_currency" and val > 100000:
85+ val /= 100.0 # offset en cents
86+ if PRICE_MIN <= val <= PRICE_MAX:
87+ return round(val, 0)
88+ return None
89+
90+
91+def _unit_type(title: str, desc: str = "") -> str:
92+ blob = f"{title} {desc}"
93+ ut = normalize_unit_type(title)
94+ if re.match(r"^\d½$|^6½\+$|^Studio$|^Loft$", ut or ""):
95+ return ut
96+ m = re.search(r"(\d+)\s*(?:bed|bedroom|chambre|cc|br)\b", blob, re.I)
97+ if m:
98+ n = int(m.group(1)) + 2
99+ return "6½+" if n >= 6 else f"{n}½"
100+ if re.search(r"\bstudio|bachelor\b", blob, re.I):
101+ return "Studio"
102+ if re.search(r"\b(?:private\s+)?room\b|chambre", blob, re.I):
103+ return "Chambre"
104+ return ""
105+
106+
107+class FacebookMarketplaceConnector(BaseConnector):
108+ source_id = "fb_marketplace"
109+ request_delay = 1.0
110+
111+ def _scrape(self, url: str) -> str:
112+ return self.get_scrapfly(url, render_js=True, asp=True, country="ca",
113+ proxy_pool="public_residential_pool",
114+ rendering_wait=5000)
115+
116+ def _parse_detail(self, html: str) -> dict:
117+ listings = _walk_listings(html)
118+ # la fiche contient l'annonce cible + des « similaires » : on prend
119+ # celle qui possède une localisation ou une description
120+ best = {}
121+ for o in listings.values():
122+ loc = o.get("location") or {}
123+ desc = (o.get("redacted_description") or {}).get("text") or ""
124+ score = (1 if loc.get("latitude") else 0) + (1 if desc else 0)
125+ if score > best.get("_score", -1):
126+ best = {"_score": score, "obj": o}
127+ o = best.get("obj") or {}
128+ out: dict = {}
129+ loc = o.get("location") or {}
130+ if loc.get("latitude") and loc.get("longitude"):
131+ out["lat"] = loc["latitude"]
132+ out["lng"] = loc["longitude"]
133+ desc = (o.get("redacted_description") or {}).get("text") or ""
134+ if desc:
135+ out["description"] = desc[:6000]
136+ lt = o.get("location_text") or {}
137+ if lt.get("text"):
138+ out["location_text"] = lt["text"]
139+ photos = []
140+ for p in o.get("listing_photos") or []:
141+ uri = ((p.get("image") or {}).get("uri")) or ""
142+ if uri and uri not in photos:
143+ photos.append(uri)
144+ if photos:
145+ out["images"] = photos
146+ return out
147+
148+ def _search(self, slug: str) -> dict:
149+ found: dict = {}
150+ for lo, hi in PRICE_BANDS:
151+ url = (f"{BASE}/{slug}/propertyrentals"
152+ f"?minPrice={lo}&maxPrice={hi}&sortBy=creation_time_descend")
153+ try:
154+ html = self._scrape(url)
155+ except Exception:
156+ continue
157+ for lid, obj in _walk_listings(html).items():
158+ found.setdefault(lid, obj)
159+ return found
160+
161+ def fetch(self) -> list[Listing]:
162+ if not os.environ.get("SCRAPFLY_KEY"):
163+ raise RuntimeError("SCRAPFLY_KEY manquant (voir .env)")
164+ cities = dict(CITIES)
165+ if CITY_LIMIT:
166+ wanted = {c.strip() for c in CITY_LIMIT.split(",")}
167+ cities = {k: v for k, v in CITIES.items() if k in wanted}
168+
169+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS,
170+ key="v1", fetch_html=self._scrape)
171+ out: dict[str, Listing] = {}
172+ try:
173+ for slug, city in cities.items():
174+ for lid, obj in self._search(slug).items():
175+ if lid in out:
176+ continue
177+ title = obj.get("marketplace_listing_title") or ""
178+ price = _price(obj)
179+ if price is None:
180+ continue # sans prix mensuel valable : ignorer
181+ loc = obj.get("location") or {}
182+ coords = loc.get("reverse_geocode") or {}
183+ obj_city = (coords.get("city")
184+ or (coords.get("city_page") or {}).get("display_name")
185+ or city)
186+ photo = ((obj.get("primary_listing_photo") or {})
187+ .get("image") or {}).get("uri") or ""
188+ url = f"{BASE}/item/{lid}/"
189+
190+ detail = cache.get(lid, url, self._parse_detail) or {}
191+ desc = detail.get("description") or ""
192+ lst = Listing(
193+ source=self.source_id,
194+ external_id=lid,
195+ url=url,
196+ title=title[:200],
197+ city=obj_city,
198+ unit_type=_unit_type(title, desc),
199+ price=price,
200+ price_label=f"{price:,.0f} $/mois".replace(",", " "),
201+ description=desc,
202+ images=detail.get("images") or ([photo] if photo else []),
203+ lat=detail.get("lat"),
204+ lng=detail.get("lng"),
205+ )
206+ if obj.get("is_sold") or obj.get("is_pending"):
207+ continue # déjà loué / en attente
208+ out[lid] = lst
209+ finally:
210+ cache.close()
211+ return list(out.values())
added louka/connectors/groupe_maurice.py +98 −0
@@ -0,0 +1,98 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/groupe_maurice.py : Le Groupe Maurice — 37 résidences pour aînés
5+# (RPA) au Québec (Montréal, Laval, Rive-Sud/Nord, Québec, Gatineau…).
6+# WordPress avec API REST maison : /wp-json/api/v1/residences retourne les
7+# 37 résidences géolocalisées (adresse, lat/lng, ville, url, vignette).
8+# Chaque page résidence (SSR) expose la grille des types d'unités dans des
9+# <article class="y-bordered"> : « Studio | 408 pi.ca à 628 pi.ca |
10+# À partir de 2 286 $ par mois ». Une annonce Lou-Ka par type × résidence,
11+# prix plancher « à partir de », étiquetée Résidence pour aînés.
12+# -----------------------------------------------------------------------------
13+from __future__ import annotations
14+
15+import re
16+
17+from bs4 import BeautifulSoup
18+
19+from ..schema import Listing, normalize_unit_type
20+from .base import BaseConnector
21+
22+API = "https://www.legroupemaurice.com/wp-json/api/v1/residences"
23+
24+_ROW_RE = re.compile(
25+ r"^(?P<type>.+?)\s+(?P<a1>[\d\s]+)\s*pi\.?\s*ca(?:\s*à\s*(?P<a2>[\d\s]+)\s*pi\.?\s*ca)?"
26+ r"\s*À partir de\s*(?P<prix>[\d\s]+)\s*\$\s*par mois", re.S)
27+_TAG_RE = re.compile(r"<[^>]+>")
28+
29+
30+class GroupeMauriceConnector(BaseConnector):
31+ source_id = "groupe_maurice"
32+ request_delay = 0.6
33+
34+ def _rows(self, page_html: str) -> list[dict]:
35+ soup = BeautifulSoup(page_html, "html.parser")
36+ out = []
37+ for art in soup.find_all("article"):
38+ if "y-bordered" not in (art.get("class") or []):
39+ continue
40+ text = re.sub(r"\s+", " ", art.get_text(" ", strip=True))
41+ m = _ROW_RE.match(text)
42+ if not m:
43+ continue
44+ try:
45+ price = float(m.group("prix").replace(" ", ""))
46+ area = float(m.group("a1").replace(" ", ""))
47+ except ValueError:
48+ continue
49+ out.append({"type": m.group("type").strip(),
50+ "price": price, "area": area})
51+ return out
52+
53+ def fetch(self) -> list[Listing]:
54+ data = self.get(API).json()
55+ residences = (data.get("result") or {})
56+ if isinstance(residences, dict):
57+ residences = list(residences.values())
58+ out: list[Listing] = []
59+ for res in residences:
60+ url = res.get("url") or ""
61+ if not url:
62+ continue
63+ slug = url.rstrip("/").rsplit("/", 1)[-1]
64+ try:
65+ rows = self._rows(self.get(url).text)
66+ except Exception:
67+ continue
68+ desc = _TAG_RE.sub("", res.get("excerpt") or "").strip()
69+ try:
70+ lat, lng = float(res.get("latitude")), float(res.get("longitude"))
71+ except (TypeError, ValueError):
72+ lat = lng = None
73+ for row in rows:
74+ raw_type = row["type"]
75+ unit_type = normalize_unit_type(raw_type)
76+ details = {"Résidence pour aînés": "oui"}
77+ if "den" in raw_type.lower() or "+" in raw_type:
78+ details["Type d'unité (résidence)"] = raw_type
79+ eid = f"{slug}-{re.sub(r'[^a-z0-9]+', '-', raw_type.lower()).strip('-')}"
80+ out.append(Listing(
81+ source=self.source_id,
82+ external_id=eid,
83+ url=url,
84+ title=f"{raw_type} — Résidence {res.get('residence', slug)}",
85+ address=res.get("address") or "",
86+ city=res.get("city") or "",
87+ unit_type=unit_type,
88+ price=row["price"],
89+ price_label=f"À partir de {row['price']:,.0f} $ par mois"
90+ .replace(",", " "),
91+ area_sqft=row["area"],
92+ description=desc,
93+ details=details,
94+ images=[res["thumbnail"]] if res.get("thumbnail") else [],
95+ lat=lat,
96+ lng=lng,
97+ ))
98+ return out
added louka/connectors/logisquebec.py +222 −0
@@ -0,0 +1,222 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/logisquebec.py : LogisQuébec (logisquebec.com) — portail locatif
5+# québécois (Réglisse Média). ~9 800 fiches location, province entière.
6+# Découverte par les sitemaps (99,7 % du parc vérifié vs compteurs des pages
7+# liste) : URLs de fiche « {type}-a-louer-{ville}-l###### ». Chaque fiche
8+# (SSR, Cloudflare tolérant) expose un JSON-LD complet : SingleFamilyResidence
9+# (adresse + code postal + GPS + téléphone) et Product (nom, description,
10+# offers.price, sku = id). Galerie lightGallery, disponibilité et
11+# caractéristiques actives (span .caracteristique-texte) dans le HTML.
12+# Fiche expirée -> HTTP 410 Gone (retrait immédiat). Cache détail avec TTL
13+# (les prix changent) via _detailutil.TtlDetailCache.
14+# -----------------------------------------------------------------------------
15+from __future__ import annotations
16+
17+import os
18+import re
19+
20+from bs4 import BeautifulSoup
21+
22+from ..schema import Listing, normalize_unit_type
23+from .base import BaseConnector
24+from . import _detailutil as du
25+
26+BASE = "https://www.logisquebec.com"
27+
28+# type d'URL -> (type d'unité par défaut, court terme ?) ; les slugs
29+# commerciaux (local-commercial, bureau, entrepot-industriel…) sont exclus
30+TYPES = {
31+ "appartement": ("", False),
32+ "condo": ("", False),
33+ "maison": ("Maison", False),
34+ "studio": ("Studio", False),
35+ "loft": ("Loft", False),
36+ "chambre-et-colocation": ("Chambre", False),
37+ "chalet": ("Chalet", False),
38+ "appartement-condo-court-terme": ("", True),
39+ "maison-court-terme": ("Maison", True),
40+ "chambre-court-terme": ("Chambre", True),
41+}
42+
43+_FICHE_RE = re.compile(
44+ r"^https://www\.logisquebec\.com/(" + "|".join(map(re.escape, TYPES)) +
45+ r")-a-louer-([a-z0-9_-]+)-l(\d+)$")
46+_LOC_RE = re.compile(r"<loc>(.*?)</loc>")
47+_PHOTO_RE = re.compile(r"https://i\.logisquebec\.com/i-a-louer/\d+/\d+/[^'\"]+")
48+_DISPO_RE = re.compile(r"Disponible à partir de\s*:?\s*\|\s*([^|]{1,60})")
49+
50+DETAIL_LIMIT = int(os.environ.get("LOUKA_LOGISQUEBEC_DETAIL_LIMIT", "1200"))
51+TTL_DAYS = float(os.environ.get("LOUKA_LOGISQUEBEC_TTL_DAYS", "7"))
52+MAX_FICHES = int(os.environ.get("LOUKA_LOGISQUEBEC_MAX", "0")) # 0 = tout
53+
54+
55+def _parse_fiche(html: str) -> dict:
56+ """Extrait le payload complet d'une page fiche LogisQuébec."""
57+ out: dict = {}
58+ for node in du.ld_nodes(html):
59+ t = node.get("@type") or ""
60+ if t == "SingleFamilyResidence":
61+ addr = node.get("address") or {}
62+ out["address"] = (addr.get("streetAddress") or "").strip()
63+ out["locality"] = (addr.get("addressLocality") or "").strip()
64+ out["postal_code"] = (addr.get("postalCode") or "").strip()
65+ geo = node.get("geo") or {}
66+ try:
67+ out["lat"] = float(geo.get("latitude"))
68+ out["lng"] = float(geo.get("longitude"))
69+ except (TypeError, ValueError):
70+ pass
71+ if node.get("telephone"):
72+ out["telephone"] = str(node["telephone"]).strip()
73+ rooms = node.get("numberOfRooms")
74+ if isinstance(rooms, (int, float)) and rooms > 0:
75+ out["rooms"] = int(rooms)
76+ elif t == "Product":
77+ out["title"] = (node.get("name") or "").strip()
78+ out["description"] = (node.get("description") or "").strip()[:6000]
79+ offers = node.get("offers") or {}
80+ try:
81+ out["price"] = float(str(offers.get("price")).replace(",", "."))
82+ except (TypeError, ValueError):
83+ pass
84+ if node.get("sku"):
85+ out["sku"] = str(node["sku"])
86+
87+ photos, seen = [], set()
88+ for u in _PHOTO_RE.findall(html):
89+ if "/thumbnail/" in u or u in seen:
90+ continue
91+ seen.add(u)
92+ photos.append(u)
93+ if photos:
94+ out["images"] = photos
95+
96+ # « Disponible à partir de : Immédiatement / 1er juillet… » (texte aplati)
97+ flat = du.flatten(html)
98+ m = _DISPO_RE.search(flat)
99+ if m:
100+ out["availability"] = m.group(1).strip(" .")
101+
102+ # caractéristiques ACTIVES = items possédant un span .caracteristique-texte
103+ # (les items sans texte sont la légende inactive des icônes)
104+ soup = BeautifulSoup(html, "html.parser")
105+ amenities = []
106+ for it in soup.select(".caracteristique-item"):
107+ if it.select_one(".caracteristique-texte") is None:
108+ continue
109+ label = it.get_text(" ", strip=True)
110+ label = re.sub(r"\s+", " ", label)
111+ # le titre est dupliqué dans le texte : « 2 chambres 2 chambres »
112+ half = len(label) // 2
113+ if len(label) % 2 == 1 and label[:half] == label[half + 1:]:
114+ label = label[:half]
115+ if label and label not in amenities:
116+ amenities.append(label)
117+ if amenities:
118+ out["amenities"] = amenities
119+ return out
120+
121+
122+class LogisQuebecConnector(BaseConnector):
123+ source_id = "logisquebec"
124+ request_delay = 0.7
125+
126+ # -- découverte ------------------------------------------------------------
127+ def _fiche_urls(self) -> list[tuple[str, str, str, str]]:
128+ """(url, type_slug, ville_slug, id) des fiches location résidentielles."""
129+ index = self.get(f"{BASE}/sitemap.xml").text
130+ subs = [u for u in _LOC_RE.findall(index) if "/sitemap/" in u]
131+ out, seen = [], set()
132+ for sub in subs:
133+ try:
134+ xml = self.get(sub).text
135+ except Exception:
136+ continue
137+ for u in _LOC_RE.findall(xml):
138+ m = _FICHE_RE.match(u)
139+ if m and m.group(3) not in seen:
140+ seen.add(m.group(3))
141+ out.append((u, m.group(1), m.group(2), m.group(3)))
142+ return out
143+
144+ # -- construction ----------------------------------------------------------
145+ def _to_listing(self, url: str, type_slug: str, payload: dict) -> Listing | None:
146+ if not payload or payload.get("gone"):
147+ return None
148+ eid = payload.get("sku") or url.rsplit("-l", 1)[-1]
149+ default_type, court_terme = TYPES[type_slug]
150+
151+ title = payload.get("title") or ""
152+ unit_type = normalize_unit_type(title)
153+ if not re.match(r"^\d½|^6½\+$", unit_type or ""):
154+ unit_type = ""
155+ if not unit_type and payload.get("rooms"):
156+ n = payload["rooms"]
157+ unit_type = "6½+" if n >= 6 else f"{n}½"
158+ if not unit_type: # « 2 chambres » des caractéristiques
159+ for a in payload.get("amenities") or []:
160+ if re.match(r"^\d+ chambres?$", a):
161+ unit_type = normalize_unit_type(a)
162+ break
163+ if not unit_type:
164+ unit_type = default_type
165+
166+ locality = payload.get("locality") or ""
167+ m = re.match(r"^(.*?)\s*\((.+)\)\s*$", locality)
168+ sector, city = (m.group(1), m.group(2)) if m else ("", locality)
169+
170+ price = payload.get("price")
171+ details: dict = {}
172+ if payload.get("postal_code"):
173+ details["Code postal"] = payload["postal_code"]
174+ if payload.get("telephone"):
175+ details["Téléphone"] = payload["telephone"]
176+ if court_terme:
177+ details["Location court terme"] = "oui"
178+
179+ amenities = payload.get("amenities") or []
180+ lst = Listing(
181+ source=self.source_id,
182+ external_id=str(eid),
183+ url=url,
184+ title=title,
185+ address=payload.get("address") or "",
186+ sector=sector,
187+ city=city,
188+ unit_type=unit_type,
189+ price=price,
190+ price_label=(f"{price:,.0f} $/mois".replace(",", " ") if price else ""),
191+ availability=payload.get("availability") or "",
192+ description=payload.get("description") or "",
193+ amenities=amenities,
194+ details=details,
195+ images=payload.get("images") or [],
196+ lat=payload.get("lat"),
197+ lng=payload.get("lng"),
198+ )
199+ low = [a.lower() for a in amenities]
200+ if any("animaux accept" in a for a in low):
201+ lst.pets = "oui"
202+ if "meublé" in low:
203+ lst.furnished = True
204+ return lst
205+
206+ def fetch(self) -> list[Listing]:
207+ fiches = self._fiche_urls()
208+ if MAX_FICHES:
209+ fiches = fiches[:MAX_FICHES]
210+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS)
211+ out: list[Listing] = []
212+ try:
213+ for url, type_slug, _ville, eid in fiches:
214+ payload = cache.get(eid, url, _parse_fiche)
215+ if payload is None: # jamais visitée + budget épuisé :
216+ continue # captée à une prochaine synchronisation
217+ lst = self._to_listing(url, type_slug, payload)
218+ if lst is not None:
219+ out.append(lst)
220+ finally:
221+ cache.close()
222+ return out
added louka/connectors/louer_ca.py +312 −0
@@ -0,0 +1,312 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/louer_ca.py : Louer.ca — portail locatif 100 % Québec du réseau
5+# Rentals.ca. Accès par l'API GraphQL interne (https://louer.ca/graphql),
6+# rétro-conçue : clé publique `rentalsGqlKey` lue dans window.appconf de la
7+# page d'accueil, mutation `acquireAuthInfo` -> JWT (scalaire JSON contenant
8+# accessToken ~1 h + refreshToken), en-tête `Authorization: Bearer <access>`.
9+# La recherche passe par la ville : typeahead -> City.id, puis
10+# `node(id){... on City{ rentalListings(first,after) }}` (curseur Relay ;
11+# le champ rentalListings top-level renvoie 0). Le détail `node(id)` fournit
12+# adresse, description, galerie (images.rentals.ca) et les floorPlans
13+# (un type/prix par plan) -> une annonce Lou-Ka par plan d'étage.
14+# ⚠️ Réseau Rentals.ca : CGU interdisant l'extraction sans accord écrit.
15+# -----------------------------------------------------------------------------
16+from __future__ import annotations
17+
18+import base64
19+import datetime
20+import json
21+import os
22+import re
23+import time
24+
25+from ..schema import Listing
26+from .base import BaseConnector
27+from . import _detailutil as du
28+
29+HOME = "https://louer.ca/"
30+GQL = "https://louer.ca/graphql"
31+FALLBACK_KEY = "ME8N-J3IX-At86-2yIi"
32+
33+# villes québécoises visées (slug interne Louer.ca) ; résolues via typeahead
34+CITIES = ["montreal", "quebec", "laval", "gatineau", "longueuil", "sherbrooke",
35+ "trois-rivieres", "levis", "terrebonne", "brossard", "saguenay",
36+ "drummondville", "granby", "saint-jean-sur-richelieu", "repentigny"]
37+
38+PAGE_SIZE = int(os.environ.get("LOUKA_LOUERCA_PAGE_SIZE", "50"))
39+DETAIL_LIMIT = int(os.environ.get("LOUKA_LOUERCA_DETAIL_LIMIT", "600"))
40+TTL_DAYS = float(os.environ.get("LOUKA_LOUERCA_TTL_DAYS", "5"))
41+MAX_PER_CITY = int(os.environ.get("LOUKA_LOUERCA_MAX_PER_CITY", "0")) # 0 = tout
42+
43+# amenity (catégorie, valeur) -> libellé FR affichable
44+_AMENITY_FR = {
45+ "laundry-facilities": "Buanderie", "in-suite-laundry": "Laveuse/sécheuse dans l'unité",
46+ "security-on-site": "Sécurité sur place", "storage-lockers": "Espace de rangement",
47+ "swimming-pool": "Piscine", "gym": "Salle d'entraînement", "elevator": "Ascenseur",
48+ "sauna": "Sauna", "on-site-staff": "Personnel sur place", "stove": "Cuisinière",
49+ "fridge": "Réfrigérateur", "balcony": "Balcon", "microwave": "Micro-ondes",
50+ "dishwasher": "Lave-vaisselle", "air-conditioning": "Air climatisé",
51+ "individual-thermostats": "Thermostats individuels", "heating": "Chauffage inclus",
52+ "water": "Eau incluse", "hydro-electricity": "Électricité incluse",
53+ "public-transit": "Transport en commun à proximité", "parking": "Stationnement",
54+ "no-smoking-allowed": "Non-fumeur",
55+}
56+
57+
58+class LouerCaConnector(BaseConnector):
59+ source_id = "louer_ca"
60+ request_delay = 0.5
61+
62+ def __init__(self) -> None:
63+ super().__init__()
64+ self.session.headers.update({
65+ "Content-Type": "application/json",
66+ "Origin": "https://louer.ca",
67+ "Referer": "https://louer.ca/",
68+ })
69+ self._token = ""
70+ self._token_time = 0.0
71+ self._api_key = FALLBACK_KEY
72+
73+ # -- auth ------------------------------------------------------------------
74+ def _ensure_token(self) -> None:
75+ if self._token and time.time() - self._token_time < 2700: # ~45 min
76+ return
77+ try:
78+ home = self.get(HOME).text
79+ m = re.search(r'"rentalsGqlKey":\s*"([^"]+)"', home)
80+ if m:
81+ self._api_key = m.group(1)
82+ except Exception:
83+ pass
84+ data = self._gql(
85+ "mutation($k:String!){acquireAuthInfo(credentials:{apiKey:$k}){jwt status}}",
86+ {"k": self._api_key}, auth=False)
87+ auth = (data or {}).get("acquireAuthInfo") or {}
88+ jwt = auth.get("jwt")
89+ if isinstance(jwt, str) and jwt.startswith("{"):
90+ jwt = json.loads(jwt)
91+ token = jwt.get("accessToken") if isinstance(jwt, dict) else jwt
92+ if not token:
93+ raise RuntimeError("Louer.ca: handshake JWT échoué")
94+ self._token = token
95+ self._token_time = time.time()
96+
97+ def _gql(self, query: str, variables: dict, auth: bool = True) -> dict:
98+ headers = {}
99+ if auth:
100+ self._ensure_token()
101+ headers["Authorization"] = f"Bearer {self._token}"
102+ resp = self.post(GQL, data=json.dumps({"query": query,
103+ "variables": variables}),
104+ headers=headers)
105+ payload = resp.json()
106+ if payload.get("errors"):
107+ msg = payload["errors"][0].get("message", "")
108+ if "AUTH" in msg.upper() and auth: # jeton expiré : on réessaie
109+ self._token = ""
110+ self._ensure_token()
111+ resp = self.post(GQL, data=json.dumps({"query": query,
112+ "variables": variables}),
113+ headers={"Authorization": f"Bearer {self._token}"})
114+ payload = resp.json()
115+ return payload.get("data") or {}
116+
117+ # -- découverte ------------------------------------------------------------
118+ def _city_id(self, slug: str) -> str | None:
119+ d = self._gql(
120+ "query($v:String!){typeahead(value:$v){nodeType node{id ... on City"
121+ "{name path regionCode listingCount}}}}", {"v": slug.replace("-", " ")})
122+ for it in d.get("typeahead") or []:
123+ node = it.get("node") or {}
124+ if node.get("path") == slug and node.get("regionCode") == "QC":
125+ return node.get("id")
126+ # à défaut, première ville québécoise proposée
127+ for it in d.get("typeahead") or []:
128+ node = it.get("node") or {}
129+ if node.get("id") and node.get("regionCode") == "QC":
130+ return node.get("id")
131+ return None
132+
133+ _LIST_FRAG = ("id name path location rentRange bedsRange bathsRange "
134+ "sizeRange type furnished petOptions amenities verified "
135+ "created modified")
136+
137+ def _city_listings(self, city_id: str) -> list[dict]:
138+ query = ("query($id:ID!,$first:PositiveInt!,$after:String){node(id:$id)"
139+ "{... on City{rentalListings(first:$first,after:$after){"
140+ "meta{totalCount} pageInfo{hasNextPage endCursor} "
141+ "edges{node{" + self._LIST_FRAG + "}}}}}}")
142+ out, after = [], None
143+ while True:
144+ d = self._gql(query, {"id": city_id, "first": PAGE_SIZE,
145+ "after": after})
146+ rl = ((d.get("node") or {}).get("rentalListings")) or {}
147+ edges = rl.get("edges") or []
148+ out.extend(e["node"] for e in edges if e.get("node"))
149+ info = rl.get("pageInfo") or {}
150+ if not info.get("hasNextPage") or not edges:
151+ break
152+ if MAX_PER_CITY and len(out) >= MAX_PER_CITY:
153+ break
154+ after = info.get("endCursor")
155+ return out
156+
157+ _DETAIL_FRAG = ("id name path location "
158+ "address{city{name regionCode} neighbourhood{name} "
159+ "postalCode street} description{plain} "
160+ "imagesCount images{scales} "
161+ "floorPlans{beds baths rent size availability furnished}")
162+
163+ def _detail(self, gid: str) -> dict:
164+ d = self._gql("query($id:ID!){node(id:$id){... on RentalListing{"
165+ + self._DETAIL_FRAG + "}}}", {"id": gid})
166+ return d.get("node") or {}
167+
168+ # -- construction ----------------------------------------------------------
169+ @staticmethod
170+ def _unit_type(beds) -> str:
171+ try:
172+ b = float(beds)
173+ except (TypeError, ValueError):
174+ return ""
175+ if b <= 0:
176+ return "Studio"
177+ n = int(b) + 2 # chambres -> pièces et demie
178+ return "6½+" if n >= 6 else f"{n}½"
179+
180+ @staticmethod
181+ def _amenities(pairs) -> list[str]:
182+ out = []
183+ for pair in pairs or []:
184+ val = pair[1] if isinstance(pair, list) and len(pair) > 1 else None
185+ label = _AMENITY_FR.get(val)
186+ if label and label not in out:
187+ out.append(label)
188+ return out
189+
190+ def _images(self, node: dict) -> list[str]:
191+ imgs = []
192+ for im in node.get("images") or []:
193+ scales = im.get("scales")
194+ if isinstance(scales, str):
195+ try:
196+ scales = json.loads(scales)
197+ except ValueError:
198+ scales = []
199+ best = ""
200+ for sc in scales or []:
201+ if sc.get("name") in ("large", "medium") and sc.get("url"):
202+ best = sc["url"]
203+ if sc["name"] == "large":
204+ break
205+ if not best and scales:
206+ best = scales[0].get("url", "")
207+ if best and best not in imgs:
208+ imgs.append(best)
209+ return imgs
210+
211+ @staticmethod
212+ def _numeric_id(gid: str) -> str:
213+ """« cmVudGFsbGlzdGluZzoxMTMyMTE2 » -> « 1132116 » (rentallisting:1132116)."""
214+ try:
215+ decoded = base64.b64decode(gid + "==").decode("utf-8", "ignore")
216+ m = re.search(r"(\d+)", decoded)
217+ if m:
218+ return m.group(1)
219+ except Exception:
220+ pass
221+ m = re.search(r"(\d+)", gid)
222+ return m.group(1) if m else gid
223+
224+ def fetch(self) -> list[Listing]:
225+ # le « détail » est une requête GraphQL, pas une page HTML : le
226+ # fetch_html du cache reçoit le gid et le renvoie tel quel, parse_fn
227+ # exécute la requête GraphQL.
228+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS,
229+ key="v1", fetch_html=lambda gid: gid)
230+ out: list[Listing] = []
231+ today = datetime.date.today().isoformat()
232+ try:
233+ for slug in CITIES:
234+ cid = self._city_id(slug)
235+ if not cid:
236+ continue
237+ for card in self._city_listings(cid):
238+ gid = card.get("id")
239+ if not gid:
240+ continue
241+ node = cache.get(gid, gid,
242+ lambda g: self._detail(g)) or {}
243+ addr = node.get("address") or {}
244+ city = (addr.get("city") or {}).get("name") or slug.title()
245+ sector = (addr.get("neighbourhood") or {}).get("name") or ""
246+ loc = card.get("location") or []
247+ lng, lat = (loc + [None, None])[:2]
248+ base_desc = ((node.get("description") or {}).get("plain")
249+ or "")[:6000]
250+ images = self._images(node)
251+ amenities = self._amenities(card.get("amenities"))
252+ common = dict(
253+ source=self.source_id, url=f"https://louer.ca/{card.get('path','')}",
254+ address=addr.get("street") or "", sector=sector, city=city,
255+ description=base_desc, amenities=amenities, images=images,
256+ lat=lat, lng=lng,
257+ )
258+ if addr.get("postalCode"):
259+ common["details"] = {"Code postal": addr["postalCode"]}
260+
261+ plans = node.get("floorPlans") or []
262+ if not plans:
263+ # pas de plan détaillé : une annonce « à partir de »
264+ rng = card.get("rentRange") or []
265+ price = rng[0] if rng else None
266+ beds = (card.get("bedsRange") or [None])[0]
267+ out.append(self._mk(common, gid, "", price,
268+ self._unit_type(beds), None, today,
269+ price_from=bool(rng)))
270+ continue
271+ for i, fp in enumerate(plans):
272+ avail = fp.get("availability") or {}
273+ adate = "now" if avail.get("now") else (
274+ avail.get("date") or "")[:10] or None
275+ if adate and adate != "now" and adate <= today:
276+ adate = "now"
277+ lst = self._mk(common, gid, f"-{i}", fp.get("rent"),
278+ self._unit_type(fp.get("beds")),
279+ fp.get("size"), today, adate=adate)
280+ if fp.get("furnished") == "yes":
281+ lst.furnished = True
282+ out.append(lst)
283+ finally:
284+ cache.close()
285+ return out
286+
287+ def _mk(self, common: dict, gid: str, suffix: str, price, unit_type,
288+ size, today, adate=None, price_from=False) -> Listing:
289+ num = self._numeric_id(gid)
290+ p = None
291+ try:
292+ p = float(price) if price is not None else None
293+ except (TypeError, ValueError):
294+ p = None
295+ details = dict(common.get("details") or {})
296+ if price_from:
297+ details["price_from"] = True
298+ lst = Listing(
299+ **{k: v for k, v in common.items() if k != "details"},
300+ external_id=f"{num}{suffix}",
301+ title=common.get("address") or "Logement à louer",
302+ unit_type=unit_type,
303+ price=p,
304+ price_label=(("À partir de " if price_from else "")
305+ + (f"{p:,.0f} $/mois".replace(",", " ") if p else "")),
306+ area_sqft=float(size) if size else None,
307+ availability_date=adate,
308+ availability=("Libre immédiatement" if adate == "now"
309+ else f"Libre le {adate}" if adate else ""),
310+ details=details,
311+ )
312+ return lst
added louka/connectors/omhm.py +86 −0
@@ -0,0 +1,86 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/omhm.py : OMHM — Office municipal d'habitation de Montréal.
5+# Le HLM classique fonctionne par liste d'attente (non agrégeable) ; SEULE
6+# la page « logements disponibles » (résidences abordables ENHARMONIE pour
7+# aînés autonomes 60 ans +) liste de vraies unités. Site Drupal SSR, aucun
8+# anti-bot. Chaque unité = un <article class="node--type-logement-disponible">
9+# avec adresse civique complète et grille de prix par type (« 3 ½ entre
10+# 1 357 $ et 1 497 $ »). Volume faible (~5) mais différenciateur social
11+# (filtre « logement abordable » de Lou-Ka).
12+# -----------------------------------------------------------------------------
13+from __future__ import annotations
14+
15+import re
16+
17+from bs4 import BeautifulSoup
18+
19+from ..schema import Listing, normalize_unit_type
20+from .base import BaseConnector
21+
22+URL = "https://www.omhm.qc.ca/fr/soumettre-une-demande/logements-disponibles"
23+
24+_POSTAL_RE = re.compile(r"[A-Z]\d[A-Z]\s?\d[A-Z]\d")
25+# « 3 ½ entre 1 357 $ et 1 497 $ » ou « studio 730 $ à 925 $ » ou « 4 ½ à 2 167 $ »
26+_UNIT_PRICE_RE = re.compile(
27+ r"(studio|\d\s*½|\d\s*1/2)\s*(?:entre|à|de|:)?\s*"
28+ r"([\d\s]+)\s*\$(?:\s*(?:et|à|-)\s*([\d\s]+)\s*\$)?", re.I)
29+
30+
31+class OmhmConnector(BaseConnector):
32+ source_id = "omhm"
33+ request_delay = 0.8
34+
35+ def fetch(self) -> list[Listing]:
36+ soup = BeautifulSoup(self.get(URL).text, "html.parser")
37+ out: list[Listing] = []
38+ for art in soup.select("article.node--type-logement-disponible"):
39+ title_el = art.find(re.compile("^h[2-4]$"))
40+ name = title_el.get_text(" ", strip=True) if title_el else ""
41+ full = re.sub(r"\s+", " ", art.get_text(" | ", strip=True))
42+
43+ # adresse : « 12100, rue Rodolphe-Forget | Montréal | H1E 7S2 »
44+ m = re.search(r"(\d+[^|]+?)\s*\|\s*([A-ZÉÈÀ][^|]+?)\s*\|\s*"
45+ r"(" + _POSTAL_RE.pattern + r")", full)
46+ address = city = postal = ""
47+ if m:
48+ address = m.group(1).strip(" ,")
49+ city = m.group(2).strip()
50+ postal = m.group(3).strip()
51+
52+ slug = re.sub(r"[^a-z0-9]+", "-",
53+ (name or address).lower()).strip("-") or "omhm"
54+ # une annonce par type d'unité offert dans la résidence
55+ seen = set()
56+ for um in _UNIT_PRICE_RE.finditer(full):
57+ unit_type = normalize_unit_type(um.group(1))
58+ if unit_type in seen:
59+ continue
60+ seen.add(unit_type)
61+ try:
62+ price = float(um.group(2).replace(" ", ""))
63+ except ValueError:
64+ continue
65+ pmax = um.group(3)
66+ label = (f"Entre {price:,.0f} $ et {float(pmax.replace(' ','')):,.0f} $"
67+ .replace(",", " ") if pmax
68+ else f"À partir de {price:,.0f} $".replace(",", " "))
69+ out.append(Listing(
70+ source=self.source_id,
71+ external_id=f"{slug}-{unit_type}",
72+ url=URL,
73+ title=f"{name} — {unit_type}" if name else unit_type,
74+ address=address,
75+ city=city or "Montréal",
76+ unit_type=unit_type,
77+ price=price,
78+ price_label=label,
79+ availability="Disponible",
80+ details={"Code postal": postal,
81+ "Logement abordable": "oui",
82+ "Résidence pour aînés": "oui"} if postal else
83+ {"Logement abordable": "oui",
84+ "Résidence pour aînés": "oui"},
85+ ))
86+ return out
added louka/connectors/roomies.py +182 −0
@@ -0,0 +1,182 @@
1+# -----------------------------------------------------------------------------
2+# Lou-Ka — Agrégateur de logements à louer (province de Québec)
3+# Auteur : Simon-Pierre Boucher — contact@spboucher.ai
4+# connectors/roomies.py : Roomies.ca — chambres et colocation (segment que les
5+# gestionnaires ne couvrent pas). Site Laravel/Livewire server-rendered,
6+# robots.txt entièrement ouvert, aucun anti-bot (curl simple = 200).
7+# Découverte par les pages villes `/{ville}-qc?page=N` (les sitemaps
8+# `sitemap/rooms.xml` répondent 504 par intermittence — non fiables).
9+# Fiche `/rooms/<id>` : JSON-LD `Room` complet (offers.price mensuel,
10+# availabilityStarts, lat/lng, petsAllowed, chambres/SDB de l'appartement,
11+# PostalAddress, amenityFeature) + description intégrale dans un div
12+# `prose` + galerie Cloudinary. On ignore les profils de chercheurs de
13+# colocataires (/profiles) : seules les chambres offertes sont des annonces.
14+# -----------------------------------------------------------------------------
15+from __future__ import annotations
16+
17+import datetime
18+import os
19+import re
20+
21+from bs4 import BeautifulSoup
22+
23+from ..schema import Listing
24+from .base import BaseConnector
25+from . import _detailutil as du
26+
27+BASE = "https://www.roomies.ca"
28+
29+# villes québécoises servies (slug -qc) ; une page absente (404) est ignorée
30+CITIES = {
31+ "montreal-qc": "Montréal", "quebec-qc": "Québec", "laval-qc": "Laval",
32+ "gatineau-qc": "Gatineau", "longueuil-qc": "Longueuil",
33+ "sherbrooke-qc": "Sherbrooke", "trois-rivieres-qc": "Trois-Rivières",
34+ "levis-qc": "Lévis", "brossard-qc": "Brossard",
35+ "terrebonne-qc": "Terrebonne", "repentigny-qc": "Repentigny",
36+ "saint-jean-sur-richelieu-qc": "Saint-Jean-sur-Richelieu",
37+ "drummondville-qc": "Drummondville", "granby-qc": "Granby",
38+ "saguenay-qc": "Saguenay",
39+}
40+MAX_PAGES = int(os.environ.get("LOUKA_ROOMIES_MAX_PAGES", "20"))
41+DETAIL_LIMIT = int(os.environ.get("LOUKA_ROOMIES_DETAIL_LIMIT", "250"))
42+TTL_DAYS = float(os.environ.get("LOUKA_ROOMIES_TTL_DAYS", "5"))
43+
44+_ROOM_LINK_RE = re.compile(r'href="(?:https://www\.roomies\.ca)?(/rooms/(\d+))"')
45+_PHOTO_RE = re.compile(
46+ r'https://cloudinary\.roomies\.pics/image/upload/[^"\\\s]+?/(\w+)')
47+
48+
49+def _parse_fiche(html: str) -> dict:
50+ out: dict = {}
51+ for node in du.ld_nodes(html):
52+ if node.get("@type") != "Room":
53+ continue
54+ name = str(node.get("name") or "")
55+ out["type_label"] = name.split("|")[0].strip() # « Furnished room in… »
56+ addr = node.get("address") or {}
57+ out["address"] = (addr.get("streetAddress") or "").strip()
58+ out["postal_code"] = (addr.get("postalCode") or "").strip()
59+ out["lat"] = node.get("latitude")
60+ out["lng"] = node.get("longitude")
61+ if node.get("petsAllowed") is not None:
62+ out["pets"] = "oui" if node["petsAllowed"] else "non"
63+ details = {}
64+ if node.get("numberOfBedrooms"):
65+ details["Chambres (logement)"] = str(node["numberOfBedrooms"])
66+ if node.get("numberOfBathroomsTotal"):
67+ details["Salles de bain"] = str(node["numberOfBathroomsTotal"])
68+ if details:
69+ out["details"] = details
70+ feats = node.get("amenityFeature") or []
71+ amenities = []
72+ for f in feats:
73+ if isinstance(f, dict) and f.get("name") and f.get("value") is not False:
74+ amenities.append(str(f["name"]))
75+ if f["name"] == "Furnished":
76+ out["furnished"] = True
77+ if amenities:
78+ out["amenities"] = amenities
79+ offers = node.get("offers") or {}
80+ try:
81+ out["price"] = float(offers.get("price"))
82+ except (TypeError, ValueError):
83+ pass
84+ starts = str(offers.get("availabilityStarts") or "")[:10]
85+ if starts:
86+ out["availability_date"] = ("now" if starts
87+ <= datetime.date.today().isoformat()
88+ else starts)
89+ if node.get("description"):
90+ out["description"] = str(node["description"]).strip()[:6000]
91+
92+ # description intégrale (le JSON-LD est tronqué à ~250 caractères)
93+ soup = BeautifulSoup(html, "html.parser")
94+ prose = soup.find("div", class_=lambda c: c and "prose" in c
95+ and "break-words" in c)
96+ if prose:
97+ full = prose.get_text("\n", strip=True)
98+ if len(full) > len(out.get("description") or ""):
99+ out["description"] = full[:6000]
100+
101+ # galerie Cloudinary : dédupliquée par identifiant public, format standard
102+ photos, seen = [], set()
103+ for pid in _PHOTO_RE.findall(html):
104+ if pid in seen or len(pid) < 12:
105+ continue
106+ seen.add(pid)
107+ photos.append("https://cloudinary.roomies.pics/image/upload/"
108+ f"c_fill,f_jpg,fl_lossy,g_auto,h_512,q_auto:good,w_896/{pid}")
109+ if photos:
110+ out["images"] = photos
111+ return out
112+
113+
114+class RoomiesConnector(BaseConnector):
115+ source_id = "roomies"
116+ request_delay = 0.8
117+
118+ def _city_room_ids(self, slug: str) -> list[str]:
119+ ids, page = [], 1
120+ while page <= MAX_PAGES:
121+ url = f"{BASE}/{slug}" + (f"?page={page}" if page > 1 else "")
122+ try:
123+ html = self.get(url).text
124+ except Exception:
125+ break
126+ found = [rid for _, rid in _ROOM_LINK_RE.findall(html)]
127+ fresh = [r for r in found if r not in ids]
128+ if not fresh:
129+ break
130+ ids.extend(fresh)
131+ page += 1
132+ return ids
133+
134+ def fetch(self) -> list[Listing]:
135+ cache = du.TtlDetailCache(self, budget=DETAIL_LIMIT, ttl_days=TTL_DAYS)
136+ out: dict[str, Listing] = {}
137+ try:
138+ for slug, city in CITIES.items():
139+ for rid in self._city_room_ids(slug):
140+ if rid in out:
141+ continue
142+ url = f"{BASE}/rooms/{rid}"
143+ payload = cache.get(rid, url, _parse_fiche)
144+ if not payload or payload.get("gone"):
145+ continue
146+ price = payload.get("price")
147+ avail = payload.get("availability_date")
148+ type_label = (payload.get("type_label") or "").lower()
149+ unit_type = ("Studio" if "studio" in type_label
150+ and "shared" not in type_label else "Chambre")
151+ lst = Listing(
152+ source=self.source_id,
153+ external_id=rid,
154+ url=url,
155+ title=(payload.get("type_label") or "Chambre")
156+ + (f" — {payload['address']}" if payload.get("address") else ""),
157+ address=payload.get("address") or "",
158+ city=city,
159+ unit_type=unit_type,
160+ price=price,
161+ price_label=(f"{price:,.0f} $/mois".replace(",", " ")
162+ if price else ""),
163+ availability_date=avail,
164+ availability=("Libre immédiatement" if avail == "now"
165+ else f"Libre le {avail}" if avail else ""),
166+ description=payload.get("description") or "",
167+ amenities=payload.get("amenities") or [],
168+ details=payload.get("details") or {},
169+ images=payload.get("images") or [],
170+ lat=payload.get("lat"),
171+ lng=payload.get("lng"),
172+ )
173+ if payload.get("postal_code"):
174+ lst.details["Code postal"] = payload["postal_code"]
175+ if payload.get("pets"):
176+ lst.pets = payload["pets"]
177+ if payload.get("furnished"):
178+ lst.furnished = True
179+ out[rid] = lst
180+ finally:
181+ cache.close()
182+ return list(out.values())
modified louka/db.py +2 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added louka/dedup.py +273 −0
Large diff (273 lines) is not loaded by default to keep the page fast.Load diff
modified louka/ingest.py +6 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
modified louka/web.py +18 −7
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere