SPB Git
spboucher.ai

spb/home-ka

Public

Home-Ka v1 — US real-estate aggregator (Immo-Ka architecture): RESO-aligned schema, PROPERTY/LISTING separation, connector families (reso/rets/xml/json/jsonld/csv/sftp/arcgis), brokerage discovery pipeline (630 seeds), admin sources/connectors/brokerages, EN/USD frontend (forest green), deployed M4M36:8099 + ngrok www.home-ka.com

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Aug 28, 2026) Browse files

60 changed files +20,001 −0

60 changed files
Load all 60 files Unified Split
added .env.example +17 −0
@@ -0,0 +1,17 @@
1 +# Home-Ka — credentials (copy to .env; NEVER commit .env)
2 +# Admin API token (optional — /api/admin/* is open when unset)
3 +HOMEKA_ADMIN_TOKEN=
4 +
5 +# Anti-bot fallback chain (optional, LAST RESORT — see connectors/_resilient.py)
6 +SCRAPFLY_KEY=
7 +FIRECRAWL_API_KEY=
8 +OXYLABS_PROXY=
9 +OXYLABS_PROXY_USER=
10 +OXYLABS_PROXY_PASS=
11 +BRIGHTDATA_API_KEY=
12 +BRIGHTDATA_ZONE=
13 +
14 +# RESO / RETS feed credentials, referenced from source configs as $ENV:NAME
15 +# MLSGRID_TOKEN=
16 +# TRESTLE_CLIENT_ID=
17 +# TRESTLE_CLIENT_SECRET=
added .gitignore +9 −0
@@ -0,0 +1,9 @@
1 +__pycache__/
2 +*.pyc
3 +.venv/
4 +node_modules/
5 +frontend/dist/
6 +frontend/*.tsbuildinfo
7 +data/homeka.db*
8 +.env
9 +.DS_Store
added README.md +78 −0
@@ -0,0 +1,78 @@
1 +<h1 align="center">Home·Ka</h1>
2 +
3 +<p align="center"><b>US real-estate aggregator — every home for sale, from direct feeds first.</b><br>
4 +<a href="https://www.home-ka.com">www.home-ka.com</a> · Groupe KA · US counterpart of <a href="https://www.immo-ka.com">Immo·Ka</a> (Canada)</p>
5 +
6 +**Home-Ka** reuses the Immo-Ka architecture end to end — connectors, normalization, ingestion, property matching, deduplication, quality gate, search API, Ka Maps frontend — adapted to the US market: **RESO Data Dictionary** as the schema reference, USD/sq ft/USPS states, US Census geocoder, and a **brokerage partnership pipeline** designed to convert sources into direct feeds instead of crawling.
7 +
8 +## Pipeline (same philosophy as Immo-Ka)
9 +
10 +```
11 +SOURCE → connector → raw ingestion → normalization → property matching
12 + → deduplication → canonical listing → search index → Home-Ka
13 +```
14 +
15 +- **PROPERTY ≠ LISTING** — a `Listing` is one publication by one source; a `Property` (`properties` table, keyed by APN or normalized address) is the physical asset and keeps existing after the listing goes off-market. Every listing is matched on ingest (`homeka/propertymatch.py`).
16 +- **Change detection** — content hash per listing, price history (`price_log`), grace period (2 syncs) before deactivation, drift detection (volume/price collapse suspends removals).
17 +- **Dedup** — precomputed (`dup_hidden`/`dup_of`): same MLS number across sources, or same property at ±1% price; most authoritative source wins (`sources.authority`: direct feed < partner API < site < classifieds).
18 +- **Quality gate** — completeness score, quarantine below threshold (`published=0`).
19 +- **Geocoding** — US Census batch geocoder (free, 5k/req) + budgeted Nominatim fallback, persistent cache.
20 +
21 +## Connector families (`homeka/connectors/`)
22 +
23 +| Family | Module | Use |
24 +|---|---|---|
25 +| `reso` | `reso/webapi.py` | **RESO Web API (OData)** — MLS Grid, Trestle, Bridge, Spark… the preferred feed |
26 +| `rets` | `rets/client.py` | legacy RETS (login + DMQL2, COMPACT-DECODED) |
27 +| `xml` | `xml/feed.py` | XML exports / syndication feeds |
28 +| `json` | `json/feed.py` | JSON partner APIs, wp-json, internal search APIs |
29 +| `jsonld` | `json/jsonld_site.py` | schema.org JSON-LD site crawler (bootstrap only) |
30 +| `csv` | `csv/feed.py` | CSV exports |
31 +| `sftp` | `sftp/drop.py` | SFTP drops (csv/xml), newest file by mtime |
32 +| `arcgis` | `public_data/arcgis.py` | county assessor/GIS parcels → feeds **properties**, not listings |
33 +| custom | `custom_broker/*.py` | one class per site when a family can't express it |
34 +
35 +A new source of a known family = **one row in the `sources` table** (`config` JSON: URL, field mapping, `$ENV:NAME` credentials), zero code. Custom connectors auto-register via `source_id`, exactly like Immo-Ka. Anti-bot policy: direct requests first; auto-escalation chain (Oxylabs → Scrapfly ASP → Bright Data, `connectors/_resilient.py`) only when blocked.
36 +
37 +## Brokerage pipeline (`/admin/brokerages`)
38 +
39 +```
40 +discover brokerage → inspect website → detect IDX/feed/provider
41 +→ find public contacts → classify connector → score → admin
42 +```
43 +
44 +- Seed: **630 real US brokerages** (`data/brokerages_seed.json`, `python run.py seed`).
45 +- `homeka/discovery.py` inspects each website (direct GET): IDX platform fingerprints (kvCORE, iHomefinder, IDX Broker, Sierra, REW, Delta Media, Ylopo…), RESO/RETS hints, MLS affiliations, contact/partnership emails; then scores `feed_probability_score` and `priority_score` (size × feed probability × coverage).
46 +- Partnership statuses: prospect → to-contact → contacted → in-discussion → feed-received → **live** (linked to a `sources` row).
47 +
48 +## Run
49 +
50 +```bash
51 +python run.py serve 8099 # API + frontend
52 +python run.py sync [source ...] # one-shot ingestion
53 +python run.py watch 60 # sync loop + geocode + img audit + discovery
54 +python run.py seed # load brokerage seed
55 +python run.py discover 25 # inspect next prospects
56 +python run.py list | geocode | quality | dedup | rescore
57 +```
58 +
59 +Admin: `/admin` (overview), `/admin/sources`, `/admin/connectors`, `/admin/brokerages` — optional `HOMEKA_ADMIN_TOKEN` (.env) required as `X-Admin-Token`.
60 +
61 +## Deployment
62 +
63 +- Node **M4M36** (`~/apps/home-ka`), port **8099**, PM2 `home-ka-web` + `home-ka-sync` + `home-ka-ngrok` (`--url=www.home-ka.com`).
64 +- Frontend: Vite build (`frontend/`, ka-maps vendored in `vendor/ka-maps`).
65 +- Git: `origin` = spbgit (`gitsrv:srv/git/home-ka.git`).
66 +
67 +## Adding a RESO feed (example)
68 +
69 +```python
70 +from homeka import db
71 +con = db.connect()
72 +db.upsert_source(con, "mlsgrid_actris", "MLS Grid — ACTRIS", "reso", config={
73 + "base_url": "https://api.mlsgrid.com/v2",
74 + "token": "$ENV:MLSGRID_TOKEN",
75 + "filter": "StandardStatus eq 'Active'",
76 + "expand": "Media", "originating_system": "ACTRIS",
77 +}, authority=0, states=["TX"])
78 +```
added data/brokerages_seed.json +10059 −0
Large diff (10,060 lines) is not loaded by default to keep the page fast.Load diff
added frontend/index.html +33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added frontend/package-lock.json +1874 −0
Large diff (1,874 lines) is not loaded by default to keep the page fast.Load diff
added frontend/package.json +26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added frontend/public/apple-touch-icon.png +0 −0

Binary file not shown.

added frontend/public/favicon.svg +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added frontend/public/og.png +0 −0

Binary file not shown.

added frontend/scripts/check-order.mjs +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/App.tsx +199 −0
Large diff (199 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/api.ts +402 −0
Large diff (402 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/components/AmenityIco.tsx +111 −0
Large diff (111 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/components/Icons.tsx +188 −0
Large diff (188 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/components/ListingCard.tsx +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/components/MapView.tsx +323 −0
Large diff (323 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/components/PropertyImg.tsx +47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/components/PropertyMap.tsx +70 −0
Large diff (70 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/ka/GroupeKaBadge.tsx +21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/ka/KaFooter.tsx +102 −0
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/ka/ecosystem.json +216 −0
Large diff (217 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/ka/tokens.css +319 −0
Large diff (319 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/kamaps/adapter.ts +101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/kamaps/config.ts +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/kamaps/theme.ts +48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/main.tsx +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Admin.tsx +478 −0
Large diff (478 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Contact.tsx +65 −0
Large diff (65 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Home.tsx +519 −0
Large diff (519 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Legal.tsx +129 −0
Large diff (129 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Listing.tsx +501 −0
Large diff (501 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Sources.tsx +92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/pages/Stats.tsx +176 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/styles.css +1874 −0
Large diff (1,874 lines) is not loaded by default to keep the page fast.Load diff
added frontend/src/vite-env.d.ts +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added frontend/tsconfig.json +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added frontend/vite.config.ts +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added homeka/__init__.py +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added homeka/brokerages.py +88 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/__init__.py +67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/_maputil.py +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/_resilient.py +318 −0
Large diff (318 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/base.py +247 −0
Large diff (247 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/csv/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/csv/feed.py +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/custom_broker/__init__.py +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/json/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/json/feed.py +76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/json/jsonld_site.py +226 −0
Large diff (226 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/public_data/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/public_data/arcgis.py +148 −0
Large diff (148 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/reso/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/reso/webapi.py +158 −0
Large diff (158 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/rets/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/rets/client.py +128 −0
Large diff (128 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/sftp/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/sftp/drop.py +88 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/xml/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added homeka/connectors/xml/feed.py +74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere