SPB Git

spb/ka4 Public

ka4 — explorateur structuré du web québécois (édition agressive, Groupe KA). Crawling concurrent, Claude Sonnet+Haiku, Firecrawl/Scrapfly, graphe de connaissances.

influencequebecscraperknowledge-graphclaudefastapi
Python 97.8% Shell 2.2%

feat: initial commit

Simon-Pierre Boucher committed 1 h ago (Aug 13, 2026)

Showing 23 changed files with +3,883 and −0

added .env.example +12 −0
@@ -0,0 +1,12 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +
4 +ANTHROPIC_API_KEY=sk-ant-...
5 +FIRECRAWL_API_KEY=fc-...
6 +
7 +KA_BOT_MODEL=claude-haiku-4-5-20251001
8 +KA_BOT_DB=ka_bot.db
9 +
10 +KA_BOT_DELAY=1.5
11 +KA_BOT_RESPECT_ROBOTS=true
12 +KA_BOT_MAX_PAGES=50
added .gitignore +17 −0
@@ -0,0 +1,17 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +.env
4 +*.db
5 +*.db-wal
6 +*.db-shm
7 +*.db-journal
8 +__pycache__/
9 +*.pyc
10 +.venv/
11 +venv/
12 +logs/
13 +exports/
14 +*.log
15 +.DS_Store
16 +.claude/
17 +deploy/ngrok-*.yml
added README.md +203 −0
@@ -0,0 +1,203 @@
1 +<div align="center">
2 +
3 +# ka4 — explorateur structuré du web québécois
4 +### Édition agressive · une plateforme du **Groupe KA**
5 +
6 +![Auteur](https://img.shields.io/badge/auteur-Simon--Pierre%20Boucher-111?style=flat-square)
7 +![Contact](https://img.shields.io/badge/contact-contact%40spboucher.ai-e11d48?style=flat-square)
8 +![by Groupe KA](https://img.shields.io/badge/by-Groupe%20KA-e11d48?style=flat-square)
9 +![Statut](https://img.shields.io/badge/statut-en%20production-brightgreen?style=flat-square)
10 +
11 +![Python](https://img.shields.io/badge/Python-3.9%2B-3776AB?style=flat-square&logo=python&logoColor=white)
12 +![FastAPI](https://img.shields.io/badge/FastAPI-uvicorn-009688?style=flat-square&logo=fastapi&logoColor=white)
13 +![Anthropic](https://img.shields.io/badge/Anthropic-Claude-8A2BE2?style=flat-square&logo=anthropic&logoColor=white)
14 +![Firecrawl](https://img.shields.io/badge/Firecrawl-scrape%20%C2%B7%20search%20%C2%B7%20map-FF7A00?style=flat-square)
15 +![Scrapfly](https://img.shields.io/badge/Scrapfly-repli%20anti--bot-1E90FF?style=flat-square)
16 +![SQLite](https://img.shields.io/badge/SQLite-WAL-003B57?style=flat-square&logo=sqlite&logoColor=white)
17 +![ngrok](https://img.shields.io/badge/ngrok-www.ka4.bot-1F1E37?style=flat-square&logo=ngrok&logoColor=white)
18 +![Licence](https://img.shields.io/badge/licence-propri%C3%A9taire-lightgrey?style=flat-square)
19 +
20 +**ka4** est un bot **scraper + recherche piloté par IA** qui cartographie le web québécois
21 +sous forme de **graphe de connaissances** : entités (entreprises, organisations, personnes,
22 +créateurs/influenceurs, agences, marques…), leurs **attributs** et leurs **relations typées**.
23 +
24 +🌐 **Dashboard en direct : https://www.ka4.bot**
25 +
26 +</div>
27 +
28 +---
29 +
30 +## ✨ Aperçu
31 +
32 +| | |
33 +|---|---|
34 +| ![Tableau de bord](assets/ka4_dashboard.png) | ![Répertoire structuré](assets/ka4_explorer.png) |
35 +| **Tableau de bord** temps réel (thème rouge) | **Répertoire structuré** — graphe global interactif |
36 +
37 +> Les captures ci-dessus utilisent `assets/ka4_dashboard.png` et `assets/ka4_explorer.png`.
38 +
39 +---
40 +
41 +## 🎯 Positionnement — Édition agressive
42 +
43 +Le Groupe KA opère trois bots jumeaux partageant le **même moteur**, à trois paliers :
44 +
45 +| Bot | Palier | Contrôleur | Concurrence | Pages/jour | Thème |
46 +|-----|--------|-----------|-------------|-----------|-------|
47 +| ka2 | léger / économique | Haiku 4.5 | 6 | 800 | 🟢 vert lime |
48 +| ka4 | agressif | Sonnet 5 | 10 | 3000 | 🔴 rouge |
49 +| ka6 | **flagship** | Sonnet 5 | 16 | 9000 | 🟣 violet |
50 +
51 +> **Ce dépôt = ka4** (Édition agressive, thème rouge).
52 +
53 +---
54 +
55 +## 🧠 Comment ça marche
56 +
57 +```
58 + ┌──────────────────────────────────────────────┐
59 + Missions ───────▶ │ Agent (Claude) — planifie, cherche, │
60 + (dashboard) │ approfondit (deep_dive), pilote via outils │
61 + └───────────────┬──────────────────────────────┘
62 + │ outils
63 + search_web / map_site / scrape_page / deep_dive / crawl_queue
64 + │
65 + ┌─────────────────────────▼─────────────────────────┐
66 + │ Firecrawl (principal) → Scrapfly (repli anti-bot) │
67 + │ robots.txt + rate-limit + fail-fast + coupe-circuit│
68 + └─────────────────────────┬─────────────────────────┘
69 + │ markdown
70 + ┌────────────▼────────────┐
71 + │ Extracteur (Haiku) │
72 + │ → GRAPHE {entités, │
73 + │ relations typées} │
74 + └────────────┬────────────┘
75 + │ résolution d'entités + provenance
76 + ┌────────────▼────────────┐
77 + │ SQLite (WAL) │
78 + │ entités · relations · │
79 + │ liens sociaux · sources │
80 + └────────────┬────────────┘
81 + │ SSE temps réel
82 + ┌────────────▼────────────┐
83 + │ Dashboard + Répertoire │
84 + │ (FastAPI, 3 vues) │
85 + └──────────────────────────┘
86 +```
87 +
88 +---
89 +
90 +## 🚀 Fonctionnalités
91 +
92 +- 🕸️ **Graphe de connaissances** — entités normalisées + **relations typées** (WORKS_AT, FOUNDER_OF,
93 + REPRESENTED_BY, COLLABORATES_WITH, SPONSORED_BY, MEMBER_OF, PARTNER_OF…), avec rôle et provenance.
94 +- 🧩 **Résolution d'entités / déduplication** (nom normalisé + domaine + région) et **régions QC canoniques**.
95 +- 👤 **Modèle créateur/influenceur** — type `creator`/`agency`/`brand`, handle, plateforme, **abonnés**
96 + (par plateforme), niche, langues.
97 +- ⚡ **Crawling concurrent** (10 pages en parallèle) + **frontière agressive** (auto-découverte de liens).
98 +- 🛟 **Robustesse** — timeouts fail-fast, **retries** par URL (dead-letter), **coupe-circuit** par domaine.
99 +- 🧭 **Agent rigoureux** (plan → découverte → collecte → vérification) + **auto-compaction** du contexte.
100 +- 🎛️ **Contrôles de mission** — arrêter la mission en cours, **lancer maintenant** (prioriser), supprimer.
101 +- 🗄️ **Archives** — snapshot des données puis remise à zéro ; historique **restaurable / exportable**.
102 +- 🖥️ **Dashboard temps réel (SSE)** — KPIs, flux d'activité, réglages en direct, thème rouge, responsive.
103 +- 🔎 **Répertoire structuré** — 3 vues : **Cartes**, **Tableau** triable, **Graphe global** (force-directed,
104 + pan/zoom), facettes (type/secteur/région), recherche, **export CSV/JSON**.
105 +- 🇶🇨 **Conforme** — robots.txt respecté, rate-limiting, note **Loi 25 (Québec) / LPRPDE**, données publiques.
106 +
107 +---
108 +
109 +## ⚙️ Configuration (`.env`)
110 +
111 +| Variable | Rôle | Valeur (ka4) |
112 +|---|---|---|
113 +| `ANTHROPIC_API_KEY` | Clé Anthropic | — |
114 +| `FIRECRAWL_API_KEY` | Clé Firecrawl | — |
115 +| `SCRAPFLY_API_KEY` | Clé Scrapfly (repli) | — |
116 +| `KA_BOT_MODEL` | Modèle contrôleur | `claude-sonnet-5` |
117 +| `KA_BOT_EXTRACT_MODEL` | Modèle extraction | `claude-haiku-4-5-20251001` |
118 +| `KA_BOT_DB` | Base SQLite | `ka4.db` |
119 +| `KA_BOT_CONCURRENCY` | Pages en parallèle | `10` |
120 +| `KA_BOT_DELAY` | Délai entre requêtes (s) | `0.4` |
121 +| `KA_BOT_MAX_PAGES` | Pages max / exécution | `220` |
122 +| `KA_BOT_FRONTIER` | Liens auto-enfilés / page | `12–16` |
123 +| `KA_BOT_MAX_ATTEMPTS` | Tentatives / URL | `2` |
124 +| `KA_BOT_DOMAIN_FAILS` | Seuil coupe-circuit | `4` |
125 +| `KA_BOT_AUTOSEED` | Pré-remplir des missions | `0` (off) |
126 +
127 +> ⚠️ Le fichier `.env` (clés API) **n'est pas** versionné (`.gitignore`).
128 +
129 +---
130 +
131 +## 🏗️ Installation & exécution
132 +
133 +```bash
134 +python3 -m venv .venv && source .venv/bin/activate
135 +pip install -r requirements.txt
136 +cp .env.example .env # renseigner les clés
137 +
138 +# Dashboard (web)
139 +uvicorn web.app:app --host 127.0.0.1 --port 8899
140 +
141 +# Bot long terme (crawler)
142 +python main.py run
143 +
144 +# Modes ponctuels
145 +python main.py pipeline --query "influenceurs beauté Québec"
146 +python main.py agent --goal "Cartographier les agences d'influence de Montréal"
147 +python main.py stats
148 +python main.py export --out exports/entites.json
149 +```
150 +
151 +## ☁️ Déploiement (cluster MacLustr — node M4M36)
152 +
153 +```bash
154 +bash deploy/deploy.sh M4M36 8899 www.ka4.bot
155 +```
156 +
157 +Crée 3 services **launchd** résilients : `com.ka4.web`, `com.ka4.bot`, `com.ka4.ngrok`
158 +(tunnel `--url=https://www.ka4.bot`, inspection ngrok sur `:4041` pour cohabiter avec les autres bots).
159 +
160 +---
161 +
162 +## 🔌 API principale
163 +
164 +| Méthode | Route | Rôle |
165 +|---|---|---|
166 +| `GET` | `/health` | état du service |
167 +| `GET` | `/api/stats` | compteurs + état session |
168 +| `GET` | `/api/stream` | flux SSE (stats + événements) |
169 +| `GET` | `/api/entities` | entités (filtres type/région/secteur, tri, pagination) |
170 +| `GET` | `/api/entity/{id}` | fiche entité + relations |
171 +| `GET` | `/api/graph` | nœuds + arêtes du graphe global |
172 +| `GET` | `/api/export.{json,csv}` | export filtré |
173 +| `POST` | `/api/missions` | ajouter une mission |
174 +| `POST` | `/api/missions/{id}/{prioritize,delete}` | lancer maintenant / supprimer |
175 +| `POST` | `/api/control/{pause,resume,skip,extend}` | contrôles bot |
176 +| `POST` | `/api/archive` · `GET /api/archives` | archiver & vider · historique |
177 +| `POST` | `/api/archive/{id}/{restore,delete}` | restaurer / supprimer archive |
178 +
179 +---
180 +
181 +## 🗃️ Modèle de données (SQLite)
182 +
183 +`entities` (type, nom, canonical/norm, domaine, secteur, région, coordonnées, **niche/handle/plateforme/abonnés**,
184 +provenance) · `relations` (from, to, type, rôle, source, confiance) · `social_links` (plateforme, url, abonnés) ·
185 +`sources` · `entity_mentions` · `crawl_queue` · `missions` · `events` · `settings` · `archives`.
186 +
187 +---
188 +
189 +## ⚖️ Conformité & usage responsable
190 +
191 +Données **publiques** uniquement · `robots.txt` respecté · rate-limiting · finalités légitimes (annuaire,
192 +veille, recherche de marché). Encadré par la **Loi 25 (Québec)** et la **LPRPDE**. Ne pas constituer de
193 +profils de personnes privées à des fins de surveillance ou de démarchage non sollicité.
194 +
195 +---
196 +
197 +<div align="center">
198 +
199 +**ka4** · une plateforme du **Groupe KA**
200 +Auteur **Simon-Pierre Boucher** · ✉️ **contact@spboucher.ai**
201 +*Zéro boîte noire — tout est automatisé, rien n'est inventé.*
202 +
203 +</div>
added assets/ka4_dashboard.png +0 −0

Binary file not shown.

added assets/ka4_explorer.png +0 −0

Binary file not shown.

added deploy/deploy.sh +100 −0
@@ -0,0 +1,100 @@
1 +#!/usr/bin/env bash
2 +# Author: Simon-Pierre Boucher
3 +# Contact: contact@spboucher.ai
4 +#
5 +# Déploie ka4 (version agressive/concurrente) sur un node macOS du cluster (défaut : M4M36).
6 +# Cohabite avec ka2 sur le même node : port + services + domaine + web-addr ngrok distincts.
7 +# - venv + dépendances
8 +# - 3 services launchd résilients : com.ka4.web / com.ka4.bot / com.ka4.ngrok
9 +# - ngrok --url=https://www.ka4.bot -> uvicorn 127.0.0.1:8899 (inspection sur :4041)
10 +#
11 +# Usage : bash deploy/deploy.sh [NODE] [PORT] [DOMAIN]
12 +set -euo pipefail
13 +
14 +NODE="${1:-M4M36}"
15 +PORT="${2:-8899}"
16 +DOMAIN="${3:-www.ka4.bot}"
17 +REMOTE_DIR="~/cluster-projects/ka4"
18 +NGROK="/opt/homebrew/bin/ngrok"
19 +
20 +echo "==> Déploiement de ka4 sur ${NODE} (port ${PORT}, domaine ${DOMAIN})"
21 +
22 +rsync -az --delete \
23 + --exclude '.venv' --exclude '.git' --exclude '*.db' --exclude '*.db-*' \
24 + --exclude '__pycache__' --exclude 'exports' --exclude 'logs' \
25 + ./ "${NODE}:${REMOTE_DIR}/"
26 +
27 +ssh "${NODE}" DOMAIN="${DOMAIN}" PORT="${PORT}" NGROK="${NGROK}" 'bash -s' <<'REMOTE'
28 +set -euo pipefail
29 +DIR="$HOME/cluster-projects/ka4"
30 +cd "$DIR"
31 +
32 +echo "--> venv + dépendances"
33 +python3 -m venv .venv
34 +./.venv/bin/python -m pip install -q --upgrade pip
35 +./.venv/bin/python -m pip install -q -r requirements.txt
36 +
37 +PY="$DIR/.venv/bin/python"
38 +UVICORN="$DIR/.venv/bin/uvicorn"
39 +LOGS="$DIR/logs"; mkdir -p "$LOGS"
40 +LA="$HOME/Library/LaunchAgents"; mkdir -p "$LA"
41 +
42 +write_plist () {
43 + local label="$1" ; shift
44 + local plist="$LA/${label}.plist"
45 + {
46 + echo '<?xml version="1.0" encoding="UTF-8"?>'
47 + echo '<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">'
48 + echo '<plist version="1.0"><dict>'
49 + echo " <key>Label</key><string>${label}</string>"
50 + echo ' <key>ProgramArguments</key><array>'
51 + for a in "$@"; do echo " <string>${a}</string>"; done
52 + echo ' </array>'
53 + echo " <key>WorkingDirectory</key><string>${DIR}</string>"
54 + echo ' <key>RunAtLoad</key><true/>'
55 + echo ' <key>KeepAlive</key><true/>'
56 + echo " <key>StandardOutPath</key><string>${LOGS}/${label}.out.log</string>"
57 + echo " <key>StandardErrorPath</key><string>${LOGS}/${label}.err.log</string>"
58 + echo ' <key>EnvironmentVariables</key><dict>'
59 + echo " <key>PATH</key><string>/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>"
60 + echo ' </dict>'
61 + echo '</dict></plist>'
62 + } > "$plist"
63 + launchctl unload "$plist" 2>/dev/null || true
64 + launchctl load "$plist"
65 + echo " launchd chargé: ${label}"
66 +}
67 +
68 +# config ngrok dédiée : inspection sur :4041 (KA2 occupe déjà :4040 sur ce node)
69 +NGCONF="$DIR/ngrok-ka4.yml"
70 +printf 'version: "3"\nagent:\n web_addr: localhost:4041\n' > "$NGCONF"
71 +DEFCONF="$HOME/Library/Application Support/ngrok/ngrok.yml"
72 +
73 +echo "--> services launchd (com.ka4.*)"
74 +write_plist "com.ka4.web" "$UVICORN" "web.app:app" "--host" "127.0.0.1" "--port" "${PORT}"
75 +write_plist "com.ka4.bot" "$PY" "main.py" "run" "--daily-budget" "3000" "--cycle-delay" "8" "--queue-batch" "24" "--session-minutes" "15"
76 +write_plist "com.ka4.ngrok" "$NGROK" "http" "--url=https://${DOMAIN}" "--config" "$DEFCONF" "--config" "$NGCONF" "${PORT}"
77 +
78 +echo "--> session : 15 min dès le déploiement (bouton « Continuer » pour +15 min)"
79 +"$PY" - <<'PYSET'
80 +import time
81 +from src.config import config
82 +from src.storage import Store
83 +s = Store(config.db_path)
84 +s.set_setting("run_until", str(time.time() + 15 * 60))
85 +s.set_setting("paused", "0")
86 +s.close()
87 +print("run_until = maintenant + 15 min")
88 +PYSET
89 +
90 +sleep 6
91 +echo "--> healthcheck local"
92 +curl -s "http://127.0.0.1:${PORT}/health" || echo "(web pas encore prêt)"
93 +echo
94 +echo "--> tunnel ngrok (inspection :4041)"
95 +curl -s http://127.0.0.1:4041/api/tunnels 2>/dev/null | head -c 400 || echo "(api ngrok non prête)"
96 +echo
97 +REMOTE
98 +
99 +echo
100 +echo "==> Déploiement terminé. Dashboard ka4 : https://${DOMAIN}"
added main.py +166 −0
@@ -0,0 +1,166 @@
1 +#!/usr/bin/env python3
2 +# Author: Simon-Pierre Boucher
3 +# Contact: contact@spboucher.ai
4 +"""CLI de ka-bot — cartographe du web québécois (Claude Haiku 4.5 + Firecrawl).
5 +
6 +Exemples :
7 + python main.py agent --goal "Cartographier les PME technos de Québec" \\
8 + --seed "https://www.quebecinternational.ca" --max-steps 20
9 + python main.py pipeline --query "agences marketing Montréal" --limit 10
10 + python main.py drain --max-pages 30
11 + python main.py stats
12 + python main.py export --out exports/entites.json
13 +"""
14 +
15 +from __future__ import annotations
16 +
17 +import argparse
18 +import json
19 +import os
20 +import sys
21 +
22 +from src.config import config
23 +from src.orchestrator import Orchestrator
24 +from src.scheduler import LongRunner
25 +
26 +
27 +def _print(obj) -> None:
28 + print(json.dumps(obj, ensure_ascii=False, indent=2))
29 +
30 +
31 +def cmd_agent(args) -> None:
32 + orch = Orchestrator(config)
33 + try:
34 + summary = orch.run_agent(args.goal, args.seed, args.max_steps)
35 + print("\n=== Résumé de mission ===")
36 + print(summary)
37 + print("\n=== Statistiques ===")
38 + _print(orch.store.stats())
39 + finally:
40 + orch.close()
41 +
42 +
43 +def cmd_pipeline(args) -> None:
44 + orch = Orchestrator(config)
45 + try:
46 + _print(orch.run_pipeline(args.query, args.limit, args.per_site_pages))
47 + finally:
48 + orch.close()
49 +
50 +
51 +def cmd_drain(args) -> None:
52 + orch = Orchestrator(config)
53 + try:
54 + _print(orch.drain_queue(args.max_pages))
55 + finally:
56 + orch.close()
57 +
58 +
59 +def cmd_run(args) -> None:
60 + runner = LongRunner(
61 + config,
62 + cycle_delay=args.cycle_delay,
63 + queue_batch=args.queue_batch,
64 + daily_page_budget=args.daily_budget,
65 + revisit_hours=args.revisit_hours,
66 + session_minutes=args.session_minutes,
67 + )
68 + print("ka4 démarré (Ctrl+C pour arrêter proprement).")
69 + runner.run(max_cycles=args.max_cycles)
70 +
71 +
72 +def cmd_missions(args) -> None:
73 + orch = Orchestrator(config)
74 + try:
75 + if args.add:
76 + mid = orch.store.add_mission(goal=args.add, seed=args.seed or "", priority=args.priority)
77 + print(f"Mission #{mid} ajoutée." if mid else "Mission déjà existante.")
78 + _print(orch.store.list_missions())
79 + finally:
80 + orch.close()
81 +
82 +
83 +def cmd_stats(_args) -> None:
84 + orch = Orchestrator(config)
85 + try:
86 + _print(orch.store.stats())
87 + finally:
88 + orch.close()
89 +
90 +
91 +def cmd_export(args) -> None:
92 + orch = Orchestrator(config)
93 + try:
94 + data = orch.store.export()
95 + if args.out:
96 + os.makedirs(os.path.dirname(args.out) or ".", exist_ok=True)
97 + with open(args.out, "w", encoding="utf-8") as f:
98 + json.dump(data, f, ensure_ascii=False, indent=2)
99 + print(f"{len(data)} entités exportées vers {args.out}")
100 + else:
101 + _print(data)
102 + finally:
103 + orch.close()
104 +
105 +
106 +def build_parser() -> argparse.ArgumentParser:
107 + p = argparse.ArgumentParser(prog="ka-bot", description="Cartographe du web québécois.")
108 + sub = p.add_subparsers(dest="cmd", required=True)
109 +
110 + a = sub.add_parser("agent", help="Mode agentique piloté par Claude Haiku")
111 + a.add_argument("--goal", required=True, help="Objectif de mission")
112 + a.add_argument("--seed", help="URL de départ optionnelle")
113 + a.add_argument("--max-steps", type=int, default=20)
114 + a.set_defaults(func=cmd_agent)
115 +
116 + pl = sub.add_parser("pipeline", help="Pipeline déterministe search->scrape->extract")
117 + pl.add_argument("--query", required=True)
118 + pl.add_argument("--limit", type=int, default=10)
119 + pl.add_argument("--per-site-pages", type=int, default=3)
120 + pl.set_defaults(func=cmd_pipeline)
121 +
122 + d = sub.add_parser("drain", help="Traiter la file d'URLs en attente")
123 + d.add_argument("--max-pages", type=int, default=None)
124 + d.set_defaults(func=cmd_drain)
125 +
126 + r = sub.add_parser("run", help="Bot LONG TERME : boucle perpétuelle de cartographie")
127 + r.add_argument("--cycle-delay", type=float, default=20.0)
128 + r.add_argument("--queue-batch", type=int, default=8)
129 + r.add_argument("--daily-budget", type=int, default=800, help="Pages max par jour")
130 + r.add_argument("--revisit-hours", type=float, default=72.0)
131 + r.add_argument("--session-minutes", type=float, default=15.0,
132 + help="Durée de session avant d'attendre « Continuer »")
133 + r.add_argument("--max-cycles", type=int, default=None, help="Limite de cycles (test)")
134 + r.set_defaults(func=cmd_run)
135 +
136 + m = sub.add_parser("missions", help="Lister / ajouter des missions")
137 + m.add_argument("--add", help="Ajouter une mission (objectif)")
138 + m.add_argument("--seed", help="URL de départ pour la mission")
139 + m.add_argument("--priority", type=int, default=5)
140 + m.set_defaults(func=cmd_missions)
141 +
142 + s = sub.add_parser("stats", help="Statistiques de la base")
143 + s.set_defaults(func=cmd_stats)
144 +
145 + e = sub.add_parser("export", help="Exporter les entités en JSON")
146 + e.add_argument("--out", help="Fichier de sortie (sinon stdout)")
147 + e.set_defaults(func=cmd_export)
148 +
149 + return p
150 +
151 +
152 +def main() -> int:
153 + args = build_parser().parse_args()
154 + try:
155 + args.func(args)
156 + return 0
157 + except RuntimeError as e:
158 + print(f"Erreur : {e}", file=sys.stderr)
159 + return 1
160 + except KeyboardInterrupt:
161 + print("\nInterrompu.", file=sys.stderr)
162 + return 130
163 +
164 +
165 +if __name__ == "__main__":
166 + raise SystemExit(main())
added requirements.txt +7 −0
@@ -0,0 +1,7 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +anthropic>=0.40.0
4 +requests>=2.31.0
5 +python-dotenv>=1.0.0
6 +fastapi>=0.110.0
7 +uvicorn[standard]>=0.29.0
added src/__init__.py +7 −0
@@ -0,0 +1,7 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""ka-bot — cartographe du web québécois piloté par Claude Haiku 4.5 + Firecrawl."""
4 +
5 +__version__ = "1.0.0"
6 +__author__ = "Simon-Pierre Boucher"
7 +__contact__ = "contact@spboucher.ai"
added src/config.py +74 −0
@@ -0,0 +1,74 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Configuration centrale de ka6 (chargée depuis .env).
4 +
5 +ka6 = bot FLAGSHIP du Groupe KA : ~3x plus robuste et performant que ka4
6 +(concurrence élevée, robustesse retries + coupe-circuit), avec un objectif initial
7 +de découverte des INFLUENCEURS et CRÉATEURS DE CONTENU québécois.
8 +"""
9 +
10 +from __future__ import annotations
11 +
12 +import os
13 +from dataclasses import dataclass, field
14 +
15 +from dotenv import load_dotenv
16 +
17 +load_dotenv()
18 +
19 +
20 +def _env_bool(key: str, default: bool) -> bool:
21 + return os.getenv(key, str(default)).strip().lower() in ("1", "true", "yes", "on")
22 +
23 +
24 +@dataclass
25 +class Config:
26 + """Paramètres runtime. Toutes les clés proviennent de l'environnement."""
27 +
28 + anthropic_api_key: str = field(default_factory=lambda: os.getenv("ANTHROPIC_API_KEY", ""))
29 + firecrawl_api_key: str = field(default_factory=lambda: os.getenv("FIRECRAWL_API_KEY", ""))
30 + scrapfly_api_key: str = field(default_factory=lambda: os.getenv("SCRAPFLY_API_KEY", ""))
31 +
32 + # Contrôleur agentique : modèle PLUS PUISSANT (planification/raisonnement)
33 + model: str = field(default_factory=lambda: os.getenv("KA_BOT_MODEL", "claude-sonnet-5"))
34 + # Extraction en masse : modèle rapide/économique
35 + extract_model: str = field(default_factory=lambda: os.getenv("KA_BOT_EXTRACT_MODEL", "claude-haiku-4-5-20251001"))
36 + db_path: str = field(default_factory=lambda: os.getenv("KA_BOT_DB", "ka6.db"))
37 + # Crawling CONCURRENT — flagship : ~3x plus de parallélisme
38 + concurrency: int = field(default_factory=lambda: int(os.getenv("KA_BOT_CONCURRENCY", "16")))
39 + # Expansion agressive : liens auto-ajoutés à la file par page
40 + frontier_per_page: int = field(default_factory=lambda: int(os.getenv("KA_BOT_FRONTIER", "16")))
41 + # Robustesse : nb max de tentatives par URL avant abandon (dead-letter)
42 + max_attempts: int = field(default_factory=lambda: int(os.getenv("KA_BOT_MAX_ATTEMPTS", "2")))
43 + # Coupe-circuit : abandon d'un domaine après N échecs consécutifs dans un run
44 + domain_fail_threshold: int = field(default_factory=lambda: int(os.getenv("KA_BOT_DOMAIN_FAILS", "4")))
45 +
46 + # Backend de scrape : auto | firecrawl | scrapfly
47 + scraper_backend: str = field(default_factory=lambda: os.getenv("KA_BOT_SCRAPER", "auto").strip().lower())
48 + scrapfly_render_js: bool = field(default_factory=lambda: _env_bool("KA_BOT_SCRAPFLY_RENDER_JS", False))
49 +
50 + firecrawl_base: str = "https://api.firecrawl.dev"
51 + scrapfly_base: str = "https://api.scrapfly.io"
52 + request_delay: float = field(default_factory=lambda: float(os.getenv("KA_BOT_DELAY", "0.35")))
53 + respect_robots: bool = field(default_factory=lambda: _env_bool("KA_BOT_RESPECT_ROBOTS", True))
54 + max_pages_per_run: int = field(default_factory=lambda: int(os.getenv("KA_BOT_MAX_PAGES", "320")))
55 + max_context_tokens: int = field(default_factory=lambda: int(os.getenv("KA_BOT_MAX_CONTEXT_TOKENS", "180000")))
56 +
57 + user_agent: str = "ka6-bot/1.0 (+contact@spboucher.ai)"
58 +
59 + def validate(self) -> "Config":
60 + missing = []
61 + if not self.anthropic_api_key:
62 + missing.append("ANTHROPIC_API_KEY")
63 + if not self.firecrawl_api_key:
64 + missing.append("FIRECRAWL_API_KEY (requis pour search/map)")
65 + if self.scraper_backend == "scrapfly" and not self.scrapfly_api_key:
66 + missing.append("SCRAPFLY_API_KEY (backend=scrapfly)")
67 + if missing:
68 + raise RuntimeError(
69 + "Clés manquantes: " + ", ".join(missing) + ". Copiez .env.example vers .env."
70 + )
71 + return self
72 +
73 +
74 +config = Config()
added src/extractor.py +133 −0
@@ -0,0 +1,133 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Extraction d'un GRAPHE DE CONNAISSANCES orienté CRÉATEURS / INFLUENCEURS (ka6).
4 +
5 +Objectif initial du flagship : cartographier l'écosystème des créateurs de contenu
6 +et influenceurs québécois — leurs plateformes, handles, nombre d'abonnés, niche,
7 +langues — et leurs relations (agence qui les représente, marques avec qui ils
8 +collaborent, plateformes où ils publient, collectifs dont ils sont membres).
9 +
10 +On extrait aussi les entités classiques (entreprises, organisations, agences).
11 +Données PUBLIQUES uniquement ; aucune donnée sensible ou privée.
12 +"""
13 +
14 +from __future__ import annotations
15 +
16 +from typing import Any
17 +
18 +from anthropic import Anthropic
19 +
20 +GRAPH_TOOL = {
21 + "name": "record_graph",
22 + "description": "Enregistre les entités (dont créateurs/influenceurs) et leurs relations.",
23 + "input_schema": {
24 + "type": "object",
25 + "properties": {
26 + "entities": {
27 + "type": "array",
28 + "items": {
29 + "type": "object",
30 + "properties": {
31 + "temp_id": {"type": "string", "description": "Identifiant local unique (e1, e2…) référencé par les relations"},
32 + "type": {"type": "string",
33 + "enum": ["creator", "business", "organization", "agency", "brand", "person", "website"],
34 + "description": "Utilise 'creator' pour un influenceur/créateur de contenu, 'agency' pour une agence d'influence/talent, 'brand' pour une marque."},
35 + "name": {"type": "string"},
36 + "canonical_name": {"type": "string"},
37 + "handle": {"type": "string", "description": "Pseudo principal (ex. @nomcreateur)"},
38 + "platform": {"type": "string", "description": "Plateforme principale (Instagram, TikTok, YouTube, Twitch, X, Facebook, Snapchat…)"},
39 + "followers": {"type": "string", "description": "Nb d'abonnés principal (ex. '125K', '1.2M', 34000)"},
40 + "niche": {"type": "string", "description": "Créneau/thématique (mode, beauté, gaming, cuisine, humour, lifestyle, sport, tech, voyage, famille, musique…)"},
41 + "languages": {"type": "array", "items": {"type": "string"}, "description": "Langues du contenu (fr, en…)"},
42 + "sector": {"type": "string", "description": "Secteur pour les entreprises/agences/marques"},
43 + "description": {"type": "string"},
44 + "website": {"type": "string"},
45 + "email": {"type": "string"},
46 + "phone": {"type": "string"},
47 + "city": {"type": "string"},
48 + "region": {"type": "string", "description": "Ville/région du Québec"},
49 + "social_links": {
50 + "type": "array",
51 + "items": {
52 + "type": "object",
53 + "properties": {
54 + "platform": {"type": "string"},
55 + "url": {"type": "string"},
56 + "followers": {"type": "string", "description": "Abonnés sur CETTE plateforme si connu"},
57 + },
58 + "required": ["platform", "url"],
59 + },
60 + },
61 + "confidence": {"type": "number", "minimum": 0, "maximum": 1},
62 + },
63 + "required": ["temp_id", "type", "name"],
64 + },
65 + },
66 + "relations": {
67 + "type": "array",
68 + "items": {
69 + "type": "object",
70 + "properties": {
71 + "from": {"type": "string"},
72 + "to": {"type": "string"},
73 + "type": {
74 + "type": "string",
75 + "enum": ["REPRESENTED_BY", "COLLABORATES_WITH", "CREATES_ON", "SPONSORED_BY",
76 + "PROMOTES", "MEMBER_OF", "MANAGES", "APPEARS_WITH",
77 + "WORKS_AT", "FOUNDER_OF", "PARTNER_OF", "AFFILIATED_WITH"],
78 + "description": "Ex : créateur REPRESENTED_BY agence ; créateur COLLABORATES_WITH marque ; créateur CREATES_ON plateforme ; créateur MEMBER_OF collectif.",
79 + },
80 + "role": {"type": "string", "description": "Précision (ex. 'ambassadrice', 'campagne 2025', 'chaîne principale')"},
81 + "confidence": {"type": "number", "minimum": 0, "maximum": 1},
82 + },
83 + "required": ["from", "to", "type"],
84 + },
85 + },
86 + },
87 + "required": ["entities"],
88 + },
89 +}
90 +
91 +SYSTEM = (
92 + "Tu es l'extracteur du bot FLAGSHIP ka6 du Groupe KA. Objectif initial : cartographier "
93 + "l'écosystème des CRÉATEURS DE CONTENU et INFLUENCEURS québécois à partir de pages publiques "
94 + "(répertoires, palmarès, rosters d'agences, articles, pages de marques).\n\n"
95 + "Pour chaque créateur/influenceur (type 'creator') extrais : nom, handle (@), plateforme "
96 + "principale, nombre d'abonnés, niche/thématique, langues, région, et TOUTES ses plateformes "
97 + "dans social_links (avec abonnés par plateforme si indiqués).\n"
98 + "Modélise les RELATIONS : REPRESENTED_BY (agence de talent/influence), COLLABORATES_WITH / "
99 + "SPONSORED_BY / PROMOTES (marques), CREATES_ON (plateformes), MEMBER_OF (collectif/réseau), "
100 + "MANAGES (gérant/agence -> créateur).\n"
101 + "Extrais aussi agences ('agency'), marques ('brand'), entreprises/organisations pertinentes.\n"
102 + "Règles : données PUBLIQUES uniquement ; priorise le Québec ; n'invente rien (laisse vide si "
103 + "absent) ; donne un temp_id unique par entité et référence-le dans les relations ; confidence "
104 + "réaliste. Réponds uniquement via l'outil record_graph."
105 +)
106 +
107 +
108 +class Extractor:
109 + def __init__(self, client: Anthropic, model: str):
110 + self.client = client
111 + self.model = model
112 +
113 + def extract_graph(self, url: str, title: str, markdown: str) -> dict[str, Any]:
114 + if not markdown or not markdown.strip():
115 + return {"entities": [], "relations": []}
116 + content = markdown[:20000]
117 + resp = self.client.messages.create(
118 + model=self.model,
119 + max_tokens=3500,
120 + system=SYSTEM,
121 + tools=[GRAPH_TOOL],
122 + tool_choice={"type": "tool", "name": "record_graph"},
123 + messages=[{
124 + "role": "user",
125 + "content": f"URL: {url}\nTitre: {title}\n\nContenu (markdown):\n{content}",
126 + }],
127 + )
128 + for block in resp.content:
129 + if block.type == "tool_use" and block.name == "record_graph":
130 + data = block.input or {}
131 + return {"entities": data.get("entities", []) or [],
132 + "relations": data.get("relations", []) or []}
133 + return {"entities": [], "relations": []}
added src/firecrawl_client.py +70 −0
@@ -0,0 +1,70 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Client REST Firecrawl : search, map, scrape.
4 +
5 +Le respect du robots.txt et le débit sont gérés en amont (voir src/robots.py et
6 +src/scraper.py). Ce client reste une couche transport minimale.
7 +"""
8 +
9 +from __future__ import annotations
10 +
11 +import time
12 +from typing import Any, Optional
13 +
14 +import requests
15 +
16 +from .util import RateLimiter
17 +
18 +
19 +class FirecrawlClient:
20 + # ka4 : fail-fast pour ne pas bloquer le crawl concurrent
21 + def __init__(self, api_key: str, base_url: str = "https://api.firecrawl.dev", delay: float = 1.5,
22 + timeout: int = 45, retries: int = 2):
23 + self.base = base_url.rstrip("/")
24 + self.session = requests.Session()
25 + self.session.headers.update(
26 + {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
27 + )
28 + self.limiter = RateLimiter(delay)
29 + self.timeout = timeout
30 + self.retries = retries
31 +
32 + def _post(self, path: str, payload: dict[str, Any], retries: Optional[int] = None) -> dict[str, Any]:
33 + url = f"{self.base}{path}"
34 + last_err: Optional[Exception] = None
35 + for attempt in range(retries or self.retries):
36 + self.limiter.wait()
37 + try:
38 + r = self.session.post(url, json=payload, timeout=self.timeout)
39 + if r.status_code == 429:
40 + time.sleep(1.5 * (attempt + 1))
41 + continue
42 + r.raise_for_status()
43 + return r.json()
44 + except Exception as e: # noqa: BLE001
45 + last_err = e
46 + time.sleep(1.0 * (attempt + 1))
47 + raise RuntimeError(f"Firecrawl {path} a échoué: {last_err}")
48 +
49 + def search(self, query: str, limit: int = 10) -> list[dict[str, Any]]:
50 + data = self._post("/v1/search", {"query": query, "limit": limit})
51 + return data.get("data", []) or []
52 +
53 + def map(self, url: str, search: Optional[str] = None, limit: int = 200) -> list[str]:
54 + payload: dict[str, Any] = {"url": url, "limit": limit}
55 + if search:
56 + payload["search"] = search
57 + data = self._post("/v1/map", payload)
58 + links = data.get("links", []) or []
59 + return [l["url"] if isinstance(l, dict) else l for l in links]
60 +
61 + def scrape(self, url: str, formats: Optional[list[str]] = None) -> dict[str, Any]:
62 + # demande aussi les liens -> alimente la frontière agressive
63 + data = self._post("/v1/scrape", {"url": url, "formats": formats or ["markdown", "links"]})
64 + d = data.get("data", {}) or {}
65 + return {
66 + "markdown": d.get("markdown", "") or "",
67 + "metadata": d.get("metadata", {}) or {},
68 + "links": d.get("links", []) or [],
69 + "backend": "firecrawl",
70 + }
added src/llm_controller.py +298 −0
@@ -0,0 +1,298 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Contrôleur agentique rigoureux : Claude Haiku 4.5 pilote la cartographie.
4 +
5 +Points clés :
6 +- Méthodologie imposée (plan -> découverte -> collecte -> vérification).
7 +- Garde anti-boucle : on rejette les actions déjà faites (mêmes URL/recherches).
8 +- Journal de progression persistant (outil note_progress).
9 +- AUTO-COMPACTION du contexte : quand l'historique dépasse un seuil de tokens, les
10 + anciens échanges sont résumés par le modèle et remplacés par un condensé, ce qui
11 + permet des missions très longues sans dépasser la fenêtre de contexte.
12 +"""
13 +
14 +from __future__ import annotations
15 +
16 +import json
17 +from typing import Any, Callable, Optional
18 +
19 +from anthropic import Anthropic
20 +
21 +TOOLS = [
22 + {
23 + "name": "update_plan",
24 + "description": "Établis ou révise ton plan de mission : étapes, secteurs/régions à couvrir, "
25 + "et prochaines actions. À appeler au début, puis quand la stratégie change.",
26 + "input_schema": {
27 + "type": "object",
28 + "properties": {
29 + "plan": {"type": "array", "items": {"type": "string"}},
30 + "rationale": {"type": "string"},
31 + },
32 + "required": ["plan"],
33 + },
34 + },
35 + {
36 + "name": "search_web",
37 + "description": "Recherche web (Firecrawl). Retourne titres, URLs et extraits.",
38 + "input_schema": {
39 + "type": "object",
40 + "properties": {
41 + "query": {"type": "string"},
42 + "limit": {"type": "integer", "default": 8},
43 + },
44 + "required": ["query"],
45 + },
46 + },
47 + {
48 + "name": "map_site",
49 + "description": "Liste rapidement les URLs d'un site (découverte). 'search' filtre par mot-clé.",
50 + "input_schema": {
51 + "type": "object",
52 + "properties": {"url": {"type": "string"}, "search": {"type": "string"}},
53 + "required": ["url"],
54 + },
55 + },
56 + {
57 + "name": "scrape_page",
58 + "description": "Récupère le contenu d'une page ET en extrait/enregistre le graphe "
59 + "(entités + relations typées avec rôles). Retourne un résumé.",
60 + "input_schema": {
61 + "type": "object",
62 + "properties": {"url": {"type": "string"}},
63 + "required": ["url"],
64 + },
65 + },
66 + {
67 + "name": "deep_dive",
68 + "description": "APPROFONDIT une organisation : mappe son site et scrape automatiquement ses "
69 + "pages clés (à propos, équipe, direction, contact) pour extraire un profil complet ET les "
70 + "relations (personnes ↔ organisation avec rôles). À privilégier sur toute org prometteuse "
71 + "pour bâtir un vrai graphe plutôt que des entités isolées.",
72 + "input_schema": {
73 + "type": "object",
74 + "properties": {
75 + "url": {"type": "string", "description": "URL ou domaine de l'organisation"},
76 + "max_pages": {"type": "integer", "default": 5},
77 + },
78 + "required": ["url"],
79 + },
80 + },
81 + {
82 + "name": "crawl_queue",
83 + "description": "Traite EN PARALLÈLE un lot d'URLs déjà en file (crawling concurrent). "
84 + "La frontière se remplit automatiquement à chaque page ; sers-t'en pour avancer vite.",
85 + "input_schema": {
86 + "type": "object",
87 + "properties": {"count": {"type": "integer", "default": 6}},
88 + },
89 + },
90 + {
91 + "name": "enqueue_urls",
92 + "description": "Ajoute des URLs prometteuses à la file d'exploration long terme.",
93 + "input_schema": {
94 + "type": "object",
95 + "properties": {"urls": {"type": "array", "items": {"type": "string"}}},
96 + "required": ["urls"],
97 + },
98 + },
99 + {
100 + "name": "note_progress",
101 + "description": "Consigne une note de progression (couverture, angles morts, pistes). "
102 + "Sers-t'en pour rester rigoureux et éviter les redites.",
103 + "input_schema": {
104 + "type": "object",
105 + "properties": {"note": {"type": "string"}},
106 + "required": ["note"],
107 + },
108 + },
109 + {
110 + "name": "finish",
111 + "description": "Termine la mission avec un bilan (couverture atteinte, entités clés, reste à faire).",
112 + "input_schema": {
113 + "type": "object",
114 + "properties": {"summary": {"type": "string"}},
115 + "required": ["summary"],
116 + },
117 + },
118 +]
119 +
120 +SYSTEM = (
121 + "Tu es ka6, l'agent FLAGSHIP du Groupe KA. Objectif initial : cartographier à grande échelle "
122 + "l'écosystème des CRÉATEURS DE CONTENU et INFLUENCEURS québécois (et l'industrie autour : agences "
123 + "de talents/influence, marques, plateformes, collectifs).\n\n"
124 + "Pour chaque créateur : nom, handle, plateformes + nombre d'abonnés, niche, langues, région, et "
125 + "les RELATIONS (représenté par une agence, collabore avec / sponsorisé par des marques, membre d'un "
126 + "collectif, publie sur telles plateformes).\n\n"
127 + "MÉTHODE (agressive, flagship) :\n"
128 + "1. PLANIFIE : update_plan (niches × plateformes, agences, palmarès, marques).\n"
129 + "2. DÉCOUVRE en volume : search_web (ex. « top influenceurs québécois mode TikTok », "
130 + "« agences d'influence Montréal », « créateurs YouTube Québec gaming », palmarès, répertoires).\n"
131 + "3. APPROFONDIS : deep_dive sur les annuaires, rosters d'agences et pages de créateurs (scrape "
132 + "parallèle) — c'est là que se trouvent handles, abonnés et relations. Outil PRINCIPAL.\n"
133 + "4. AVANCE VITE : crawl_queue traite en parallèle la frontière qui se remplit automatiquement.\n"
134 + "5. COMPLÈTE : scrape_page pour une page précise; enqueue_urls pour de nouvelles pistes.\n"
135 + "6. VÉRIFIE : note_progress (couverture par niche/plateforme, qualité des relations).\n"
136 + "7. finish quand la limite est atteinte.\n\n"
137 + "RÈGLES : données PUBLIQUES uniquement; priorise le Québec et les LIENS créateur↔agence↔marque; "
138 + "ne répète pas une action; une action à la fois; raisonne brièvement. robots.txt/débit gérés par les outils."
139 +)
140 +
141 +ToolFn = Callable[[dict[str, Any]], dict[str, Any]]
142 +EventFn = Callable[[str, str], None]
143 +
144 +
145 +class Controller:
146 + def __init__(
147 + self,
148 + client: Anthropic,
149 + model: str,
150 + tool_impls: dict[str, ToolFn],
151 + on_event: Optional[EventFn] = None,
152 + max_context_tokens: int = 120_000,
153 + keep_last_messages: int = 6,
154 + ):
155 + self.client = client
156 + self.model = model
157 + self.tools = tool_impls
158 + self.on_event = on_event or (lambda kind, msg: None)
159 + self.max_context_tokens = max_context_tokens
160 + self.keep_last_messages = keep_last_messages
161 + # garde anti-boucle
162 + self._done_actions: set[str] = set()
163 +
164 + # -- boucle principale -------------------------------------------------
165 + def run(self, goal: str, seed: str | None = None, max_steps: int = 40,
166 + should_abort: Optional[Callable[[], bool]] = None) -> str:
167 + user = f"Objectif de mission : {goal}"
168 + if seed:
169 + user += f"\nPoint de départ suggéré : {seed}"
170 + messages: list[dict[str, Any]] = [{"role": "user", "content": user}]
171 +
172 + for step in range(max_steps):
173 + if should_abort and should_abort():
174 + self.on_event("agent", "Mission interrompue par l'utilisateur.")
175 + return "Interrompu par l'utilisateur."
176 + messages = self._maybe_compact(messages)
177 + resp = self.client.messages.create(
178 + model=self.model,
179 + max_tokens=1600,
180 + system=SYSTEM,
181 + tools=TOOLS,
182 + messages=messages,
183 + )
184 + messages.append({"role": "assistant", "content": resp.content})
185 +
186 + tool_uses = [b for b in resp.content if b.type == "tool_use"]
187 + for b in resp.content:
188 + if b.type == "text" and b.text.strip():
189 + self.on_event("agent", b.text.strip()[:500])
190 +
191 + if not tool_uses:
192 + return self._final(resp)
193 +
194 + results = []
195 + for tu in tool_uses:
196 + if tu.name == "finish":
197 + summary = tu.input.get("summary", "Mission terminée.")
198 + self.on_event("agent", f"FIN : {summary[:300]}")
199 + return summary
200 +
201 + out = self._dispatch(tu.name, tu.input)
202 + results.append(
203 + {"type": "tool_result", "tool_use_id": tu.id, "content": _stringify(out)}
204 + )
205 + messages.append({"role": "user", "content": results})
206 +
207 + return "Budget d'étapes épuisé (mission non conclue explicitement)."
208 +
209 + # -- exécution d'un outil (avec garde anti-boucle) --------------------
210 + def _dispatch(self, name: str, inp: dict[str, Any]) -> Any:
211 + key = name + ":" + json.dumps(inp, ensure_ascii=False, sort_keys=True)
212 + if name in ("search_web", "map_site", "scrape_page", "deep_dive") and key in self._done_actions:
213 + return {"skipped": "action déjà effectuée — change d'angle ou d'URL."}
214 + self._done_actions.add(key)
215 +
216 + self.on_event("agent", f"→ {name}({_short(inp)})")
217 + fn = self.tools.get(name)
218 + if not fn:
219 + return {"error": f"outil inconnu: {name}"}
220 + try:
221 + return fn(inp)
222 + except Exception as e: # noqa: BLE001
223 + self.on_event("error", f"{name}: {e}")
224 + return {"error": str(e)}
225 +
226 + # -- auto-compaction ---------------------------------------------------
227 + def _maybe_compact(self, messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
228 + if self._estimate_tokens(messages) < self.max_context_tokens:
229 + return messages
230 + if len(messages) <= self.keep_last_messages + 2:
231 + return messages
232 +
233 + cut = self._safe_cut(messages)
234 + if cut <= 1:
235 + return messages
236 +
237 + head, tail = messages[:cut], messages[cut:]
238 + summary = self._summarize(head)
239 + self.on_event("info", f"Contexte compacté : {len(head)} messages résumés.")
240 + return [{"role": "user", "content": f"[MÉMOIRE COMPACTÉE — progrès à ce jour]\n{summary}"}] + tail
241 +
242 + def _safe_cut(self, messages: list[dict[str, Any]]) -> int:
243 + """Coupe à une frontière valide : juste après un message user (tool_result),
244 + pour que le premier message conservé soit un assistant et que toute paire
245 + tool_use/tool_result reste intacte."""
246 + target = len(messages) - self.keep_last_messages
247 + cut = 0
248 + for i in range(min(target, len(messages) - 1)):
249 + if messages[i]["role"] == "user":
250 + cut = i + 1
251 + return cut
252 +
253 + def _summarize(self, head: list[dict[str, Any]]) -> str:
254 + transcript = _stringify(head, limit=40000)
255 + try:
256 + resp = self.client.messages.create(
257 + model=self.model,
258 + max_tokens=1200,
259 + system=(
260 + "Résume l'avancement d'une mission de cartographie web pour poursuivre sans perdre "
261 + "le contexte. Conserve : plan courant, secteurs/régions couverts, URLs et domaines "
262 + "déjà traités, entités clés trouvées, pistes en attente, angles morts. Sois dense et factuel."
263 + ),
264 + messages=[{"role": "user", "content": f"Historique à résumer :\n{transcript}"}],
265 + )
266 + return "".join(b.text for b in resp.content if b.type == "text").strip() or "(résumé vide)"
267 + except Exception as e: # noqa: BLE001
268 + return f"(échec du résumé: {e})"
269 +
270 + def _estimate_tokens(self, messages: list[dict[str, Any]]) -> int:
271 + return len(_stringify(messages, limit=10**9)) // 3
272 +
273 + def _final(self, resp) -> str:
274 + return "".join(b.text for b in resp.content if b.type == "text").strip() or "(fin sans appel d'outil)"
275 +
276 +
277 +def _short(d: dict[str, Any]) -> str:
278 + s = json.dumps(d, ensure_ascii=False)
279 + return s if len(s) <= 120 else s[:117] + "..."
280 +
281 +
282 +def _stringify(obj: Any, limit: int = 6000) -> str:
283 + try:
284 + s = json.dumps(obj, ensure_ascii=False, default=_json_default)
285 + except Exception:
286 + s = str(obj)
287 + return s[:limit]
288 +
289 +
290 +def _json_default(o: Any) -> Any:
291 + # Les blocs de contenu Anthropic ne sont pas JSON-sérialisables nativement.
292 + for attr in ("model_dump", "dict", "to_dict"):
293 + if hasattr(o, attr):
294 + try:
295 + return getattr(o, attr)()
296 + except Exception:
297 + pass
298 + return str(o)
added src/normalize.py +82 −0
@@ -0,0 +1,82 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Normalisation pour la résolution d'entités et le regroupement géographique.
4 +
5 +- normalize_name : clé canonique d'un nom (accents, ponctuation, suffixes légaux retirés).
6 +- canonical_location : regroupe les variantes ("Montreal"/"Montréal"/"Montréal, QC") en un
7 + libellé unique, et rattache les grandes villes du Québec à un libellé stable.
8 +- domain : domaine racine d'une URL (pour relier des entités du même site).
9 +"""
10 +
11 +from __future__ import annotations
12 +
13 +import re
14 +import unicodedata
15 +from urllib.parse import urlparse
16 +
17 +_LEGAL = {
18 + "inc", "ltee", "ltd", "limited", "limitee", "enr", "senc", "sencrl", "srl",
19 + "corp", "corporation", "cie", "co", "llc", "llp", "sec", "sa", "sas",
20 +}
21 +
22 +
23 +def strip_accents(s: str) -> str:
24 + return "".join(c for c in unicodedata.normalize("NFD", s or "") if unicodedata.category(c) != "Mn")
25 +
26 +
27 +def normalize_name(name: str) -> str:
28 + """Clé de résolution : minuscules, sans accents/ponctuation ni suffixe légal."""
29 + s = strip_accents((name or "").lower())
30 + s = re.sub(r"[^a-z0-9 ]+", " ", s)
31 + toks = [t for t in s.split() if t and t not in _LEGAL]
32 + return " ".join(toks).strip()
33 +
34 +
35 +def domain(url: str | None) -> str:
36 + if not url:
37 + return ""
38 + u = url.strip()
39 + if "://" not in u:
40 + u = "http://" + u
41 + try:
42 + net = urlparse(u).netloc.lower()
43 + except Exception:
44 + return ""
45 + if net.startswith("www."):
46 + net = net[4:]
47 + return net
48 +
49 +
50 +# Grandes villes / variantes fréquentes -> libellé canonique
51 +_CITY_MAP = {
52 + "montreal": "Montréal", "mtl": "Montréal", "ville-marie": "Montréal",
53 + "quebec": "Québec", "quebec city": "Québec", "ville de quebec": "Québec",
54 + "laval": "Laval", "gatineau": "Gatineau", "hull": "Gatineau",
55 + "sherbrooke": "Sherbrooke", "trois rivieres": "Trois-Rivières",
56 + "saguenay": "Saguenay", "chicoutimi": "Saguenay", "jonquiere": "Saguenay",
57 + "levis": "Lévis", "longueuil": "Longueuil", "terrebonne": "Terrebonne",
58 + "brossard": "Brossard", "repentigny": "Repentigny", "drummondville": "Drummondville",
59 + "saint jean sur richelieu": "Saint-Jean-sur-Richelieu", "granby": "Granby",
60 + "blainville": "Blainville", "saint jerome": "Saint-Jérôme", "mirabel": "Mirabel",
61 + "rimouski": "Rimouski", "victoriaville": "Victoriaville", "shawinigan": "Shawinigan",
62 + "rouyn noranda": "Rouyn-Noranda", "sept iles": "Sept-Îles", "val d or": "Val-d'Or",
63 + "boucherville": "Boucherville", "mascouche": "Mascouche", "salaberry": "Salaberry-de-Valleyfield",
64 + "chateauguay": "Châteauguay", "saint hyacinthe": "Saint-Hyacinthe", "sorel": "Sorel-Tracy",
65 + "joliette": "Joliette", "magog": "Magog", "alma": "Alma", "thetford": "Thetford Mines",
66 +}
67 +
68 +
69 +def canonical_location(text: str | None) -> str:
70 + """Retourne un libellé de lieu canonique (ville QC connue, sinon 1er segment nettoyé)."""
71 + if not text:
72 + return ""
73 + base = strip_accents(text.lower())
74 + base = re.sub(r"[^a-z0-9 ]+", " ", base)
75 + base = re.sub(r"\s+", " ", base).strip()
76 + for kw, label in _CITY_MAP.items():
77 + if re.search(rf"\b{re.escape(kw)}\b", base):
78 + return label
79 + # sinon : 1er segment avant virgule, sans mentions province/pays
80 + seg = text.split(",")[0].strip()
81 + seg = re.sub(r"(?i)\b(qc|québec|quebec|canada)\b", "", seg).strip(" ,-")
82 + return seg[:60]
added src/orchestrator.py +273 −0
@@ -0,0 +1,273 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Orchestrateur ka4 — version plus large / agressive / capable.
4 +
5 +Nouveautés vs ka2 :
6 +- Crawling CONCURRENT : fetch (Firecrawl/Scrapfly) + extraction (Haiku) en parallèle
7 + via un pool de threads ; les écritures SQLite restent sérialisées sur le thread
8 + principal (une seule connexion) pour rester sûres.
9 +- Frontière AGRESSIVE : chaque page auto-alimente la file avec les liens découverts
10 + (nouveaux domaines priorisés) pour élargir la couverture en continu.
11 +- Contrôleur agentique sur un modèle PLUS PUISSANT (Sonnet), extraction sur Haiku.
12 +"""
13 +
14 +from __future__ import annotations
15 +
16 +from concurrent.futures import ThreadPoolExecutor, as_completed
17 +from typing import Any
18 +from urllib.parse import urlparse
19 +
20 +from anthropic import Anthropic
21 +
22 +from .config import Config
23 +from .extractor import Extractor
24 +from .llm_controller import Controller
25 +from .scraper import Scraper
26 +from .storage import Store
27 +
28 +_SKIP_HOSTS = ("facebook.", "instagram.", "twitter.", "x.com", "linkedin.", "youtube.",
29 + "tiktok.", "pinterest.", "google.", "maps.", "apple.", "amazon.")
30 +_SKIP_EXT = (".jpg", ".jpeg", ".png", ".pdf", ".zip", ".mp4", ".svg", ".gif", ".webp",
31 + ".css", ".js", ".xml", ".ico", ".woff", ".woff2")
32 +
33 +
34 +class Orchestrator:
35 + def __init__(self, cfg: Config):
36 + cfg.validate()
37 + self.cfg = cfg
38 + self.anthropic = Anthropic(api_key=cfg.anthropic_api_key)
39 + self.fc = Scraper(cfg)
40 + self.store = Store(cfg.db_path)
41 + # extraction sur le modèle rapide ; contrôleur sur cfg.model (plus puissant)
42 + self.extractor = Extractor(self.anthropic, cfg.extract_model)
43 + self._pages_scraped = 0
44 + self._domain_fails: dict[str, int] = {} # coupe-circuit par domaine (robustesse)
45 +
46 + # -- fetch pur (THREAD-SAFE : réseau + LLM, aucun accès DB) -------------
47 + def _fetch_only(self, url: str) -> dict[str, Any]:
48 + data = self.fc.scrape(url)
49 + if data.get("blocked"):
50 + return {"url": url, "blocked": True}
51 + if data.get("error"):
52 + return {"url": url, "error": data["error"]}
53 + md = data.get("markdown", "") or ""
54 + meta = data.get("metadata", {}) or {}
55 + title = meta.get("title", "") or ""
56 + graph = (self.extractor.extract_graph(url, title, md)
57 + if md.strip() else {"entities": [], "relations": []})
58 + return {"url": url, "title": title, "backend": data.get("backend"),
59 + "markdown": md, "links": data.get("links", []) or [], "graph": graph}
60 +
61 + # -- persistance (THREAD PRINCIPAL uniquement) -------------------------
62 + def _save_result(self, r: dict[str, Any]) -> dict[str, Any]:
63 + url = r["url"]
64 + if r.get("blocked"):
65 + self.store.log_event("info", "robots.txt interdit", url)
66 + return {"entities_found": 0, "relations_found": 0, "blocked": True}
67 + if r.get("error"):
68 + self._domain_fails[urlparse(url).netloc.lower()] = \
69 + self._domain_fails.get(urlparse(url).netloc.lower(), 0) + 1
70 + self.store.log_event("error", r["error"], url)
71 + return {"entities_found": 0, "relations_found": 0, "error": r["error"]}
72 +
73 + self.store.record_source(url, r.get("title", ""), r.get("markdown", ""))
74 + self.store.log_event("scrape", f"{r.get('title') or url} [{r.get('backend')}]", url,
75 + {"chars": len(r.get("markdown", ""))})
76 + graph = r.get("graph", {}) or {}
77 + # robustesse : le LLM peut occasionnellement renvoyer un item non-dict -> on filtre
78 + entities = [e for e in (graph.get("entities") or []) if isinstance(e, dict)]
79 + relations = [x for x in (graph.get("relations") or []) if isinstance(x, dict)]
80 + idmap: dict[str, int] = {}
81 + for e in entities:
82 + eid = self.store.upsert_entity(e, source_url=url)
83 + if e.get("temp_id"):
84 + idmap[e["temp_id"]] = eid
85 + rel = 0
86 + for rr in relations:
87 + fid, tid = idmap.get(rr.get("from")), idmap.get(rr.get("to"))
88 + if fid and tid:
89 + self.store.add_relation(fid, tid, rr.get("type", ""), rr.get("role", ""),
90 + source_url=url, confidence=float(rr.get("confidence", 0.6) or 0.6))
91 + rel += 1
92 + self._enqueue_frontier(url, r.get("links", []))
93 + self.store.log_event("extract", f"{len(idmap)} entité(s), {rel} relation(s)", url,
94 + {"relations": rel})
95 + return {"entities_found": len(idmap), "relations_found": rel,
96 + "links_on_page": (r.get("links", []) or [])[:25]}
97 +
98 + def _enqueue_frontier(self, url: str, links: list) -> None:
99 + if self.cfg.frontier_per_page <= 0 or not links:
100 + return
101 + base = urlparse(url).netloc.lower()
102 + picked = []
103 + for l in links:
104 + u = l["url"] if isinstance(l, dict) else l
105 + if not isinstance(u, str) or not u.startswith("http"):
106 + continue
107 + h = urlparse(u).netloc.lower()
108 + if not h or any(s in h for s in _SKIP_HOSTS) or u.lower().endswith(_SKIP_EXT):
109 + continue
110 + picked.append(u)
111 + # priorise les NOUVEAUX domaines (élargit la couverture)
112 + picked.sort(key=lambda u: urlparse(u).netloc.lower() == base)
113 + self.store.enqueue(picked[: self.cfg.frontier_per_page])
114 +
115 + # -- crawling CONCURRENT ----------------------------------------------
116 + def _tripped(self, url: str) -> bool:
117 + """Coupe-circuit : True si le domaine a trop échoué dans ce run."""
118 + return self._domain_fails.get(urlparse(url).netloc.lower(), 0) >= self.cfg.domain_fail_threshold
119 +
120 + def crawl_batch(self, urls: list[str]) -> dict[str, Any]:
121 + seen, todo = set(), []
122 + for u in urls:
123 + if not u or u in seen:
124 + continue
125 + seen.add(u)
126 + if self._tripped(u):
127 + continue # domaine coupé (trop d'échecs)
128 + if not self.store.source_seen(u):
129 + todo.append(u)
130 + remaining = max(0, self.cfg.max_pages_per_run - self._pages_scraped)
131 + todo = todo[:remaining]
132 + if not todo:
133 + return {"pages": 0, "entities_found": 0, "relations_found": 0}
134 + results = []
135 + with ThreadPoolExecutor(max_workers=max(1, self.cfg.concurrency)) as ex:
136 + futs = [ex.submit(self._fetch_only, u) for u in todo]
137 + for f in as_completed(futs):
138 + try:
139 + results.append(f.result())
140 + except Exception as e: # noqa: BLE001
141 + self.store.log_event("error", f"fetch: {e}", "")
142 + tot_e = tot_r = pages = 0
143 + for r in results:
144 + out = self._save_result(r)
145 + if not (r.get("blocked") or r.get("error")):
146 + self._pages_scraped += 1
147 + pages += 1
148 + tot_e += out.get("entities_found", 0)
149 + tot_r += out.get("relations_found", 0)
150 + return {"pages": pages, "entities_found": tot_e, "relations_found": tot_r}
151 +
152 + # -- scrape unitaire (utilisé par l'agent) ----------------------------
153 + def _scrape_and_extract(self, url: str) -> dict[str, Any]:
154 + if self._pages_scraped >= self.cfg.max_pages_per_run:
155 + return {"stopped": "limite de pages atteinte", "entities_found": 0}
156 + if self.store.source_seen(url):
157 + return {"skipped": "déjà scrapé", "entities_found": 0}
158 + r = self._fetch_only(url)
159 + out = self._save_result(r)
160 + if not (r.get("blocked") or r.get("error")):
161 + self._pages_scraped += 1
162 + return {**out, "url": url, "title": r.get("title", ""), "backend": r.get("backend")}
163 +
164 + # -- crawling centré-entité (concurrent) ------------------------------
165 + _DEEP_KEYWORDS = ("about", "a-propos", "apropos", "propos", "equipe", "team", "notre-equipe",
166 + "leadership", "direction", "gouvernance", "membres", "conseil", "contact",
167 + "coordonnees", "nous-joindre", "qui-sommes", "notre-histoire", "carrieres")
168 +
169 + def deep_dive(self, url: str, max_pages: int = 8) -> dict[str, Any]:
170 + parts = urlparse(url if "://" in url else "http://" + url)
171 + root = f"{parts.scheme or 'https'}://{parts.netloc or parts.path}"
172 + try:
173 + links = self.fc.map(root)
174 + except Exception as e: # noqa: BLE001
175 + links = [root]
176 + self.store.log_event("error", f"map deep_dive: {e}", root)
177 +
178 + def score(u: str) -> int:
179 + lu = u.lower()
180 + return sum(2 if k in lu else 0 for k in self._DEEP_KEYWORDS)
181 + targets = [t for t in sorted(set(links), key=score, reverse=True) if score(t) > 0][: max_pages - 1]
182 + targets = [root] + targets
183 + self.store.log_event("agent", f"deep_dive {parts.netloc}: {len(targets)} page(s), concurrent", root)
184 + res = self.crawl_batch(targets)
185 + return {"domain": parts.netloc, "pages": res["pages"],
186 + "entities_found": res["entities_found"], "relations_found": res["relations_found"]}
187 +
188 + # -- mode agent (contrôleur = modèle plus puissant) -------------------
189 + def run_agent(self, goal: str, seed: str | None, max_steps: int, should_abort=None) -> str:
190 + self.store.log_event("agent", f"Mission: {goal}", "")
191 +
192 + def t_search(inp: dict[str, Any]) -> dict[str, Any]:
193 + res = self.fc.search(inp["query"], int(inp.get("limit", 10)))
194 + self.store.log_event("search", inp["query"], "", {"results": len(res)})
195 + return {"results": [{"url": r.get("url"), "title": r.get("title"),
196 + "desc": r.get("description")} for r in res]}
197 +
198 + def t_map(inp: dict[str, Any]) -> dict[str, Any]:
199 + links = self.fc.map(inp["url"], inp.get("search"))
200 + self.store.enqueue(links[:150])
201 + self.store.log_event("map", inp["url"], inp["url"], {"links": len(links)})
202 + return {"count": len(links), "sample": links[:40]}
203 +
204 + def t_scrape(inp: dict[str, Any]) -> dict[str, Any]:
205 + return self._scrape_and_extract(inp["url"])
206 +
207 + def t_deep(inp: dict[str, Any]) -> dict[str, Any]:
208 + return self.deep_dive(inp["url"], int(inp.get("max_pages", 8)))
209 +
210 + def t_enqueue(inp: dict[str, Any]) -> dict[str, Any]:
211 + return {"enqueued": self.store.enqueue(inp.get("urls", []))}
212 +
213 + def t_drain(inp: dict[str, Any]) -> dict[str, Any]:
214 + return self.crawl_batch([r["url"] for r in self.store.pending_batch(
215 + int(inp.get("count", self.cfg.concurrency)))])
216 +
217 + def t_plan(inp: dict[str, Any]) -> dict[str, Any]:
218 + self.store.log_event("agent", "PLAN: " + " | ".join(inp.get("plan", [])), "",
219 + {"rationale": inp.get("rationale", "")})
220 + return {"ack": True}
221 +
222 + def t_note(inp: dict[str, Any]) -> dict[str, Any]:
223 + self.store.log_event("agent", "NOTE: " + inp.get("note", ""), "")
224 + return {"ack": True}
225 +
226 + controller = Controller(
227 + self.anthropic, self.cfg.model,
228 + {"search_web": t_search, "map_site": t_map, "scrape_page": t_scrape,
229 + "deep_dive": t_deep, "crawl_queue": t_drain, "enqueue_urls": t_enqueue,
230 + "update_plan": t_plan, "note_progress": t_note},
231 + on_event=lambda kind, msg: self.store.log_event(kind, msg, ""),
232 + max_context_tokens=self.cfg.max_context_tokens,
233 + )
234 + summary = controller.run(goal=goal, seed=seed, max_steps=max_steps, should_abort=should_abort)
235 + self.store.log_event("agent", "Mission conclue", "", {"summary": summary})
236 + return summary
237 +
238 + # -- pipeline (concurrent) --------------------------------------------
239 + def run_pipeline(self, query: str, limit: int = 10, per_site_pages: int = 3) -> dict[str, Any]:
240 + results = self.fc.search(query, limit)
241 + urls = [r.get("url") for r in results if r.get("url")]
242 + res = self.crawl_batch(urls)
243 + # approfondir les domaines découverts
244 + for u in urls[:per_site_pages]:
245 + if self._pages_scraped >= self.cfg.max_pages_per_run:
246 + break
247 + self.deep_dive(u, max_pages=per_site_pages + 2)
248 + return {"query": query, "pages_scraped": self._pages_scraped,
249 + "entities_found": res["entities_found"], "stats": self.store.stats()}
250 +
251 + # -- file d'attente (drainée en concurrence) --------------------------
252 + def drain_queue(self, max_pages: int | None = None, should_abort=None) -> dict[str, Any]:
253 + budget = max_pages or self.cfg.max_pages_per_run
254 + processed = 0
255 + while processed < budget and self._pages_scraped < self.cfg.max_pages_per_run:
256 + if should_abort and should_abort():
257 + break
258 + rows = self.store.pending_batch(min(self.cfg.concurrency, budget - processed),
259 + max_attempts=self.cfg.max_attempts)
260 + if not rows:
261 + break
262 + self.crawl_batch([r["url"] for r in rows])
263 + # robustesse : succès -> done ; échec -> retry (attempts++) puis dead-letter
264 + for r in rows:
265 + if self.store.source_seen(r["url"]):
266 + self.store.mark(r["id"], "done")
267 + else:
268 + self.store.bump_attempt(r["id"], self.cfg.max_attempts)
269 + processed += len(rows)
270 + return {"processed": processed, "stats": self.store.stats()}
271 +
272 + def close(self) -> None:
273 + self.store.close()
added src/robots.py +59 −0
@@ -0,0 +1,59 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Cache robots.txt robuste.
4 +
5 +`urllib.robotparser.RobotFileParser.read()` récupère le robots.txt avec le
6 +User-Agent `Python-urllib`, souvent bloqué (403) par les serveurs, ce qui bascule
7 +le parseur en `disallow_all=True` (faux négatif). On récupère donc le fichier nous-
8 +mêmes via `requests` avec un vrai User-Agent, puis on le parse.
9 +"""
10 +
11 +from __future__ import annotations
12 +
13 +import urllib.robotparser
14 +from typing import Optional
15 +from urllib.parse import urlparse
16 +
17 +import requests
18 +
19 +
20 +class RobotsCache:
21 + def __init__(self, user_agent: str, enabled: bool = True, timeout: int = 15):
22 + self.user_agent = user_agent
23 + self.enabled = enabled
24 + self.timeout = timeout
25 + self._cache: dict[str, Optional[urllib.robotparser.RobotFileParser]] = {}
26 +
27 + def _load(self, root: str) -> Optional[urllib.robotparser.RobotFileParser]:
28 + if root in self._cache:
29 + return self._cache[root]
30 + rp: Optional[urllib.robotparser.RobotFileParser] = None
31 + try:
32 + r = requests.get(
33 + f"{root}/robots.txt",
34 + headers={"User-Agent": self.user_agent},
35 + timeout=self.timeout,
36 + )
37 + if r.status_code == 200 and r.text.strip():
38 + rp = urllib.robotparser.RobotFileParser()
39 + rp.parse(r.text.splitlines())
40 + # 4xx/5xx ou vide -> pas de règles connues -> on autorise (rp=None)
41 + except Exception:
42 + rp = None # robots injoignable -> on autorise, la politesse vient du rate-limit
43 + self._cache[root] = rp
44 + return rp
45 +
46 + def allowed(self, url: str) -> bool:
47 + if not self.enabled:
48 + return True
49 + try:
50 + parts = urlparse(url)
51 + if not parts.scheme or not parts.netloc:
52 + return True
53 + root = f"{parts.scheme}://{parts.netloc}"
54 + rp = self._load(root)
55 + if rp is None:
56 + return True
57 + return rp.can_fetch(self.user_agent, url)
58 + except Exception:
59 + return True
added src/scheduler.py +223 −0
@@ -0,0 +1,223 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Bot LONG TERME : boucle persistante, reprise sur incident, budget quotidien.
4 +
5 +Cycle :
6 + 1. Draine une partie de la file d'URLs découvertes (exploration en largeur).
7 + 2. Sinon, prend la prochaine mission « due » et laisse l'agent Haiku la mener.
8 + 3. Revisite périodique des missions (crawl perpétuel).
9 +Respecte un budget de pages/jour et s'arrête proprement (SIGINT/SIGTERM).
10 +L'état vit dans SQLite : le bot peut être tué et redémarré sans rien perdre.
11 +"""
12 +
13 +from __future__ import annotations
14 +
15 +import signal
16 +import time
17 +from typing import Optional
18 +
19 +from .config import Config
20 +from .orchestrator import Orchestrator
21 +
22 +# Amorçage FLAGSHIP ka6 : découverte des CRÉATEURS / INFLUENCEURS québécois.
23 +NICHES = [
24 + "mode", "beauté", "maquillage", "gaming", "cuisine", "humour", "lifestyle", "sport",
25 + "fitness", "techno", "voyage", "famille et parentalité", "musique", "art et illustration",
26 + "décoration et rénovation", "finances personnelles", "plein air", "food et restaurants",
27 + "danse", "actualité et politique",
28 +]
29 +PLATFORMS = ["TikTok", "Instagram", "YouTube", "Twitch"]
30 +# Requêtes annuaire/agences (missions à haute densité de créateurs)
31 +DIRECTORY_MISSIONS = [
32 + "Cartographier les agences d'influence et de gestion de talents au Québec et les créateurs qu'elles représentent (handle, plateformes, abonnés, niche).",
33 + "Répertorier les palmarès et listes des meilleurs influenceurs et créateurs de contenu québécois (tous réseaux).",
34 + "Découvrir les créateurs de contenu franco-québécois sur YouTube : chaînes, abonnés, thématique, collaborations.",
35 + "Découvrir les influenceurs québécois sur TikTok par niche : handle, abonnés, marques partenaires.",
36 + "Cartographier les plateformes et collectifs de créateurs au Québec (réseaux, MCN, coopératives de contenu).",
37 + "Identifier les marques québécoises qui font du marketing d'influence et leurs créateurs ambassadeurs.",
38 +]
39 +DEFAULT_REGIONS = ["Québec"] # objectif initial : tout le Québec
40 +
41 +
42 +class LongRunner:
43 + # ka6 flagship : cadence très agressive et budget élevé
44 + def __init__(self, cfg: Config, cycle_delay: float = 6.0, queue_batch: int = 40,
45 + daily_page_budget: int = 9000, revisit_hours: float = 36.0,
46 + session_minutes: float = 15.0):
47 + self.cfg = cfg
48 + self.cycle_delay = cycle_delay
49 + self.queue_batch = queue_batch
50 + self.daily_page_budget = daily_page_budget
51 + self.revisit_seconds = revisit_hours * 3600
52 + self.session_minutes = session_minutes
53 + self._stop = False
54 + self._idle_logged = False
55 +
56 + def _install_signals(self) -> None:
57 + def handler(signum, _frame):
58 + self._stop = True
59 + for sig in (signal.SIGINT, signal.SIGTERM):
60 + try:
61 + signal.signal(sig, handler)
62 + except Exception:
63 + pass
64 +
65 + def seed_missions(self, orch: Orchestrator, limit: int = 120) -> int:
66 + # Pré-remplissage désactivé par défaut : le bot démarre VIDE et attend des missions
67 + # ajoutées via le dashboard. Activer avec KA_BOT_AUTOSEED=1 pour réamorcer.
68 + import os
69 + if os.getenv("KA_BOT_AUTOSEED", "0").strip().lower() not in ("1", "true", "yes", "on"):
70 + return 0
71 + if orch.store.list_missions():
72 + return 0
73 + n = 0
74 + # 1) missions annuaires/agences (haute priorité — riches en créateurs)
75 + for goal in DIRECTORY_MISSIONS:
76 + orch.store.add_mission(goal=goal, sector="influence", region="Québec", priority=1)
77 + n += 1
78 + # 2) niche × plateforme
79 + for niche in NICHES:
80 + for platform in PLATFORMS:
81 + if n >= limit:
82 + break
83 + goal = (f"Découvrir les créateurs de contenu et influenceurs québécois en « {niche} » "
84 + f"sur {platform} : nom, handle, plateformes, nombre d'abonnés, niche, langues, "
85 + f"agence qui les représente et marques avec qui ils collaborent.")
86 + orch.store.add_mission(goal=goal, sector=niche, region=platform, priority=3)
87 + n += 1
88 + orch.store.log_event("info", f"{n} missions créateurs/influenceurs initiales créées.")
89 + return n
90 +
91 + def _apply_live_settings(self, orch: Orchestrator) -> None:
92 + """Réglages modifiables en direct depuis le dashboard (table settings)."""
93 + s = orch.store
94 + def num(key, cur, cast):
95 + v = s.get_setting(key, "")
96 + try:
97 + return cast(v) if v != "" else cur
98 + except ValueError:
99 + return cur
100 + self.daily_page_budget = num("cfg_daily_budget", self.daily_page_budget, int)
101 + self.cycle_delay = num("cfg_cycle_delay", self.cycle_delay, float)
102 + self.queue_batch = num("cfg_queue_batch", self.queue_batch, int)
103 + self.session_minutes = num("cfg_session_minutes", self.session_minutes, float)
104 + orch.cfg.max_pages_per_run = num("cfg_max_pages", orch.cfg.max_pages_per_run, int)
105 + # backend + robots + délai appliqués en direct au scraper
106 + backend = s.get_setting("cfg_backend", "")
107 + if backend in ("auto", "firecrawl", "scrapfly"):
108 + orch.fc.backend = backend
109 + robots = s.get_setting("cfg_respect_robots", "")
110 + if robots in ("0", "1"):
111 + orch.fc.robots.enabled = robots == "1"
112 + delay = num("cfg_delay", None, float)
113 + if delay is not None:
114 + orch.fc.firecrawl.limiter.delay = max(0.0, delay)
115 + if orch.fc.scrapfly:
116 + orch.fc.scrapfly.limiter.delay = max(0.0, delay)
117 +
118 + def _start_of_day(self) -> float:
119 + lt = time.localtime()
120 + return time.mktime((lt.tm_year, lt.tm_mon, lt.tm_mday, 0, 0, 0, 0, 0, -1))
121 +
122 + def run(self, max_cycles: Optional[int] = None) -> None:
123 + self._install_signals()
124 + orch = Orchestrator(self.cfg)
125 + orch.store.requeue_stale_missions()
126 + self.seed_missions(orch)
127 +
128 + # Fenêtre de session : dès le démarrage, le bot travaille `session_minutes`.
129 + # Le bouton « Continuer » du dashboard prolonge run_until (voir web/app.py).
130 + if not orch.store.get_setting("run_until", ""):
131 + orch.store.set_setting("run_until", str(time.time() + self.session_minutes * 60))
132 + orch.store.log_event("info", f"Session de {int(self.session_minutes)} min démarrée dès le déploiement.")
133 + orch.store.log_event("info", "ka6 démarré.")
134 +
135 + cycles = 0
136 + try:
137 + while not self._stop:
138 + if max_cycles is not None and cycles >= max_cycles:
139 + break
140 + cycles += 1
141 + self._apply_live_settings(orch)
142 + orch.store.set_setting("bot_busy", "0") # au repos par défaut (heartbeat)
143 +
144 + # Pause pilotée depuis le dashboard
145 + if orch.store.get_setting("paused", "0") == "1":
146 + self._sleep(self.cycle_delay)
147 + continue
148 +
149 + # Fenêtre de session : au-delà de run_until, on attend « Continuer »
150 + try:
151 + run_until = float(orch.store.get_setting("run_until", "0") or 0)
152 + except ValueError:
153 + run_until = 0.0
154 + if time.time() >= run_until:
155 + if not self._idle_logged:
156 + orch.store.log_event(
157 + "info", "Session écoulée — cliquez « Continuer » pour 15 min de plus."
158 + )
159 + self._idle_logged = True
160 + self._sleep(self.cycle_delay)
161 + continue
162 + self._idle_logged = False
163 +
164 + # Budget quotidien
165 + done_today = orch.store.pages_since(self._start_of_day())
166 + if done_today >= self.daily_page_budget:
167 + orch.store.log_event("info", f"Budget quotidien atteint ({done_today}). Pause.")
168 + self._sleep(min(1800, self.cycle_delay * 30))
169 + continue
170 +
171 + orch._pages_scraped = 0 # réinitialise la limite par cycle
172 + # STOP = arrêt demandé (skip/archive) OU pause : interrompt le travail EN COURS
173 + stop_fn = lambda: (orch.store.get_setting("abort_current", "0") == "1"
174 + or orch.store.get_setting("paused", "0") == "1")
175 +
176 + # 0) mission FORCÉE ("lancer maintenant", priorité 0) : passe avant tout
177 + mission = orch.store.next_forced()
178 +
179 + # 1) sinon, drainer la file découverte (interruptible)
180 + if mission is None and orch.store.next_pending() is not None and not stop_fn():
181 + orch.store.set_setting("bot_busy", "1")
182 + res = orch.drain_queue(max_pages=self.queue_batch, should_abort=stop_fn)
183 + orch.store.set_setting("bot_busy", "0")
184 + orch.store.log_event("info", f"File drainée: {res['processed']} page(s).")
185 + orch.store.set_setting("abort_current", "0")
186 + self._sleep(self.cycle_delay)
187 + continue
188 +
189 + # 2) sinon, mission due
190 + if mission is None:
191 + mission = orch.store.due_mission(self.revisit_seconds)
192 + orch.store.set_setting("abort_current", "0") # nouvelle activité → drapeau propre
193 + if mission is None:
194 + orch.store.log_event("info", "Aucune mission due — veille.")
195 + self._sleep(min(1800, self.cycle_delay * 15))
196 + continue
197 +
198 + mid = mission["id"]
199 + orch.store.set_mission_status(mid, "running")
200 + orch.store.set_setting("active_mission_id", str(mid))
201 + orch.store.set_setting("bot_busy", "1")
202 + orch.store.log_event("info", f"Mission #{mid} démarrée: {mission['goal'][:120]}")
203 + try:
204 + summary = orch.run_agent(mission["goal"], mission["seed"], max_steps=40,
205 + should_abort=stop_fn)
206 + st = "pending" if summary == "Interrompu par l'utilisateur." else "done"
207 + orch.store.set_mission_status(mid, st, bump_run=True)
208 + except Exception as e: # noqa: BLE001
209 + orch.store.set_mission_status(mid, "pending", bump_run=True)
210 + orch.store.log_event("error", f"Mission #{mid}: {e}")
211 + finally:
212 + orch.store.set_setting("bot_busy", "0")
213 + orch.store.set_setting("active_mission_id", "")
214 + orch.store.set_setting("abort_current", "0")
215 + self._sleep(self.cycle_delay)
216 + finally:
217 + orch.store.log_event("info", "ka6 long terme arrêté.")
218 + orch.close()
219 +
220 + def _sleep(self, seconds: float) -> None:
221 + end = time.time() + seconds
222 + while time.time() < end and not self._stop:
223 + time.sleep(min(1.0, end - time.time()))
added src/scraper.py +71 −0
@@ -0,0 +1,71 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Dispatcher de scrape multi-backend + garde robots.txt.
4 +
5 +- search / map : toujours via Firecrawl.
6 +- scrape : selon le backend choisi (auto | firecrawl | scrapfly).
7 + En mode `auto`, Firecrawl est tenté d'abord, puis Scrapfly en repli si le
8 + résultat est vide ou en erreur.
9 +"""
10 +
11 +from __future__ import annotations
12 +
13 +from typing import Any, Optional
14 +
15 +from .config import Config
16 +from .firecrawl_client import FirecrawlClient
17 +from .robots import RobotsCache
18 +from .scrapfly_client import ScrapflyClient
19 +
20 +
21 +class Scraper:
22 + def __init__(self, cfg: Config):
23 + self.cfg = cfg
24 + self.firecrawl = FirecrawlClient(cfg.firecrawl_api_key, cfg.firecrawl_base, cfg.request_delay)
25 + self.scrapfly = (
26 + ScrapflyClient(cfg.scrapfly_api_key, cfg.scrapfly_base, cfg.request_delay, cfg.scrapfly_render_js)
27 + if cfg.scrapfly_api_key
28 + else None
29 + )
30 + self.robots = RobotsCache(cfg.user_agent, enabled=cfg.respect_robots)
31 + self.backend = cfg.scraper_backend
32 +
33 + # -- découverte (Firecrawl) -------------------------------------------
34 + def search(self, query: str, limit: int = 10) -> list[dict[str, Any]]:
35 + return self.firecrawl.search(query, limit)
36 +
37 + def map(self, url: str, search: Optional[str] = None) -> list[str]:
38 + return self.firecrawl.map(url, search)
39 +
40 + # -- scrape ------------------------------------------------------------
41 + def scrape(self, url: str) -> dict[str, Any]:
42 + if not self.robots.allowed(url):
43 + return {"blocked": True, "markdown": "", "metadata": {}, "links": [], "backend": None}
44 +
45 + order = self._order()
46 + last_err: Optional[Exception] = None
47 + for backend in order:
48 + try:
49 + data = self._scrape_with(backend, url)
50 + if data.get("markdown", "").strip():
51 + return data
52 + except Exception as e: # noqa: BLE001
53 + last_err = e
54 + if last_err:
55 + return {"error": str(last_err), "markdown": "", "metadata": {}, "links": [], "backend": None}
56 + return {"markdown": "", "metadata": {}, "links": [], "backend": order[-1] if order else None}
57 +
58 + def _order(self) -> list[str]:
59 + if self.backend == "firecrawl":
60 + return ["firecrawl"]
61 + if self.backend == "scrapfly":
62 + return ["scrapfly"] if self.scrapfly else ["firecrawl"]
63 + # auto : Firecrawl puis Scrapfly en repli
64 + return ["firecrawl", "scrapfly"] if self.scrapfly else ["firecrawl"]
65 +
66 + def _scrape_with(self, backend: str, url: str) -> dict[str, Any]:
67 + if backend == "scrapfly":
68 + if not self.scrapfly:
69 + raise RuntimeError("Scrapfly non configuré (SCRAPFLY_API_KEY manquant)")
70 + return self.scrapfly.scrape(url)
71 + return self.firecrawl.scrape(url)
added src/scrapfly_client.py +58 −0
@@ -0,0 +1,58 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Client REST Scrapfly (https://scrapfly.io) — backend de scrape alternatif.
4 +
5 +Atouts : rendu JavaScript, contournement anti-bot (ASP), proxies. Utile quand
6 +Firecrawl échoue ou renvoie une page vide (SPA, protection anti-scraping).
7 +"""
8 +
9 +from __future__ import annotations
10 +
11 +import time
12 +from typing import Any, Optional
13 +
14 +import requests
15 +
16 +from .util import RateLimiter
17 +
18 +
19 +class ScrapflyClient:
20 + def __init__(self, api_key: str, base_url: str = "https://api.scrapfly.io", delay: float = 1.5,
21 + render_js: bool = False):
22 + self.api_key = api_key
23 + self.base = base_url.rstrip("/")
24 + self.render_js = render_js
25 + self.session = requests.Session()
26 + self.limiter = RateLimiter(delay)
27 +
28 + def scrape(self, url: str, render_js: Optional[bool] = None, retries: int = 2) -> dict[str, Any]:
29 + params = {
30 + "key": self.api_key,
31 + "url": url,
32 + "format": "markdown",
33 + "asp": "true", # Anti Scraping Protection bypass
34 + "render_js": "true" if (self.render_js if render_js is None else render_js) else "false",
35 + }
36 + last_err: Optional[Exception] = None
37 + for attempt in range(retries):
38 + self.limiter.wait()
39 + try:
40 + r = self.session.get(f"{self.base}/scrape", params=params, timeout=70)
41 + if r.status_code == 429:
42 + time.sleep(1.5 * (attempt + 1))
43 + continue
44 + r.raise_for_status()
45 + result = (r.json() or {}).get("result", {}) or {}
46 + content = result.get("content", "") or ""
47 + meta = {
48 + "title": (result.get("metadata") or {}).get("title", "")
49 + if isinstance(result.get("metadata"), dict)
50 + else "",
51 + "status_code": result.get("status_code"),
52 + "url": result.get("url", url),
53 + }
54 + return {"markdown": content, "metadata": meta, "links": [], "backend": "scrapfly"}
55 + except Exception as e: # noqa: BLE001
56 + last_err = e
57 + time.sleep(1.5 * (attempt + 1))
58 + raise RuntimeError(f"Scrapfly scrape a échoué: {last_err}")
added src/storage.py +671 −0
@@ -0,0 +1,671 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Persistance SQLite — graphe de connaissances du web québécois (v2).
4 +
5 +Entités enrichies et normalisées, relations typées (avec rôle + provenance),
6 +mentions (provenance multi-sources), déduplication par résolution d'entités,
7 +plus missions / événements / réglages / file de crawl pour le bot long terme.
8 +"""
9 +
10 +from __future__ import annotations
11 +
12 +import hashlib
13 +import json
14 +import sqlite3
15 +import time
16 +from typing import Any, Iterable, Optional
17 +
18 +from .normalize import canonical_location, domain, normalize_name
19 +
20 +
21 +def _to_int(v: Any) -> Optional[int]:
22 + """Parse un nombre d'abonnés : 12000, '12,5 K', '3.2M', '1 200 abonnés' -> int."""
23 + if v is None or v == "":
24 + return None
25 + if isinstance(v, (int, float)):
26 + return int(v)
27 + s = str(v).lower().replace(" ", " ").strip()
28 + import re as _re
29 + m = _re.search(r"([\d]+(?:[.,]\d+)?)\s*([km])?", s.replace(" ", ""))
30 + if not m:
31 + return None
32 + num = float(m.group(1).replace(",", "."))
33 + mult = {"k": 1_000, "m": 1_000_000}.get(m.group(2) or "", 1)
34 + try:
35 + return int(num * mult)
36 + except (ValueError, OverflowError):
37 + return None
38 +
39 +
40 +REL_TYPES = {
41 + "WORKS_AT", "FOUNDER_OF", "OWNS", "MEMBER_OF", "PARTNER_OF",
42 + "SUBSIDIARY_OF", "PARENT_OF", "AFFILIATED_WITH", "LOCATED_IN", "SUPPLIER_OF",
43 + # relations créateurs / influenceurs
44 + "REPRESENTED_BY", "COLLABORATES_WITH", "CREATES_ON", "SPONSORED_BY", "PROMOTES",
45 + "MANAGES", "APPEARS_WITH",
46 +}
47 +
48 +SCHEMA = """
49 +CREATE TABLE IF NOT EXISTS entities (
50 + id INTEGER PRIMARY KEY AUTOINCREMENT,
51 + type TEXT NOT NULL,
52 + name TEXT NOT NULL,
53 + description TEXT,
54 + url TEXT,
55 + location TEXT,
56 + email TEXT,
57 + phone TEXT,
58 + confidence REAL DEFAULT 0.5,
59 + raw_json TEXT,
60 + created_at REAL NOT NULL,
61 + updated_at REAL NOT NULL,
62 + UNIQUE(type, name, url)
63 +);
64 +
65 +CREATE TABLE IF NOT EXISTS social_links (
66 + id INTEGER PRIMARY KEY AUTOINCREMENT,
67 + entity_id INTEGER NOT NULL, platform TEXT NOT NULL, url TEXT NOT NULL,
68 + UNIQUE(entity_id, url),
69 + FOREIGN KEY(entity_id) REFERENCES entities(id) ON DELETE CASCADE
70 +);
71 +
72 +CREATE TABLE IF NOT EXISTS relations (
73 + id INTEGER PRIMARY KEY AUTOINCREMENT,
74 + from_entity INTEGER NOT NULL, to_entity INTEGER NOT NULL, relation_type TEXT NOT NULL,
75 + UNIQUE(from_entity, to_entity, relation_type)
76 +);
77 +
78 +CREATE TABLE IF NOT EXISTS entity_mentions (
79 + id INTEGER PRIMARY KEY AUTOINCREMENT,
80 + entity_id INTEGER NOT NULL, source_url TEXT NOT NULL, seen_at REAL NOT NULL,
81 + UNIQUE(entity_id, source_url),
82 + FOREIGN KEY(entity_id) REFERENCES entities(id) ON DELETE CASCADE
83 +);
84 +
85 +CREATE TABLE IF NOT EXISTS sources (
86 + id INTEGER PRIMARY KEY AUTOINCREMENT,
87 + url TEXT NOT NULL UNIQUE, title TEXT, content_hash TEXT, scraped_at REAL NOT NULL
88 +);
89 +
90 +CREATE TABLE IF NOT EXISTS crawl_queue (
91 + id INTEGER PRIMARY KEY AUTOINCREMENT,
92 + url TEXT NOT NULL UNIQUE, status TEXT NOT NULL DEFAULT 'pending',
93 + depth INTEGER DEFAULT 0, added_at REAL NOT NULL
94 +);
95 +
96 +CREATE TABLE IF NOT EXISTS missions (
97 + id INTEGER PRIMARY KEY AUTOINCREMENT,
98 + goal TEXT NOT NULL, seed TEXT, sector TEXT, region TEXT,
99 + priority INTEGER DEFAULT 5, status TEXT NOT NULL DEFAULT 'pending',
100 + runs_count INTEGER DEFAULT 0, last_run REAL, created_at REAL NOT NULL,
101 + UNIQUE(goal, region)
102 +);
103 +
104 +CREATE TABLE IF NOT EXISTS events (
105 + id INTEGER PRIMARY KEY AUTOINCREMENT,
106 + ts REAL NOT NULL, kind TEXT NOT NULL, message TEXT, url TEXT, data_json TEXT
107 +);
108 +
109 +CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT);
110 +
111 +CREATE TABLE IF NOT EXISTS archives (
112 + id INTEGER PRIMARY KEY AUTOINCREMENT, label TEXT, created_at REAL NOT NULL,
113 + entities INTEGER, relations INTEGER, sources INTEGER, payload TEXT
114 +);
115 +
116 +CREATE INDEX IF NOT EXISTS idx_entities_type ON entities(type);
117 +CREATE INDEX IF NOT EXISTS idx_queue_status ON crawl_queue(status);
118 +CREATE INDEX IF NOT EXISTS idx_events_ts ON events(ts);
119 +CREATE INDEX IF NOT EXISTS idx_rel_from ON relations(from_entity);
120 +CREATE INDEX IF NOT EXISTS idx_rel_to ON relations(to_entity);
121 +"""
122 +
123 +# Colonnes ajoutées par migration aux DB existantes
124 +_ENTITY_COLS = [
125 + ("canonical_name", "TEXT"), ("norm_name", "TEXT"), ("domain", "TEXT"), ("sector", "TEXT"),
126 + ("address", "TEXT"), ("city", "TEXT"), ("region", "TEXT"), ("postal_code", "TEXT"),
127 + ("neq", "TEXT"), ("founded", "TEXT"), ("size", "TEXT"), ("tags", "TEXT"),
128 + ("source_url", "TEXT"), ("first_seen", "REAL"), ("last_seen", "REAL"),
129 + # champs CRÉATEURS / INFLUENCEURS (ka6)
130 + ("niche", "TEXT"), ("handle", "TEXT"), ("platform", "TEXT"), ("followers", "INTEGER"),
131 + ("languages", "TEXT"),
132 +]
133 +_RELATION_COLS = [("role", "TEXT"), ("source_url", "TEXT"), ("confidence", "REAL")]
134 +
135 +
136 +class Store:
137 + def __init__(self, path: str):
138 + self.conn = sqlite3.connect(path, timeout=30, check_same_thread=False)
139 + self.conn.row_factory = sqlite3.Row
140 + self.conn.execute("PRAGMA foreign_keys = ON")
141 + self.conn.execute("PRAGMA journal_mode = WAL")
142 + self.conn.execute("PRAGMA busy_timeout = 30000")
143 + self.conn.executescript(SCHEMA)
144 + self._migrate()
145 + self.conn.execute("CREATE INDEX IF NOT EXISTS idx_entities_norm ON entities(norm_name)")
146 + self.conn.execute("CREATE INDEX IF NOT EXISTS idx_entities_domain ON entities(domain)")
147 + self.conn.commit()
148 +
149 + def _migrate(self) -> None:
150 + ecols = {r["name"] for r in self.conn.execute("PRAGMA table_info(entities)")}
151 + for name, typ in _ENTITY_COLS:
152 + if name not in ecols:
153 + self.conn.execute(f"ALTER TABLE entities ADD COLUMN {name} {typ}")
154 + rcols = {r["name"] for r in self.conn.execute("PRAGMA table_info(relations)")}
155 + for name, typ in _RELATION_COLS:
156 + if name not in rcols:
157 + self.conn.execute(f"ALTER TABLE relations ADD COLUMN {name} {typ}")
158 + # social_links.followers (abonnés par plateforme) + crawl_queue.attempts (robustesse)
159 + scols = {r["name"] for r in self.conn.execute("PRAGMA table_info(social_links)")}
160 + if "followers" not in scols:
161 + self.conn.execute("ALTER TABLE social_links ADD COLUMN followers INTEGER")
162 + qcols = {r["name"] for r in self.conn.execute("PRAGMA table_info(crawl_queue)")}
163 + if "attempts" not in qcols:
164 + self.conn.execute("ALTER TABLE crawl_queue ADD COLUMN attempts INTEGER DEFAULT 0")
165 + self.conn.commit()
166 + self._backfill()
167 +
168 + def _backfill(self) -> None:
169 + """Normalise les entités héritées (norm_name/domain/region) pour la dédup et l'explorer."""
170 + rows = self.conn.execute(
171 + "SELECT id, name, url, location FROM entities WHERE norm_name IS NULL"
172 + ).fetchall()
173 + for r in rows:
174 + nn = normalize_name(r["name"]) or domain(r["url"])
175 + self.conn.execute(
176 + "UPDATE entities SET norm_name=?, domain=?, region=COALESCE(region,?) WHERE id=?",
177 + (nn, domain(r["url"]) or None, canonical_location(r["location"]) or None, r["id"]),
178 + )
179 + if rows:
180 + self.conn.commit()
181 +
182 + # -- résolution + upsert d'entités -------------------------------------
183 + def _resolve(self, etype: str, nn: str, dom: str, region: str) -> Optional[int]:
184 + if not nn:
185 + return None
186 + c = self.conn
187 + if dom:
188 + r = c.execute(
189 + "SELECT id FROM entities WHERE type=? AND norm_name=? AND domain=? LIMIT 1",
190 + (etype, nn, dom),
191 + ).fetchone()
192 + if r:
193 + return r["id"]
194 + if region:
195 + r = c.execute(
196 + "SELECT id FROM entities WHERE type=? AND norm_name=? AND region=? "
197 + "AND (domain IS NULL OR domain='') LIMIT 1",
198 + (etype, nn, region),
199 + ).fetchone()
200 + if r:
201 + return r["id"]
202 + r = c.execute(
203 + "SELECT id FROM entities WHERE type=? AND norm_name=? "
204 + "AND (domain IS NULL OR domain='') AND (region IS NULL OR region='') LIMIT 1",
205 + (etype, nn),
206 + ).fetchone()
207 + return r["id"] if r else None
208 +
209 + def upsert_entity(self, e: dict[str, Any], source_url: Optional[str] = None) -> int:
210 + now = time.time()
211 + etype = (e.get("type") or "website").strip().lower()
212 + name = (e.get("name") or "").strip()
213 + website = (e.get("website") or e.get("url") or "").strip() or None
214 + dom = domain(website)
215 + region = canonical_location(e.get("region") or e.get("city") or e.get("location") or "")
216 + city = (e.get("city") or "").strip() or None
217 + nn = normalize_name(name) or dom
218 + if not name:
219 + name = website or "inconnu"
220 + conf = float(e.get("confidence", 0.5) or 0.5)
221 + tags = json.dumps(e.get("tags"), ensure_ascii=False) if e.get("tags") else None
222 + langs = e.get("languages")
223 + languages = ", ".join(langs) if isinstance(langs, list) else (langs or None)
224 + followers = _to_int(e.get("followers"))
225 +
226 + eid = self._resolve(etype, nn, dom, region)
227 + vals = {
228 + "canonical_name": e.get("canonical_name") or name,
229 + "description": e.get("description"), "url": website, "domain": dom or None,
230 + "sector": e.get("sector"), "address": e.get("address"), "city": city,
231 + "region": region or None, "location": e.get("location") or region or city,
232 + "email": e.get("email"), "phone": e.get("phone"), "postal_code": e.get("postal_code"),
233 + "neq": e.get("neq"), "founded": e.get("founded"), "size": e.get("size"),
234 + "tags": tags, "confidence": conf, "source_url": source_url,
235 + "niche": e.get("niche"), "handle": e.get("handle"), "platform": e.get("platform"),
236 + "followers": followers, "languages": languages,
237 + }
238 + if eid:
239 + self.conn.execute(
240 + """UPDATE entities SET
241 + canonical_name=COALESCE(canonical_name,?), description=COALESCE(description,?),
242 + url=COALESCE(url,?), domain=COALESCE(NULLIF(domain,''),?), sector=COALESCE(sector,?),
243 + address=COALESCE(address,?), city=COALESCE(city,?), region=COALESCE(region,?),
244 + location=COALESCE(location,?), email=COALESCE(email,?), phone=COALESCE(phone,?),
245 + postal_code=COALESCE(postal_code,?), neq=COALESCE(neq,?), founded=COALESCE(founded,?),
246 + size=COALESCE(size,?), tags=COALESCE(tags,?), confidence=MAX(IFNULL(confidence,0),?),
247 + source_url=COALESCE(source_url,?), niche=COALESCE(niche,?), handle=COALESCE(handle,?),
248 + platform=COALESCE(platform,?), followers=MAX(IFNULL(followers,0),?),
249 + languages=COALESCE(languages,?), last_seen=?, updated_at=? WHERE id=?""",
250 + (
251 + vals["canonical_name"], vals["description"], vals["url"], vals["domain"],
252 + vals["sector"], vals["address"], vals["city"], vals["region"], vals["location"],
253 + vals["email"], vals["phone"], vals["postal_code"], vals["neq"], vals["founded"],
254 + vals["size"], vals["tags"], vals["confidence"], vals["source_url"],
255 + vals["niche"], vals["handle"], vals["platform"], followers or 0, vals["languages"],
256 + now, now, eid,
257 + ),
258 + )
259 + else:
260 + cur = self.conn.execute(
261 + """INSERT INTO entities(type, name, canonical_name, norm_name, description, url, domain,
262 + sector, address, city, region, location, email, phone, postal_code, neq, founded,
263 + size, tags, confidence, raw_json, source_url, niche, handle, platform, followers,
264 + languages, first_seen, last_seen, created_at, updated_at)
265 + VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
266 + (
267 + etype, name, vals["canonical_name"], nn, vals["description"], vals["url"], vals["domain"],
268 + vals["sector"], vals["address"], vals["city"], vals["region"], vals["location"],
269 + vals["email"], vals["phone"], vals["postal_code"], vals["neq"], vals["founded"],
270 + vals["size"], vals["tags"], conf, json.dumps(e, ensure_ascii=False), source_url,
271 + vals["niche"], vals["handle"], vals["platform"], followers, vals["languages"],
272 + now, now, now, now,
273 + ),
274 + )
275 + eid = cur.lastrowid
276 +
277 + for link in e.get("social_links", []) or []:
278 + self.add_social_link(eid, link.get("platform", "?"), link.get("url", ""),
279 + _to_int(link.get("followers")))
280 + if source_url:
281 + self.conn.execute(
282 + "INSERT OR IGNORE INTO entity_mentions(entity_id, source_url, seen_at) VALUES(?,?,?)",
283 + (eid, source_url, now),
284 + )
285 + self.conn.commit()
286 + return eid
287 +
288 + def add_social_link(self, entity_id: int, platform: str, url: str,
289 + followers: Optional[int] = None) -> None:
290 + url = (url or "").strip()
291 + if not url:
292 + return
293 + self.conn.execute(
294 + "INSERT INTO social_links(entity_id, platform, url, followers) VALUES(?,?,?,?) "
295 + "ON CONFLICT(entity_id, url) DO UPDATE SET followers=COALESCE(excluded.followers, followers)",
296 + (entity_id, (platform or "?").strip().lower(), url, followers),
297 + )
298 +
299 + # -- relations ---------------------------------------------------------
300 + def add_relation(self, from_id: int, to_id: int, rel_type: str, role: str = "",
301 + source_url: str = "", confidence: float = 0.6) -> None:
302 + rel_type = (rel_type or "").strip().upper()
303 + if not from_id or not to_id or from_id == to_id or rel_type not in REL_TYPES:
304 + return
305 + self.conn.execute(
306 + """INSERT INTO relations(from_entity, to_entity, relation_type, role, source_url, confidence)
307 + VALUES(?,?,?,?,?,?)
308 + ON CONFLICT(from_entity, to_entity, relation_type) DO UPDATE SET
309 + role=COALESCE(NULLIF(excluded.role,''), role),
310 + source_url=COALESCE(NULLIF(excluded.source_url,''), source_url),
311 + confidence=MAX(IFNULL(confidence,0), excluded.confidence)""",
312 + (from_id, to_id, rel_type, role or None, source_url or None, confidence),
313 + )
314 + self.conn.commit()
315 +
316 + def entity_relations(self, entity_id: int) -> list[dict[str, Any]]:
317 + rows = self.conn.execute(
318 + """SELECT r.relation_type, r.role, r.confidence, r.from_entity, r.to_entity,
319 + e.id oid, e.name oname, e.type otype, e.region oregion, e.sector osector
320 + FROM relations r
321 + JOIN entities e ON e.id = CASE WHEN r.from_entity=? THEN r.to_entity ELSE r.from_entity END
322 + WHERE r.from_entity=? OR r.to_entity=?
323 + ORDER BY r.confidence DESC""",
324 + (entity_id, entity_id, entity_id),
325 + ).fetchall()
326 + out = []
327 + for r in rows:
328 + outgoing = r["from_entity"] == entity_id
329 + out.append({
330 + "id": r["oid"], "name": r["oname"], "type": r["otype"],
331 + "region": r["oregion"], "sector": r["osector"],
332 + "relation": r["relation_type"], "role": r["role"],
333 + "direction": "out" if outgoing else "in",
334 + "confidence": r["confidence"],
335 + })
336 + return out
337 +
338 + # -- sources / file ----------------------------------------------------
339 + def record_source(self, url: str, title: str, content: str) -> None:
340 + h = hashlib.sha256((content or "").encode("utf-8")).hexdigest()
341 + self.conn.execute(
342 + "INSERT OR REPLACE INTO sources(url, title, content_hash, scraped_at) VALUES(?,?,?,?)",
343 + (url, title, h, time.time()),
344 + )
345 + self.conn.commit()
346 +
347 + def source_seen(self, url: str) -> bool:
348 + return self.conn.execute("SELECT 1 FROM sources WHERE url=?", (url,)).fetchone() is not None
349 +
350 + def enqueue(self, urls: Iterable[str], depth: int = 0) -> int:
351 + n = 0
352 + for u in urls:
353 + u = (u or "").strip()
354 + if not u:
355 + continue
356 + n += self.conn.execute(
357 + "INSERT OR IGNORE INTO crawl_queue(url, status, depth, added_at) VALUES(?,?,?,?)",
358 + (u, "pending", depth, time.time()),
359 + ).rowcount
360 + self.conn.commit()
361 + return n
362 +
363 + def next_pending(self) -> Optional[sqlite3.Row]:
364 + return self.conn.execute(
365 + "SELECT * FROM crawl_queue WHERE status='pending' ORDER BY depth, id LIMIT 1"
366 + ).fetchone()
367 +
368 + def pending_batch(self, n: int, max_attempts: int = 99) -> list[sqlite3.Row]:
369 + return self.conn.execute(
370 + "SELECT * FROM crawl_queue WHERE status='pending' AND IFNULL(attempts,0) < ? "
371 + "ORDER BY depth, id LIMIT ?", (max_attempts, max(1, n)),
372 + ).fetchall()
373 +
374 + def mark(self, queue_id: int, status: str) -> None:
375 + self.conn.execute("UPDATE crawl_queue SET status=? WHERE id=?", (status, queue_id))
376 + self.conn.commit()
377 +
378 + def bump_attempt(self, queue_id: int, max_attempts: int) -> None:
379 + """Incrémente le compteur de tentatives ; passe en 'error' au-delà du seuil (dead-letter)."""
380 + self.conn.execute(
381 + "UPDATE crawl_queue SET attempts=IFNULL(attempts,0)+1, "
382 + "status=CASE WHEN IFNULL(attempts,0)+1 >= ? THEN 'error' ELSE 'pending' END WHERE id=?",
383 + (max_attempts, queue_id),
384 + )
385 + self.conn.commit()
386 +
387 + # -- settings ----------------------------------------------------------
388 + def get_setting(self, key: str, default: str = "") -> str:
389 + row = self.conn.execute("SELECT value FROM settings WHERE key=?", (key,)).fetchone()
390 + return row["value"] if row else default
391 +
392 + def set_setting(self, key: str, value: str) -> None:
393 + self.conn.execute(
394 + "INSERT INTO settings(key, value) VALUES(?,?) "
395 + "ON CONFLICT(key) DO UPDATE SET value=excluded.value",
396 + (key, value),
397 + )
398 + self.conn.commit()
399 +
400 + # -- missions ----------------------------------------------------------
401 + def add_mission(self, goal: str, seed: str = "", sector: str = "", region: str = "",
402 + priority: int = 5) -> int:
403 + cur = self.conn.execute(
404 + """INSERT OR IGNORE INTO missions(goal, seed, sector, region, priority, status, created_at)
405 + VALUES(?,?,?,?,?, 'pending', ?)""",
406 + (goal, seed or None, sector or None, region or None, priority, time.time()),
407 + )
408 + self.conn.commit()
409 + return cur.lastrowid or 0
410 +
411 + def next_mission(self) -> Optional[sqlite3.Row]:
412 + return self.conn.execute(
413 + "SELECT * FROM missions WHERE status='pending' ORDER BY priority, last_run IS NOT NULL, "
414 + "IFNULL(last_run, 0), id LIMIT 1"
415 + ).fetchone()
416 +
417 + def due_mission(self, revisit_seconds: float) -> Optional[sqlite3.Row]:
418 + threshold = time.time() - revisit_seconds
419 + return self.conn.execute(
420 + "SELECT * FROM missions WHERE status IN ('pending','done') "
421 + "AND (last_run IS NULL OR last_run <= ?) "
422 + "ORDER BY priority, IFNULL(last_run, 0), id LIMIT 1",
423 + (threshold,),
424 + ).fetchone()
425 +
426 + def set_mission_status(self, mission_id: int, status: str, bump_run: bool = False) -> None:
427 + if bump_run:
428 + self.conn.execute(
429 + "UPDATE missions SET status=?, runs_count=runs_count+1, last_run=? WHERE id=?",
430 + (status, time.time(), mission_id),
431 + )
432 + else:
433 + self.conn.execute("UPDATE missions SET status=? WHERE id=?", (status, mission_id))
434 + self.conn.commit()
435 +
436 + def list_missions(self) -> list[dict[str, Any]]:
437 + return [dict(r) for r in self.conn.execute("SELECT * FROM missions ORDER BY priority, id")]
438 +
439 + def delete_mission(self, mission_id: int) -> None:
440 + self.conn.execute("DELETE FROM missions WHERE id=?", (mission_id,))
441 + self.conn.commit()
442 +
443 + def prioritize_mission(self, mission_id: int) -> None:
444 + """Passe la mission tout en haut de la file et la rend « due » immédiatement."""
445 + self.conn.execute(
446 + "UPDATE missions SET priority=0, status='pending', last_run=NULL WHERE id=?",
447 + (mission_id,),
448 + )
449 + self.conn.commit()
450 +
451 + def next_forced(self) -> Optional[sqlite3.Row]:
452 + """Mission « lancer maintenant » (priorité 0) à exécuter avant tout le reste."""
453 + return self.conn.execute(
454 + "SELECT * FROM missions WHERE status='pending' AND priority <= 0 "
455 + "ORDER BY IFNULL(last_run, 0), id LIMIT 1"
456 + ).fetchone()
457 +
458 + def requeue_stale_missions(self) -> None:
459 + self.conn.execute("UPDATE missions SET status='pending' WHERE status='running'")
460 + self.conn.commit()
461 +
462 + def pages_since(self, since_ts: float) -> int:
463 + return self.conn.execute(
464 + "SELECT COUNT(*) n FROM sources WHERE scraped_at >= ?", (since_ts,)
465 + ).fetchone()["n"]
466 +
467 + # -- événements --------------------------------------------------------
468 + def log_event(self, kind: str, message: str = "", url: str = "", data: Any = None) -> None:
469 + self.conn.execute(
470 + "INSERT INTO events(ts, kind, message, url, data_json) VALUES(?,?,?,?,?)",
471 + (time.time(), kind, message, url or None,
472 + json.dumps(data, ensure_ascii=False) if data is not None else None),
473 + )
474 + self.conn.commit()
475 +
476 + def recent_events(self, limit: int = 100, after_id: int = 0) -> list[dict[str, Any]]:
477 + rows = self.conn.execute(
478 + "SELECT * FROM events WHERE id > ? ORDER BY id DESC LIMIT ?", (after_id, limit)
479 + ).fetchall()
480 + return [dict(r) for r in rows]
481 +
482 + # -- lecture / agrégats ------------------------------------------------
483 + def get_entity(self, entity_id: int) -> Optional[dict[str, Any]]:
484 + e = self.conn.execute("SELECT * FROM entities WHERE id=?", (entity_id,)).fetchone()
485 + if not e:
486 + return None
487 + d = dict(e)
488 + d["social_links"] = [
489 + {"platform": r["platform"], "url": r["url"], "followers": r["followers"]}
490 + for r in self.conn.execute(
491 + "SELECT platform, url, followers FROM social_links WHERE entity_id=?", (entity_id,)
492 + )
493 + ]
494 + d["sources"] = [
495 + r["source_url"] for r in self.conn.execute(
496 + "SELECT source_url FROM entity_mentions WHERE entity_id=? ORDER BY seen_at DESC LIMIT 15",
497 + (entity_id,),
498 + )
499 + ]
500 + return d
501 +
502 + def distinct_locations(self) -> list[dict[str, Any]]:
503 + rows = self.conn.execute(
504 + "SELECT COALESCE(NULLIF(region,''), location) loc, COUNT(*) n FROM entities "
505 + "WHERE COALESCE(NULLIF(region,''), location) IS NOT NULL "
506 + "AND COALESCE(NULLIF(region,''), location)<>'' GROUP BY loc ORDER BY n DESC"
507 + ).fetchall()
508 + return [{"location": r["loc"], "count": r["n"]} for r in rows]
509 +
510 + def platform_counts(self) -> list[dict[str, Any]]:
511 + rows = self.conn.execute(
512 + "SELECT platform, COUNT(*) n FROM social_links GROUP BY platform ORDER BY n DESC"
513 + ).fetchall()
514 + return [{"platform": r["platform"], "count": r["n"]} for r in rows]
515 +
516 + def degrees(self) -> dict[int, int]:
517 + deg: dict[int, int] = {}
518 + for r in self.conn.execute("SELECT from_entity a, to_entity b FROM relations"):
519 + deg[r["a"]] = deg.get(r["a"], 0) + 1
520 + deg[r["b"]] = deg.get(r["b"], 0) + 1
521 + return deg
522 +
523 + def top_connected(self, limit: int = 8) -> list[dict[str, Any]]:
524 + deg = self.degrees()
525 + if not deg:
526 + return []
527 + top = sorted(deg.items(), key=lambda kv: kv[1], reverse=True)[:limit]
528 + out = []
529 + for eid, d in top:
530 + e = self.conn.execute(
531 + "SELECT id, name, type, sector, region FROM entities WHERE id=?", (eid,)
532 + ).fetchone()
533 + if e:
534 + out.append({**dict(e), "degree": d})
535 + return out
536 +
537 + def graph(self, limit: int = 140, etype: str = "", region: str = "", sector: str = "",
538 + connected_only: bool = True) -> dict[str, Any]:
539 + q = "SELECT id, name, type, sector, region, location FROM entities WHERE 1=1"
540 + args: list[Any] = []
541 + if etype:
542 + q += " AND type=?"
543 + args.append(etype)
544 + if region:
545 + q += " AND COALESCE(NULLIF(region,''), location)=?"
546 + args.append(region)
547 + if sector:
548 + q += " AND sector LIKE ?"
549 + args.append(f"%{sector}%")
550 + rows = self.conn.execute(q, args).fetchall()
551 + deg = self.degrees()
552 + if connected_only:
553 + rows = [r for r in rows if deg.get(r["id"], 0) > 0]
554 + rows = sorted(rows, key=lambda r: deg.get(r["id"], 0), reverse=True)[:limit]
555 + ids = {r["id"] for r in rows}
556 + nodes = [{"id": r["id"], "name": r["name"], "type": r["type"],
557 + "degree": deg.get(r["id"], 0)} for r in rows]
558 + edges = []
559 + for r in self.conn.execute(
560 + "SELECT from_entity a, to_entity b, relation_type t, role FROM relations"
561 + ):
562 + if r["a"] in ids and r["b"] in ids:
563 + edges.append({"s": r["a"], "t": r["b"], "type": r["t"], "role": r["role"]})
564 + return {"nodes": nodes, "edges": edges, "truncated": len(ids)}
565 +
566 + def sector_counts(self) -> list[dict[str, Any]]:
567 + rows = self.conn.execute(
568 + "SELECT sector, COUNT(*) n FROM entities WHERE sector IS NOT NULL AND sector<>'' "
569 + "GROUP BY sector ORDER BY n DESC"
570 + ).fetchall()
571 + return [{"sector": r["sector"], "count": r["n"]} for r in rows]
572 +
573 + def stats(self) -> dict[str, Any]:
574 + c = self.conn
575 + by_type = {r["type"]: r["n"] for r in c.execute("SELECT type, COUNT(*) n FROM entities GROUP BY type")}
576 + return {
577 + "entities": c.execute("SELECT COUNT(*) n FROM entities").fetchone()["n"],
578 + "by_type": by_type,
579 + "relations": c.execute("SELECT COUNT(*) n FROM relations").fetchone()["n"],
580 + "social_links": c.execute("SELECT COUNT(*) n FROM social_links").fetchone()["n"],
581 + "sources": c.execute("SELECT COUNT(*) n FROM sources").fetchone()["n"],
582 + "queue_pending": c.execute(
583 + "SELECT COUNT(*) n FROM crawl_queue WHERE status='pending'"
584 + ).fetchone()["n"],
585 + }
586 +
587 + def export(self) -> list[dict[str, Any]]:
588 + out = []
589 + for e in self.conn.execute("SELECT * FROM entities ORDER BY type, name"):
590 + d = dict(e)
591 + d["social_links"] = [
592 + {"platform": r["platform"], "url": r["url"], "followers": r["followers"]}
593 + for r in self.conn.execute(
594 + "SELECT platform, url, followers FROM social_links WHERE entity_id=?", (e["id"],)
595 + )
596 + ]
597 + out.append(d)
598 + return out
599 +
600 + # -- archives (snapshot + reset + réutilisation) -----------------------
601 + def _snapshot(self) -> dict[str, Any]:
602 + c = self.conn
603 + return {
604 + "entities": self.export(),
605 + "relations": [dict(r) for r in c.execute(
606 + "SELECT from_entity, to_entity, relation_type, role, source_url, confidence FROM relations")],
607 + "missions": [dict(r) for r in c.execute(
608 + "SELECT goal, seed, sector, region, priority FROM missions")],
609 + "sources": [dict(r) for r in c.execute("SELECT url, title FROM sources")],
610 + }
611 +
612 + def archive_and_reset(self, label: str) -> dict[str, Any]:
613 + """Sauvegarde tout le graphe dans l'historique puis remet l'espace de travail à VIDE."""
614 + st = self.stats()
615 + payload = json.dumps(self._snapshot(), ensure_ascii=False)
616 + lab = (label or "").strip() or time.strftime("Archive %Y-%m-%d %H:%M", time.localtime())
617 + cur = self.conn.execute(
618 + "INSERT INTO archives(label, created_at, entities, relations, sources, payload) VALUES(?,?,?,?,?,?)",
619 + (lab, time.time(), st["entities"], st["relations"], st["sources"], payload),
620 + )
621 + for t in ("relations", "social_links", "entity_mentions", "entities", "sources",
622 + "crawl_queue", "events", "missions"):
623 + self.conn.execute(f"DELETE FROM {t}")
624 + self.conn.execute("DELETE FROM settings WHERE key IN ('active_mission_id','abort_current')")
625 + self.conn.commit()
626 + return {"id": cur.lastrowid, "label": lab, "entities": st["entities"], "relations": st["relations"]}
627 +
628 + def list_archives(self) -> list[dict[str, Any]]:
629 + return [dict(r) for r in self.conn.execute(
630 + "SELECT id, label, created_at, entities, relations, sources FROM archives ORDER BY id DESC")]
631 +
632 + def get_archive(self, archive_id: int) -> Optional[dict[str, Any]]:
633 + r = self.conn.execute("SELECT * FROM archives WHERE id=?", (archive_id,)).fetchone()
634 + if not r:
635 + return None
636 + d = dict(r)
637 + d["data"] = json.loads(d.pop("payload") or "{}")
638 + return d
639 +
640 + def delete_archive(self, archive_id: int) -> None:
641 + self.conn.execute("DELETE FROM archives WHERE id=?", (archive_id,))
642 + self.conn.commit()
643 +
644 + def restore_archive(self, archive_id: int) -> dict[str, Any]:
645 + """Recharge une archive dans l'espace de travail (remap des identifiants)."""
646 + a = self.get_archive(archive_id)
647 + if not a:
648 + return {"ok": False}
649 + data = a["data"]
650 + idmap: dict[Any, int] = {}
651 + for e in data.get("entities", []):
652 + old = e.get("id")
653 + nid = self.upsert_entity(dict(e))
654 + if old is not None:
655 + idmap[old] = nid
656 + rel = 0
657 + for r in data.get("relations", []):
658 + f, t = idmap.get(r.get("from_entity")), idmap.get(r.get("to_entity"))
659 + if f and t:
660 + self.add_relation(f, t, r.get("relation_type", ""), r.get("role") or "",
661 + r.get("source_url") or "", float(r.get("confidence") or 0.6))
662 + rel += 1
663 + for m in data.get("missions", []):
664 + self.add_mission(m.get("goal", ""), m.get("seed") or "", m.get("sector") or "",
665 + m.get("region") or "", int(m.get("priority") or 5))
666 + for s in data.get("sources", []):
667 + self.record_source(s.get("url", ""), s.get("title") or "", "")
668 + return {"ok": True, "entities": len(idmap), "relations": rel}
669 +
670 + def close(self) -> None:
671 + self.conn.close()
added src/util.py +21 −0
@@ -0,0 +1,21 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Petits utilitaires partagés."""
4 +
5 +from __future__ import annotations
6 +
7 +import time
8 +
9 +
10 +class RateLimiter:
11 + """Impose un délai minimal entre deux requêtes (politesse serveur)."""
12 +
13 + def __init__(self, delay: float):
14 + self.delay = max(0.0, delay)
15 + self._last = 0.0
16 +
17 + def wait(self) -> None:
18 + elapsed = time.time() - self._last
19 + if elapsed < self.delay:
20 + time.sleep(self.delay - elapsed)
21 + self._last = time.time()
added web/__init__.py +3 −0
@@ -0,0 +1,3 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""Plateforme web de suivi de ka2."""
added web/app.py +1335 −0
@@ -0,0 +1,1335 @@
1 +# Author: Simon-Pierre Boucher
2 +# Contact: contact@spboucher.ai
3 +"""API + dashboard de suivi de ka4 (FastAPI).
4 +
5 +Interface alignée sur le système de design du Groupe KA (néo-brutalisme éditorial :
6 +papier chaud, encre, accent lime, bordures nettes, ombres dures décalées).
7 +Temps réel via SSE, réglages personnalisables (backend, budget, session, accent).
8 +Le bot de crawl tourne dans un processus séparé partageant la même base SQLite (WAL).
9 +"""
10 +
11 +from __future__ import annotations
12 +
13 +import asyncio
14 +import json
15 +import time
16 +from typing import Optional
17 +
18 +from fastapi import FastAPI, Request
19 +from fastapi.responses import HTMLResponse, JSONResponse, StreamingResponse
20 +
21 +from src.config import config
22 +from src.storage import Store
23 +
24 +app = FastAPI(title="ka4 — cartographe du web québécois", version="2.0.0")
25 +
26 +SESSION_SECONDS = 15 * 60 # bloc de session (bouton « Continuer »)
27 +
28 +# Réglages personnalisables : nom public -> (clé en base, défaut effectif, type)
29 +CONFIG_KEYS = {
30 + "daily_budget": ("cfg_daily_budget", 800, int),
31 + "cycle_delay": ("cfg_cycle_delay", 20.0, float),
32 + "queue_batch": ("cfg_queue_batch", 8, int),
33 + "session_minutes": ("cfg_session_minutes", 15.0, float),
34 + "max_pages": ("cfg_max_pages", config.max_pages_per_run, int),
35 + "backend": ("cfg_backend", config.scraper_backend, str),
36 + "respect_robots": ("cfg_respect_robots", "1" if config.respect_robots else "0", str),
37 + "delay": ("cfg_delay", config.request_delay, float),
38 +}
39 +
40 +
41 +def store() -> Store:
42 + return Store(config.db_path)
43 +
44 +
45 +def _domain(url: Optional[str]) -> str:
46 + """Domaine racine simplifié d'une URL (pour relier entités du même site)."""
47 + if not url:
48 + return ""
49 + u = url.strip().lower()
50 + for pref in ("https://", "http://"):
51 + if u.startswith(pref):
52 + u = u[len(pref):]
53 + u = u.split("/")[0]
54 + if u.startswith("www."):
55 + u = u[4:]
56 + return u
57 +
58 +
59 +def _session(s: Store) -> dict:
60 + try:
61 + run_until = float(s.get_setting("run_until", "0") or 0)
62 + except ValueError:
63 + run_until = 0.0
64 + remaining = max(0, int(run_until - time.time()))
65 + return {"run_until": run_until, "session_remaining": remaining, "session_active": remaining > 0}
66 +
67 +
68 +@app.get("/health")
69 +def health():
70 + return {"status": "ok", "service": "ka4"}
71 +
72 +
73 +@app.get("/api/stats")
74 +def api_stats():
75 + s = store()
76 + try:
77 + data = s.stats()
78 + data["paused"] = s.get_setting("paused", "0") == "1"
79 + data["active_mission_id"] = s.get_setting("active_mission_id", "")
80 + data.update(_session(s))
81 + return data
82 + finally:
83 + s.close()
84 +
85 +
86 +@app.get("/api/events")
87 +def api_events(after_id: int = 0, limit: int = 80):
88 + s = store()
89 + try:
90 + return {"events": s.recent_events(limit=limit, after_id=after_id)}
91 + finally:
92 + s.close()
93 +
94 +
95 +def _filter_entities(s: Store, type=None, q=None, region=None, sector=None):
96 + rows = s.export()
97 + if type:
98 + rows = [r for r in rows if r["type"] == type]
99 + if region:
100 + rl = region.lower()
101 + rows = [r for r in rows if rl in ((r.get("region") or r.get("location") or "").lower())]
102 + if sector:
103 + sl = sector.lower()
104 + rows = [r for r in rows if sl in ((r.get("sector") or "").lower())]
105 + if q:
106 + ql = q.lower()
107 + rows = [r for r in rows if ql in (r["name"] or "").lower()
108 + or ql in (r.get("description") or "").lower()
109 + or ql in (r.get("sector") or "").lower()]
110 + deg = s.degrees()
111 + for r in rows:
112 + r["domain"] = r.get("domain") or _domain(r.get("url"))
113 + r["relations_count"] = deg.get(r["id"], 0)
114 + r["social_count"] = len(r.get("social_links") or [])
115 + return rows
116 +
117 +
118 +@app.get("/api/entities")
119 +def api_entities(type: Optional[str] = None, q: Optional[str] = None, region: Optional[str] = None,
120 + sector: Optional[str] = None, sort: str = "name", offset: int = 0, limit: int = 60):
121 + s = store()
122 + try:
123 + rows = _filter_entities(s, type, q, region, sector)
124 + if sort == "degree":
125 + rows.sort(key=lambda r: (r["relations_count"], r.get("social_count", 0)), reverse=True)
126 + elif sort == "recent":
127 + rows.sort(key=lambda r: (r.get("last_seen") or r.get("updated_at") or 0), reverse=True)
128 + else:
129 + rows.sort(key=lambda r: (r.get("name") or "").lower())
130 + total = len(rows)
131 + return {"count": total, "entities": rows[offset:offset + limit],
132 + "offset": offset, "limit": limit}
133 + finally:
134 + s.close()
135 +
136 +
137 +@app.get("/api/export.json")
138 +def export_json(type: Optional[str] = None, q: Optional[str] = None,
139 + region: Optional[str] = None, sector: Optional[str] = None):
140 + s = store()
141 + try:
142 + rows = _filter_entities(s, type, q, region, sector)
143 + finally:
144 + s.close()
145 + import json as _json
146 + return StreamingResponse(
147 + iter([_json.dumps(rows, ensure_ascii=False, indent=2)]),
148 + media_type="application/json",
149 + headers={"Content-Disposition": "attachment; filename=ka4-entites.json"},
150 + )
151 +
152 +
153 +@app.get("/api/export.csv")
154 +def export_csv(type: Optional[str] = None, q: Optional[str] = None,
155 + region: Optional[str] = None, sector: Optional[str] = None):
156 + s = store()
157 + try:
158 + rows = _filter_entities(s, type, q, region, sector)
159 + finally:
160 + s.close()
161 + import csv as _csv
162 + import io as _io
163 + cols = ["id", "type", "name", "sector", "region", "city", "url", "email", "phone",
164 + "founded", "size", "neq", "relations_count", "social_count"]
165 + buf = _io.StringIO()
166 + w = _csv.writer(buf)
167 + w.writerow(cols)
168 + for r in rows:
169 + w.writerow([r.get(c, "") for c in cols])
170 + return StreamingResponse(
171 + iter([buf.getvalue()]), media_type="text/csv",
172 + headers={"Content-Disposition": "attachment; filename=ka4-entites.csv"},
173 + )
174 +
175 +
176 +@app.get("/api/explore/summary")
177 +def explore_summary():
178 + s = store()
179 + try:
180 + st = s.stats()
181 + return {
182 + "total": st["entities"],
183 + "relations": st["relations"],
184 + "by_type": st["by_type"],
185 + "regions": s.distinct_locations()[:40],
186 + "sectors": s.sector_counts()[:40],
187 + "platforms": s.platform_counts()[:16],
188 + "top_connected": s.top_connected(10),
189 + }
190 + finally:
191 + s.close()
192 +
193 +
194 +@app.get("/api/graph")
195 +def api_graph(type: Optional[str] = None, region: Optional[str] = None,
196 + sector: Optional[str] = None, limit: int = 140):
197 + s = store()
198 + try:
199 + return s.graph(limit=limit, etype=type or "", region=region or "",
200 + sector=sector or "", connected_only=True)
201 + finally:
202 + s.close()
203 +
204 +
205 +@app.get("/api/entity/{entity_id}")
206 +def api_entity(entity_id: int):
207 + s = store()
208 + try:
209 + e = s.get_entity(entity_id)
210 + if not e:
211 + return JSONResponse({"error": "introuvable"}, status_code=404)
212 + e["domain"] = e.get("domain") or _domain(e.get("url"))
213 + # Relations RÉELLES extraites (typées, avec rôle) — le cœur du graphe
214 + e["relations"] = s.entity_relations(entity_id)
215 + # Contexte secondaire : autres entités de la même région
216 + region = (e.get("region") or e.get("location") or "").strip()
217 + same_region = []
218 + if region:
219 + linked = {r["id"] for r in e["relations"]}
220 + for r in s.export():
221 + if r["id"] == entity_id or r["id"] in linked:
222 + continue
223 + if (r.get("region") or r.get("location") or "").strip() == region:
224 + same_region.append({"id": r["id"], "name": r["name"], "type": r["type"]})
225 + e["same_region"] = same_region[:20]
226 + finally:
227 + s.close()
228 + return e
229 +
230 +
231 +@app.get("/api/missions")
232 +def api_missions():
233 + s = store()
234 + try:
235 + return {"missions": s.list_missions()}
236 + finally:
237 + s.close()
238 +
239 +
240 +@app.post("/api/missions")
241 +def add_mission(payload: dict):
242 + s = store()
243 + try:
244 + mid = s.add_mission(
245 + goal=payload.get("goal", "").strip(),
246 + seed=payload.get("seed", ""),
247 + sector=payload.get("sector", ""),
248 + region=payload.get("region", ""),
249 + priority=int(payload.get("priority", 5)),
250 + )
251 + return {"id": mid, "ok": bool(mid)}
252 + finally:
253 + s.close()
254 +
255 +
256 +@app.post("/api/missions/{mission_id}/{op}")
257 +def mission_op(mission_id: int, op: str):
258 + s = store()
259 + try:
260 + if op == "delete":
261 + s.delete_mission(mission_id)
262 + s.log_event("info", f"Mission #{mission_id} supprimée.")
263 + elif op == "prioritize":
264 + # « Lancer maintenant » : priorité max + interrompt l'activité en cours
265 + s.prioritize_mission(mission_id)
266 + s.set_setting("abort_current", "1")
267 + s.set_setting("paused", "0")
268 + s.log_event("info", f"Mission #{mission_id} priorisée (lancer maintenant).")
269 + else:
270 + return JSONResponse({"error": "op inconnue"}, status_code=400)
271 + return {"ok": True}
272 + finally:
273 + s.close()
274 +
275 +
276 +@app.post("/api/archive")
277 +def archive_now(payload: dict):
278 + s = store()
279 + try:
280 + # 1) stopper l'activité en cours (pause + interruption) pour éviter la course
281 + s.set_setting("paused", "1")
282 + s.set_setting("abort_current", "1")
283 + # 2) attendre que le bot confirme qu'il est au repos (≤20s) avant de vider
284 + for _ in range(40):
285 + if s.get_setting("bot_busy", "0") != "1":
286 + break
287 + time.sleep(0.5)
288 + # 3) snapshot + reset
289 + res = s.archive_and_reset(payload.get("label", ""))
290 + s.log_event("info", f"Archive « {res['label']} » créée ({res['entities']} entités). Espace de travail vidé.")
291 + return res
292 + finally:
293 + s.close()
294 +
295 +
296 +@app.get("/api/archives")
297 +def api_archives():
298 + s = store()
299 + try:
300 + return {"archives": s.list_archives()}
301 + finally:
302 + s.close()
303 +
304 +
305 +@app.post("/api/archive/{archive_id}/{op}")
306 +def archive_op(archive_id: int, op: str):
307 + s = store()
308 + try:
309 + if op == "restore":
310 + return s.restore_archive(archive_id)
311 + if op == "delete":
312 + s.delete_archive(archive_id)
313 + return {"ok": True}
314 + return JSONResponse({"error": "op inconnue"}, status_code=400)
315 + finally:
316 + s.close()
317 +
318 +
319 +@app.get("/api/archive/{archive_id}/export.json")
320 +def archive_export(archive_id: int):
321 + s = store()
322 + try:
323 + a = s.get_archive(archive_id)
324 + finally:
325 + s.close()
326 + if not a:
327 + return JSONResponse({"error": "introuvable"}, status_code=404)
328 + import json as _json
329 + return StreamingResponse(
330 + iter([_json.dumps(a["data"], ensure_ascii=False, indent=2)]),
331 + media_type="application/json",
332 + headers={"Content-Disposition": f"attachment; filename=ka-archive-{archive_id}.json"},
333 + )
334 +
335 +
336 +@app.post("/api/control/{action}")
337 +def control(action: str):
338 + s = store()
339 + try:
340 + if action == "pause":
341 + s.set_setting("paused", "1")
342 + elif action == "resume":
343 + s.set_setting("paused", "0")
344 + elif action == "skip":
345 + # arrête l'agent / le drainage en cours et passe à la suite
346 + s.set_setting("abort_current", "1")
347 + s.log_event("info", "Interruption demandée (arrêter l'activité en cours).")
348 + elif action == "extend":
349 + try:
350 + cur = float(s.get_setting("run_until", "0") or 0)
351 + except ValueError:
352 + cur = 0.0
353 + s.set_setting("run_until", str(max(cur, time.time()) + SESSION_SECONDS))
354 + s.set_setting("paused", "0")
355 + s.log_event("info", "Session prolongée de 15 min (bouton Continuer).")
356 + elif action == "stop":
357 + s.set_setting("run_until", str(time.time()))
358 + else:
359 + return JSONResponse({"error": "action inconnue"}, status_code=400)
360 + out = {"paused": s.get_setting("paused", "0") == "1"}
361 + out.update(_session(s))
362 + return out
363 + finally:
364 + s.close()
365 +
366 +
367 +@app.get("/api/config")
368 +def get_config():
369 + s = store()
370 + try:
371 + out = {}
372 + for name, (key, default, cast) in CONFIG_KEYS.items():
373 + raw = s.get_setting(key, "")
374 + try:
375 + out[name] = cast(raw) if raw != "" else default
376 + except ValueError:
377 + out[name] = default
378 + return out
379 + finally:
380 + s.close()
381 +
382 +
383 +@app.post("/api/config")
384 +def set_config(payload: dict):
385 + s = store()
386 + try:
387 + applied = {}
388 + for name, value in payload.items():
389 + if name not in CONFIG_KEYS:
390 + continue
391 + key, _default, cast = CONFIG_KEYS[name]
392 + try:
393 + val = cast(value)
394 + except (ValueError, TypeError):
395 + continue
396 + s.set_setting(key, str(val))
397 + applied[name] = val
398 + if applied:
399 + s.log_event("info", "Réglages: " + ", ".join(f"{k}={v}" for k, v in applied.items()))
400 + return {"applied": applied}
401 + finally:
402 + s.close()
403 +
404 +
405 +@app.get("/api/stream")
406 +async def stream(request: Request):
407 + """Flux SSE : pousse stats + nouveaux événements (remplace le polling lourd)."""
408 +
409 + async def gen():
410 + last_id = 0
411 + while True:
412 + if await request.is_disconnected():
413 + break
414 + s = store()
415 + try:
416 + stats = s.stats()
417 + stats["paused"] = s.get_setting("paused", "0") == "1"
418 + stats.update(_session(s))
419 + events = s.recent_events(limit=40, after_id=last_id)
420 + if events:
421 + last_id = max(e["id"] for e in events)
422 + finally:
423 + s.close()
424 + payload = {"stats": stats, "events": list(reversed(events))}
425 + yield f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
426 + await asyncio.sleep(2.5)
427 +
428 + return StreamingResponse(
429 + gen(),
430 + media_type="text/event-stream",
431 + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "Connection": "keep-alive"},
432 + )
433 +
434 +
435 +@app.get("/ka4.css")
436 +def ka4_css():
437 + # CSS partagé (extrait du dashboard) — source unique pour les deux pages.
438 + css = HTML.split("<style>", 1)[1].split("</style>", 1)[0]
439 + return StreamingResponse(iter([css]), media_type="text/css",
440 + headers={"Cache-Control": "max-age=300"})
441 +
442 +
443 +@app.get("/", response_class=HTMLResponse)
444 +def index():
445 + return HTML
446 +
447 +
448 +@app.get("/explore", response_class=HTMLResponse)
449 +def explore():
450 + return EXPLORE_HTML
451 +
452 +
453 +# =========================================================================
454 +# Dashboard mono-fichier — design system Groupe KA (néo-brutalisme éditorial)
455 +# =========================================================================
456 +HTML = r"""<!doctype html>
457 +<html lang="fr"><head>
458 +<meta charset="utf-8">
459 +<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
460 +<meta name="theme-color" content="#f5f3ee">
461 +<title>ka4 · explorateur structuré du web — Groupe KA</title>
462 +<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='64' height='64'%3E%3Crect width='64' height='64' rx='12' fill='%23141814'/%3E%3Ctext x='50%25' y='54%25' font-family='Arial Black,Arial' font-weight='900' font-size='30' fill='%23e11d48' text-anchor='middle' dominant-baseline='central'%3Ek4%3C/text%3E%3C/svg%3E">
463 +<link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
464 +<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
465 +<style>
466 +:root{
467 + --paper:#f5f3ee;--surface:#ffffff;--surface-2:#faf9f5;
468 + --ink:#141814;--ink-2:#4d5551;--ink-3:#8b928c;
469 + --line:rgba(20,24,20,.14);--line-strong:rgba(20,24,20,.85);
470 + --green:#9f1239;--green-deep:#7f1d33;
471 + --lime:#e11d48;--lime-soft:#fce7ec;
472 + --amber:#e8a33d;--amber-soft:#fdf3e2;--danger:#b3423a;--danger-soft:#fbe9e7;
473 + --r-card:10px;--r-ctl:6px;
474 + --shadow-flat:0 1px 2px rgba(20,24,20,.05);
475 + --shadow-off:6px 6px 0 var(--ink);
476 + --shadow-off-soft:8px 8px 0 rgba(20,24,20,.08);
477 + --font-display:'Space Grotesk',system-ui,sans-serif;
478 + --font-body:'Inter',system-ui,sans-serif;
479 + --font-mono:'JetBrains Mono',ui-monospace,monospace;
480 +}
481 +*{box-sizing:border-box}
482 +html,body{margin:0}
483 +body{font-family:var(--font-body);font-size:15px;line-height:1.55;background:var(--paper);color:var(--ink);
484 + -webkit-font-smoothing:antialiased;position:relative;min-height:100vh}
485 +body::before{content:"";position:fixed;inset:0;pointer-events:none;z-index:0;opacity:.35;
486 + background-image:url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='140' height='140'%3E%3Cfilter id='n'%3E%3CfeTurbulence type='fractalNoise' baseFrequency='0.9' numOctaves='2' stitchTiles='stitch'/%3E%3CfeColorMatrix type='saturate' values='0'/%3E%3C/filter%3E%3Crect width='100%25' height='100%25' filter='url(%23n)' opacity='0.5'/%3E%3C/svg%3E")}
487 +::selection{background:var(--lime);color:var(--ink)}
488 +a{color:var(--green);text-decoration:none}a:hover{text-decoration:underline}
489 +.wrap{max-width:1240px;margin:0 auto;padding:0 24px;position:relative;z-index:1}
490 +h1,h2,h3{font-family:var(--font-display);letter-spacing:-.03em;margin:0}
491 +.kicker{font-family:var(--font-mono);font-size:11px;font-weight:500;text-transform:uppercase;letter-spacing:.13em;
492 + color:var(--green);display:inline-flex;align-items:center;gap:9px}
493 +.kicker::before{content:"";width:22px;height:2px;background:var(--green);display:inline-block}
494 +.mono{font-family:var(--font-mono)}
495 +
496 +/* Header */
497 +header{position:sticky;top:0;z-index:30;background:rgba(245,243,238,.88);backdrop-filter:saturate(180%) blur(14px);
498 + border-bottom:2px solid var(--ink)}
499 +.hdr{display:flex;align-items:center;gap:16px;height:64px}
500 +.brand{display:flex;align-items:center;gap:12px;min-width:0;text-decoration:none;color:inherit}
501 +.logo{width:40px;height:40px;border-radius:10px;background:var(--ink);color:var(--lime);display:grid;place-items:center;
502 + font-family:var(--font-display);font-weight:700;font-size:18px;flex:0 0 auto;transform:rotate(-2deg);transition:.15s}
503 +.brand:hover .logo{transform:rotate(0)}
504 +.wordmark{font-family:var(--font-display);font-weight:700;font-size:22px;letter-spacing:-.04em;line-height:1;display:flex;align-items:baseline;gap:1px}
505 +.wordmark .kx{background:var(--ink);color:var(--lime);border-radius:6px;padding:0 6px 2px;transform:rotate(-2deg);display:inline-block}
506 +.wordmark .kn{color:var(--lime);-webkit-text-stroke:.7px var(--ink);paint-order:stroke fill}
507 +.brand .sub{font-family:var(--font-mono);font-size:10px;font-weight:500;color:var(--ink-3);text-transform:uppercase;letter-spacing:.1em;margin-top:3px}
508 +.brand .sub b{color:var(--green);font-weight:700}
509 +.hdr .right{margin-left:auto;display:flex;align-items:center;gap:8px;flex-wrap:wrap;justify-content:flex-end}
510 +
511 +/* Contrôles */
512 +.status{display:inline-flex;align-items:center;gap:8px;font-family:var(--font-mono);font-size:12px;font-weight:500;
513 + padding:8px 12px;border:1.5px solid var(--ink);border-radius:999px;background:var(--surface);text-transform:uppercase;letter-spacing:.06em}
514 +.dot{width:9px;height:9px;border-radius:50%;background:var(--green);position:relative}
515 +.dot.live::after{content:"";position:absolute;inset:-5px;border-radius:50%;background:var(--lime);opacity:.5;animation:pulse 2.4s ease-out infinite;z-index:-1}
516 +.dot.paused{background:var(--amber)}
517 +@keyframes pulse{0%{transform:scale(.6);opacity:.6}70%{transform:scale(1.8);opacity:0}100%{opacity:0}}
518 +.btn{font-family:var(--font-display);font-weight:700;font-size:13.5px;min-height:40px;padding:8px 16px;cursor:pointer;
519 + border:1.5px solid var(--ink);border-radius:var(--r-ctl);background:var(--surface);color:var(--ink);transition:.13s ease}
520 +.btn:hover{transform:translate(-2px,-2px);box-shadow:4px 4px 0 var(--ink)}
521 +.btn:active{transform:translate(2px,2px);box-shadow:none!important}
522 +.btn-primary{background:var(--ink);color:var(--lime);box-shadow:4px 4px 0 rgba(20,24,20,.25)}
523 +.btn-primary:hover{background:var(--green-deep);color:var(--lime)}
524 +.btn-lime{background:var(--lime);color:var(--ink)}
525 +.btn-icon{padding:8px 12px;font-size:16px}
526 +
527 +/* Ticker */
528 +.ticker{background:var(--ink);color:var(--lime);overflow:hidden;border-bottom:2px solid var(--ink);white-space:nowrap}
529 +.ticker .run{display:inline-block;padding:7px 0;font-family:var(--font-mono);font-size:11.5px;font-weight:500;
530 + text-transform:uppercase;letter-spacing:.1em;animation:tick 42s linear infinite}
531 +.ticker .run span{padding:0 18px;opacity:.92}
532 +@keyframes tick{from{transform:translateX(0)}to{transform:translateX(-50%)}}
533 +
534 +main{padding:26px 0 60px}
535 +
536 +/* KPIs */
537 +.kpis{display:grid;grid-template-columns:repeat(auto-fit,minmax(160px,1fr));gap:16px;margin-bottom:22px}
538 +.kpi{background:var(--surface);border:1.5px solid var(--ink);border-radius:var(--r-card);padding:16px;box-shadow:var(--shadow-off-soft);
539 + transition:.15s ease}
540 +.kpi:hover{transform:translate(-3px,-3px);box-shadow:var(--shadow-off)}
541 +.kpi .lab{font-family:var(--font-mono);font-size:9.5px;font-weight:700;text-transform:uppercase;letter-spacing:.12em;color:var(--ink-3)}
542 +.kpi .val{font-family:var(--font-display);font-weight:700;font-size:34px;letter-spacing:-.03em;line-height:1.05;margin-top:8px}
543 +.kpi .bar{height:4px;border-radius:2px;margin-top:10px;background:var(--accent,var(--lime))}
544 +
545 +/* Grille */
546 +.grid{display:grid;grid-template-columns:1fr;gap:18px}
547 +@media(min-width:980px){.grid{grid-template-columns:1.3fr 1fr}}
548 +.card{background:var(--surface);border:1.5px solid var(--ink);border-radius:var(--r-card);box-shadow:var(--shadow-off-soft);overflow:hidden}
549 +.card>.head{display:flex;align-items:center;gap:12px;padding:15px 18px;border-bottom:1.5px solid var(--ink)}
550 +.card>.head .count{margin-left:auto;font-family:var(--font-mono);font-size:11px;font-weight:700;color:var(--ink-3);text-transform:uppercase;letter-spacing:.08em}
551 +
552 +/* Feed */
553 +#feed{max-height:64vh;overflow:auto;padding:6px}
554 +.ev{display:grid;grid-template-columns:58px 92px 1fr;gap:10px;align-items:baseline;padding:9px 12px;border-radius:6px}
555 +.ev:hover{background:var(--surface-2)}
556 +.ev .t{font-family:var(--font-mono);font-size:11px;color:var(--ink-3);font-variant-numeric:tabular-nums}
557 +.ev .msg{font-size:13.5px;word-break:break-word}
558 +.badge{font-family:var(--font-mono);font-size:9.5px;font-weight:700;text-transform:uppercase;letter-spacing:.06em;
559 + padding:3px 8px;border-radius:999px;text-align:center;border:1.5px solid var(--ink)}
560 +.b-scrape{background:var(--surface);color:var(--ink)}
561 +.b-extract{background:var(--lime);color:var(--ink)}
562 +.b-agent{background:var(--ink);color:var(--lime)}
563 +.b-search,.b-map{background:var(--amber-soft);color:#8a5a12}
564 +.b-error{background:var(--danger-soft);color:var(--danger)}
565 +.b-info{background:var(--lime-soft);color:var(--green-deep)}
566 +
567 +/* Tableaux */
568 +.tblwrap{overflow:auto;max-height:56vh}
569 +table{width:100%;border-collapse:collapse;font-size:13.5px}
570 +th,td{text-align:left;padding:11px 16px;vertical-align:top}
571 +th{position:sticky;top:0;background:var(--surface-2);font-family:var(--font-mono);font-size:9.5px;font-weight:700;
572 + text-transform:uppercase;letter-spacing:.1em;color:var(--ink-2);border-bottom:1.5px solid var(--ink);z-index:1}
573 +td{border-bottom:1px solid var(--line)}
574 +tr:hover td{background:var(--surface-2)}
575 +.tag{display:inline-block;font-family:var(--font-mono);font-size:10px;font-weight:700;text-transform:uppercase;letter-spacing:.05em;
576 + padding:3px 9px;border-radius:999px;border:1.5px solid var(--ink);background:var(--lime-soft)}
577 +.tag.person{background:var(--ink);color:var(--lime)}
578 +.tag.st-running{background:var(--lime);color:var(--ink)}.tag.st-pending{background:var(--surface-2)}
579 +.tag.st-done{background:var(--lime-soft);color:var(--green-deep)}.tag.st-error{background:var(--danger-soft);color:var(--danger)}
580 +.name{font-family:var(--font-display);font-weight:600}
581 +.desc{color:var(--ink-2);font-size:12px;margin-top:2px}
582 +.chip{display:inline-block;font-family:var(--font-mono);font-size:11px;font-weight:500;padding:2px 9px;border:1.5px solid var(--ink);
583 + border-radius:999px;margin:2px 3px 0 0;color:var(--ink);background:var(--surface);box-shadow:2px 2px 0 rgba(20,24,20,.1)}
584 +.chip:hover{text-decoration:none;background:var(--lime-soft)}
585 +
586 +/* Filtres / formulaires */
587 +.filters,.mform{display:flex;gap:9px;flex-wrap:wrap;padding:13px 18px;border-bottom:1.5px solid var(--ink)}
588 +.mform input{flex:1;min-width:200px}
589 +input,select{font-family:var(--font-body);font-size:14px;background:var(--surface);border:1.5px solid var(--ink);color:var(--ink);
590 + border-radius:var(--r-ctl);padding:9px 13px;outline:none;box-shadow:3px 3px 0 rgba(20,24,20,.08)}
591 +input:focus,select:focus{outline:2px solid var(--green);outline-offset:1px;box-shadow:3px 3px 0 rgba(28,92,65,.25)}
592 +input::placeholder{color:var(--ink-3)}
593 +
594 +/* Drawer réglages */
595 +.scrim{position:fixed;inset:0;background:rgba(20,24,20,.35);backdrop-filter:blur(2px);z-index:40;opacity:0;pointer-events:none;transition:.2s}
596 +.scrim.open{opacity:1;pointer-events:auto}
597 +.drawer{position:fixed;top:0;right:0;height:100%;width:min(400px,92vw);background:var(--paper);border-left:2px solid var(--ink);
598 + z-index:41;transform:translateX(100%);transition:.25s cubic-bezier(.2,.8,.2,1);overflow-y:auto;box-shadow:-10px 0 0 rgba(20,24,20,.06)}
599 +.drawer.open{transform:translateX(0)}
600 +.drawer .dhead{display:flex;align-items:center;padding:18px 20px;border-bottom:2px solid var(--ink);position:sticky;top:0;background:var(--paper)}
601 +.drawer h2{font-size:19px}.drawer .close{margin-left:auto;cursor:pointer;font-size:22px;line-height:1;background:none;border:none}
602 +.dsec{padding:18px 20px;border-bottom:1.5px dashed var(--line)}
603 +.dsec label{display:block;font-family:var(--font-mono);font-size:10px;font-weight:700;text-transform:uppercase;letter-spacing:.1em;color:var(--ink-2);margin-bottom:6px}
604 +.dsec .row{display:flex;flex-direction:column;gap:6px;margin-bottom:14px}
605 +.dsec input,.dsec select{width:100%}
606 +.swatches{display:flex;gap:9px;flex-wrap:wrap}
607 +.sw{width:34px;height:34px;border-radius:8px;border:1.5px solid var(--ink);cursor:pointer;box-shadow:2px 2px 0 rgba(20,24,20,.12);transition:.12s}
608 +.sw:hover{transform:translate(-2px,-2px);box-shadow:4px 4px 0 var(--ink)}
609 +.sw.sel{outline:3px solid var(--ink);outline-offset:2px}
610 +.toggle{display:flex;align-items:center;gap:10px}
611 +.toggle input{width:auto;box-shadow:none}
612 +
613 +footer{border-top:2px solid var(--ink);background:var(--ink);color:rgba(245,243,238,.75);margin-top:20px;position:relative;z-index:1}
614 +footer .wrap{padding:26px 24px;font-family:var(--font-mono);font-size:11px;line-height:1.9;text-transform:uppercase;letter-spacing:.06em}
615 +footer b{color:var(--lime)}
616 +footer .lk{color:var(--lime)}
617 +
618 +/* Barre d'action mobile (thumb-friendly) */
619 +.mobilebar{display:none}
620 +.only-mobile{display:none}
621 +@media(max-width:760px){
622 + .hide-mobile{display:none!important}
623 + .only-mobile{display:inline-flex!important}
624 + .mobilebar{display:flex;gap:9px;position:fixed;left:0;right:0;bottom:0;z-index:35;
625 + padding:10px 14px calc(10px + env(safe-area-inset-bottom));
626 + background:rgba(245,243,238,.94);backdrop-filter:blur(12px);border-top:2px solid var(--ink)}
627 + .mobilebar .btn{flex:1;min-height:46px}
628 + body{padding-bottom:78px}
629 +}
630 +@media(max-width:600px){
631 + .wrap{padding:0 16px}
632 + .brand .sub{display:none}
633 + .hdr{height:56px;flex-wrap:nowrap}
634 + .wordmark{font-size:20px}.logo{width:36px;height:36px}
635 + .status{padding:7px 10px}
636 + .ev{grid-template-columns:46px 76px 1fr;gap:8px}
637 + .ev .t{font-size:10px}
638 + .kpis{grid-template-columns:1fr 1fr;gap:12px}
639 + .kpi{padding:13px}.kpi .val{font-size:26px}
640 + th,td{padding:9px 11px}
641 + input,select{font-size:16px}/* anti-zoom iOS */
642 +}
643 +@media(max-width:430px){
644 + #state{display:none}/* garde le point + le minuteur, gagne de la place */
645 + .kpi:nth-child(n){}
646 +}
647 +@media(prefers-reduced-motion:reduce){*{animation-duration:.01ms!important;transition-duration:.01ms!important}}
648 +</style></head><body>
649 +
650 +<header>
651 + <div class="wrap hdr">
652 + <a class="brand" href="/">
653 + <div class="logo">k4</div>
654 + <div>
655 + <div class="wordmark"><span class="kn">ka</span><span class="kx">4</span></div>
656 + <div class="sub">by <b>groupe·ka</b> · explorateur structuré du web</div>
657 + </div>
658 + </a>
659 + <div class="right">
660 + <span class="status"><span class="dot live" id="live"></span><span id="state">connexion…</span></span>
661 + <span class="status" id="timerChip" title="Temps restant de la session"><span id="timer">--:--</span></span>
662 + <button class="btn hide-mobile" id="btnPause" onclick="ctl('pause')">⏸ Pause</button>
663 + <button class="btn hide-mobile" id="btnResume" onclick="ctl('resume')" style="display:none">▶ Reprendre</button>
664 + <button class="btn btn-primary hide-mobile" id="btnContinue" onclick="ctl('extend')">▶ Continuer +15 min</button>
665 + <a class="btn hide-mobile" href="/explore" title="Explorer les données">▦ Données</a>
666 + <a class="btn btn-icon only-mobile" href="/explore" title="Explorer les données">▦</a>
667 + <button class="btn btn-icon" title="Réglages" onclick="openDrawer()">⚙</button>
668 + </div>
669 + </div>
670 +</header>
671 +<div class="ticker"><div class="run" id="ticker"></div></div>
672 +
673 +<main class="wrap">
674 + <div class="kpis" id="kpis"></div>
675 +
676 + <div class="grid">
677 + <section class="card">
678 + <div class="head"><span class="kicker">Activité en direct</span><span class="count" id="feedCount"></span></div>
679 + <div id="feed"></div>
680 + </section>
681 +
682 + <section class="card">
683 + <div class="head"><span class="kicker">Entités découvertes</span><span class="count" id="entCount"></span></div>
684 + <div class="filters">
685 + <select id="ftype" onchange="loadEntities()">
686 + <option value="">Tous les types</option><option value="creator">Créateurs</option>
687 + <option value="agency">Agences</option><option value="brand">Marques</option>
688 + <option value="business">Entreprises</option><option value="organization">Organisations</option>
689 + <option value="person">Personnes</option><option value="website">Sites web</option>
690 + </select>
691 + <input id="fq" placeholder="Rechercher…" style="flex:1;min-width:130px">
692 + </div>
693 + <div class="tblwrap"><table id="ent"></table></div>
694 + </section>
695 + </div>
696 +
697 + <section class="card" style="margin-top:18px">
698 + <div class="head"><span class="kicker">Missions</span><span class="count" id="misCount"></span>
699 + <button class="btn" style="margin-left:auto" onclick="ctl('skip')" title="Arrête l'agent/le crawl en cours et passe à la suite">⏹ Arrêter la mission en cours</button>
700 + <button class="btn btn-lime" onclick="archiveNow()" title="Sauvegarde les données recueillies dans l'historique puis remet l'espace de travail à vide">🗄 Archiver &amp; vider</button>
701 + </div>
702 + <div class="mform">
703 + <input id="mgoal" placeholder="Nouvel objectif (ex. influenceurs beauté QC, PME techno Laval…)">
704 + <input id="mseed" placeholder="URL de départ (optionnel)" style="flex:0 0 210px">
705 + <button class="btn btn-primary" onclick="addMission()">+ Ajouter</button>
706 + </div>
707 + <div class="tblwrap"><table id="mis"></table></div>
708 + </section>
709 +
710 + <section class="card" style="margin-top:18px">
711 + <div class="head"><span class="kicker">Archives (historique)</span><span class="count" id="arcCount"></span></div>
712 + <div class="tblwrap"><table id="arc"></table></div>
713 + </section>
714 +</main>
715 +
716 +<div class="mobilebar">
717 + <button class="btn" id="mToggle" onclick="ctl('pause')">⏸ Pause</button>
718 + <button class="btn btn-primary" id="mContinue" onclick="ctl('extend')">▶ Continuer +15 min</button>
719 +</div>
720 +
721 +<footer><div class="wrap">
722 + <b>ka·4</b> — bot IA de cartographie · propulsé par Claude Haiku 4.5 + Firecrawl + Scrapfly<br>
723 + Auteur Simon-Pierre Boucher · <span class="lk">contact@spboucher.ai</span> · une plateforme du <b>Groupe KA</b><br>
724 + Données publiques uniquement · conforme Loi 25 (Québec) / LPRPDE · robots.txt respecté · zéro boîte noire
725 +</div></footer>
726 +
727 +<div class="scrim" id="scrim" onclick="closeDrawer()"></div>
728 +<aside class="drawer" id="drawer">
729 + <div class="dhead"><h2>Réglages</h2><button class="close" onclick="closeDrawer()">✕</button></div>
730 + <div class="dsec">
731 + <label>Accent de marque</label>
732 + <div class="swatches" id="swatches"></div>
733 + </div>
734 + <div class="dsec">
735 + <label style="margin-bottom:12px">Paramètres du bot</label>
736 + <div class="row"><label>Budget quotidien (pages)</label><input id="c_daily_budget" type="number" min="0"></div>
737 + <div class="row"><label>Durée de session (min)</label><input id="c_session_minutes" type="number" min="1"></div>
738 + <div class="row"><label>Délai entre requêtes (s)</label><input id="c_delay" type="number" min="0" step="0.5"></div>
739 + <div class="row"><label>Pages max par cycle</label><input id="c_max_pages" type="number" min="1"></div>
740 + <div class="row"><label>Backend de scrape</label>
741 + <select id="c_backend"><option value="auto">auto (Firecrawl → Scrapfly)</option>
742 + <option value="firecrawl">firecrawl</option><option value="scrapfly">scrapfly</option></select></div>
743 + <div class="row"><label>robots.txt</label>
744 + <div class="toggle"><input id="c_respect_robots" type="checkbox"><span class="mono" style="font-size:12px">Respecter robots.txt</span></div></div>
745 + <button class="btn btn-primary" style="width:100%" onclick="saveConfig()">Enregistrer</button>
746 + </div>
747 + <div class="dsec" style="border-bottom:none">
748 + <label>Session</label>
749 + <div style="display:flex;gap:8px;flex-wrap:wrap">
750 + <button class="btn btn-lime" onclick="ctl('extend')">+15 min</button>
751 + <button class="btn" onclick="ctl('stop')">Terminer</button>
752 + </div>
753 + </div>
754 +</aside>
755 +
756 +<script>
757 +const ACCENTS=[["#e11d48","rouge · défaut"],["#ff2d55","rose"],["#9f1239","bordeaux"],["#f59e0b","ambre"],["#0ea5e9","cyan"],["#7c3aed","violet"]];
758 +const $=id=>document.getElementById(id);
759 +const esc=s=>(s||'').replace(/[&<>]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;'}[c]));
760 +const hhmm=ts=>new Date(ts*1000).toLocaleTimeString('fr-CA',{hour12:false,hour:'2-digit',minute:'2-digit',second:'2-digit'});
761 +const fmt=n=>(n||0).toLocaleString('fr-CA');
762 +const mmss=s=>{s=Math.max(0,s|0);return String((s/60)|0).padStart(2,'0')+':'+String(s%60).padStart(2,'0')};
763 +async function j(u,o){const r=await fetch(u,o);return r.json()}
764 +let remaining=0,lastId=0,lastEntCount=-1,feedEls=0,activeMid='';
765 +
766 +/* Accent personnalisable */
767 +function applyAccent(hex){
768 + document.documentElement.style.setProperty('--lime',hex);
769 + document.documentElement.style.setProperty('--lime-soft','color-mix(in srgb,'+hex+' 22%, white)');
770 + localStorage.setItem('ka4-accent',hex);
771 + document.querySelectorAll('.sw').forEach(e=>e.classList.toggle('sel',e.dataset.hex===hex));
772 +}
773 +function buildSwatches(){
774 + $('swatches').innerHTML=ACCENTS.map(([h,t])=>`<div class="sw" data-hex="${h}" title="${t}" style="background:${h}" onclick="applyAccent('${h}')"></div>`).join('');
775 + applyAccent(localStorage.getItem('ka4-accent')||ACCENTS[0][0]);
776 +}
777 +
778 +/* KPIs + ticker + état (depuis le flux) */
779 +const KPIS=[['entities','Entités','var(--lime)'],['relations','Relations','#6d28d9'],
780 + ['types','Types','#0ea5e9'],['social_links','Liens sociaux','var(--amber)'],
781 + ['sources','Pages','var(--green-deep)'],['queue_pending','File','var(--ink-3)']];
782 +function renderStats(s){
783 + const bt=s.by_type||{};
784 + const v={entities:s.entities,relations:s.relations||0,social_links:s.social_links,sources:s.sources,
785 + queue_pending:s.queue_pending,types:Object.keys(bt).length};
786 + $('kpis').innerHTML=KPIS.map(([k,lab,c])=>
787 + `<div class="kpi"><div class="lab">${lab}</div><div class="val">${fmt(v[k])}</div><div class="bar" style="background:${c}"></div></div>`).join('');
788 + activeMid=String(s.active_mission_id||'');
789 + remaining=s.session_remaining||0;
790 + const running=s.session_active&&!s.paused;
791 + $('state').textContent=s.paused?'En pause':(s.session_active?'Actif':'Session terminée');
792 + $('live').className='dot'+(running?' live':' paused');
793 + $('btnPause').style.display=running?'':'none';
794 + $('btnResume').style.display=(s.paused&&s.session_active)?'':'none';
795 + // barre mobile : un seul bouton bascule Pause/Reprendre (masqué si session terminée)
796 + const mt=$('mToggle');
797 + mt.style.display=s.session_active?'':'none';
798 + mt.textContent=running?'⏸ Pause':'▶ Reprendre';
799 + mt.onclick=()=>ctl(running?'pause':'resume');
800 + renderTimer();
801 + const parts=[`${fmt(v.entities)} entités`,`${fmt(v.business)} entreprises`,`${fmt(v.person)} personnes`,
802 + `${fmt(v.sources)} pages scrapées`,`${fmt(v.social_links)} liens sociaux`,`file: ${fmt(v.queue_pending)}`,
803 + running?`session ${mmss(remaining)}`:'session en veille','groupe ka · zéro boîte noire'];
804 + const line=parts.map(p=>`<span>◆ ${p}</span>`).join('');
805 + $('ticker').innerHTML=line+line;
806 + if(v.entities!==lastEntCount){lastEntCount=v.entities;loadEntities();}
807 +}
808 +function renderTimer(){
809 + $('timer').textContent=mmss(remaining);
810 + $('timerChip').style.color=remaining<=0?'var(--amber)':'';
811 + const lbl=remaining<=0?'▶ Continuer (terminée)':'▶ Continuer +15 min';
812 + $('btnContinue').textContent=lbl;$('mContinue').textContent=lbl;
813 +}
814 +function pushEvents(events){
815 + const feed=$('feed');
816 + events.forEach(e=>{
817 + if(e.id<=lastId)return; lastId=e.id;
818 + const div=document.createElement('div');div.className='ev';
819 + div.innerHTML=`<span class="t">${hhmm(e.ts)}</span><span class="badge b-${e.kind}">${e.kind}</span><span class="msg">${esc(e.message)}</span>`;
820 + feed.prepend(div);feedEls++;
821 + });
822 + while(feed.childNodes.length>200){feed.removeChild(feed.lastChild);}
823 + $('feedCount').textContent=feed.childNodes.length+' évén.';
824 +}
825 +
826 +/* Entités + missions (rafraîchis quand utile) */
827 +let entTimer=null;
828 +async function loadEntities(){
829 + const t=$('ftype').value,q=$('fq').value;
830 + const d=await j(`/api/entities?type=${t}&q=${encodeURIComponent(q)}&limit=100`);
831 + $('entCount').textContent=fmt(d.count)+' au total';
832 + const rows=d.entities.map(e=>{
833 + const soc=(e.social_links||[]).map(l=>`<a class="chip" href="${esc(l.url)}" target="_blank" rel="noopener">${esc(l.platform)}</a>`).join('');
834 + const site=e.url?`<a class="chip" href="${esc(e.url)}" target="_blank" rel="noopener">site ↗</a>`:'';
835 + return `<tr><td><span class="tag ${esc(e.type)}">${esc(e.type)}</span></td>
836 + <td><div class="name">${esc(e.name)}</div><div class="desc">${esc((e.description||'').slice(0,110))}</div></td>
837 + <td>${site}${soc}</td><td class="mono" style="font-size:12px">${esc(e.location||'')}</td></tr>`;
838 + }).join('');
839 + $('ent').innerHTML=`<thead><tr><th>Type</th><th>Nom</th><th>Liens</th><th>Lieu</th></tr></thead><tbody>${rows}</tbody>`;
840 +}
841 +async function loadMissions(){
842 + const d=await j('/api/missions');
843 + $('misCount').textContent=d.missions.length+' mission(s)';
844 + const rows=d.missions.map(m=>{
845 + const active=String(m.id)===activeMid;
846 + const stat=active?'<span class="tag st-running">▶ en cours</span>':`<span class="tag st-${esc(m.status)}">${esc(m.status)}</span>`;
847 + const act=`<button class="btn" style="padding:5px 9px;min-height:0" title="Lancer maintenant" onclick="missionOp(${m.id},'prioritize')">▶</button>
848 + <button class="btn" style="padding:5px 9px;min-height:0" title="Supprimer" onclick="missionOp(${m.id},'delete')">✕</button>`;
849 + return `<tr><td class="mono">#${m.id}</td><td>${esc(m.goal)}</td><td>${stat}</td>
850 + <td class="mono">${m.runs_count||0}</td><td style="white-space:nowrap">${act}</td></tr>`;
851 + }).join('');
852 + $('mis').innerHTML=`<thead><tr><th>ID</th><th>Objectif</th><th>Statut</th><th>Exéc.</th><th>Actions</th></tr></thead><tbody>${rows}</tbody>`
853 + +(d.missions.length?'':'<div class="pad muted">Aucune mission. Ajoute un objectif ci-dessus (n\'importe quel sujet) pour lancer une exploration.</div>');
854 +}
855 +async function missionOp(id,op){
856 + if(op==='delete'&&!confirm('Supprimer cette mission ?'))return;
857 + await j('/api/missions/'+id+'/'+op,{method:'POST'});
858 + loadMissions();
859 +}
860 +async function addMission(){
861 + const g=$('mgoal');if(!g.value.trim())return;
862 + await j('/api/missions',{method:'POST',headers:{'Content-Type':'application/json'},
863 + body:JSON.stringify({goal:g.value.trim(),seed:$('mseed').value})});
864 + g.value='';$('mseed').value='';loadMissions();
865 +}
866 +async function ctl(a){const r=await j('/api/control/'+a,{method:'POST'});if(r.session_remaining!=null){remaining=r.session_remaining;renderTimer();}}
867 +
868 +/* Archives : archiver+vider, historique, restaurer/exporter/supprimer */
869 +async function archiveNow(){
870 + const lab=prompt("Nom de l'archive (les données seront sauvegardées puis l'espace de travail sera vidé) :",
871 + new Date().toLocaleString('fr-CA'));
872 + if(lab===null)return;
873 + const r=await j('/api/archive',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({label:lab})});
874 + alert('Archivé : '+(r.label||'')+' — '+(r.entities||0)+' entités sauvegardées. Espace de travail vidé.');
875 + loadMissions();loadArchives();loadEntities();lastEntCount=-1;
876 +}
877 +async function loadArchives(){
878 + const d=await j('/api/archives');
879 + $('arcCount').textContent=d.archives.length+' archive(s)';
880 + const rows=d.archives.map(a=>`<tr><td class="mono">#${a.id}</td><td>${esc(a.label||'')}</td>
881 + <td class="mono">${hhmm(a.created_at)}</td><td class="mono">${fmt(a.entities)}</td><td class="mono">${fmt(a.relations)}</td>
882 + <td style="white-space:nowrap">
883 + <button class="btn" style="padding:5px 9px;min-height:0" title="Restaurer dans l'espace de travail" onclick="archiveOp(${a.id},'restore')">↺</button>
884 + <a class="btn" style="padding:5px 9px;min-height:0" title="Exporter (JSON)" href="/api/archive/${a.id}/export.json">↓</a>
885 + <button class="btn" style="padding:5px 9px;min-height:0" title="Supprimer l'archive" onclick="archiveOp(${a.id},'delete')">✕</button>
886 + </td></tr>`).join('');
887 + $('arc').innerHTML=`<thead><tr><th>ID</th><th>Nom</th><th>Créée</th><th>Entités</th><th>Relations</th><th>Actions</th></tr></thead><tbody>${rows}</tbody>`
888 + +(d.archives.length?'':'<div class="pad muted">Aucune archive. Clique « Archiver &amp; vider » pour sauvegarder l\'exploration en cours et repartir à zéro.</div>');
889 +}
890 +async function archiveOp(id,op){
891 + if(op==='delete'&&!confirm("Supprimer définitivement cette archive ?"))return;
892 + if(op==='restore'&&!confirm("Restaurer cette archive dans l'espace de travail actuel ?"))return;
893 + const r=await j('/api/archive/'+id+'/'+op,{method:'POST'});
894 + if(op==='restore')alert('Restauré : '+(r.entities||0)+' entités, '+(r.relations||0)+' relations.');
895 + loadArchives();loadMissions();lastEntCount=-1;
896 +}
897 +
898 +/* Réglages */
899 +function openDrawer(){$('drawer').classList.add('open');$('scrim').classList.add('open');loadConfig();}
900 +function closeDrawer(){$('drawer').classList.remove('open');$('scrim').classList.remove('open');}
901 +async function loadConfig(){
902 + const c=await j('/api/config');
903 + $('c_daily_budget').value=c.daily_budget;$('c_session_minutes').value=c.session_minutes;
904 + $('c_delay').value=c.delay;$('c_max_pages').value=c.max_pages;$('c_backend').value=c.backend;
905 + $('c_respect_robots').checked=String(c.respect_robots)==='1';
906 +}
907 +async function saveConfig(){
908 + const body={daily_budget:+$('c_daily_budget').value,session_minutes:+$('c_session_minutes').value,
909 + delay:+$('c_delay').value,max_pages:+$('c_max_pages').value,backend:$('c_backend').value,
910 + respect_robots:$('c_respect_robots').checked?'1':'0'};
911 + await j('/api/config',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body)});
912 + closeDrawer();
913 +}
914 +
915 +/* Recherche debouncée */
916 +$('fq').addEventListener('input',()=>{clearTimeout(entTimer);entTimer=setTimeout(loadEntities,300);});
917 +
918 +/* Temps réel : SSE, repli sur polling si indisponible */
919 +function startSSE(){
920 + try{
921 + const es=new EventSource('/api/stream');
922 + es.onmessage=ev=>{const d=JSON.parse(ev.data);if(d.stats)renderStats(d.stats);if(d.events)pushEvents(d.events);};
923 + es.onerror=()=>{es.close();startPolling();};
924 + }catch(e){startPolling();}
925 +}
926 +function startPolling(){
927 + async function tick(){const s=await j('/api/stats');renderStats(s);
928 + const d=await j('/api/events?after_id='+lastId+'&limit=60');pushEvents(d.events.reverse());}
929 + tick();setInterval(tick,3000);
930 +}
931 +buildSwatches();loadMissions();loadArchives();startSSE();
932 +setInterval(loadMissions,20000);setInterval(loadArchives,30000);
933 +setInterval(()=>{if(remaining>0){remaining--;renderTimer();}},1000);
934 +</script></body></html>"""
935 +
936 +
937 +
938 +
939 +# =========================================================================
940 +# Page « Répertoire structuré » v3 — 3 vues (cartes / tableau / graphe global)
941 +# + facettes (type, secteur, région), tri, insights, export CSV/JSON.
942 +# =========================================================================
943 +EXPLORE_HTML = r"""<!doctype html>
944 +<html lang="fr"><head>
945 +<meta charset="utf-8">
946 +<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
947 +<meta name="theme-color" content="#f5f3ee">
948 +<title>ka4 · Répertoire structuré — graphe du web québécois</title>
949 +<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='64' height='64'%3E%3Crect width='64' height='64' rx='12' fill='%23141814'/%3E%3Ctext x='50%25' y='54%25' font-family='Arial Black,Arial' font-weight='900' font-size='30' fill='%23e11d48' text-anchor='middle' dominant-baseline='central'%3Ek4%3C/text%3E%3C/svg%3E">
950 +<link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
951 +<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
952 +<link rel="stylesheet" href="/ka4.css">
953 +<style>
954 +.muted{color:var(--ink-3)}
955 +.subhead{padding:22px 0 4px}
956 +.subhead h1{font-size:clamp(26px,5vw,42px);text-transform:uppercase;letter-spacing:-.035em;line-height:1;margin-top:10px}
957 +.subhead h1 .hl{background:var(--lime);border-radius:8px;padding:0 10px;transform:rotate(-1deg);display:inline-block}
958 +.insights{display:flex;gap:10px;flex-wrap:wrap;margin:16px 0 6px}
959 +.ins{font-family:var(--font-mono);font-size:12px;padding:8px 14px;border:1.5px solid var(--ink);border-radius:999px;background:var(--surface);box-shadow:2px 2px 0 rgba(20,24,20,.1)}
960 +.ins b{font-family:var(--font-display);font-size:15px}
961 +.bar2{display:flex;gap:10px;align-items:center;flex-wrap:wrap;margin:14px 0 18px}
962 +.tabs{display:inline-flex;gap:4px;border:1.5px solid var(--ink);border-radius:999px;padding:4px;background:var(--surface);box-shadow:var(--shadow-off-soft)}
963 +.tab{font-family:var(--font-display);font-weight:700;font-size:13px;padding:7px 15px;border-radius:999px;cursor:pointer;border:none;background:transparent;color:var(--ink)}
964 +.tab.on{background:var(--ink);color:var(--lime)}
965 +.bar2 .exp{margin-left:auto;display:flex;gap:8px}
966 +.bar2 .cnt{font-family:var(--font-mono);font-size:11px;color:var(--ink-3);text-transform:uppercase;letter-spacing:.08em}
967 +.layout{display:grid;grid-template-columns:1fr;gap:18px;padding-bottom:40px}
968 +@media(min-width:980px){.layout{grid-template-columns:274px 1fr;align-items:start}}
969 +.side{background:var(--surface);border:1.5px solid var(--ink);border-radius:var(--r-card);box-shadow:var(--shadow-off-soft);overflow:hidden}
970 +@media(min-width:980px){.side{position:sticky;top:80px}}
971 +.side .sec{padding:14px 16px;border-bottom:1.5px dashed var(--line)}
972 +.side .sec:last-child{border-bottom:none}
973 +.side label{font-family:var(--font-mono);font-size:9.5px;font-weight:700;text-transform:uppercase;letter-spacing:.12em;color:var(--ink-3);display:block;margin-bottom:8px}
974 +.side input,.side select{width:100%}
975 +.facets{display:flex;flex-direction:column;gap:3px;max-height:190px;overflow:auto;margin:-2px}
976 +.f{display:flex;align-items:center;gap:8px;padding:6px 8px;border-radius:6px;cursor:pointer;font-size:13px}
977 +.f:hover{background:var(--surface-2)}
978 +.f.on{background:var(--ink);color:var(--lime)}
979 +.f .c{margin-left:auto;font-family:var(--font-mono);font-size:11px;color:var(--ink-3)}
980 +.f.on .c{color:var(--lime)}
981 +.topc{display:flex;flex-direction:column;gap:4px}
982 +.topc a{display:flex;gap:8px;font-size:12.5px;padding:4px 6px;border-radius:6px}
983 +.topc a:hover{background:var(--surface-2);text-decoration:none}
984 +.topc .d{margin-left:auto;font-family:var(--font-mono);font-size:11px;color:var(--green)}
985 +.filterbtn{display:none}
986 +/* Vue cartes */
987 +.egrid{display:grid;grid-template-columns:repeat(auto-fill,minmax(280px,1fr));gap:16px}
988 +.ecard{background:var(--surface);border:1.5px solid var(--ink);border-radius:var(--r-card);box-shadow:var(--shadow-off-soft);padding:15px;cursor:pointer;transition:.15s}
989 +.ecard:hover{transform:translate(-3px,-3px);box-shadow:var(--shadow-off)}
990 +.ecard .top{display:flex;align-items:center;gap:11px}
991 +.ecard .ini{width:40px;height:40px;border-radius:9px;background:var(--ink);color:var(--lime);display:grid;place-items:center;font-family:var(--font-display);font-weight:700;font-size:17px;flex:0 0 auto;transform:rotate(-2deg)}
992 +.ecard.person .ini{background:var(--lime);color:var(--ink)}
993 +.ecard h3{font-family:var(--font-display);font-size:16px;letter-spacing:-.02em;line-height:1.1}
994 +.ecard .meta{font-family:var(--font-mono);font-size:10px;color:var(--ink-3);text-transform:uppercase;letter-spacing:.05em;margin-top:3px}
995 +.ecard p{font-size:12.5px;color:var(--ink-2);margin:8px 0 6px}
996 +.ecard .foot{display:flex;align-items:center;gap:6px;flex-wrap:wrap;margin-top:8px}
997 +.rbadge{font-family:var(--font-mono);font-size:10px;font-weight:700;padding:3px 8px;border-radius:999px;background:var(--lime-soft);color:var(--green-deep);border:1.5px solid var(--green)}
998 +/* Vue tableau */
999 +.dtable{background:var(--surface);border:1.5px solid var(--ink);border-radius:var(--r-card);box-shadow:var(--shadow-off-soft);overflow:auto}
1000 +.dtable table{width:100%;border-collapse:collapse;font-size:13px}
1001 +.dtable th{position:sticky;top:0;background:var(--surface-2);font-family:var(--font-mono);font-size:9.5px;text-transform:uppercase;letter-spacing:.08em;color:var(--ink-2);border-bottom:1.5px solid var(--ink);padding:10px 12px;text-align:left;cursor:pointer;white-space:nowrap}
1002 +.dtable th.on{color:var(--green)}
1003 +.dtable td{border-bottom:1px solid var(--line);padding:10px 12px;vertical-align:top}
1004 +.dtable tr{cursor:pointer}.dtable tr:hover td{background:var(--surface-2)}
1005 +/* Vue graphe */
1006 +.gwrap{position:relative;background:var(--surface);border:1.5px solid var(--ink);border-radius:var(--r-card);box-shadow:var(--shadow-off-soft);overflow:hidden}
1007 +#net{width:100%;height:min(72vh,680px);display:block;touch-action:none;cursor:grab;background:
1008 + radial-gradient(circle at 1px 1px, rgba(20,24,20,.06) 1px, transparent 0) 0 0/22px 22px}
1009 +#net:active{cursor:grabbing}
1010 +.gctrl{position:absolute;top:12px;right:12px;display:flex;flex-direction:column;gap:6px}
1011 +.gctrl button{width:38px;height:38px;font-size:18px;padding:0;display:grid;place-items:center}
1012 +.glegend{position:absolute;left:12px;bottom:12px;display:flex;gap:12px;flex-wrap:wrap;background:rgba(255,255,255,.9);
1013 + border:1.5px solid var(--ink);border-radius:8px;padding:7px 10px;font-family:var(--font-mono);font-size:10px;text-transform:uppercase;letter-spacing:.06em}
1014 +.glegend i{display:inline-block;width:11px;height:11px;border-radius:50%;border:1.5px solid var(--ink);vertical-align:-1px;margin-right:5px}
1015 +.gnode circle{cursor:pointer}.gnode:hover circle{stroke-width:3}
1016 +/* Drawer détail */
1017 +.drawer.wide{width:min(580px,96vw)}
1018 +.det h2{font-size:22px;letter-spacing:-.03em}
1019 +.graph{width:100%;height:290px;border:1.5px solid var(--ink);border-radius:10px;background:var(--surface-2);box-shadow:var(--shadow-off-soft)}
1020 +.legend{display:flex;gap:16px;flex-wrap:wrap;font-family:var(--font-mono);font-size:10px;text-transform:uppercase;letter-spacing:.08em;color:var(--ink-2);margin-top:10px}
1021 +.legend i{display:inline-block;width:18px;height:0;border-top:2px solid var(--ink);vertical-align:middle;margin-right:6px}
1022 +.legend i.reg{border-top:2px dashed rgba(20,24,20,.4)}
1023 +.det-row{display:flex;gap:10px;margin:7px 0;font-size:14px}
1024 +.det-row .k{font-family:var(--font-mono);font-size:10px;text-transform:uppercase;letter-spacing:.08em;color:var(--ink-3);flex:0 0 96px;padding-top:3px}
1025 +.det-row .v{flex:1;word-break:break-word}
1026 +.conn h4{font-family:var(--font-mono);font-size:10px;text-transform:uppercase;letter-spacing:.1em;color:var(--green);margin:16px 0 8px}
1027 +.conn .chip{cursor:pointer}
1028 +@media(max-width:979px){
1029 + .filterbtn{display:inline-flex}
1030 + .side{display:none}.side.open{display:block}
1031 +}
1032 +</style></head><body>
1033 +
1034 +<header>
1035 + <div class="wrap hdr">
1036 + <a class="brand" href="/">
1037 + <div class="logo">k4</div>
1038 + <div><div class="wordmark"><span class="kn">ka</span><span class="kx">4</span></div>
1039 + <div class="sub">by <b>groupe·ka</b> · répertoire structuré</div></div>
1040 + </a>
1041 + <div class="right"><a class="btn btn-primary" href="/">← Tableau de bord</a></div>
1042 + </div>
1043 +</header>
1044 +
1045 +<main class="wrap">
1046 + <div class="subhead">
1047 + <span class="kicker">Répertoire structuré</span>
1048 + <h1>Répertoire <span class="hl">structuré</span></h1>
1049 + <div class="insights" id="insights"></div>
1050 + </div>
1051 +
1052 + <div class="bar2">
1053 + <div class="tabs">
1054 + <button class="tab on" data-v="cards" onclick="setView('cards')">▦ Cartes</button>
1055 + <button class="tab" data-v="table" onclick="setView('table')">▤ Tableau</button>
1056 + <button class="tab" data-v="graph" onclick="setView('graph')">◉ Graphe</button>
1057 + </div>
1058 + <button class="btn filterbtn" onclick="document.getElementById('side').classList.toggle('open')">⚙ Filtres</button>
1059 + <span class="cnt" id="count"></span>
1060 + <div class="exp">
1061 + <a class="btn" id="expCsv" href="/api/export.csv">↓ CSV</a>
1062 + <a class="btn" id="expJson" href="/api/export.json">↓ JSON</a>
1063 + </div>
1064 + </div>
1065 +
1066 + <div class="layout">
1067 + <aside class="side" id="side">
1068 + <div class="sec"><label>Recherche</label><input id="fq" placeholder="Nom, secteur, description…"></div>
1069 + <div class="sec"><label>Trier par</label>
1070 + <select id="fsort" onchange="state.sort=this.value;reload()">
1071 + <option value="degree">Plus connectées</option>
1072 + <option value="name">Nom (A→Z)</option>
1073 + <option value="recent">Récentes</option>
1074 + </select></div>
1075 + <div class="sec"><label>Type</label><div class="facets" id="fType"></div></div>
1076 + <div class="sec"><label>Secteur</label><div class="facets" id="fSector"></div></div>
1077 + <div class="sec"><label>Région</label><div class="facets" id="fRegion"></div></div>
1078 + <div class="sec"><label>Plus connectées</label><div class="topc" id="topc"></div></div>
1079 + <div class="sec"><button class="btn" style="width:100%" onclick="resetFilters()">Réinitialiser</button></div>
1080 + </aside>
1081 +
1082 + <section>
1083 + <div id="v-cards" class="egrid"></div>
1084 + <div id="v-table" class="dtable" style="display:none"></div>
1085 + <div id="v-graph" class="gwrap" style="display:none">
1086 + <svg id="net" viewBox="0 0 900 620" preserveAspectRatio="xMidYMid meet"><g id="gwrap"></g></svg>
1087 + <div class="gctrl">
1088 + <button class="btn" onclick="zoom(1.25)">+</button>
1089 + <button class="btn" onclick="zoom(0.8)">−</button>
1090 + <button class="btn" onclick="fitGraph()" title="Recentrer">⤢</button>
1091 + </div>
1092 + <div class="glegend">
1093 + <span><i style="background:#141814"></i>Entreprise</span>
1094 + <span><i style="background:#e8a33d"></i>Organisation</span>
1095 + <span><i style="background:#9f1239"></i>Personne</span>
1096 + <span><i style="background:#8b928c"></i>Site web</span>
1097 + </div>
1098 + </div>
1099 + <div style="text-align:center;margin-top:20px">
1100 + <button class="btn" id="moreBtn" style="display:none" onclick="loadMore()">Charger plus</button>
1101 + </div>
1102 + </section>
1103 + </div>
1104 +</main>
1105 +
1106 +<footer><div class="wrap">
1107 + <b>ka·4</b> — répertoire structuré · toute entité, ses attributs &amp; ses relations<br>
1108 + Auteur Simon-Pierre Boucher · <span class="lk">contact@spboucher.ai</span> · une plateforme du <b>Groupe KA</b><br>
1109 + Données publiques uniquement · conforme Loi 25 (Québec) / LPRPDE
1110 +</div></footer>
1111 +
1112 +<div class="scrim" id="scrim" onclick="closeD()"></div>
1113 +<aside class="drawer wide" id="drawer"><div class="det" id="det"></div></aside>
1114 +
1115 +<script>
1116 +const $=id=>document.getElementById(id);
1117 +const esc=s=>(s||'').replace(/[&<>"]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
1118 +const fmt=n=>(n||0).toLocaleString('fr-CA');
1119 +const TYPES={creator:'Créateur',agency:'Agence',brand:'Marque',business:'Entreprise',organization:'Organisation',person:'Personne',website:'Site web'};
1120 +const fmtFol=n=>{n=+n||0;return n>=1e6?(n/1e6).toFixed(1).replace('.0','')+'M':n>=1e3?(n/1e3).toFixed(1).replace('.0','')+'K':(n||'')};
1121 +const REL={WORKS_AT:'travaille chez',FOUNDER_OF:'fondateur·trice de',OWNS:'propriétaire de',
1122 + MEMBER_OF:'membre de',PARTNER_OF:'partenaire de',SUBSIDIARY_OF:'filiale de',PARENT_OF:'société mère de',
1123 + AFFILIATED_WITH:'affilié·e à',SUPPLIER_OF:'fournisseur de',LOCATED_IN:'situé·e à'};
1124 +const COL={business:'#141814',organization:'#e8a33d',person:'#9f1239',website:'#8b928c'};
1125 +const ini=n=>{const t=(n||'?').trim();return (t[0]||'?').toUpperCase()+(t.split(' ')[1]?t.split(' ')[1][0].toUpperCase():'')};
1126 +const _dom=u=>{try{return new URL(u).hostname.replace(/^www\./,'')}catch(e){return ''}};
1127 +async function j(u){const r=await fetch(u);return r.json()}
1128 +let state={view:'cards',type:'',sector:'',region:'',q:'',sort:'degree'},items=[],offset=0,total=0,tmr=null;
1129 +const LIMIT=60;
1130 +
1131 +function qs(){const p=new URLSearchParams();
1132 + if(state.type)p.set('type',state.type);if(state.sector)p.set('sector',state.sector);
1133 + if(state.region)p.set('region',state.region);if(state.q)p.set('q',state.q);return p.toString();}
1134 +
1135 +/* Résumé / facettes / insights */
1136 +async function loadSummary(){
1137 + const d=await j('/api/explore/summary');
1138 + $('insights').innerHTML=[
1139 + ['Entités',d.total],['Relations',d.relations],['Secteurs',(d.sectors||[]).length],['Régions',(d.regions||[]).length]
1140 + ].map(([k,v])=>`<span class="ins">${k} <b>${fmt(v)}</b></span>`).join('');
1141 + const bt=d.by_type||{};
1142 + facetGroup('fType','type',Object.keys(TYPES).map(k=>[k,TYPES[k],bt[k]||0]));
1143 + facetGroup('fSector','sector',(d.sectors||[]).slice(0,30).map(x=>[x.sector,x.sector,x.count]));
1144 + facetGroup('fRegion','region',(d.regions||[]).slice(0,30).map(x=>[x.location,x.location,x.count]));
1145 + $('topc').innerHTML=(d.top_connected||[]).map(e=>
1146 + `<a onclick="openEntity(${e.id})">${esc(e.name)}<span class="d">${e.degree}</span></a>`).join('')||'<span class="muted" style="font-size:12px">—</span>';
1147 +}
1148 +function facetGroup(elId,key,arr){
1149 + $(elId).innerHTML=arr.map(([val,lab,n])=>
1150 + `<div class="f${state[key]===val?' on':''}" onclick="setFacet('${key}',${JSON.stringify(val).replace(/"/g,'&quot;')})">
1151 + <span>${esc(lab)}</span><span class="c">${fmt(n)}</span></div>`).join('')||'<span class="muted" style="font-size:12px">—</span>';
1152 +}
1153 +function setFacet(key,val){state[key]=(state[key]===val)?'':val;loadSummary();reload();}
1154 +function resetFilters(){state.type=state.sector=state.region=state.q='';$('fq').value='';loadSummary();reload();}
1155 +
1156 +/* Vues */
1157 +function setView(v){state.view=v;
1158 + document.querySelectorAll('.tab').forEach(t=>t.classList.toggle('on',t.dataset.v===v));
1159 + $('v-cards').style.display=v==='cards'?'':'none';
1160 + $('v-table').style.display=v==='table'?'':'none';
1161 + $('v-graph').style.display=v==='graph'?'':'none';
1162 + $('moreBtn').style.display=(v!=='graph'&&offset<total)?'':'none';
1163 + if(v==='graph')loadGraph();else render();
1164 +}
1165 +function updateExport(){const q=qs();$('expCsv').href='/api/export.csv?'+q;$('expJson').href='/api/export.json?'+q;}
1166 +
1167 +async function reload(){updateExport();offset=0;items=[];
1168 + if(state.view==='graph'){loadGraph();return;}
1169 + await loadPage();}
1170 +async function loadPage(){
1171 + const d=await j(`/api/entities?${qs()}&sort=${state.sort}&offset=${offset}&limit=${LIMIT}`);
1172 + total=d.count;items=items.concat(d.entities);offset+=d.entities.length;
1173 + $('count').textContent=`${fmt(items.length)} / ${fmt(total)}`;
1174 + $('moreBtn').style.display=(state.view!=='graph'&&offset<total)?'':'none';
1175 + render();
1176 +}
1177 +function loadMore(){loadPage();}
1178 +function render(){state.view==='table'?renderTable():renderCards();}
1179 +
1180 +function emptyState(){
1181 + const q=(state.q||state.type||state.sector||state.region);
1182 + return `<div style="grid-column:1/-1;text-align:center;padding:60px 20px">
1183 + <div style="font-size:44px;line-height:1">🗺️</div>
1184 + <h3 style="font-family:var(--font-display);font-size:22px;margin:14px 0 6px">${q?'Aucune entité pour ce filtre':'Le répertoire est vide'}</h3>
1185 + <p class="muted" style="max-width:460px;margin:0 auto">${q?'Essaie d\'élargir ou de réinitialiser les filtres.':'Lance une exploration : ouvre le <b>Tableau de bord</b>, ajoute une mission (n\'importe quel sujet), puis démarre le bot. Les entités et leurs relations apparaîtront ici.'}</p>
1186 + <div style="margin-top:16px"><a class="btn btn-primary" href="/">Aller au tableau de bord</a></div></div>`;
1187 +}
1188 +function renderCards(){
1189 + $('v-cards').innerHTML=items.map(e=>{
1190 + const meta=[TYPES[e.type]||esc(e.type),e.niche?esc(e.niche):(e.sector?esc(e.sector):''),
1191 + e.platform?esc(e.platform):'',esc(e.region||e.location||'')].filter(Boolean).join(' · ');
1192 + const rb=e.relations_count?`<span class="rbadge">${e.relations_count} lien${e.relations_count>1?'s':''}</span>`:'';
1193 + const fb=e.followers?`<span class="rbadge" style="background:var(--lime-soft);color:var(--green-deep);border-color:var(--lime)">${fmtFol(e.followers)} abonnés</span>`:'';
1194 + const hb=e.handle?`<span class="chip">${esc(e.handle)}</span>`:'';
1195 + const site=e.domain?`<a class="chip" href="${esc(e.url)}" target="_blank" rel="noopener" onclick="event.stopPropagation()">${esc(e.domain)}</a>`:'';
1196 + return `<div class="ecard ${esc(e.type)}" onclick="openEntity(${e.id})">
1197 + <div class="top"><div class="ini">${esc(ini(e.name))}</div>
1198 + <div><h3>${esc(e.name)}</h3><div class="meta">${meta}</div></div></div>
1199 + ${e.description?`<p>${esc(e.description.slice(0,120))}</p>`:''}
1200 + <div class="foot">${fb}${rb}${hb}${site}</div></div>`;
1201 + }).join('')||emptyState();
1202 +}
1203 +function th(key,lab){return `<th class="${state.sort===key?'on':''}" onclick="state.sort='${key}';reload()">${lab}</th>`}
1204 +function renderTable(){
1205 + const rows=items.map(e=>`<tr onclick="openEntity(${e.id})">
1206 + <td><span class="tag ${esc(e.type)}">${TYPES[e.type]||esc(e.type)}</span></td>
1207 + <td class="name">${esc(e.name)}</td><td>${esc(e.sector||'')}</td>
1208 + <td>${esc(e.region||e.location||'')}</td>
1209 + <td class="mono">${e.relations_count||0}</td><td class="mono">${e.social_count||0}</td>
1210 + <td>${e.domain?`<a href="${esc(e.url)}" target="_blank" rel="noopener" onclick="event.stopPropagation()">↗</a>`:''}</td></tr>`).join('');
1211 + $('v-table').innerHTML=`<table><thead><tr>
1212 + <th>Type</th>${th('name','Nom')}<th>Secteur</th><th>Région</th>
1213 + ${th('degree','Relations')}<th>Liens</th><th></th></tr></thead><tbody>${rows}</tbody></table>`;
1214 +}
1215 +
1216 +/* ---------- Vue GRAPHE GLOBAL (force-directed) ---------- */
1217 +let gt={x:0,y:0,s:1},NODES=[],POS={};
1218 +async function loadGraph(){
1219 + const d=await j(`/api/graph?${qs()}&limit=140`);
1220 + NODES=d.nodes;const edges=d.edges;
1221 + $('count').textContent=`${fmt(NODES.length)} nœuds · ${fmt(edges.length)} liens`;
1222 + if(!NODES.length){$('gwrap').innerHTML='<text x="450" y="300" text-anchor="middle" font-family="Inter" fill="#8b928c">Aucune relation à afficher pour ce filtre — le graphe s\'enrichit au fil du crawl.</text>';return;}
1223 + POS=layout(NODES,edges,900,620,260);
1224 + const byId={};NODES.forEach(n=>byId[n.id]=n);
1225 + let e='',nd='';
1226 + edges.forEach(g=>{const a=POS[g.s],b=POS[g.t];if(!a||!b)return;
1227 + e+=`<line x1="${a.x.toFixed(1)}" y1="${a.y.toFixed(1)}" x2="${b.x.toFixed(1)}" y2="${b.y.toFixed(1)}" stroke="rgba(20,24,20,.22)" stroke-width="1.2"/>`;});
1228 + NODES.forEach(n=>{const p=POS[n.id];const r=6+Math.min(16,n.degree*1.6);const c=COL[n.type]||'#8b928c';
1229 + nd+=`<g class="gnode" onclick="openEntity(${n.id})"><title>${esc(n.name)} · ${TYPES[n.type]||n.type} · ${n.degree} lien(s)</title>`+
1230 + `<circle cx="${p.x.toFixed(1)}" cy="${p.y.toFixed(1)}" r="${r.toFixed(1)}" fill="${c}" stroke="#141814" stroke-width="1.5"/>`+
1231 + (n.degree>=4?`<text x="${p.x.toFixed(1)}" y="${(p.y-r-3).toFixed(1)}" font-family="'Space Grotesk',sans-serif" font-size="11" font-weight="600" fill="#141814" text-anchor="middle">${esc(n.name.slice(0,22))}</text>`:'')+
1232 + `</g>`;});
1233 + $('gwrap').innerHTML=e+nd;
1234 + fitGraph();
1235 +}
1236 +function layout(nodes,edges,W,H,iters){
1237 + const pos={};nodes.forEach((n,i)=>{const a=i/nodes.length*2*Math.PI;pos[n.id]={x:W/2+Math.cos(a)*W/6+(i%5),y:H/2+Math.sin(a)*H/6+(i%3)};});
1238 + const k=Math.sqrt((W*H)/Math.max(1,nodes.length))*0.62;
1239 + for(let it=0;it<iters;it++){
1240 + const disp={};nodes.forEach(n=>disp[n.id]={x:0,y:0});
1241 + for(let i=0;i<nodes.length;i++)for(let j2=i+1;j2<nodes.length;j2++){
1242 + const a=pos[nodes[i].id],b=pos[nodes[j2].id];let dx=a.x-b.x,dy=a.y-b.y;let dd=Math.hypot(dx,dy)||.01;
1243 + const f=k*k/dd,ux=dx/dd,uy=dy/dd;
1244 + disp[nodes[i].id].x+=ux*f;disp[nodes[i].id].y+=uy*f;disp[nodes[j2].id].x-=ux*f;disp[nodes[j2].id].y-=uy*f;}
1245 + edges.forEach(g=>{const a=pos[g.s],b=pos[g.t];if(!a||!b)return;let dx=a.x-b.x,dy=a.y-b.y;let dd=Math.hypot(dx,dy)||.01;
1246 + const f=dd*dd/k,ux=dx/dd,uy=dy/dd;disp[g.s].x-=ux*f;disp[g.s].y-=uy*f;disp[g.t].x+=ux*f;disp[g.t].y+=uy*f;});
1247 + // gravité vers le centre : garde les composants déconnectés groupés (évite la dérive aux bords)
1248 + nodes.forEach(n=>{const p=pos[n.id];disp[n.id].x+=(W/2-p.x)*0.06;disp[n.id].y+=(H/2-p.y)*0.06;});
1249 + const t=Math.max(2,(1-it/iters)*k*1.5);
1250 + nodes.forEach(n=>{const dd2=disp[n.id];let d=Math.hypot(dd2.x,dd2.y)||.01;const lim=Math.min(d,t);
1251 + pos[n.id].x+=dd2.x/d*lim;pos[n.id].y+=dd2.y/d*lim;});}
1252 + return pos;
1253 +}
1254 +function applyT(){$('gwrap').setAttribute('transform',`translate(${gt.x},${gt.y}) scale(${gt.s})`);}
1255 +function zoom(f){gt.s=Math.max(.3,Math.min(4,gt.s*f));applyT();}
1256 +function fitGraph(){
1257 + const ids=Object.keys(POS);if(!ids.length){gt={x:0,y:0,s:1};applyT();return;}
1258 + let mnx=1e9,mny=1e9,mxx=-1e9,mxy=-1e9;
1259 + ids.forEach(i=>{const p=POS[i];mnx=Math.min(mnx,p.x);mny=Math.min(mny,p.y);mxx=Math.max(mxx,p.x);mxy=Math.max(mxy,p.y);});
1260 + const bw=Math.max(1,mxx-mnx),bh=Math.max(1,mxy-mny),pad=70;
1261 + gt.s=Math.max(.3,Math.min(2.4,Math.min((900-pad)/bw,(620-pad)/bh)));
1262 + gt.x=(900-(mnx+mxx)*gt.s)/2;gt.y=(620-(mny+mxy)*gt.s)/2;applyT();
1263 +}
1264 +(function(){const svg=$('net');let drag=false,lx,ly;
1265 + svg.addEventListener('mousedown',e=>{drag=true;lx=e.clientX;ly=e.clientY;});
1266 + window.addEventListener('mouseup',()=>drag=false);
1267 + window.addEventListener('mousemove',e=>{if(!drag)return;const r=svg.getBoundingClientRect();const sc=900/r.width;
1268 + gt.x+=(e.clientX-lx)*sc;gt.y+=(e.clientY-ly)*sc;lx=e.clientX;ly=e.clientY;applyT();});
1269 + svg.addEventListener('wheel',e=>{e.preventDefault();zoom(e.deltaY<0?1.12:0.9);},{passive:false});
1270 + let pd=0;
1271 + svg.addEventListener('touchstart',e=>{if(e.touches.length===1){drag=true;lx=e.touches[0].clientX;ly=e.touches[0].clientY;}
1272 + else if(e.touches.length===2){pd=Math.hypot(e.touches[0].clientX-e.touches[1].clientX,e.touches[0].clientY-e.touches[1].clientY);}},{passive:true});
1273 + svg.addEventListener('touchmove',e=>{const r=svg.getBoundingClientRect();const sc=900/r.width;
1274 + if(e.touches.length===1&&drag){gt.x+=(e.touches[0].clientX-lx)*sc;gt.y+=(e.touches[0].clientY-ly)*sc;lx=e.touches[0].clientX;ly=e.touches[0].clientY;applyT();}
1275 + else if(e.touches.length===2){const nd=Math.hypot(e.touches[0].clientX-e.touches[1].clientX,e.touches[0].clientY-e.touches[1].clientY);if(pd)zoom(nd/pd);pd=nd;}},{passive:true});
1276 + svg.addEventListener('touchend',()=>{drag=false;pd=0;});
1277 +})();
1278 +
1279 +/* ---------- Détail (drawer) + ego-graph ---------- */
1280 +function chip(url,label,stop){return `<a class="chip" href="${esc(url)}" target="_blank" rel="noopener"${stop?' onclick="event.stopPropagation()"':''}>${esc(label)}</a>`}
1281 +function egoGraph(e){
1282 + const rm={};(e.relations||[]).forEach(r=>{const k=r.id;if(!rm[k])rm[k]=Object.assign({kind:'rel',roles:[]},r);rm[k].roles.push(r.role||REL[r.relation]||'');});
1283 + const rels=Object.values(rm).map(n=>Object.assign(n,{role:n.roles.filter(Boolean).join(' · ')}));
1284 + const reg=(e.same_region||[]).map(n=>Object.assign({kind:'region'},n));
1285 + const neigh=rels.concat(reg).slice(0,12);
1286 + if(!neigh.length) return '<p class="muted" style="margin:10px 0">Aucune relation extraite pour l\'instant.</p>';
1287 + const W=560,H=300,cx=W/2,cy=H/2,R=112;let ed='',lb='',no='';
1288 + neigh.forEach((n,i)=>{const a=(i/neigh.length)*2*Math.PI-Math.PI/2,x=cx+R*Math.cos(a),y=cy+R*Math.sin(a);
1289 + const isRel=n.kind==='rel',col=isRel?'#141814':'rgba(20,24,20,.3)',dash=isRel?'':' stroke-dasharray="4 3"';
1290 + ed+=`<line x1="${cx}" y1="${cy}" x2="${x.toFixed(0)}" y2="${y.toFixed(0)}" stroke="${col}" stroke-width="${isRel?2:1.2}"${dash}/>`;
1291 + if(isRel){const lab=(n.role||REL[n.relation]||'').slice(0,20);if(lab)lb+=`<text x="${((cx+x)/2).toFixed(0)}" y="${((cy+y)/2-3).toFixed(0)}" font-family="'JetBrains Mono',monospace" font-size="8" fill="#9f1239" text-anchor="middle">${esc(lab)}</text>`;}
1292 + const fill=n.type==='person'?'#e11d48':'#ffffff';
1293 + no+=`<g class="gnode" onclick="openEntity(${n.id})"><title>${esc(n.name)}${n.role?' — '+esc(n.role):''}</title>`+
1294 + `<circle cx="${x.toFixed(0)}" cy="${y.toFixed(0)}" r="17" fill="${fill}" stroke="#141814" stroke-width="1.5"/>`+
1295 + `<text x="${x.toFixed(0)}" y="${y.toFixed(0)}" font-family="'JetBrains Mono',monospace" font-size="10" font-weight="700" fill="#141814" text-anchor="middle" dominant-baseline="central">${esc(ini(n.name))}</text></g>`;});
1296 + const center=`<circle cx="${cx}" cy="${cy}" r="28" fill="#141814"/><text x="${cx}" y="${cy}" font-family="'Space Grotesk',sans-serif" font-size="15" font-weight="700" fill="#e11d48" text-anchor="middle" dominant-baseline="central">${esc(ini(e.name))}</text>`;
1297 + return `<svg viewBox="0 0 ${W} ${H}" class="graph" preserveAspectRatio="xMidYMid meet">${ed}${lb}${center}${no}</svg>`+
1298 + `<div class="legend"><span><i></i>relation extraite (${rels.length})</span><span><i class="reg"></i>même région (${reg.length})</span></div>`;
1299 +}
1300 +async function openEntity(id){
1301 + const e=await j('/api/entity/'+id);if(e.error)return;
1302 + const rows=[];const add=(k,v)=>{if(v)rows.push(`<div class="det-row"><div class="k">${k}</div><div class="v">${v}</div></div>`)};
1303 + add('Type',TYPES[e.type]||esc(e.type));
1304 + add('Niche',e.niche?esc(e.niche):'');add('Handle',e.handle?esc(e.handle):'');
1305 + add('Plateforme',e.platform?esc(e.platform):'');
1306 + add('Abonnés',e.followers?fmtFol(e.followers)+' ('+fmt(e.followers)+')':'');
1307 + add('Langues',e.languages?esc(e.languages):'');
1308 + add('Secteur',e.sector?esc(e.sector):'');
1309 + add('Description',e.description?esc(e.description):'');
1310 + add('Site web',e.url?`<a href="${esc(e.url)}" target="_blank" rel="noopener">${esc(e.domain||e.url)} ↗</a>`:'');
1311 + add('Région',esc(e.region||e.location||''));add('Adresse',esc(e.address||''));
1312 + add('Fondée',esc(e.founded||''));add('Taille',esc(e.size||''));add('NEQ',esc(e.neq||''));
1313 + add('Courriel',e.email?`<a href="mailto:${esc(e.email)}">${esc(e.email)}</a>`:'');
1314 + add('Téléphone',e.phone?esc(e.phone):'');
1315 + add('Médias sociaux',(e.social_links||[]).map(l=>chip(l.url,l.platform+(l.followers?' ('+fmtFol(l.followers)+')':''))).join(' '));
1316 + add('Sources',(e.sources||[]).slice(0,6).map(u=>chip(u,_dom(u)||'source',1)).join(' '));
1317 + const rels=e.relations||[];let conn='';
1318 + if(rels.length){conn+=`<h4>Relations extraites — ${rels.length}</h4><div>`+rels.map(r=>{
1319 + const verb=r.role?esc(r.role):(REL[r.relation]||esc(r.relation));
1320 + return `<span class="chip" onclick="openEntity(${r.id})">${esc(r.name)} <span class="muted">· ${verb}</span></span>`;}).join(' ')+`</div>`;}
1321 + if((e.same_region||[]).length){conn+=`<h4>Même région — ${e.same_region.length}</h4><div>`+
1322 + e.same_region.map(n=>`<span class="chip" onclick="openEntity(${n.id})">${esc(n.name)} · ${TYPES[n.type]||esc(n.type)}</span>`).join(' ')+`</div>`;}
1323 + $('det').innerHTML=`<div class="dhead"><h2>${esc(e.name)}</h2><button class="close" onclick="closeD()">✕</button></div>
1324 + <div class="dsec"><span class="kicker">Carte de connexions</span><div style="margin-top:12px">${egoGraph(e)}</div></div>
1325 + <div class="dsec">${rows.join('')||'<p class="muted">Aucune coordonnée.</p>'}</div>
1326 + <div class="dsec conn" style="border-bottom:none">${conn||'<p class="muted">Aucune connexion pour l\'instant.</p>'}</div>`;
1327 + $('drawer').classList.add('open');$('scrim').classList.add('open');
1328 +}
1329 +function closeD(){$('drawer').classList.remove('open');$('scrim').classList.remove('open');}
1330 +
1331 +$('fq').addEventListener('input',e=>{clearTimeout(tmr);state.q=e.target.value;tmr=setTimeout(reload,300);});
1332 +document.addEventListener('keydown',e=>{if(e.key==='Escape')closeD();});
1333 +$('fsort').value=state.sort;
1334 +loadSummary();updateExport();reload();
1335 +</script></body></html>"""
1336