API-KA — plateforme centrale : collecte quotidienne des 8 services KA + API publique
Collecteurs louka/immoka/foodka/autoka/fabrika/restoka/sortika/creaka (resto-ka avec fiches détaillées : menus, plats, options et prix réels), PostgreSQL append-only, backups horodatés, scheduler 02:00 + backfill, FastAPI /api/v1 sur www.api-ka.com, plateforme web de documentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
56 changed files +4,098 −0
added
.env.example
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : .env.example | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +# Copier ce fichier vers .env sur m3u96b et remplir les valeurs. | |
| 10 | +# Ne JAMAIS commiter le fichier .env. | |
| 11 | + | |
| 12 | +APP_ENV=production | |
| 13 | +NODE_NAME=m3u96b | |
| 14 | +DATABASE_URL=postgresql://apika_user:@localhost:5432/apika | |
| 15 | +NGROK_AUTHTOKEN= | |
| 16 | +NGROK_DOMAIN=www.api-ka.com | |
| 17 | +API_PORT=8000 | |
| 18 | +DAILY_RUN_HOUR=02 | |
| 19 | +BACKUP_RETENTION_DAYS=90 | |
| 20 | +RATE_LIMIT_PER_MINUTE=120 | |
| 21 | +LOUKA_SOURCE_URL= | |
| 22 | +IMMOKA_SOURCE_URL= | |
| 23 | +FOODKA_SOURCE_URL= | |
| 24 | +AUTOKA_SOURCE_URL= | |
| 25 | +FABRIKA_SOURCE_URL= | |
| 26 | +RESTOKA_SOURCE_URL= | |
| 27 | +SORTIKA_SOURCE_URL= | |
| 28 | +CREAKA_SOURCE_URL= | |
added
.gitignore
+19 −0
@@ -0,0 +1,19 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : .gitignore | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +.env | |
| 11 | +venv/ | |
| 12 | +__pycache__/ | |
| 13 | +*.pyc | |
| 14 | +.pytest_cache/ | |
| 15 | +.ruff_cache/ | |
| 16 | +logs/* | |
| 17 | +!logs/.gitkeep | |
| 18 | +data/* | |
| 19 | +!data/backups/.gitkeep | |
added
CLAUDE.md
+275 −0
@@ -0,0 +1,275 @@ | ||
| 1 | +# CLAUDE.md — Projet API-KA | |
| 2 | + | |
| 3 | +## Vision du projet | |
| 4 | + | |
| 5 | +**API-KA** est la plateforme centrale légendaire de l'écosystème KA. Elle collecte, sauvegarde et centralise **chaque jour** les données de tous les services KA (lou-ka, immo-ka, food-ka, auto-ka, fabri-ka, resto-ka, sorti-ka, crea-ka) dans une base de données unifiée, puis les expose via une API publique accessible sur **www.api-ka.com** (tunnel ngrok). | |
| 6 | + | |
| 7 | +**Tout le déploiement se fait sur le node `m3u96b`.** Aucun service de production ne tourne ailleurs. | |
| 8 | + | |
| 9 | +## Auteur — OBLIGATOIRE dans chaque fichier | |
| 10 | + | |
| 11 | +Chaque fichier de code (Python, JS, SQL, shell, config, etc.) DOIT commencer par un en-tête d'auteur. Règle non négociable — aucun fichier sans en-tête. | |
| 12 | + | |
| 13 | +**Python / Shell / SQL / YAML :** | |
| 14 | +``` | |
| 15 | +# ============================================ | |
| 16 | +# Projet : API-KA | |
| 17 | +# Fichier : <nom_du_fichier> | |
| 18 | +# Node : m3u96b | |
| 19 | +# Author : Simon-Pierre Boucher | |
| 20 | +# Contact : contact@spboucher.ai | |
| 21 | +# Date : <date de création> | |
| 22 | +# ============================================ | |
| 23 | +``` | |
| 24 | + | |
| 25 | +**JavaScript / TypeScript :** | |
| 26 | +```javascript | |
| 27 | +/** | |
| 28 | + * ============================================ | |
| 29 | + * Projet : API-KA | |
| 30 | + * Fichier : <nom_du_fichier> | |
| 31 | + * Node : m3u96b | |
| 32 | + * Author : Simon-Pierre Boucher | |
| 33 | + * Contact : contact@spboucher.ai | |
| 34 | + * Date : <date de création> | |
| 35 | + * ============================================ | |
| 36 | + */ | |
| 37 | +``` | |
| 38 | + | |
| 39 | +## Node de déploiement : m3u96b | |
| 40 | + | |
| 41 | +- **Node cible unique** : `m3u96b`. Tous les services (API, scheduler, base de données, tunnel ngrok) tournent sur ce node. | |
| 42 | +- Répertoire de déploiement : `/opt/api-ka/` sur m3u96b. | |
| 43 | +- Données et backups : `/opt/api-ka/data/` et `/opt/api-ka/data/backups/`. | |
| 44 | +- Logs centralisés : `/opt/api-ka/logs/`. | |
| 45 | +- Les services sont gérés par **systemd** sur m3u96b (voir section "Services systemd"). | |
| 46 | +- Toute commande de déploiement, cron ou service doit référencer explicitement m3u96b dans sa documentation et ses logs (`hostname` vérifié au démarrage : si le hostname n'est pas `m3u96b`, le service refuse de démarrer en mode production et log un avertissement). | |
| 47 | + | |
| 48 | +### Vérification du node au démarrage (obligatoire dans chaque service) | |
| 49 | + | |
| 50 | +```python | |
| 51 | +import socket, os | |
| 52 | + | |
| 53 | +REQUIRED_NODE = "m3u96b" | |
| 54 | + | |
| 55 | +def verify_node(): | |
| 56 | + hostname = socket.gethostname() | |
| 57 | + if os.getenv("APP_ENV") == "production" and hostname != REQUIRED_NODE: | |
| 58 | + raise RuntimeError(f"API-KA doit tourner sur {REQUIRED_NODE}, node actuel : {hostname}") | |
| 59 | +``` | |
| 60 | + | |
| 61 | +## Sources de données (les 8 services KA) | |
| 62 | + | |
| 63 | +| Service | Table | Variable source | Fréquence | | |
| 64 | +|-----------|------------------|----------------------|-------------| | |
| 65 | +| lou-ka | `louka_data` | `LOUKA_SOURCE_URL` | Quotidienne | | |
| 66 | +| immo-ka | `immoka_data` | `IMMOKA_SOURCE_URL` | Quotidienne | | |
| 67 | +| food-ka | `foodka_data` | `FOODKA_SOURCE_URL` | Quotidienne | | |
| 68 | +| auto-ka | `autoka_data` | `AUTOKA_SOURCE_URL` | Quotidienne | | |
| 69 | +| fabri-ka | `fabrika_data` | `FABRIKA_SOURCE_URL` | Quotidienne | | |
| 70 | +| resto-ka | `restoka_data` | `RESTOKA_SOURCE_URL` | Quotidienne | | |
| 71 | +| sorti-ka | `sortika_data` | `SORTIKA_SOURCE_URL` | Quotidienne | | |
| 72 | +| crea-ka | `creaka_data` | `CREAKA_SOURCE_URL` | Quotidienne | | |
| 73 | + | |
| 74 | +**RÈGLE CRITIQUE : les données de chaque service DOIVENT être sauvegardées tous les jours, sans exception.** | |
| 75 | +- Chaque exécution quotidienne est journalisée. | |
| 76 | +- Tout échec de collecte est détecté, loggé et relancé automatiquement (3 tentatives, backoff exponentiel : 30s → 2min → 10min). | |
| 77 | +- Si un service échoue après 3 tentatives, une entrée `status=failed` est écrite dans `collection_runs` et une alerte est ajoutée à `logs/alerts.log`. Le lendemain, le job tente automatiquement un rattrapage (backfill) des dates manquées. | |
| 78 | + | |
| 79 | +## Architecture | |
| 80 | + | |
| 81 | +``` | |
| 82 | +/opt/api-ka/ # sur le node m3u96b | |
| 83 | +├── CLAUDE.md | |
| 84 | +├── README.md | |
| 85 | +├── .env # jamais commité | |
| 86 | +├── .env.example | |
| 87 | +├── requirements.txt | |
| 88 | +├── src/ | |
| 89 | +│ ├── config.py # chargement .env + vérification node m3u96b | |
| 90 | +│ ├── collectors/ | |
| 91 | +│ │ ├── base_collector.py # classe abstraite : fetch, validate, save, retry | |
| 92 | +│ │ ├── louka_collector.py | |
| 93 | +│ │ ├── immoka_collector.py | |
| 94 | +│ │ ├── foodka_collector.py | |
| 95 | +│ │ ├── autoka_collector.py | |
| 96 | +│ │ ├── fabrika_collector.py | |
| 97 | +│ │ ├── restoka_collector.py | |
| 98 | +│ │ ├── sortika_collector.py | |
| 99 | +│ │ └── creaka_collector.py | |
| 100 | +│ ├── database/ | |
| 101 | +│ │ ├── models.py # SQLAlchemy : 8 tables données + collection_runs | |
| 102 | +│ │ ├── db.py # engine, session, init, healthcheck | |
| 103 | +│ │ └── migrations/ # Alembic | |
| 104 | +│ ├── api/ | |
| 105 | +│ │ ├── main.py # FastAPI, monté derrière ngrok | |
| 106 | +│ │ ├── routes/ | |
| 107 | +│ │ │ ├── services.py # /api/v1/{service}... | |
| 108 | +│ │ │ ├── runs.py # /api/v1/runs | |
| 109 | +│ │ │ └── health.py # /health (inclut node, dernière collecte, db) | |
| 110 | +│ │ └── middleware/ | |
| 111 | +│ │ ├── logging.py # log de chaque requête | |
| 112 | +│ │ └── ratelimit.py # protection basique de l'API publique | |
| 113 | +│ ├── scheduler/ | |
| 114 | +│ │ ├── daily_job.py # orchestre les 8 collecteurs | |
| 115 | +│ │ └── backfill.py # rattrapage des dates manquées | |
| 116 | +│ └── utils/ | |
| 117 | +│ ├── logger.py # logs JSON structurés, rotation quotidienne | |
| 118 | +│ ├── retry.py # décorateur retry avec backoff | |
| 119 | +│ └── backup.py # dump quotidien horodaté | |
| 120 | +├── scripts/ | |
| 121 | +│ ├── deploy_m3u96b.sh # déploiement complet sur le node | |
| 122 | +│ ├── run_daily_backup.sh | |
| 123 | +│ ├── start_ngrok.sh # tunnel www.api-ka.com avec auto-restart | |
| 124 | +│ └── healthcheck.sh # utilisé par systemd / monitoring | |
| 125 | +├── systemd/ | |
| 126 | +│ ├── apika-api.service | |
| 127 | +│ ├── apika-scheduler.service | |
| 128 | +│ └── apika-ngrok.service | |
| 129 | +├── data/ | |
| 130 | +│ └── backups/YYYY-MM-DD/ # dumps quotidiens par service | |
| 131 | +├── logs/ | |
| 132 | +└── tests/ | |
| 133 | +``` | |
| 134 | + | |
| 135 | +## Base de données | |
| 136 | + | |
| 137 | +- **Moteur** : PostgreSQL 16 sur m3u96b (SQLite acceptable uniquement en dev local). | |
| 138 | +- Base : `apika`, utilisateur dédié `apika_user`, accès restreint à localhost sur m3u96b. | |
| 139 | + | |
| 140 | +### Schéma des tables de données (identique pour les 8 services) | |
| 141 | + | |
| 142 | +| Colonne | Type | Détail | | |
| 143 | +|----------------|-------------|------------------------------------------| | |
| 144 | +| `id` | BIGSERIAL | Clé primaire | | |
| 145 | +| `payload` | JSONB | Données brutes du service | | |
| 146 | +| `source` | TEXT | Nom du service (ex. `louka`) | | |
| 147 | +| `collected_at` | TIMESTAMPTZ | Horodatage exact de la collecte | | |
| 148 | +| `date_key` | DATE | Date logique de la collecte (index) | | |
| 149 | +| `checksum` | TEXT | SHA-256 du payload (déduplication) | | |
| 150 | + | |
| 151 | +Index : `(date_key)`, `(source, date_key)`, unique sur `(source, date_key, checksum)`. | |
| 152 | + | |
| 153 | +### Table `collection_runs` | |
| 154 | + | |
| 155 | +`id`, `service`, `date_key`, `status` (`success` / `failed` / `retried`), `records_count`, `duration_seconds`, `error_message`, `node` (toujours `m3u96b`), `started_at`, `finished_at`. | |
| 156 | + | |
| 157 | +### Règles | |
| 158 | +- **Append-only** : aucune suppression destructive, l'historique complet est conservé. | |
| 159 | +- **Backup quotidien** : `pg_dump` horodaté par service dans `data/backups/YYYY-MM-DD/`, compressé (`.sql.gz`), conservé 90 jours minimum. | |
| 160 | +- Vérification d'intégrité hebdomadaire : comparaison du nombre de `date_key` distincts vs jours écoulés — toute journée manquante déclenche un backfill. | |
| 161 | + | |
| 162 | +## Collecte quotidienne (scheduler) | |
| 163 | + | |
| 164 | +- Job planifié **tous les jours à 02:00 (heure du node m3u96b)** via `apika-scheduler.service` (APScheduler) — cron système en fallback. | |
| 165 | +- Les 8 collecteurs s'exécutent en parallèle mais de façon **indépendante** : l'échec d'un service ne bloque jamais les autres. | |
| 166 | +- Pipeline par collecteur : `fetch → validate → checksum → insert → backup → log run`. | |
| 167 | +- À la fin du run global : résumé dans `collection_runs` + `logs/daily_YYYY-MM-DD.log` + mise à jour de `/health`. | |
| 168 | +- `backfill.py` s'exécute juste après le job quotidien et rattrape automatiquement toute date manquée des 7 derniers jours. | |
| 169 | + | |
| 170 | +## API publique (www.api-ka.com via ngrok) | |
| 171 | + | |
| 172 | +- Framework : **FastAPI** (docs auto sur `/docs`, OpenAPI sur `/openapi.json`). | |
| 173 | +- Serveur : `uvicorn` sur `127.0.0.1:8000` (jamais exposé directement — seul ngrok est public). | |
| 174 | +- Tunnel : `ngrok http --domain=www.api-ka.com 8000` — le domaine doit être réservé dans le dashboard ngrok (compte payant requis pour un domaine personnalisé) et le `NGROK_AUTHTOKEN` configuré sur m3u96b. | |
| 175 | +- `apika-ngrok.service` redémarre le tunnel automatiquement en cas de coupure (`Restart=always`). | |
| 176 | + | |
| 177 | +### Endpoints | |
| 178 | + | |
| 179 | +| Méthode | Route | Description | | |
| 180 | +|---------|-----------------------------------------|----------------------------------------------------| | |
| 181 | +| GET | `/` | Statut de la plateforme + version | | |
| 182 | +| GET | `/health` | Node (m3u96b), état DB, dernière collecte par service | | |
| 183 | +| GET | `/api/v1/{service}` | Données paginées d'un service | | |
| 184 | +| GET | `/api/v1/{service}/latest` | Dernière collecte du service | | |
| 185 | +| GET | `/api/v1/{service}/date/{YYYY-MM-DD}` | Données d'une date précise | | |
| 186 | +| GET | `/api/v1/{service}/stats` | Nb d'enregistrements par jour, dernière réussite | | |
| 187 | +| GET | `/api/v1/runs` | Historique des collectes (filtrable par service/statut) | | |
| 188 | + | |
| 189 | +- `{service}` ∈ `louka`, `immoka`, `foodka`, `autoka`, `fabrika`, `restoka`, `sortika`, `creaka` — toute autre valeur → 404. | |
| 190 | +- Versionnement `/api/v1/`, pagination `?page=&limit=` (limit max 500). | |
| 191 | +- Format de réponse uniforme : | |
| 192 | +```json | |
| 193 | +{ "success": true, "data": [...], "meta": { "page": 1, "limit": 100, "total": 4200, "node": "m3u96b" } } | |
| 194 | +``` | |
| 195 | +- Rate limiting basique (ex. 120 req/min/IP) car l'API est publique via ngrok. | |
| 196 | + | |
| 197 | +## Services systemd (sur m3u96b) | |
| 198 | + | |
| 199 | +| Service | Rôle | Restart | | |
| 200 | +|--------------------------|-----------------------------------|----------| | |
| 201 | +| `apika-api.service` | uvicorn FastAPI :8000 | always | | |
| 202 | +| `apika-scheduler.service`| Job quotidien 02:00 + backfill | always | | |
| 203 | +| `apika-ngrok.service` | Tunnel www.api-ka.com | always | | |
| 204 | + | |
| 205 | +Commandes : `sudo systemctl enable --now apika-api apika-scheduler apika-ngrok`, statut via `systemctl status apika-*`. | |
| 206 | + | |
| 207 | +## Configuration (.env sur m3u96b) | |
| 208 | + | |
| 209 | +``` | |
| 210 | +APP_ENV=production | |
| 211 | +NODE_NAME=m3u96b | |
| 212 | +DATABASE_URL=postgresql://apika_user:***@localhost:5432/apika | |
| 213 | +NGROK_AUTHTOKEN=... | |
| 214 | +NGROK_DOMAIN=www.api-ka.com | |
| 215 | +API_PORT=8000 | |
| 216 | +DAILY_RUN_HOUR=02 | |
| 217 | +BACKUP_RETENTION_DAYS=90 | |
| 218 | +LOUKA_SOURCE_URL=... | |
| 219 | +IMMOKA_SOURCE_URL=... | |
| 220 | +FOODKA_SOURCE_URL=... | |
| 221 | +AUTOKA_SOURCE_URL=... | |
| 222 | +FABRIKA_SOURCE_URL=... | |
| 223 | +RESTOKA_SOURCE_URL=... | |
| 224 | +SORTIKA_SOURCE_URL=... | |
| 225 | +CREAKA_SOURCE_URL=... | |
| 226 | +``` | |
| 227 | + | |
| 228 | +Ne jamais commiter `.env` — seulement `.env.example` avec des valeurs vides. | |
| 229 | + | |
| 230 | +## Standards de code | |
| 231 | + | |
| 232 | +- Python 3.11+, type hints partout, `black` + `ruff`, docstrings sur toute fonction publique. | |
| 233 | +- Logging structuré JSON via `utils/logger.py` avec rotation quotidienne — jamais de `print()` en production. | |
| 234 | +- Tests `pytest` dans `tests/` : chaque collecteur, chaque route API, la logique de retry et de backfill. | |
| 235 | +- Gestion d'erreurs explicite : jamais de `except: pass`. | |
| 236 | +- Commits descriptifs ; branche `main` = état déployé sur m3u96b. | |
| 237 | +- **Rappel : en-tête d'auteur (Simon-Pierre Boucher / contact@spboucher.ai / Node m3u96b) dans CHAQUE fichier.** | |
| 238 | + | |
| 239 | +## Commandes utiles | |
| 240 | + | |
| 241 | +```bash | |
| 242 | +# Déploiement complet sur m3u96b | |
| 243 | +bash scripts/deploy_m3u96b.sh | |
| 244 | + | |
| 245 | +# Initialiser la base | |
| 246 | +python -m src.database.db --init | |
| 247 | + | |
| 248 | +# Collecte manuelle immédiate (tous les services) | |
| 249 | +python -m src.scheduler.daily_job --now | |
| 250 | + | |
| 251 | +# Backfill d'une date manquée | |
| 252 | +python -m src.scheduler.backfill --date 2026-08-15 | |
| 253 | + | |
| 254 | +# API en local (dev) | |
| 255 | +uvicorn src.api.main:app --reload --port 8000 | |
| 256 | + | |
| 257 | +# Tunnel public | |
| 258 | +bash scripts/start_ngrok.sh | |
| 259 | + | |
| 260 | +# Vérifier la santé | |
| 261 | +curl http://127.0.0.1:8000/health | |
| 262 | +``` | |
| 263 | + | |
| 264 | +## Priorités absolues (dans l'ordre) | |
| 265 | + | |
| 266 | +1. **Fiabilité de la sauvegarde quotidienne** — aucune journée sans données pour aucun des 8 services. | |
| 267 | +2. Intégrité de la base (append-only, checksums, backups horodatés, rétention 90 jours). | |
| 268 | +3. Tout tourne sur le node **m3u96b** — vérification du hostname au démarrage. | |
| 269 | +4. Disponibilité de l'API publique sur www.api-ka.com (auto-restart ngrok + uvicorn). | |
| 270 | +5. En-tête d'auteur présent dans chaque fichier. | |
| 271 | +6. Logs, alertes et traçabilité de chaque collecte et chaque requête. | |
| 272 | + | |
| 273 | +--- | |
| 274 | + | |
| 275 | +*Plateforme légendaire API-KA — Node m3u96b — créée par Simon-Pierre Boucher (contact@spboucher.ai)* | |
added
README.md
+67 −0
@@ -0,0 +1,67 @@ | ||
| 1 | +# API-KA — Plateforme centrale de l'écosystème KA | |
| 2 | + | |
| 3 | +API-KA collecte, sauvegarde et centralise **chaque jour** les données des 8 services KA | |
| 4 | +(**lou-ka, immo-ka, food-ka, auto-ka, fabri-ka**) dans une base PostgreSQL unifiée, puis | |
| 5 | +les expose via une API publique sur **www.api-ka.com** (tunnel ngrok). | |
| 6 | + | |
| 7 | +**Tout le déploiement se fait sur le node `m3u96b`** (`/opt/api-ka/`). Chaque service | |
| 8 | +vérifie le hostname au démarrage et refuse de tourner ailleurs en production. | |
| 9 | + | |
| 10 | +## Démarrage rapide | |
| 11 | + | |
| 12 | +```bash | |
| 13 | +# Déploiement complet sur m3u96b | |
| 14 | +bash scripts/deploy_m3u96b.sh | |
| 15 | + | |
| 16 | +# Initialiser la base | |
| 17 | +python -m src.database.db --init | |
| 18 | + | |
| 19 | +# Collecte manuelle immédiate (tous les services) | |
| 20 | +python -m src.scheduler.daily_job --now | |
| 21 | + | |
| 22 | +# Backfill d'une date manquée | |
| 23 | +python -m src.scheduler.backfill --date 2026-08-15 | |
| 24 | + | |
| 25 | +# API en local (dev) | |
| 26 | +uvicorn src.api.main:app --reload --port 8000 | |
| 27 | + | |
| 28 | +# Tunnel public | |
| 29 | +bash scripts/start_ngrok.sh | |
| 30 | + | |
| 31 | +# Vérifier la santé | |
| 32 | +curl http://127.0.0.1:8000/health | |
| 33 | +``` | |
| 34 | + | |
| 35 | +## Composants | |
| 36 | + | |
| 37 | +| Composant | Rôle | | |
| 38 | +|-----------|------| | |
| 39 | +| `src/collectors/` | 8 collecteurs (fetch → validate → checksum → insert → backup → log run), retry 3× avec backoff 30s → 2min → 10min | | |
| 40 | +| `src/scheduler/daily_job.py` | Job quotidien à 02:00 (APScheduler), collecteurs en parallèle et indépendants | | |
| 41 | +| `src/scheduler/backfill.py` | Rattrapage automatique des dates manquées (7 derniers jours) | | |
| 42 | +| `src/api/` | FastAPI sur 127.0.0.1:8000, exposée uniquement via ngrok (www.api-ka.com) | | |
| 43 | +| `src/utils/backup.py` | Dump quotidien horodaté par service (`data/backups/YYYY-MM-DD/`, rétention 90 j) | | |
| 44 | +| `systemd/` | `apika-api`, `apika-scheduler`, `apika-ngrok` (Restart=always) | | |
| 45 | + | |
| 46 | +## Endpoints | |
| 47 | + | |
| 48 | +- `GET /` — statut de la plateforme + version | |
| 49 | +- `GET /health` — node (m3u96b), état DB, dernière collecte par service | |
| 50 | +- `GET /api/v1/{service}` — données paginées (`?page=&limit=`, limit max 500) | |
| 51 | +- `GET /api/v1/{service}/latest` — dernière collecte | |
| 52 | +- `GET /api/v1/{service}/date/{YYYY-MM-DD}` — données d'une date précise | |
| 53 | +- `GET /api/v1/{service}/stats` — enregistrements par jour, dernière réussite | |
| 54 | +- `GET /api/v1/runs` — historique des collectes (filtrable par service/statut) | |
| 55 | + | |
| 56 | +`{service}` ∈ `louka`, `immoka`, `foodka`, `autoka`, `fabrika`, `restoka`, `sortika`, `creaka`. | |
| 57 | + | |
| 58 | +## Tests | |
| 59 | + | |
| 60 | +```bash | |
| 61 | +pip install -r requirements.txt | |
| 62 | +pytest tests/ -v | |
| 63 | +``` | |
| 64 | + | |
| 65 | +--- | |
| 66 | + | |
| 67 | +*Plateforme légendaire API-KA — Node m3u96b — créée par Simon-Pierre Boucher (contact@spboucher.ai)* | |
added
alembic.ini
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : alembic.ini | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +[alembic] | |
| 11 | +script_location = src/database/migrations | |
| 12 | +prepend_sys_path = . | |
| 13 | +# L'URL réelle est injectée depuis le .env par migrations/env.py. | |
| 14 | +sqlalchemy.url = | |
| 15 | + | |
| 16 | +[loggers] | |
| 17 | +keys = root,sqlalchemy,alembic | |
| 18 | + | |
| 19 | +[handlers] | |
| 20 | +keys = console | |
| 21 | + | |
| 22 | +[formatters] | |
| 23 | +keys = generic | |
| 24 | + | |
| 25 | +[logger_root] | |
| 26 | +level = WARN | |
| 27 | +handlers = console | |
| 28 | +qualname = | |
| 29 | + | |
| 30 | +[logger_sqlalchemy] | |
| 31 | +level = WARN | |
| 32 | +handlers = | |
| 33 | +qualname = sqlalchemy.engine | |
| 34 | + | |
| 35 | +[logger_alembic] | |
| 36 | +level = INFO | |
| 37 | +handlers = | |
| 38 | +qualname = alembic | |
| 39 | + | |
| 40 | +[handler_console] | |
| 41 | +class = StreamHandler | |
| 42 | +args = (sys.stderr,) | |
| 43 | +level = NOTSET | |
| 44 | +formatter = generic | |
| 45 | + | |
| 46 | +[formatter_generic] | |
| 47 | +format = %(levelname)-5.5s [%(name)s] %(message)s | |
| 48 | +datefmt = %H:%M:%S | |
added
logs/.gitkeep
+0 −0
added
pyproject.toml
+25 −0
@@ -0,0 +1,25 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : pyproject.toml | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +[tool.black] | |
| 11 | +line-length = 88 | |
| 12 | +target-version = ["py311"] | |
| 13 | + | |
| 14 | +[tool.ruff] | |
| 15 | +line-length = 88 | |
| 16 | +target-version = "py311" | |
| 17 | + | |
| 18 | +[tool.ruff.lint] | |
| 19 | +# E/W = pycodestyle, F = pyflakes, I = isort, UP = pyupgrade, B = bugbear | |
| 20 | +select = ["E", "W", "F", "I", "UP", "B"] | |
| 21 | +# E501 géré par black ; B008 = Depends() dans les signatures FastAPI (idiome officiel) | |
| 22 | +ignore = ["E501", "B008"] | |
| 23 | + | |
| 24 | +[tool.pytest.ini_options] | |
| 25 | +testpaths = ["tests"] | |
added
requirements.txt
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : requirements.txt | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +fastapi>=0.115 | |
| 11 | +uvicorn[standard]>=0.30 | |
| 12 | +sqlalchemy>=2.0 | |
| 13 | +psycopg2-binary>=2.9 | |
| 14 | +alembic>=1.13 | |
| 15 | +apscheduler>=3.10 | |
| 16 | +httpx>=0.27 | |
| 17 | +python-dotenv>=1.0 | |
| 18 | + | |
| 19 | +# Développement / tests | |
| 20 | +pytest>=8.0 | |
| 21 | +black>=24.0 | |
| 22 | +ruff>=0.5 | |
added
scripts/deploy_m3u96b.sh
+66 −0
@@ -0,0 +1,66 @@ | ||
| 1 | +#!/usr/bin/env bash | |
| 2 | +# ============================================ | |
| 3 | +# Projet : API-KA | |
| 4 | +# Fichier : scripts/deploy_m3u96b.sh | |
| 5 | +# Node : m3u96b | |
| 6 | +# Author : Simon-Pierre Boucher | |
| 7 | +# Contact : contact@spboucher.ai | |
| 8 | +# Date : 2026-08-16 | |
| 9 | +# ============================================ | |
| 10 | +# Déploiement complet d'API-KA sur le node m3u96b (/opt/api-ka). | |
| 11 | +# À exécuter depuis le laptop, à la racine du projet. | |
| 12 | + | |
| 13 | +set -euo pipefail | |
| 14 | + | |
| 15 | +NODE="m3u96b" | |
| 16 | +REMOTE_DIR="/opt/api-ka" | |
| 17 | +PROJECT_DIR="$(cd "$(dirname "$0")/.." && pwd)" | |
| 18 | + | |
| 19 | +echo "==> [1/6] Déploiement API-KA vers ${NODE}:${REMOTE_DIR}" | |
| 20 | +ssh "$NODE" "sudo mkdir -p '$REMOTE_DIR' && sudo chown \"\$(whoami)\" '$REMOTE_DIR'" | |
| 21 | + | |
| 22 | +echo "==> [2/6] Synchronisation des fichiers (rsync)" | |
| 23 | +rsync -az --delete \ | |
| 24 | + --exclude '.env' \ | |
| 25 | + --exclude 'venv/' \ | |
| 26 | + --exclude '__pycache__/' \ | |
| 27 | + --exclude '.pytest_cache/' \ | |
| 28 | + --exclude '.ruff_cache/' \ | |
| 29 | + --exclude '.git/' \ | |
| 30 | + --exclude 'logs/' \ | |
| 31 | + --exclude 'data/' \ | |
| 32 | + "$PROJECT_DIR/" "$NODE:$REMOTE_DIR/" | |
| 33 | + | |
| 34 | +echo "==> [3/6] Environnement virtuel + dépendances" | |
| 35 | +ssh "$NODE" "cd '$REMOTE_DIR' \ | |
| 36 | + && mkdir -p logs data/backups \ | |
| 37 | + && (python3.11 -m venv venv 2>/dev/null || python3 -m venv venv) \ | |
| 38 | + && ./venv/bin/pip install --quiet --upgrade pip \ | |
| 39 | + && ./venv/bin/pip install --quiet -r requirements.txt" | |
| 40 | + | |
| 41 | +echo "==> [4/6] Vérification du .env" | |
| 42 | +if ! ssh "$NODE" "test -f '$REMOTE_DIR/.env'"; then | |
| 43 | + echo "⚠️ $REMOTE_DIR/.env manquant sur $NODE — copier .env.example et le remplir avant de démarrer les services." | |
| 44 | +fi | |
| 45 | + | |
| 46 | +echo "==> [5/6] Installation des services (systemd si disponible)" | |
| 47 | +ssh "$NODE" "if command -v systemctl >/dev/null 2>&1; then \ | |
| 48 | + sudo cp '$REMOTE_DIR'/systemd/apika-*.service /etc/systemd/system/ \ | |
| 49 | + && sudo systemctl daemon-reload \ | |
| 50 | + && sudo systemctl enable --now apika-api apika-scheduler apika-ngrok \ | |
| 51 | + && sudo systemctl restart apika-api apika-scheduler apika-ngrok; \ | |
| 52 | +else \ | |
| 53 | + echo '⚠️ systemd indisponible sur ce node (macOS) — démarrer les services manuellement :'; \ | |
| 54 | + echo ' cd $REMOTE_DIR && ./venv/bin/uvicorn src.api.main:app --host 127.0.0.1 --port 8000 &'; \ | |
| 55 | + echo ' cd $REMOTE_DIR && ./venv/bin/python -m src.scheduler.daily_job &'; \ | |
| 56 | + echo ' bash $REMOTE_DIR/scripts/start_ngrok.sh &'; \ | |
| 57 | +fi" | |
| 58 | + | |
| 59 | +echo "==> [6/6] Healthcheck" | |
| 60 | +sleep 3 | |
| 61 | +if ssh "$NODE" "curl -fsS --max-time 10 http://127.0.0.1:8000/health" >/dev/null 2>&1; then | |
| 62 | + echo "✅ API-KA déployée et en bonne santé sur $NODE." | |
| 63 | +else | |
| 64 | + echo "⚠️ /health ne répond pas encore sur $NODE — vérifier les services (systemctl status apika-*)." | |
| 65 | + exit 1 | |
| 66 | +fi | |
added
scripts/healthcheck.sh
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +#!/usr/bin/env bash | |
| 2 | +# ============================================ | |
| 3 | +# Projet : API-KA | |
| 4 | +# Fichier : scripts/healthcheck.sh | |
| 5 | +# Node : m3u96b | |
| 6 | +# Author : Simon-Pierre Boucher | |
| 7 | +# Contact : contact@spboucher.ai | |
| 8 | +# Date : 2026-08-16 | |
| 9 | +# ============================================ | |
| 10 | +# Vérification de santé utilisée par systemd / monitoring. | |
| 11 | +# Exit 0 si l'API répond et que la base est OK, exit 1 sinon. | |
| 12 | + | |
| 13 | +set -uo pipefail | |
| 14 | + | |
| 15 | +URL="http://127.0.0.1:${API_PORT:-8000}/health" | |
| 16 | + | |
| 17 | +RESPONSE="$(curl -fsS --max-time 10 "$URL" 2>/dev/null)" || { | |
| 18 | + echo "❌ API-KA injoignable sur $URL" | |
| 19 | + exit 1 | |
| 20 | +} | |
| 21 | + | |
| 22 | +if echo "$RESPONSE" | grep -q '"database": *"ok"'; then | |
| 23 | + echo "✅ API-KA en bonne santé ($URL)" | |
| 24 | + exit 0 | |
| 25 | +fi | |
| 26 | + | |
| 27 | +echo "❌ API-KA dégradée : $RESPONSE" | |
| 28 | +exit 1 | |
added
scripts/run_daily_backup.sh
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +#!/usr/bin/env bash | |
| 2 | +# ============================================ | |
| 3 | +# Projet : API-KA | |
| 4 | +# Fichier : scripts/run_daily_backup.sh | |
| 5 | +# Node : m3u96b | |
| 6 | +# Author : Simon-Pierre Boucher | |
| 7 | +# Contact : contact@spboucher.ai | |
| 8 | +# Date : 2026-08-16 | |
| 9 | +# ============================================ | |
| 10 | +# Backup quotidien horodaté des 5 services + purge des backups expirés (rétention 90 j). | |
| 11 | + | |
| 12 | +set -euo pipefail | |
| 13 | + | |
| 14 | +PROJECT_DIR="$(cd "$(dirname "$0")/.." && pwd)" | |
| 15 | +cd "$PROJECT_DIR" | |
| 16 | + | |
| 17 | +PYTHON="./venv/bin/python" | |
| 18 | +[ -x "$PYTHON" ] || PYTHON="python3" | |
| 19 | + | |
| 20 | +"$PYTHON" -m src.utils.backup --cleanup | |
| 21 | +echo "✅ Backup quotidien terminé ($(date '+%Y-%m-%d %H:%M:%S'))." | |
added
scripts/start_ngrok.sh
+34 −0
@@ -0,0 +1,34 @@ | ||
| 1 | +#!/usr/bin/env bash | |
| 2 | +# ============================================ | |
| 3 | +# Projet : API-KA | |
| 4 | +# Fichier : scripts/start_ngrok.sh | |
| 5 | +# Node : m3u96b | |
| 6 | +# Author : Simon-Pierre Boucher | |
| 7 | +# Contact : contact@spboucher.ai | |
| 8 | +# Date : 2026-08-16 | |
| 9 | +# ============================================ | |
| 10 | +# Tunnel public www.api-ka.com → 127.0.0.1:8000 avec redémarrage automatique. | |
| 11 | +# Le domaine doit être réservé dans le dashboard ngrok et NGROK_AUTHTOKEN configuré. | |
| 12 | + | |
| 13 | +set -uo pipefail | |
| 14 | + | |
| 15 | +PROJECT_DIR="$(cd "$(dirname "$0")/.." && pwd)" | |
| 16 | +cd "$PROJECT_DIR" | |
| 17 | + | |
| 18 | +# shellcheck disable=SC1091 | |
| 19 | +[ -f .env ] && set -a && source .env && set +a | |
| 20 | + | |
| 21 | +DOMAIN="${NGROK_DOMAIN:-www.api-ka.com}" | |
| 22 | +PORT="${API_PORT:-8000}" | |
| 23 | +mkdir -p logs | |
| 24 | + | |
| 25 | +if [ -n "${NGROK_AUTHTOKEN:-}" ]; then | |
| 26 | + ngrok config add-authtoken "$NGROK_AUTHTOKEN" >/dev/null 2>&1 || true | |
| 27 | +fi | |
| 28 | + | |
| 29 | +while true; do | |
| 30 | + echo "$(date '+%Y-%m-%dT%H:%M:%S%z') [ngrok] démarrage du tunnel ${DOMAIN} → 127.0.0.1:${PORT}" >> logs/ngrok.log | |
| 31 | + ngrok http --domain="$DOMAIN" "$PORT" --log stdout >> logs/ngrok.log 2>&1 | |
| 32 | + echo "$(date '+%Y-%m-%dT%H:%M:%S%z') [ngrok] tunnel interrompu — redémarrage dans 5 s" >> logs/ngrok.log | |
| 33 | + sleep 5 | |
| 34 | +done | |
added
src/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Package racine d'API-KA — plateforme centrale de l'écosystème KA (node m3u96b).""" | |
added
src/api/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""API publique FastAPI d'API-KA, exposée sur www.api-ka.com via ngrok.""" | |
added
src/api/main.py
+87 −0
@@ -0,0 +1,87 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/main.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Application FastAPI d'API-KA — servie par uvicorn sur 127.0.0.1:8000, | |
| 10 | +exposée publiquement uniquement via le tunnel ngrok www.api-ka.com.""" | |
| 11 | + | |
| 12 | +from __future__ import annotations | |
| 13 | + | |
| 14 | +from collections.abc import AsyncIterator | |
| 15 | +from contextlib import asynccontextmanager | |
| 16 | +from pathlib import Path | |
| 17 | +from typing import Any | |
| 18 | + | |
| 19 | +from fastapi import FastAPI, Request | |
| 20 | +from fastapi.responses import FileResponse | |
| 21 | + | |
| 22 | +from src.api.middleware.logging import RequestLoggingMiddleware | |
| 23 | +from src.api.middleware.ratelimit import RateLimitMiddleware | |
| 24 | +from src.api.routes import envelope, health, runs, services | |
| 25 | +from src.config import SERVICES, get_settings, verify_node | |
| 26 | +from src.database.db import init_db | |
| 27 | +from src.utils.logger import get_logger | |
| 28 | + | |
| 29 | +VERSION = "1.0.0" | |
| 30 | + | |
| 31 | + | |
| 32 | +@asynccontextmanager | |
| 33 | +async def lifespan(app: FastAPI) -> AsyncIterator[None]: | |
| 34 | + """Au démarrage : vérifie le node m3u96b et initialise le schéma DB.""" | |
| 35 | + verify_node() | |
| 36 | + init_db() | |
| 37 | + get_logger("apika.api").info( | |
| 38 | + "API-KA démarrée", | |
| 39 | + extra={"version": VERSION, "node": get_settings().node_name}, | |
| 40 | + ) | |
| 41 | + yield | |
| 42 | + | |
| 43 | + | |
| 44 | +app = FastAPI( | |
| 45 | + title="API-KA", | |
| 46 | + description=( | |
| 47 | + "Plateforme centrale de l'écosystème KA — données quotidiennes des services " | |
| 48 | + "lou-ka, immo-ka, food-ka, auto-ka et fabri-ka. Node : m3u96b." | |
| 49 | + ), | |
| 50 | + version=VERSION, | |
| 51 | + lifespan=lifespan, | |
| 52 | +) | |
| 53 | + | |
| 54 | +# Le rate limiting s'applique en premier sur la requête entrante ; la | |
| 55 | +# journalisation englobe tout (elle logge aussi les réponses 429). | |
| 56 | +app.add_middleware(RateLimitMiddleware) | |
| 57 | +app.add_middleware(RequestLoggingMiddleware) | |
| 58 | + | |
| 59 | +# /api/v1/runs doit être enregistré avant les routes génériques /api/v1/{service}. | |
| 60 | +app.include_router(health.router) | |
| 61 | +app.include_router(runs.router) | |
| 62 | +app.include_router(services.router) | |
| 63 | + | |
| 64 | + | |
| 65 | +WEB_INDEX = Path(__file__).resolve().parent / "web" / "index.html" | |
| 66 | + | |
| 67 | + | |
| 68 | +@app.get("/", tags=["root"]) | |
| 69 | +def root(request: Request) -> Any: | |
| 70 | + """Statut de la plateforme + version. | |
| 71 | + | |
| 72 | + Négociation de contenu : les navigateurs (Accept: text/html) reçoivent la | |
| 73 | + plateforme web de documentation avec playground ; les clients API reçoivent | |
| 74 | + le statut JSON uniforme. | |
| 75 | + """ | |
| 76 | + accept = request.headers.get("accept", "") | |
| 77 | + if "text/html" in accept and WEB_INDEX.is_file(): | |
| 78 | + return FileResponse(WEB_INDEX, media_type="text/html") | |
| 79 | + return envelope( | |
| 80 | + { | |
| 81 | + "platform": "API-KA", | |
| 82 | + "status": "ok", | |
| 83 | + "version": VERSION, | |
| 84 | + "services": list(SERVICES), | |
| 85 | + "docs": "/docs", | |
| 86 | + } | |
| 87 | + ) | |
added
src/api/middleware/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/middleware/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Middlewares de l'API publique : journalisation des requêtes et rate limiting.""" | |
added
src/api/middleware/logging.py
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/middleware/logging.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Middleware de journalisation : chaque requête HTTP est loggée en JSON.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import time | |
| 14 | + | |
| 15 | +from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint | |
| 16 | +from starlette.requests import Request | |
| 17 | +from starlette.responses import Response | |
| 18 | + | |
| 19 | +from src.utils.logger import get_logger | |
| 20 | + | |
| 21 | + | |
| 22 | +class RequestLoggingMiddleware(BaseHTTPMiddleware): | |
| 23 | + """Logge méthode, chemin, statut, durée et IP client de chaque requête.""" | |
| 24 | + | |
| 25 | + async def dispatch( | |
| 26 | + self, request: Request, call_next: RequestResponseEndpoint | |
| 27 | + ) -> Response: | |
| 28 | + start = time.perf_counter() | |
| 29 | + response = await call_next(request) | |
| 30 | + duration_ms = round((time.perf_counter() - start) * 1000, 2) | |
| 31 | + get_logger("apika.api").info( | |
| 32 | + "http_request", | |
| 33 | + extra={ | |
| 34 | + "method": request.method, | |
| 35 | + "path": request.url.path, | |
| 36 | + "status_code": response.status_code, | |
| 37 | + "duration_ms": duration_ms, | |
| 38 | + "client_ip": request.client.host if request.client else "unknown", | |
| 39 | + }, | |
| 40 | + ) | |
| 41 | + return response | |
added
src/api/middleware/ratelimit.py
+55 −0
@@ -0,0 +1,55 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/middleware/ratelimit.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Rate limiting basique par IP (défaut : 120 req/min) — l'API est publique via ngrok.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import time | |
| 14 | +from collections import defaultdict, deque | |
| 15 | + | |
| 16 | +from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint | |
| 17 | +from starlette.requests import Request | |
| 18 | +from starlette.responses import JSONResponse, Response | |
| 19 | + | |
| 20 | +from src.config import get_settings | |
| 21 | + | |
| 22 | +WINDOW_SECONDS = 60.0 | |
| 23 | +EXEMPT_PATHS = ("/health",) | |
| 24 | + | |
| 25 | + | |
| 26 | +class RateLimitMiddleware(BaseHTTPMiddleware): | |
| 27 | + """Fenêtre glissante de 60 s par IP ; au-delà de la limite → HTTP 429.""" | |
| 28 | + | |
| 29 | + def __init__(self, app, limit_per_minute: int | None = None) -> None: # type: ignore[no-untyped-def] | |
| 30 | + super().__init__(app) | |
| 31 | + self.limit = limit_per_minute or get_settings().rate_limit_per_minute | |
| 32 | + self._hits: dict[str, deque[float]] = defaultdict(deque) | |
| 33 | + | |
| 34 | + async def dispatch( | |
| 35 | + self, request: Request, call_next: RequestResponseEndpoint | |
| 36 | + ) -> Response: | |
| 37 | + if request.url.path in EXEMPT_PATHS: | |
| 38 | + return await call_next(request) | |
| 39 | + | |
| 40 | + client_ip = request.client.host if request.client else "unknown" | |
| 41 | + now = time.monotonic() | |
| 42 | + hits = self._hits[client_ip] | |
| 43 | + while hits and now - hits[0] > WINDOW_SECONDS: | |
| 44 | + hits.popleft() | |
| 45 | + if len(hits) >= self.limit: | |
| 46 | + return JSONResponse( | |
| 47 | + status_code=429, | |
| 48 | + content={ | |
| 49 | + "success": False, | |
| 50 | + "error": f"Limite dépassée : {self.limit} requêtes/minute par IP", | |
| 51 | + "meta": {"node": get_settings().node_name}, | |
| 52 | + }, | |
| 53 | + ) | |
| 54 | + hits.append(now) | |
| 55 | + return await call_next(request) | |
added
src/api/routes/__init__.py
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/routes/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Routes de l'API publique + enveloppe de réponse uniforme.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from typing import Any | |
| 14 | + | |
| 15 | +from src.config import get_settings | |
| 16 | + | |
| 17 | + | |
| 18 | +def envelope( | |
| 19 | + data: Any, | |
| 20 | + *, | |
| 21 | + page: int | None = None, | |
| 22 | + limit: int | None = None, | |
| 23 | + total: int | None = None, | |
| 24 | + extra_meta: dict[str, Any] | None = None, | |
| 25 | +) -> dict[str, Any]: | |
| 26 | + """Construit la réponse uniforme ``{success, data, meta}`` de l'API. | |
| 27 | + | |
| 28 | + Le ``meta`` contient toujours le node (m3u96b) et, si fournis, la | |
| 29 | + pagination (``page``, ``limit``, ``total``). | |
| 30 | + """ | |
| 31 | + meta: dict[str, Any] = {} | |
| 32 | + if page is not None: | |
| 33 | + meta["page"] = page | |
| 34 | + if limit is not None: | |
| 35 | + meta["limit"] = limit | |
| 36 | + if total is not None: | |
| 37 | + meta["total"] = total | |
| 38 | + meta["node"] = get_settings().node_name | |
| 39 | + if extra_meta: | |
| 40 | + meta.update(extra_meta) | |
| 41 | + return {"success": True, "data": data, "meta": meta} | |
added
src/api/routes/health.py
+68 −0
@@ -0,0 +1,68 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/routes/health.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Route /health : node (m3u96b), état de la base, dernière collecte par service.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import socket | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +from fastapi import APIRouter, Depends | |
| 17 | +from sqlalchemy import select | |
| 18 | +from sqlalchemy.orm import Session | |
| 19 | + | |
| 20 | +from src.api.routes import envelope | |
| 21 | +from src.config import REQUIRED_NODE, SERVICES | |
| 22 | +from src.database.db import get_db, healthcheck | |
| 23 | +from src.database.models import CollectionRun | |
| 24 | + | |
| 25 | +router = APIRouter(tags=["health"]) | |
| 26 | + | |
| 27 | + | |
| 28 | +@router.get("/health") | |
| 29 | +def health(db: Session = Depends(get_db)) -> dict[str, Any]: | |
| 30 | + """État complet de la plateforme : node, base de données, dernières collectes.""" | |
| 31 | + hostname = socket.gethostname() | |
| 32 | + db_ok = healthcheck() | |
| 33 | + | |
| 34 | + last_collections: dict[str, Any] = {} | |
| 35 | + for service in SERVICES: | |
| 36 | + last_run = ( | |
| 37 | + db.execute( | |
| 38 | + select(CollectionRun) | |
| 39 | + .where( | |
| 40 | + CollectionRun.service == service, | |
| 41 | + CollectionRun.status.in_(("success", "retried")), | |
| 42 | + ) | |
| 43 | + .order_by(CollectionRun.finished_at.desc()) | |
| 44 | + .limit(1) | |
| 45 | + ) | |
| 46 | + .scalars() | |
| 47 | + .first() | |
| 48 | + ) | |
| 49 | + last_collections[service] = ( | |
| 50 | + { | |
| 51 | + "date_key": last_run.date_key.isoformat(), | |
| 52 | + "status": last_run.status, | |
| 53 | + "records_count": last_run.records_count, | |
| 54 | + "finished_at": last_run.finished_at.isoformat(), | |
| 55 | + } | |
| 56 | + if last_run | |
| 57 | + else None | |
| 58 | + ) | |
| 59 | + | |
| 60 | + data = { | |
| 61 | + "status": "ok" if db_ok else "degraded", | |
| 62 | + "node": hostname, | |
| 63 | + "required_node": REQUIRED_NODE, | |
| 64 | + "node_ok": hostname.split(".")[0].lower() == REQUIRED_NODE, | |
| 65 | + "database": "ok" if db_ok else "error", | |
| 66 | + "last_collections": last_collections, | |
| 67 | + } | |
| 68 | + return envelope(data) | |
added
src/api/routes/runs.py
+78 −0
@@ -0,0 +1,78 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/routes/runs.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Route /api/v1/runs : historique des collectes, filtrable par service et statut.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from typing import Any | |
| 14 | + | |
| 15 | +from fastapi import APIRouter, Depends, HTTPException, Query | |
| 16 | +from sqlalchemy import func, select | |
| 17 | +from sqlalchemy.orm import Session | |
| 18 | + | |
| 19 | +from src.api.routes import envelope | |
| 20 | +from src.config import SERVICES | |
| 21 | +from src.database.db import get_db | |
| 22 | +from src.database.models import CollectionRun | |
| 23 | + | |
| 24 | +router = APIRouter(prefix="/api/v1", tags=["runs"]) | |
| 25 | + | |
| 26 | +VALID_STATUSES = ("success", "failed", "retried") | |
| 27 | + | |
| 28 | + | |
| 29 | +@router.get("/runs") | |
| 30 | +def list_runs( | |
| 31 | + service: str | None = Query(None, description="Filtrer par service KA"), | |
| 32 | + status: str | None = Query(None, description="success / failed / retried"), | |
| 33 | + page: int = Query(1, ge=1), | |
| 34 | + limit: int = Query(100, ge=1, le=500), | |
| 35 | + db: Session = Depends(get_db), | |
| 36 | +) -> dict[str, Any]: | |
| 37 | + """Historique des runs de collecte, du plus récent au plus ancien.""" | |
| 38 | + if service is not None and service not in SERVICES: | |
| 39 | + raise HTTPException(status_code=404, detail=f"Service inconnu : {service}") | |
| 40 | + if status is not None and status not in VALID_STATUSES: | |
| 41 | + raise HTTPException( | |
| 42 | + status_code=422, | |
| 43 | + detail=f"Statut invalide : {status}. Valides : {', '.join(VALID_STATUSES)}", | |
| 44 | + ) | |
| 45 | + | |
| 46 | + base = select(CollectionRun) | |
| 47 | + if service: | |
| 48 | + base = base.where(CollectionRun.service == service) | |
| 49 | + if status: | |
| 50 | + base = base.where(CollectionRun.status == status) | |
| 51 | + | |
| 52 | + total = db.execute(select(func.count()).select_from(base.subquery())).scalar() or 0 | |
| 53 | + rows = ( | |
| 54 | + db.execute( | |
| 55 | + base.order_by(CollectionRun.started_at.desc()) | |
| 56 | + .offset((page - 1) * limit) | |
| 57 | + .limit(limit) | |
| 58 | + ) | |
| 59 | + .scalars() | |
| 60 | + .all() | |
| 61 | + ) | |
| 62 | + | |
| 63 | + data = [ | |
| 64 | + { | |
| 65 | + "id": run.id, | |
| 66 | + "service": run.service, | |
| 67 | + "date_key": run.date_key.isoformat(), | |
| 68 | + "status": run.status, | |
| 69 | + "records_count": run.records_count, | |
| 70 | + "duration_seconds": run.duration_seconds, | |
| 71 | + "error_message": run.error_message, | |
| 72 | + "node": run.node, | |
| 73 | + "started_at": run.started_at.isoformat(), | |
| 74 | + "finished_at": run.finished_at.isoformat(), | |
| 75 | + } | |
| 76 | + for run in rows | |
| 77 | + ] | |
| 78 | + return envelope(data, page=page, limit=limit, total=total) | |
added
src/api/routes/services.py
+194 −0
@@ -0,0 +1,194 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/api/routes/services.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Routes /api/v1/{service} : données paginées, latest, par date, stats.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import datetime | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +from fastapi import APIRouter, Depends, HTTPException, Path, Query | |
| 17 | +from sqlalchemy import func, select | |
| 18 | +from sqlalchemy.orm import Session | |
| 19 | + | |
| 20 | +from src.api.routes import envelope | |
| 21 | +from src.config import SERVICES | |
| 22 | +from src.database.db import get_db | |
| 23 | +from src.database.models import DATA_MODELS, CollectionRun | |
| 24 | + | |
| 25 | +router = APIRouter(prefix="/api/v1", tags=["services"]) | |
| 26 | + | |
| 27 | +MAX_LIMIT = 500 | |
| 28 | + | |
| 29 | + | |
| 30 | +def _model_or_404(service: str) -> type: | |
| 31 | + """Retourne le modèle du service ou lève un 404 si le service est inconnu.""" | |
| 32 | + if service not in SERVICES: | |
| 33 | + raise HTTPException( | |
| 34 | + status_code=404, | |
| 35 | + detail=f"Service inconnu : {service}. Services valides : {', '.join(SERVICES)}", | |
| 36 | + ) | |
| 37 | + return DATA_MODELS[service] | |
| 38 | + | |
| 39 | + | |
| 40 | +def _serialize(row: Any) -> dict[str, Any]: | |
| 41 | + """Sérialise un enregistrement de données en JSON.""" | |
| 42 | + return { | |
| 43 | + "id": row.id, | |
| 44 | + "source": row.source, | |
| 45 | + "date_key": row.date_key.isoformat(), | |
| 46 | + "collected_at": row.collected_at.isoformat() if row.collected_at else None, | |
| 47 | + "checksum": row.checksum, | |
| 48 | + "payload": row.payload, | |
| 49 | + } | |
| 50 | + | |
| 51 | + | |
| 52 | +@router.get("/{service}/latest") | |
| 53 | +def get_latest( | |
| 54 | + service: str = Path(..., description="Service KA"), | |
| 55 | + db: Session = Depends(get_db), | |
| 56 | +) -> dict[str, Any]: | |
| 57 | + """Dernière collecte du service (toutes les lignes de la date la plus récente).""" | |
| 58 | + model = _model_or_404(service) | |
| 59 | + max_date = db.execute( | |
| 60 | + select(func.max(model.date_key)).where(model.source == service) | |
| 61 | + ).scalar() | |
| 62 | + if max_date is None: | |
| 63 | + return envelope([], total=0, extra_meta={"service": service, "date_key": None}) | |
| 64 | + rows = ( | |
| 65 | + db.execute( | |
| 66 | + select(model) | |
| 67 | + .where(model.source == service, model.date_key == max_date) | |
| 68 | + .order_by(model.collected_at.desc()) | |
| 69 | + ) | |
| 70 | + .scalars() | |
| 71 | + .all() | |
| 72 | + ) | |
| 73 | + return envelope( | |
| 74 | + [_serialize(r) for r in rows], | |
| 75 | + total=len(rows), | |
| 76 | + extra_meta={"service": service, "date_key": max_date.isoformat()}, | |
| 77 | + ) | |
| 78 | + | |
| 79 | + | |
| 80 | +@router.get("/{service}/date/{date_str}") | |
| 81 | +def get_by_date( | |
| 82 | + service: str = Path(..., description="Service KA"), | |
| 83 | + date_str: str = Path(..., description="Date au format YYYY-MM-DD"), | |
| 84 | + page: int = Query(1, ge=1), | |
| 85 | + limit: int = Query(100, ge=1, le=MAX_LIMIT), | |
| 86 | + db: Session = Depends(get_db), | |
| 87 | +) -> dict[str, Any]: | |
| 88 | + """Données d'un service pour une date précise, paginées.""" | |
| 89 | + model = _model_or_404(service) | |
| 90 | + try: | |
| 91 | + date_key = datetime.date.fromisoformat(date_str) | |
| 92 | + except ValueError as exc: | |
| 93 | + raise HTTPException( | |
| 94 | + status_code=422, detail=f"Date invalide : {date_str} (format YYYY-MM-DD)" | |
| 95 | + ) from exc | |
| 96 | + base = select(model).where(model.source == service, model.date_key == date_key) | |
| 97 | + total = db.execute(select(func.count()).select_from(base.subquery())).scalar() or 0 | |
| 98 | + rows = ( | |
| 99 | + db.execute( | |
| 100 | + base.order_by(model.collected_at.desc()) | |
| 101 | + .offset((page - 1) * limit) | |
| 102 | + .limit(limit) | |
| 103 | + ) | |
| 104 | + .scalars() | |
| 105 | + .all() | |
| 106 | + ) | |
| 107 | + return envelope( | |
| 108 | + [_serialize(r) for r in rows], | |
| 109 | + page=page, | |
| 110 | + limit=limit, | |
| 111 | + total=total, | |
| 112 | + extra_meta={"service": service, "date_key": date_key.isoformat()}, | |
| 113 | + ) | |
| 114 | + | |
| 115 | + | |
| 116 | +@router.get("/{service}/stats") | |
| 117 | +def get_stats( | |
| 118 | + service: str = Path(..., description="Service KA"), | |
| 119 | + db: Session = Depends(get_db), | |
| 120 | +) -> dict[str, Any]: | |
| 121 | + """Nombre d'enregistrements par jour et dernière collecte réussie.""" | |
| 122 | + model = _model_or_404(service) | |
| 123 | + counts = db.execute( | |
| 124 | + select(model.date_key, func.count()) | |
| 125 | + .where(model.source == service) | |
| 126 | + .group_by(model.date_key) | |
| 127 | + .order_by(model.date_key.desc()) | |
| 128 | + ).all() | |
| 129 | + last_success = ( | |
| 130 | + db.execute( | |
| 131 | + select(CollectionRun) | |
| 132 | + .where( | |
| 133 | + CollectionRun.service == service, | |
| 134 | + CollectionRun.status.in_(("success", "retried")), | |
| 135 | + ) | |
| 136 | + .order_by(CollectionRun.finished_at.desc()) | |
| 137 | + .limit(1) | |
| 138 | + ) | |
| 139 | + .scalars() | |
| 140 | + .first() | |
| 141 | + ) | |
| 142 | + data = { | |
| 143 | + "service": service, | |
| 144 | + "total_records": sum(count for _, count in counts), | |
| 145 | + "days": [ | |
| 146 | + {"date_key": day.isoformat(), "records": count} for day, count in counts | |
| 147 | + ], | |
| 148 | + "last_success": ( | |
| 149 | + { | |
| 150 | + "date_key": last_success.date_key.isoformat(), | |
| 151 | + "status": last_success.status, | |
| 152 | + "records_count": last_success.records_count, | |
| 153 | + "finished_at": last_success.finished_at.isoformat(), | |
| 154 | + } | |
| 155 | + if last_success | |
| 156 | + else None | |
| 157 | + ), | |
| 158 | + } | |
| 159 | + return envelope(data, extra_meta={"service": service}) | |
| 160 | + | |
| 161 | + | |
| 162 | +@router.get("/{service}") | |
| 163 | +def list_service_data( | |
| 164 | + service: str = Path(..., description="Service KA"), | |
| 165 | + page: int = Query(1, ge=1), | |
| 166 | + limit: int = Query(100, ge=1, le=MAX_LIMIT), | |
| 167 | + db: Session = Depends(get_db), | |
| 168 | +) -> dict[str, Any]: | |
| 169 | + """Données paginées d'un service, des plus récentes aux plus anciennes.""" | |
| 170 | + model = _model_or_404(service) | |
| 171 | + total = ( | |
| 172 | + db.execute( | |
| 173 | + select(func.count()).select_from(model).where(model.source == service) | |
| 174 | + ).scalar() | |
| 175 | + or 0 | |
| 176 | + ) | |
| 177 | + rows = ( | |
| 178 | + db.execute( | |
| 179 | + select(model) | |
| 180 | + .where(model.source == service) | |
| 181 | + .order_by(model.date_key.desc(), model.collected_at.desc()) | |
| 182 | + .offset((page - 1) * limit) | |
| 183 | + .limit(limit) | |
| 184 | + ) | |
| 185 | + .scalars() | |
| 186 | + .all() | |
| 187 | + ) | |
| 188 | + return envelope( | |
| 189 | + [_serialize(r) for r in rows], | |
| 190 | + page=page, | |
| 191 | + limit=limit, | |
| 192 | + total=total, | |
| 193 | + extra_meta={"service": service}, | |
| 194 | + ) | |
added
src/api/web/index.html
+563 −0
@@ -0,0 +1,563 @@ | ||
| 1 | +<!DOCTYPE html> | |
| 2 | +<!-- | |
| 3 | +============================================ | |
| 4 | +Projet : API-KA | |
| 5 | +Fichier : src/api/web/index.html | |
| 6 | +Node : m3u96b | |
| 7 | +Author : Simon-Pierre Boucher | |
| 8 | +Contact : contact@spboucher.ai | |
| 9 | +Date : 2026-08-16 | |
| 10 | +============================================ | |
| 11 | +--> | |
| 12 | +<html lang="fr"> | |
| 13 | +<head> | |
| 14 | +<meta charset="utf-8"> | |
| 15 | +<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"> | |
| 16 | +<title>API-KA — La plateforme de données de l'écosystème KA</title> | |
| 17 | +<meta name="description" content="API publique centralisant chaque jour les données des services lou-ka, immo-ka, food-ka, auto-ka et fabri-ka. Documentation complète et playground interactif."> | |
| 18 | +<meta name="theme-color" content="#ffffff"> | |
| 19 | +<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 64 64'%3E%3Crect width='64' height='64' rx='14' fill='%230B1330'/%3E%3Cellipse cx='32' cy='18' rx='17' ry='7' fill='none' stroke='%23FF6A00' stroke-width='5'/%3E%3Cpath d='M15 18v28c0 3.9 7.6 7 17 7s17-3.1 17-7V18' fill='none' stroke='%23FF6A00' stroke-width='5'/%3E%3Cpath d='M15 32c0 3.9 7.6 7 17 7s17-3.1 17-7' fill='none' stroke='%23ffffff' stroke-width='4'/%3E%3C/svg%3E"> | |
| 20 | +<link rel="preconnect" href="https://fonts.googleapis.com"> | |
| 21 | +<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> | |
| 22 | +<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet"> | |
| 23 | +<style> | |
| 24 | +:root{ | |
| 25 | + --navy:#0b1330; --blue:#2258ff; --orange:#ff6a00; --orange-deep:#e55f00; | |
| 26 | + --orange-soft:#fff1e6; --blue-soft:#e9efff; --navy-soft:#eef0f6; --grey:#f2f2f2; | |
| 27 | + --white:#ffffff; --surface:var(--white); --surface-2:#f7f7f9; | |
| 28 | + --ink:var(--navy); --ink-2:#3e4560; --ink-3:#8a90a5; | |
| 29 | + --line:rgba(11,19,48,.1); --line-strong:rgba(11,19,48,.16); --danger:#d64545; | |
| 30 | + --r-card:18px; --r-ctl:12px; --r-pill:999px; | |
| 31 | + --shadow-flat:0 1px 3px rgba(11,19,48,.06); | |
| 32 | + --shadow-off:0 16px 40px rgba(11,19,48,.14); | |
| 33 | + --shadow-off-soft:0 8px 24px rgba(11,19,48,.08); | |
| 34 | + --font:"Inter","SF Pro Display",system-ui,sans-serif; | |
| 35 | + --mono:"JetBrains Mono",ui-monospace,monospace; | |
| 36 | +} | |
| 37 | +*{box-sizing:border-box;margin:0;padding:0} | |
| 38 | +html{scroll-behavior:smooth;scroll-padding-top:86px} | |
| 39 | +body{background:var(--white);color:var(--ink);font-family:var(--font);font-size:15.5px;line-height:1.62;-webkit-font-smoothing:antialiased} | |
| 40 | +a{color:inherit} | |
| 41 | +.wrap{max-width:1120px;margin:0 auto;padding:0 24px} | |
| 42 | +/* ================= topbar ================= */ | |
| 43 | +.topbar{position:sticky;top:0;z-index:60;background:rgba(255,255,255,.92);backdrop-filter:blur(10px);-webkit-backdrop-filter:blur(10px);border-bottom:1px solid var(--line)} | |
| 44 | +.topbar .wrap{display:flex;align-items:center;gap:26px;height:68px} | |
| 45 | +.brand{font-weight:800;font-size:24px;letter-spacing:-.03em;color:var(--navy);text-decoration:none;display:flex;align-items:center;gap:10px;line-height:1} | |
| 46 | +.brand svg{width:30px;height:30px;flex:none} | |
| 47 | +.brand .hl{color:var(--orange)} | |
| 48 | +.brand>span:first-of-type{white-space:nowrap} | |
| 49 | +.brand-tag{font-size:11px;font-weight:500;color:var(--ink-3);letter-spacing:.02em;margin-top:3px} | |
| 50 | +.topnav{display:flex;gap:4px;list-style:none;margin-left:auto} | |
| 51 | +.topnav a{text-decoration:none;font-size:13.5px;font-weight:600;color:var(--ink-2);padding:8px 14px;border-radius:var(--r-pill);transition:all .13s ease} | |
| 52 | +.topnav a:hover{background:var(--blue-soft);color:var(--blue)} | |
| 53 | +.live-chip{display:inline-flex;align-items:center;gap:8px;border:1px solid var(--line);border-radius:var(--r-pill);padding:8px 16px;background:var(--surface);font-size:12.5px;font-weight:600;color:var(--ink-2);box-shadow:var(--shadow-flat);white-space:nowrap} | |
| 54 | +.pulse{width:8px;height:8px;border-radius:50%;background:var(--ink-3)} | |
| 55 | +.pulse.ok{background:var(--orange);box-shadow:0 0 0 4px var(--orange-soft);animation:pulse 2.4s ease infinite} | |
| 56 | +.pulse.err{background:var(--danger);box-shadow:0 0 0 4px #fde8e8} | |
| 57 | +@keyframes pulse{0%,100%{box-shadow:0 0 0 4px var(--orange-soft)}50%{box-shadow:0 0 0 7px #ff6a002e}} | |
| 58 | +@media(max-width:900px){.topnav{display:none}.brand-tag{display:none}} | |
| 59 | +/* ================= hero ================= */ | |
| 60 | +.hero{padding:64px 0 56px;background: | |
| 61 | + radial-gradient(900px 420px at 85% -10%, var(--blue-soft) 0%, transparent 60%), | |
| 62 | + radial-gradient(700px 380px at -5% 15%, var(--orange-soft) 0%, transparent 55%), var(--white)} | |
| 63 | +.hero-grid{display:grid;grid-template-columns:1.15fr .85fr;gap:48px;align-items:center} | |
| 64 | +@media(max-width:960px){.hero-grid{grid-template-columns:1fr}} | |
| 65 | +.kicker{display:inline-flex;align-items:center;gap:8px;border:1px solid rgba(255,106,0,.35);background:var(--orange-soft);color:var(--orange-deep);border-radius:var(--r-pill);padding:6px 16px;font-size:12px;font-weight:700;letter-spacing:.06em;text-transform:uppercase} | |
| 66 | +h1{font-size:clamp(38px,5.4vw,60px);font-weight:800;line-height:1.05;letter-spacing:-.03em;color:var(--navy);margin:18px 0 0;max-width:640px} | |
| 67 | +h1 .hl{color:var(--orange)} | |
| 68 | +.lede{margin-top:20px;max-width:56ch;color:var(--ink-2);font-size:16.5px} | |
| 69 | +.hero-cta{display:flex;flex-wrap:wrap;gap:12px;margin-top:28px} | |
| 70 | +.btn{display:inline-flex;align-items:center;justify-content:center;gap:8px;border:1px solid transparent;border-radius:var(--r-pill);padding:12px 26px;font-family:var(--font);font-weight:700;font-size:14.5px;cursor:pointer;text-decoration:none;min-height:46px;transition:transform .12s ease,box-shadow .12s ease,background .15s ease} | |
| 71 | +.btn-primary{background:var(--orange);color:var(--white);box-shadow:0 6px 18px #ff6a0052} | |
| 72 | +.btn-primary:hover{background:var(--orange-deep);transform:translateY(-1px);box-shadow:0 10px 24px #ff6a0060} | |
| 73 | +.btn-ghost{background:transparent;color:var(--navy);border-color:var(--line-strong)} | |
| 74 | +.btn-ghost:hover{background:var(--navy-soft);transform:translateY(-1px)} | |
| 75 | +.stat-chips{display:flex;flex-wrap:wrap;gap:10px;margin-top:32px} | |
| 76 | +.stat-chip{background:var(--surface);border:1px solid var(--line);border-radius:var(--r-pill);padding:9px 18px;font-size:12.5px;font-weight:500;color:var(--ink-2);display:inline-flex;gap:8px;align-items:center;box-shadow:var(--shadow-flat);white-space:nowrap;transition:background .13s ease} | |
| 77 | +.stat-chip:hover{background:var(--blue-soft)} | |
| 78 | +.stat-chip b{color:var(--navy);font-weight:700;font-variant-numeric:tabular-nums} | |
| 79 | +/* terminal navy */ | |
| 80 | +.hero-tile{background:var(--navy);border-radius:var(--r-card);box-shadow:var(--shadow-off);overflow:hidden;transform:rotate(.6deg)} | |
| 81 | +.tile-bar{display:flex;align-items:center;gap:8px;padding:13px 18px;background:#101a3d;border-bottom:1px solid rgba(255,255,255,.08)} | |
| 82 | +.tile-bar i{width:11px;height:11px;border-radius:50%;background:#26325c;display:inline-block} | |
| 83 | +.tile-bar i:first-child{background:var(--orange)} | |
| 84 | +.tile-bar span{font-family:var(--mono);font-size:11.5px;color:#8a90a5;margin-left:8px} | |
| 85 | +.tile-body{padding:20px 22px;font-family:var(--mono);font-size:12.3px;line-height:1.75;color:#e9efff;white-space:pre-wrap;word-break:break-word;min-height:280px} | |
| 86 | +.tile-body .tk{color:#7ea0ff}.tile-body .ts{color:#ffb277}.tile-body .tn{color:#7dd3a7}.tile-body .tc{color:#8a90a5} | |
| 87 | +/* ================= sections ================= */ | |
| 88 | +section{padding:76px 0} | |
| 89 | +section.alt{background:var(--surface-2);border-top:1px solid var(--line);border-bottom:1px solid var(--line)} | |
| 90 | +.sec-kicker{color:var(--orange);font-size:12px;font-weight:800;letter-spacing:.1em;text-transform:uppercase} | |
| 91 | +h2{font-size:clamp(28px,3.6vw,40px);font-weight:800;letter-spacing:-.03em;line-height:1.1;color:var(--navy);margin:8px 0 12px} | |
| 92 | +h2 .hl{color:var(--orange)} | |
| 93 | +.sec-sub{color:var(--ink-2);max-width:66ch;margin-bottom:36px;font-size:15.5px} | |
| 94 | +.sec-sub code, .conv code{font-family:var(--mono);font-size:.82em;background:var(--navy-soft);border-radius:6px;padding:2px 7px;color:var(--navy)} | |
| 95 | +/* ================= services ================= */ | |
| 96 | +.svc-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(200px,1fr));gap:16px} | |
| 97 | +.svc{background:var(--surface);border:1px solid var(--line);border-radius:var(--r-card);padding:24px 22px;box-shadow:var(--shadow-flat);transition:transform .16s ease,box-shadow .16s ease;display:flex;flex-direction:column} | |
| 98 | +.svc:hover{transform:translateY(-3px);box-shadow:var(--shadow-off-soft)} | |
| 99 | +.svc h3{font-size:18px;font-weight:800;letter-spacing:-.02em;color:var(--navy)} | |
| 100 | +.svc h3 .hl{color:var(--orange)} | |
| 101 | +.svc .path{font-family:var(--mono);font-size:11.5px;color:var(--blue);margin:5px 0 12px} | |
| 102 | +.svc p{font-size:13px;color:var(--ink-2);line-height:1.55;flex:1} | |
| 103 | +.svc b{display:block;margin-top:16px;font-size:22px;font-weight:800;letter-spacing:-.02em;color:var(--navy);font-variant-numeric:tabular-nums} | |
| 104 | +.svc small{color:var(--ink-3);font-size:11.5px;font-weight:500} | |
| 105 | +.svc a{display:inline-flex;margin-top:12px;font-size:12.5px;font-weight:700;color:var(--orange);text-decoration:none} | |
| 106 | +.svc a:hover{color:var(--orange-deep)} | |
| 107 | +/* ================= endpoints ================= */ | |
| 108 | +.ep{background:var(--surface);border:1px solid var(--line);border-radius:var(--r-card);padding:26px 28px;margin-bottom:18px;box-shadow:var(--shadow-flat);transition:transform .16s ease,box-shadow .16s ease} | |
| 109 | +.ep:hover{transform:translateY(-2px);box-shadow:var(--shadow-off-soft)} | |
| 110 | +.ep-top{display:flex;flex-wrap:wrap;align-items:center;gap:12px} | |
| 111 | +.ep-num{font-family:var(--mono);font-size:11px;font-weight:700;color:var(--ink-3)} | |
| 112 | +.method{border:1px solid rgba(34,88,255,.3);background:var(--blue-soft);color:var(--blue);border-radius:var(--r-pill);padding:4px 13px;font-size:11px;font-weight:800;letter-spacing:.05em} | |
| 113 | +.ep-path{font-family:var(--mono);font-size:15px;font-weight:700;color:var(--navy);letter-spacing:-.01em} | |
| 114 | +.try{margin-left:auto;font-size:12.5px;font-weight:700;color:var(--orange);text-decoration:none;border:1px solid rgba(255,106,0,.35);border-radius:var(--r-pill);padding:6px 16px;background:var(--orange-soft);transition:all .13s ease} | |
| 115 | +.try:hover{background:var(--orange);color:var(--white)} | |
| 116 | +.ep-desc{margin:12px 0 0;color:var(--ink-2);max-width:72ch;font-size:14px} | |
| 117 | +table.params{width:100%;border-collapse:collapse;margin-top:18px;font-size:13.5px} | |
| 118 | +table.params th{text-align:left;font-weight:700;font-size:10.5px;text-transform:uppercase;letter-spacing:.08em;color:var(--ink-3);padding:8px 14px 8px 0;border-bottom:1px solid var(--line)} | |
| 119 | +table.params td{padding:10px 14px 10px 0;border-bottom:1px solid rgba(11,19,48,.06);vertical-align:top;color:var(--ink-2)} | |
| 120 | +table.params code{font-family:var(--mono);font-size:12px;color:var(--navy);font-weight:500} | |
| 121 | +.req{font-family:var(--mono);font-size:10px;font-weight:700;border-radius:var(--r-pill);padding:2px 9px;background:var(--orange-soft);color:var(--orange-deep);letter-spacing:.04em;text-transform:uppercase} | |
| 122 | +.opt{font-family:var(--mono);font-size:10px;font-weight:700;border-radius:var(--r-pill);padding:2px 9px;background:var(--navy-soft);color:var(--ink-3);letter-spacing:.04em;text-transform:uppercase} | |
| 123 | +/* code blocks — terminal navy */ | |
| 124 | +.codeblock{margin-top:18px;border-radius:var(--r-ctl);overflow:hidden;background:var(--navy);box-shadow:var(--shadow-flat)} | |
| 125 | +.code-tabs{display:flex;align-items:center;background:#101a3d;padding:0 10px;border-bottom:1px solid rgba(255,255,255,.08)} | |
| 126 | +.code-tabs button{font-family:var(--mono);font-size:11.5px;font-weight:500;background:none;border:none;padding:11px 13px;cursor:pointer;color:#8a90a5;border-bottom:2px solid transparent;transition:color .12s ease} | |
| 127 | +.code-tabs button:hover{color:#e9efff} | |
| 128 | +.code-tabs button.on{color:var(--white);border-bottom-color:var(--orange)} | |
| 129 | +.copy{margin-left:auto;font-family:var(--mono);font-size:10.5px;font-weight:500;background:none;border:1px solid rgba(255,255,255,.18);border-radius:7px;padding:4px 12px;cursor:pointer;color:#8a90a5;transition:all .12s ease} | |
| 130 | +.copy:hover{color:var(--white);border-color:var(--orange)} | |
| 131 | +pre{background:var(--navy);color:#e9efff;padding:17px 20px;overflow-x:auto;font-family:var(--mono);font-size:12.3px;line-height:1.7;white-space:pre} | |
| 132 | +/* ================= playground ================= */ | |
| 133 | +.pg{background:var(--surface);border:1px solid var(--line);border-radius:var(--r-card);padding:32px;box-shadow:var(--shadow-off-soft)} | |
| 134 | +.pg-grid{display:grid;grid-template-columns:340px 1fr;gap:34px} | |
| 135 | +@media(max-width:880px){.pg-grid{grid-template-columns:1fr}} | |
| 136 | +.field{margin-bottom:16px} | |
| 137 | +.field label{display:block;font-size:11px;font-weight:700;text-transform:uppercase;letter-spacing:.08em;color:var(--ink-3);margin-bottom:7px} | |
| 138 | +.field select,.field input{width:100%;font-family:var(--mono);font-size:13px;padding:11px 14px;border:1px solid var(--line-strong);border-radius:var(--r-ctl);background:var(--surface);color:var(--navy);transition:border-color .12s ease,box-shadow .12s ease;appearance:auto} | |
| 139 | +.field select:focus,.field input:focus{outline:none;border-color:var(--orange);box-shadow:0 0 0 4px var(--orange-soft)} | |
| 140 | +.send{width:100%;font-family:var(--font);font-weight:700;font-size:15px;background:var(--orange);color:var(--white);border:none;border-radius:var(--r-pill);padding:13px;cursor:pointer;margin-top:6px;min-height:48px;box-shadow:0 6px 18px #ff6a0052;transition:transform .12s ease,background .15s ease,box-shadow .12s ease} | |
| 141 | +.send:hover{background:var(--orange-deep);transform:translateY(-1px);box-shadow:0 10px 24px #ff6a0060} | |
| 142 | +.send:disabled{opacity:.55;cursor:wait;transform:none} | |
| 143 | +.pg-url{font-family:var(--mono);font-size:12px;background:var(--surface-2);border:1px solid var(--line);border-radius:var(--r-ctl);padding:12px 14px;word-break:break-all;margin-bottom:12px;display:flex;gap:12px;align-items:flex-start;color:var(--navy)} | |
| 144 | +.pg-url span{flex:1} | |
| 145 | +.pg-url .copy{border-color:var(--line-strong);color:var(--ink-3)} | |
| 146 | +.pg-url .copy:hover{color:var(--navy);border-color:var(--orange)} | |
| 147 | +.pg-status{display:flex;gap:10px;margin-bottom:12px;flex-wrap:wrap} | |
| 148 | +.pg-status span:not(:empty){font-family:var(--mono);font-size:11.5px;font-weight:700;border-radius:var(--r-pill);padding:4px 13px;background:var(--navy-soft);color:var(--ink-2)} | |
| 149 | +.pg-status .s-ok{background:var(--blue-soft);color:var(--blue)} | |
| 150 | +.pg-status .s-err{background:#fde8e8;color:var(--danger)} | |
| 151 | +#pg-out{background:var(--navy);color:#e9efff;border-radius:var(--r-ctl);padding:18px;font-family:var(--mono);font-size:12.3px;line-height:1.6;max-height:500px;overflow:auto;white-space:pre-wrap;word-break:break-word;min-height:220px;box-shadow:var(--shadow-flat)} | |
| 152 | +/* ================= conventions ================= */ | |
| 153 | +.conv-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(250px,1fr));gap:16px} | |
| 154 | +.conv{background:var(--surface);border:1px solid var(--line);border-radius:var(--r-card);padding:24px;box-shadow:var(--shadow-flat);transition:transform .16s ease,box-shadow .16s ease} | |
| 155 | +.conv:hover{transform:translateY(-3px);box-shadow:var(--shadow-off-soft)} | |
| 156 | +.conv h3{font-size:15.5px;font-weight:800;letter-spacing:-.01em;color:var(--navy);margin-bottom:8px;display:flex;align-items:center;gap:9px} | |
| 157 | +.conv h3 .ico{width:26px;height:26px;flex:none;border-radius:8px;background:var(--orange-soft);color:var(--orange);display:inline-flex;align-items:center;justify-content:center;font-size:13px} | |
| 158 | +.conv p,.conv li{font-size:13.5px;color:var(--ink-2)} | |
| 159 | +.conv ul{padding-left:18px} | |
| 160 | +/* ================= footer ================= */ | |
| 161 | +footer{background:var(--navy);color:#8a90a5;padding:44px 0;font-size:13.5px} | |
| 162 | +footer .wrap{display:flex;flex-wrap:wrap;gap:14px;justify-content:space-between;align-items:center} | |
| 163 | +footer b{color:var(--white);font-weight:700} | |
| 164 | +footer a{color:var(--orange);text-decoration:none;font-weight:600} | |
| 165 | +footer a:hover{color:#ffb277} | |
| 166 | +/* reveal on scroll */ | |
| 167 | +.reveal{opacity:0;transform:translateY(14px);transition:opacity .5s ease,transform .5s ease} | |
| 168 | +.reveal.in{opacity:1;transform:none} | |
| 169 | +@media(prefers-reduced-motion:reduce){.reveal{opacity:1;transform:none;transition:none}.hero-tile{transform:none}.pulse.ok{animation:none}} | |
| 170 | +</style> | |
| 171 | +</head> | |
| 172 | +<body> | |
| 173 | + | |
| 174 | +<div class="topbar"><div class="wrap"> | |
| 175 | + <a class="brand" href="#"> | |
| 176 | + <svg viewBox="0 0 64 64"><rect width="64" height="64" rx="14" fill="#0B1330"/><ellipse cx="32" cy="18" rx="17" ry="7" fill="none" stroke="#FF6A00" stroke-width="5"/><path d="M15 18v28c0 3.9 7.6 7 17 7s17-3.1 17-7V18" fill="none" stroke="#FF6A00" stroke-width="5"/><path d="M15 32c0 3.9 7.6 7 17 7s17-3.1 17-7" fill="none" stroke="#ffffff" stroke-width="4"/></svg> | |
| 177 | + <span>API-<span class="hl">KA</span></span><span class="brand-tag">Les données de l'écosystème KA</span> | |
| 178 | + </a> | |
| 179 | + <ul class="topnav"> | |
| 180 | + <li><a href="#services">Services</a></li> | |
| 181 | + <li><a href="#endpoints">Endpoints</a></li> | |
| 182 | + <li><a href="#playground">Playground</a></li> | |
| 183 | + <li><a href="#conventions">Conventions</a></li> | |
| 184 | + <li><a href="/docs">Swagger</a></li> | |
| 185 | + </ul> | |
| 186 | + <span class="live-chip"><span class="pulse" id="live-dot"></span><span id="live-txt">vérification…</span></span> | |
| 187 | +</div></div> | |
| 188 | + | |
| 189 | +<header class="hero"><div class="wrap"><div class="hero-grid"> | |
| 190 | + <div> | |
| 191 | + <span class="kicker">Plateforme centrale · Node m3u96b</span> | |
| 192 | + <h1>Toutes les données <span class="hl">KA</span>,<br>centralisées <span class="hl">chaque jour</span>.</h1> | |
| 193 | + <p class="lede">API-KA collecte, sauvegarde et historise quotidiennement les données des huit | |
| 194 | + services de l'écosystème KA — logements, propriétés, épicerie, véhicules, produits | |
| 195 | + québécois, restaurants, sorties et créateurs — puis les expose via une API publique, | |
| 196 | + versionnée et documentée.</p> | |
| 197 | + <div class="hero-cta"> | |
| 198 | + <a class="btn btn-primary" href="#playground">Essayer le playground</a> | |
| 199 | + <a class="btn btn-ghost" href="/docs">Documentation Swagger</a> | |
| 200 | + </div> | |
| 201 | + <div class="stat-chips"> | |
| 202 | + <span class="stat-chip"><b>8</b> services collectés</span> | |
| 203 | + <span class="stat-chip"><b id="f-records">…</b> enregistrements</span> | |
| 204 | + <span class="stat-chip">collecte quotidienne <b>02:00</b></span> | |
| 205 | + <span class="stat-chip">dernière collecte <b id="f-last">…</b></span> | |
| 206 | + </div> | |
| 207 | + </div> | |
| 208 | + <div class="hero-tile reveal"> | |
| 209 | + <div class="tile-bar"><i></i><i></i><i></i><span>GET https://www.api-ka.com/health</span></div> | |
| 210 | + <div class="tile-body" id="tile-json"><span class="tc">// chargement de l'état réel de la plateforme…</span></div> | |
| 211 | + </div> | |
| 212 | +</div></div></header> | |
| 213 | + | |
| 214 | +<section id="services"><div class="wrap"> | |
| 215 | + <span class="sec-kicker">01 — Sources</span> | |
| 216 | + <h2>Les huit <span class="hl">services</span></h2> | |
| 217 | + <p class="sec-sub">Chaque service est interrogé tous les jours à 02:00. Les données brutes sont | |
| 218 | + validées, dédupliquées par empreinte SHA-256, historisées en mode append-only et sauvegardées.</p> | |
| 219 | + <div class="svc-grid" id="svc-grid"></div> | |
| 220 | +</div></section> | |
| 221 | + | |
| 222 | +<section id="endpoints" class="alt"><div class="wrap"> | |
| 223 | + <span class="sec-kicker">02 — Référence</span> | |
| 224 | + <h2>Les <span class="hl">endpoints</span></h2> | |
| 225 | + <p class="sec-sub">Base : <code>https://www.api-ka.com</code> — toutes les réponses suivent | |
| 226 | + l'enveloppe uniforme <code>{ success, data, meta }</code>. | |
| 227 | + <code>{service}</code> ∈ louka · immoka · foodka · autoka · fabrika · restoka · sortika · creaka.</p> | |
| 228 | + <div id="ep-list"></div> | |
| 229 | +</div></section> | |
| 230 | + | |
| 231 | +<section id="playground"><div class="wrap"> | |
| 232 | + <span class="sec-kicker">03 — Essayer</span> | |
| 233 | + <h2>Play<span class="hl">ground</span></h2> | |
| 234 | + <p class="sec-sub">Composez une requête réelle contre l'API en production et inspectez la | |
| 235 | + réponse — statut, latence et corps JSON.</p> | |
| 236 | + <div class="pg reveal"><div class="pg-grid"> | |
| 237 | + <div> | |
| 238 | + <div class="field"><label for="pg-ep">Endpoint</label><select id="pg-ep"></select></div> | |
| 239 | + <div id="pg-fields"></div> | |
| 240 | + <button class="send" id="pg-send">Envoyer la requête →</button> | |
| 241 | + </div> | |
| 242 | + <div> | |
| 243 | + <div class="pg-url"><span id="pg-url">—</span><button class="copy" id="pg-copy-url">copier</button></div> | |
| 244 | + <div class="pg-status"><span id="pg-code"></span><span id="pg-time"></span><span id="pg-size"></span></div> | |
| 245 | + <div id="pg-out">La réponse s'affichera ici.</div> | |
| 246 | + </div> | |
| 247 | + </div></div> | |
| 248 | +</div></section> | |
| 249 | + | |
| 250 | +<section id="conventions" class="alt"><div class="wrap"> | |
| 251 | + <span class="sec-kicker">04 — Contrat</span> | |
| 252 | + <h2>Conven<span class="hl">tions</span></h2> | |
| 253 | + <div class="conv-grid"> | |
| 254 | + <div class="conv reveal"><h3><span class="ico">{ }</span>Enveloppe uniforme</h3> | |
| 255 | + <p>Toute réponse contient <code>success</code>, <code>data</code> et <code>meta</code>. | |
| 256 | + Le <code>meta</code> inclut toujours le node (<code>m3u96b</code>) et, pour les listes, | |
| 257 | + <code>page</code>, <code>limit</code> et <code>total</code>.</p></div> | |
| 258 | + <div class="conv reveal"><h3><span class="ico">⇥</span>Pagination</h3> | |
| 259 | + <p>Paramètres <code>?page=</code> (≥ 1) et <code>?limit=</code> (1 à 500, défaut 100). | |
| 260 | + Les enregistrements sont triés du plus récent au plus ancien.</p></div> | |
| 261 | + <div class="conv reveal"><h3><span class="ico">⏱</span>Limites d'usage</h3> | |
| 262 | + <p>120 requêtes / minute / IP. Au-delà : <code>429</code> avec un corps JSON explicite. | |
| 263 | + <code>/health</code> n'est pas limité.</p></div> | |
| 264 | + <div class="conv reveal"><h3><span class="ico">!</span>Erreurs</h3> | |
| 265 | + <ul><li><code>404</code> — service inconnu</li> | |
| 266 | + <li><code>422</code> — paramètre invalide (date, limit…)</li> | |
| 267 | + <li><code>429</code> — limite de débit dépassée</li></ul></div> | |
| 268 | + <div class="conv reveal"><h3><span class="ico">#</span>Intégrité des données</h3> | |
| 269 | + <p>Historique append-only : rien n'est jamais supprimé. Chaque enregistrement porte un | |
| 270 | + <code>checksum</code> SHA-256 et une <code>date_key</code> logique. Déduplication par | |
| 271 | + <code>(source, date_key, checksum)</code>.</p></div> | |
| 272 | + <div class="conv reveal"><h3><span class="ico">↻</span>Fiabilité</h3> | |
| 273 | + <p>Relance automatique (3 tentatives, backoff 30 s → 2 min → 10 min), rattrapage (backfill) | |
| 274 | + des 7 derniers jours, sauvegardes quotidiennes horodatées conservées 90 jours.</p></div> | |
| 275 | + </div> | |
| 276 | +</div></section> | |
| 277 | + | |
| 278 | +<footer><div class="wrap"> | |
| 279 | + <span>API-<b>KA</b> — plateforme légendaire de l'écosystème KA · Node <b>m3u96b</b></span> | |
| 280 | + <span>Créée par <b>Simon-Pierre Boucher</b> · <a href="mailto:contact@spboucher.ai">contact@spboucher.ai</a></span> | |
| 281 | +</div></footer> | |
| 282 | + | |
| 283 | +<script> | |
| 284 | +"use strict"; | |
| 285 | +const BASE = location.origin; | |
| 286 | +const NF = new Intl.NumberFormat("fr-CA"); | |
| 287 | + | |
| 288 | +/* ---------- catalogue des services ---------- */ | |
| 289 | +const SERVICES = [ | |
| 290 | + {id:"louka", nom:"lou-<span class='hl'>ka</span>", site:"https://www.lou-ka.com", desc:"Logements locatifs au Québec — Québec, Lévis, Grand Montréal."}, | |
| 291 | + {id:"immoka", nom:"immo-<span class='hl'>ka</span>", site:"https://www.immo-ka.com", desc:"Propriétés à vendre au Québec, toutes agences confondues."}, | |
| 292 | + {id:"foodka", nom:"food-<span class='hl'>ka</span>", site:"https://www.food-ka.com", desc:"Produits d'épicerie québécois et suivi des prix."}, | |
| 293 | + {id:"autoka", nom:"auto-<span class='hl'>ka</span>", site:"https://www.auto-ka.com", desc:"Véhicules usagés des concessionnaires du Québec."}, | |
| 294 | + {id:"fabrika",nom:"fabri-<span class='hl'>ka</span>", site:"https://www.fabri-ka.com", desc:"Produits des boutiques et fabricants québécois."}, | |
| 295 | + {id:"restoka",nom:"resto-<span class='hl'>ka</span>", site:"https://www.resto-ka.com", desc:"Restaurants du Québec — menus complets et prix réels."}, | |
| 296 | + {id:"sortika",nom:"sorti-<span class='hl'>ka</span>", site:"https://www.sorti-ka.com", desc:"Sorties et événements dans les 17 régions du Québec."}, | |
| 297 | + {id:"creaka", nom:"crea-<span class='hl'>ka</span>", site:"https://www.crea-ka.com", desc:"Annuaire des créateurs de contenu québécois et de leurs comptes."}, | |
| 298 | +]; | |
| 299 | + | |
| 300 | +/* ---------- catalogue des endpoints ---------- */ | |
| 301 | +const ENDPOINTS = [ | |
| 302 | + {id:"root", path:"/", titre:"Statut de la plateforme", | |
| 303 | + desc:"Statut global et version. Les clients API reçoivent du JSON ; les navigateurs reçoivent cette plateforme web.", | |
| 304 | + params:[], build:()=>"/", fields:[]}, | |
| 305 | + {id:"health", path:"/health", titre:"Santé de la plateforme", | |
| 306 | + desc:"Node (m3u96b), état de la base de données et horodatage de la dernière collecte réussie de chaque service. Non soumis au rate limiting.", | |
| 307 | + params:[], build:()=>"/health", fields:[]}, | |
| 308 | + {id:"list", path:"/api/v1/{service}", titre:"Données paginées d'un service", | |
| 309 | + desc:"L'historique complet d'un service, paginé, du plus récent au plus ancien. Chaque enregistrement contient le payload brut, sa date logique et son checksum.", | |
| 310 | + params:[ | |
| 311 | + ["service","path","chemin — louka, immoka, foodka, autoka, fabrika, restoka, sortika ou creaka",true], | |
| 312 | + ["page","int","numéro de page, ≥ 1 (défaut 1)",false], | |
| 313 | + ["limit","int","taille de page, 1 à 500 (défaut 100)",false]], | |
| 314 | + fields:["service","page","limit"], | |
| 315 | + build:v=>`/api/v1/${v.service}?page=${v.page||1}&limit=${v.limit||100}`}, | |
| 316 | + {id:"latest", path:"/api/v1/{service}/latest", titre:"Dernière collecte", | |
| 317 | + desc:"Tous les enregistrements de la date de collecte la plus récente du service.", | |
| 318 | + params:[["service","path","chemin — service KA",true]], | |
| 319 | + fields:["service"], build:v=>`/api/v1/${v.service}/latest`}, | |
| 320 | + {id:"bydate", path:"/api/v1/{service}/date/{YYYY-MM-DD}", titre:"Données d'une date précise", | |
| 321 | + desc:"Les enregistrements collectés pour une date logique donnée, paginés.", | |
| 322 | + params:[ | |
| 323 | + ["service","path","chemin — service KA",true], | |
| 324 | + ["date","path","date logique au format YYYY-MM-DD",true], | |
| 325 | + ["page","int","numéro de page (défaut 1)",false], | |
| 326 | + ["limit","int","taille de page, max 500 (défaut 100)",false]], | |
| 327 | + fields:["service","date","page","limit"], | |
| 328 | + build:v=>`/api/v1/${v.service}/date/${v.date}?page=${v.page||1}&limit=${v.limit||100}`}, | |
| 329 | + {id:"stats", path:"/api/v1/{service}/stats", titre:"Statistiques d'un service", | |
| 330 | + desc:"Nombre d'enregistrements par jour, total historisé et détail de la dernière collecte réussie.", | |
| 331 | + params:[["service","path","chemin — service KA",true]], | |
| 332 | + fields:["service"], build:v=>`/api/v1/${v.service}/stats`}, | |
| 333 | + {id:"runs", path:"/api/v1/runs", titre:"Historique des collectes", | |
| 334 | + desc:"Le journal de toutes les exécutions de collecte : statut (success, failed, retried), volumétrie, durée et messages d'erreur éventuels.", | |
| 335 | + params:[ | |
| 336 | + ["service","query","filtrer par service KA",false], | |
| 337 | + ["status","query","success, failed ou retried",false], | |
| 338 | + ["page","int","numéro de page (défaut 1)",false], | |
| 339 | + ["limit","int","taille de page, max 500 (défaut 100)",false]], | |
| 340 | + fields:["service_opt","status","page","limit"], | |
| 341 | + build:v=>{ | |
| 342 | + const q=[["service",v.service_opt],["status",v.status],["page",v.page||1],["limit",v.limit||100]] | |
| 343 | + .filter(([,x])=>x).map(([k,x])=>`${k}=${x}`).join("&"); | |
| 344 | + return `/api/v1/runs?${q}`;}}, | |
| 345 | +]; | |
| 346 | + | |
| 347 | +function snippets(url){ | |
| 348 | + const full = BASE + url; | |
| 349 | + return { | |
| 350 | + curl: `curl -s "${full}"`, | |
| 351 | + python: `import httpx\n\nr = httpx.get("${full}")\nr.raise_for_status()\nprint(r.json())`, | |
| 352 | + js: `const r = await fetch("${full}");\nconst body = await r.json();\nconsole.log(body);`, | |
| 353 | + }; | |
| 354 | +} | |
| 355 | + | |
| 356 | +/* ---------- rendu des cartes endpoints ---------- */ | |
| 357 | +const epList = document.getElementById("ep-list"); | |
| 358 | +ENDPOINTS.forEach((ep, i) => { | |
| 359 | + const ex = ep.build({service:"louka", date:"2026-08-16", page:1, limit:100}); | |
| 360 | + const sn = snippets(ex); | |
| 361 | + const card = document.createElement("article"); | |
| 362 | + card.className = "ep reveal"; | |
| 363 | + card.innerHTML = ` | |
| 364 | + <div class="ep-top"> | |
| 365 | + <span class="ep-num">${String(i+1).padStart(2,"0")}</span> | |
| 366 | + <span class="method">GET</span> | |
| 367 | + <span class="ep-path">${ep.path}</span> | |
| 368 | + <a class="try" href="#playground" data-ep="${ep.id}">Essayer ↓</a> | |
| 369 | + </div> | |
| 370 | + <p class="ep-desc">${ep.desc}</p> | |
| 371 | + ${ep.params.length ? `<table class="params"><tr><th>Paramètre</th><th>Type</th><th></th><th>Description</th></tr> | |
| 372 | + ${ep.params.map(([n,t,d,req])=>`<tr><td><code>${n}</code></td><td><code>${t}</code></td> | |
| 373 | + <td>${req?'<span class="req">requis</span>':'<span class="opt">optionnel</span>'}</td><td>${d}</td></tr>`).join("")} | |
| 374 | + </table>`:""} | |
| 375 | + <div class="codeblock"> | |
| 376 | + <div class="code-tabs"> | |
| 377 | + <button class="on" data-lang="curl">curl</button> | |
| 378 | + <button data-lang="python">Python</button> | |
| 379 | + <button data-lang="js">JavaScript</button> | |
| 380 | + <button class="copy">copier</button> | |
| 381 | + </div> | |
| 382 | + <pre></pre> | |
| 383 | + </div>`; | |
| 384 | + const pre = card.querySelector("pre"); | |
| 385 | + pre.textContent = sn.curl; | |
| 386 | + card.querySelectorAll(".code-tabs button[data-lang]").forEach(btn => { | |
| 387 | + btn.addEventListener("click", () => { | |
| 388 | + card.querySelectorAll(".code-tabs button[data-lang]").forEach(b=>b.classList.remove("on")); | |
| 389 | + btn.classList.add("on"); | |
| 390 | + pre.textContent = sn[btn.dataset.lang]; | |
| 391 | + }); | |
| 392 | + }); | |
| 393 | + card.querySelector(".copy").addEventListener("click", e => { | |
| 394 | + navigator.clipboard.writeText(pre.textContent); | |
| 395 | + e.target.textContent = "copié ✓"; setTimeout(()=>e.target.textContent="copier", 1400); | |
| 396 | + }); | |
| 397 | + card.querySelector(".try").addEventListener("click", () => { | |
| 398 | + document.getElementById("pg-ep").value = ep.id; renderFields(); | |
| 399 | + }); | |
| 400 | + epList.appendChild(card); | |
| 401 | +}); | |
| 402 | + | |
| 403 | +/* ---------- playground ---------- */ | |
| 404 | +const pgEp = document.getElementById("pg-ep"); | |
| 405 | +ENDPOINTS.forEach(ep => { | |
| 406 | + const o = document.createElement("option"); | |
| 407 | + o.value = ep.id; o.textContent = `GET ${ep.path}`; | |
| 408 | + pgEp.appendChild(o); | |
| 409 | +}); | |
| 410 | +pgEp.value = "list"; | |
| 411 | + | |
| 412 | +const FIELD_DEFS = { | |
| 413 | + service: {label:"Service", type:"select", options:SERVICES.map(s=>s.id)}, | |
| 414 | + service_opt: {label:"Service (optionnel)", type:"select", options:["", ...SERVICES.map(s=>s.id)]}, | |
| 415 | + status: {label:"Statut (optionnel)", type:"select", options:["","success","failed","retried"]}, | |
| 416 | + date: {label:"Date (YYYY-MM-DD)", type:"date"}, | |
| 417 | + page: {label:"Page", type:"number", value:1, min:1}, | |
| 418 | + limit: {label:"Limit (max 500)", type:"number", value:25, min:1, max:500}, | |
| 419 | +}; | |
| 420 | + | |
| 421 | +function renderFields(){ | |
| 422 | + const ep = ENDPOINTS.find(e => e.id === pgEp.value); | |
| 423 | + const zone = document.getElementById("pg-fields"); | |
| 424 | + zone.innerHTML = ""; | |
| 425 | + ep.fields.forEach(f => { | |
| 426 | + const def = FIELD_DEFS[f]; | |
| 427 | + const div = document.createElement("div"); | |
| 428 | + div.className = "field"; | |
| 429 | + const id = "pgf-" + f; | |
| 430 | + if (def.type === "select") { | |
| 431 | + div.innerHTML = `<label for="${id}">${def.label}</label><select id="${id}"> | |
| 432 | + ${def.options.map(o=>`<option value="${o}">${o||"— tous —"}</option>`).join("")}</select>`; | |
| 433 | + } else if (def.type === "date") { | |
| 434 | + const today = new Date().toISOString().slice(0,10); | |
| 435 | + div.innerHTML = `<label for="${id}">${def.label}</label><input id="${id}" type="date" value="${today}">`; | |
| 436 | + } else { | |
| 437 | + div.innerHTML = `<label for="${id}">${def.label}</label> | |
| 438 | + <input id="${id}" type="number" value="${def.value}" min="${def.min||0}" ${def.max?`max="${def.max}"`:""}>`; | |
| 439 | + } | |
| 440 | + zone.appendChild(div); | |
| 441 | + }); | |
| 442 | + updateUrl(); | |
| 443 | + zone.querySelectorAll("select,input").forEach(el => el.addEventListener("input", updateUrl)); | |
| 444 | +} | |
| 445 | + | |
| 446 | +function pgValues(){ | |
| 447 | + const ep = ENDPOINTS.find(e => e.id === pgEp.value); | |
| 448 | + const v = {}; | |
| 449 | + ep.fields.forEach(f => { v[f] = (document.getElementById("pgf-"+f)||{}).value || ""; }); | |
| 450 | + return {ep, v}; | |
| 451 | +} | |
| 452 | +function updateUrl(){ | |
| 453 | + const {ep, v} = pgValues(); | |
| 454 | + document.getElementById("pg-url").textContent = BASE + ep.build(v); | |
| 455 | +} | |
| 456 | +pgEp.addEventListener("change", renderFields); | |
| 457 | +renderFields(); | |
| 458 | + | |
| 459 | +document.getElementById("pg-copy-url").addEventListener("click", e => { | |
| 460 | + navigator.clipboard.writeText(document.getElementById("pg-url").textContent); | |
| 461 | + e.target.textContent = "copié ✓"; setTimeout(()=>e.target.textContent="copier", 1400); | |
| 462 | +}); | |
| 463 | + | |
| 464 | +document.getElementById("pg-send").addEventListener("click", async () => { | |
| 465 | + const btn = document.getElementById("pg-send"); | |
| 466 | + const {ep, v} = pgValues(); | |
| 467 | + const url = ep.build(v); | |
| 468 | + const out = document.getElementById("pg-out"); | |
| 469 | + const codeEl = document.getElementById("pg-code"); | |
| 470 | + const timeEl = document.getElementById("pg-time"); | |
| 471 | + const sizeEl = document.getElementById("pg-size"); | |
| 472 | + btn.disabled = true; out.textContent = "…"; codeEl.textContent = ""; timeEl.textContent = ""; sizeEl.textContent = ""; | |
| 473 | + codeEl.className = ""; | |
| 474 | + const t0 = performance.now(); | |
| 475 | + try { | |
| 476 | + const r = await fetch(BASE + url, {headers:{Accept:"application/json"}}); | |
| 477 | + const txt = await r.text(); | |
| 478 | + const ms = Math.round(performance.now() - t0); | |
| 479 | + codeEl.textContent = `HTTP ${r.status}`; | |
| 480 | + codeEl.className = r.ok ? "s-ok" : "s-err"; | |
| 481 | + timeEl.textContent = `${ms} ms`; | |
| 482 | + sizeEl.textContent = `${NF.format(txt.length)} octets`; | |
| 483 | + try { out.textContent = JSON.stringify(JSON.parse(txt), null, 2); } | |
| 484 | + catch { out.textContent = txt.slice(0, 20000); } | |
| 485 | + } catch (err) { | |
| 486 | + codeEl.textContent = "ÉCHEC"; codeEl.className = "s-err"; | |
| 487 | + out.textContent = String(err); | |
| 488 | + } finally { btn.disabled = false; } | |
| 489 | +}); | |
| 490 | + | |
| 491 | +/* ---------- coloration JSON du tile héros ---------- */ | |
| 492 | +function colorJson(obj){ | |
| 493 | + const json = JSON.stringify(obj, null, 2); | |
| 494 | + return json | |
| 495 | + .replace(/&/g,"&").replace(/</g,"<") | |
| 496 | + .replace(/"([^"]+)":/g, '<span class="tk">"$1"</span>:') | |
| 497 | + .replace(/: "([^"]*)"/g, ': <span class="ts">"$1"</span>') | |
| 498 | + .replace(/: (\d+(\.\d+)?)/g, ': <span class="tn">$1</span>') | |
| 499 | + .replace(/: (true|false|null)/g, ': <span class="tn">$1</span>'); | |
| 500 | +} | |
| 501 | + | |
| 502 | +/* ---------- données live (santé + stats) ---------- */ | |
| 503 | +function relTime(iso){ | |
| 504 | + if (!iso) return "—"; | |
| 505 | + const mins = Math.round((Date.now() - new Date(iso).getTime()) / 60000); | |
| 506 | + if (mins < 1) return "à l'instant"; | |
| 507 | + if (mins < 60) return `il y a ${mins} min`; | |
| 508 | + const h = Math.round(mins/60); | |
| 509 | + if (h < 48) return `il y a ${h} h`; | |
| 510 | + return `il y a ${Math.round(h/24)} j`; | |
| 511 | +} | |
| 512 | + | |
| 513 | +(async () => { | |
| 514 | + const dot = document.getElementById("live-dot"), txt = document.getElementById("live-txt"); | |
| 515 | + try { | |
| 516 | + const h = await (await fetch(BASE + "/health")).json(); | |
| 517 | + const d = h.data; | |
| 518 | + dot.className = "pulse " + (d.database === "ok" && d.node_ok ? "ok" : "err"); | |
| 519 | + txt.textContent = d.database === "ok" ? "API opérationnelle · " + d.node : "API dégradée"; | |
| 520 | + const lasts = Object.values(d.last_collections || {}).filter(Boolean) | |
| 521 | + .map(x => x.finished_at).sort().reverse(); | |
| 522 | + document.getElementById("f-last").textContent = relTime(lasts[0]); | |
| 523 | + document.getElementById("tile-json").innerHTML = colorJson(h); | |
| 524 | + } catch { | |
| 525 | + dot.className = "pulse err"; txt.textContent = "API injoignable"; | |
| 526 | + document.getElementById("tile-json").innerHTML = '<span class="tc">// impossible de joindre /health</span>'; | |
| 527 | + } | |
| 528 | + | |
| 529 | + const grid = document.getElementById("svc-grid"); | |
| 530 | + let grand = 0, done = 0; | |
| 531 | + for (const s of SERVICES) { | |
| 532 | + const card = document.createElement("div"); | |
| 533 | + card.className = "svc reveal"; card.id = "svc-" + s.id; | |
| 534 | + card.innerHTML = `<h3>${s.nom}</h3><div class="path">/api/v1/${s.id}</div> | |
| 535 | + <p>${s.desc}</p><b>…</b><small>enregistrements historisés</small> | |
| 536 | + <a href="${s.site}" target="_blank" rel="noopener">${s.site.replace("https://","")} ↗</a>`; | |
| 537 | + grid.appendChild(card); | |
| 538 | + } | |
| 539 | + observeReveals(); | |
| 540 | + await Promise.all(SERVICES.map(async s => { | |
| 541 | + try { | |
| 542 | + const st = await (await fetch(`${BASE}/api/v1/${s.id}/stats`)).json(); | |
| 543 | + const n = st.data.total_records || 0; | |
| 544 | + grand += n; done++; | |
| 545 | + document.querySelector(`#svc-${s.id} b`).textContent = NF.format(n); | |
| 546 | + } catch { | |
| 547 | + document.querySelector(`#svc-${s.id} b`).textContent = "—"; | |
| 548 | + } | |
| 549 | + })); | |
| 550 | + document.getElementById("f-records").textContent = done ? NF.format(grand) : "—"; | |
| 551 | +})(); | |
| 552 | + | |
| 553 | +/* ---------- reveal on scroll ---------- */ | |
| 554 | +function observeReveals(){ | |
| 555 | + const io = new IntersectionObserver(entries => { | |
| 556 | + entries.forEach(e => { if (e.isIntersecting) { e.target.classList.add("in"); io.unobserve(e.target); } }); | |
| 557 | + }, {threshold:.12}); | |
| 558 | + document.querySelectorAll(".reveal:not(.in)").forEach(el => io.observe(el)); | |
| 559 | +} | |
| 560 | +observeReveals(); | |
| 561 | +</script> | |
| 562 | +</body> | |
| 563 | +</html> | |
added
src/collectors/__init__.py
+49 −0
@@ -0,0 +1,49 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteurs des 8 services KA et fabriques associées.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.autoka_collector import AutokaCollector | |
| 14 | +from src.collectors.base_collector import BaseCollector | |
| 15 | +from src.collectors.creaka_collector import CreakaCollector | |
| 16 | +from src.collectors.fabrika_collector import FabrikaCollector | |
| 17 | +from src.collectors.foodka_collector import FoodkaCollector | |
| 18 | +from src.collectors.immoka_collector import ImmokaCollector | |
| 19 | +from src.collectors.louka_collector import LoukaCollector | |
| 20 | +from src.collectors.restoka_collector import RestokaCollector | |
| 21 | +from src.collectors.sortika_collector import SortikaCollector | |
| 22 | + | |
| 23 | +COLLECTORS: dict[str, type[BaseCollector]] = { | |
| 24 | + "louka": LoukaCollector, | |
| 25 | + "immoka": ImmokaCollector, | |
| 26 | + "foodka": FoodkaCollector, | |
| 27 | + "autoka": AutokaCollector, | |
| 28 | + "fabrika": FabrikaCollector, | |
| 29 | + "restoka": RestokaCollector, | |
| 30 | + "sortika": SortikaCollector, | |
| 31 | + "creaka": CreakaCollector, | |
| 32 | +} | |
| 33 | + | |
| 34 | + | |
| 35 | +def get_collector(service: str) -> BaseCollector: | |
| 36 | + """Instancie le collecteur d'un service donné. | |
| 37 | + | |
| 38 | + Raises: | |
| 39 | + ValueError: Si le service est inconnu. | |
| 40 | + """ | |
| 41 | + try: | |
| 42 | + return COLLECTORS[service]() | |
| 43 | + except KeyError as exc: | |
| 44 | + raise ValueError(f"Service inconnu : {service}") from exc | |
| 45 | + | |
| 46 | + | |
| 47 | +def build_collectors() -> list[BaseCollector]: | |
| 48 | + """Instancie les 8 collecteurs, dans l'ordre canonique des services.""" | |
| 49 | + return [cls() for cls in COLLECTORS.values()] | |
added
src/collectors/autoka_collector.py
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/autoka_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service auto-ka (table autoka_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class AutokaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes d'auto-ka depuis AUTOKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "autoka" | |
| 20 | + items_key = "vehicles" | |
| 21 | + pagination = "offset" | |
| 22 | + page_size = 500 # maximum accepté par l'API auto-ka | |
added
src/collectors/base_collector.py
+247 −0
@@ -0,0 +1,247 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/base_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Classe abstraite des collecteurs KA : fetch → validate → checksum → insert → backup → log run. | |
| 10 | + | |
| 11 | +Chaque collecteur relance automatiquement en cas d'échec (3 tentatives, | |
| 12 | +backoff 30s → 2min → 10min). Après épuisement des tentatives, une entrée | |
| 13 | +``status=failed`` est écrite dans ``collection_runs`` et une alerte est | |
| 14 | +ajoutée à ``logs/alerts.log``. | |
| 15 | +""" | |
| 16 | + | |
| 17 | +from __future__ import annotations | |
| 18 | + | |
| 19 | +import abc | |
| 20 | +import datetime | |
| 21 | +import hashlib | |
| 22 | +import json | |
| 23 | +from collections.abc import Sequence | |
| 24 | +from typing import Any | |
| 25 | + | |
| 26 | +import httpx | |
| 27 | +from sqlalchemy import select | |
| 28 | + | |
| 29 | +from src.config import get_settings | |
| 30 | +from src.database.db import session_scope | |
| 31 | +from src.database.models import DATA_MODELS, CollectionRun | |
| 32 | +from src.utils.backup import backup_service | |
| 33 | +from src.utils.logger import alert, get_logger | |
| 34 | +from src.utils.retry import DEFAULT_ATTEMPTS, DEFAULT_DELAYS, retry_call | |
| 35 | + | |
| 36 | +FETCH_TIMEOUT_SECONDS = 60.0 | |
| 37 | + | |
| 38 | + | |
| 39 | +class BaseCollector(abc.ABC): | |
| 40 | + """Collecteur abstrait ; chaque sous-classe définit l'attribut ``service``. | |
| 41 | + | |
| 42 | + La collecte pagine l'API source jusqu'à épuisement : | |
| 43 | + - ``items_key`` : clé de la liste d'items dans la réponse JSON | |
| 44 | + (``None`` = la réponse est directement la liste ou un objet unique) ; | |
| 45 | + - ``pagination`` : ``"offset"`` (paramètres limit/offset) ou ``"page"`` | |
| 46 | + (paramètres page/per_page) ; | |
| 47 | + - ``page_size`` : taille de page, bornée par le maximum de l'API source. | |
| 48 | + """ | |
| 49 | + | |
| 50 | + service: str | |
| 51 | + items_key: str | None = None | |
| 52 | + pagination: str = "offset" | |
| 53 | + page_size: int = 500 | |
| 54 | + max_pages: int = 2000 | |
| 55 | + | |
| 56 | + def __init__( | |
| 57 | + self, | |
| 58 | + attempts: int = DEFAULT_ATTEMPTS, | |
| 59 | + delays: Sequence[float] = DEFAULT_DELAYS, | |
| 60 | + ) -> None: | |
| 61 | + if self.service not in DATA_MODELS: | |
| 62 | + raise ValueError(f"Service inconnu : {self.service!r}") | |
| 63 | + self.attempts = attempts | |
| 64 | + self.delays = tuple(delays) | |
| 65 | + self.logger = get_logger(f"apika.collector.{self.service}") | |
| 66 | + | |
| 67 | + @property | |
| 68 | + def model(self) -> type: | |
| 69 | + """Modèle SQLAlchemy de la table de données du service.""" | |
| 70 | + return DATA_MODELS[self.service] | |
| 71 | + | |
| 72 | + @property | |
| 73 | + def source_url(self) -> str: | |
| 74 | + """URL source du service, lue depuis le .env.""" | |
| 75 | + url = get_settings().source_urls.get(self.service, "") | |
| 76 | + if not url: | |
| 77 | + raise RuntimeError( | |
| 78 | + f"Variable source manquante pour {self.service} " | |
| 79 | + f"(voir .env : {self.service.upper()}_SOURCE_URL)" | |
| 80 | + ) | |
| 81 | + return url | |
| 82 | + | |
| 83 | + def fetch(self) -> Any: | |
| 84 | + """Récupère TOUTES les données de la source en paginant jusqu'à épuisement. | |
| 85 | + | |
| 86 | + Returns: | |
| 87 | + Liste complète des items bruts du service. | |
| 88 | + """ | |
| 89 | + items: list[Any] = [] | |
| 90 | + total: int | None = None | |
| 91 | + with httpx.Client( | |
| 92 | + timeout=FETCH_TIMEOUT_SECONDS, follow_redirects=True | |
| 93 | + ) as client: | |
| 94 | + for page_index in range(self.max_pages): | |
| 95 | + if self.pagination == "page": | |
| 96 | + params: dict[str, int] = { | |
| 97 | + "page": page_index + 1, | |
| 98 | + "per_page": self.page_size, | |
| 99 | + } | |
| 100 | + else: | |
| 101 | + params = { | |
| 102 | + "limit": self.page_size, | |
| 103 | + "offset": page_index * self.page_size, | |
| 104 | + } | |
| 105 | + response = client.get(self.source_url, params=params) | |
| 106 | + response.raise_for_status() | |
| 107 | + payload = response.json() | |
| 108 | + | |
| 109 | + if isinstance(payload, dict) and self.items_key: | |
| 110 | + batch = payload.get(self.items_key) or [] | |
| 111 | + raw_total = payload.get("total") | |
| 112 | + if isinstance(raw_total, int): | |
| 113 | + total = raw_total | |
| 114 | + elif isinstance(payload, list): | |
| 115 | + batch = payload | |
| 116 | + else: | |
| 117 | + batch = [payload] | |
| 118 | + | |
| 119 | + items.extend(batch) | |
| 120 | + if not batch or len(batch) < self.page_size: | |
| 121 | + break | |
| 122 | + if total is not None and len(items) >= total: | |
| 123 | + break | |
| 124 | + return items | |
| 125 | + | |
| 126 | + def validate(self, payload: Any) -> list[dict[str, Any]]: | |
| 127 | + """Valide le payload et le normalise en liste d'objets JSON. | |
| 128 | + | |
| 129 | + Raises: | |
| 130 | + ValueError: Si le payload est vide ou ``None``. | |
| 131 | + """ | |
| 132 | + if payload is None: | |
| 133 | + raise ValueError(f"Payload vide (None) pour {self.service}") | |
| 134 | + items = payload if isinstance(payload, list) else [payload] | |
| 135 | + if not items: | |
| 136 | + raise ValueError(f"Payload vide (liste vide) pour {self.service}") | |
| 137 | + return [item if isinstance(item, dict) else {"value": item} for item in items] | |
| 138 | + | |
| 139 | + @staticmethod | |
| 140 | + def checksum(item: dict[str, Any]) -> str: | |
| 141 | + """SHA-256 du JSON canonique d'un enregistrement (déduplication).""" | |
| 142 | + canonical = json.dumps( | |
| 143 | + item, sort_keys=True, ensure_ascii=False, separators=(",", ":") | |
| 144 | + ) | |
| 145 | + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() | |
| 146 | + | |
| 147 | + def save(self, items: list[dict[str, Any]], date_key: datetime.date) -> int: | |
| 148 | + """Insère les enregistrements nouveaux (dédupliqués par checksum). | |
| 149 | + | |
| 150 | + Returns: | |
| 151 | + Nombre d'enregistrements réellement insérés. | |
| 152 | + """ | |
| 153 | + model = self.model | |
| 154 | + collected_at = datetime.datetime.now(tz=datetime.UTC) | |
| 155 | + with session_scope() as session: | |
| 156 | + existing: set[str] = set( | |
| 157 | + session.execute( | |
| 158 | + select(model.checksum).where( | |
| 159 | + model.source == self.service, model.date_key == date_key | |
| 160 | + ) | |
| 161 | + ).scalars() | |
| 162 | + ) | |
| 163 | + inserted = 0 | |
| 164 | + for item in items: | |
| 165 | + digest = self.checksum(item) | |
| 166 | + if digest in existing: | |
| 167 | + continue | |
| 168 | + existing.add(digest) | |
| 169 | + session.add( | |
| 170 | + model( | |
| 171 | + payload=item, | |
| 172 | + source=self.service, | |
| 173 | + collected_at=collected_at, | |
| 174 | + date_key=date_key, | |
| 175 | + checksum=digest, | |
| 176 | + ) | |
| 177 | + ) | |
| 178 | + inserted += 1 | |
| 179 | + return inserted | |
| 180 | + | |
| 181 | + def run(self, date_key: datetime.date | None = None) -> dict[str, Any]: | |
| 182 | + """Exécute le pipeline complet et journalise le run dans collection_runs. | |
| 183 | + | |
| 184 | + Returns: | |
| 185 | + Résumé du run : service, date_key, status, records_count, error. | |
| 186 | + """ | |
| 187 | + settings = get_settings() | |
| 188 | + date_key = date_key or datetime.date.today() | |
| 189 | + started_at = datetime.datetime.now(tz=datetime.UTC) | |
| 190 | + retries = {"count": 0} | |
| 191 | + status = "success" | |
| 192 | + error_message: str | None = None | |
| 193 | + records_count = 0 | |
| 194 | + | |
| 195 | + def _pipeline() -> int: | |
| 196 | + payload = self.fetch() | |
| 197 | + items = self.validate(payload) | |
| 198 | + return self.save(items, date_key) | |
| 199 | + | |
| 200 | + def _on_retry(attempt: int, exc: BaseException, delay: float) -> None: | |
| 201 | + retries["count"] = attempt | |
| 202 | + | |
| 203 | + try: | |
| 204 | + records_count = retry_call( | |
| 205 | + _pipeline, | |
| 206 | + attempts=self.attempts, | |
| 207 | + delays=self.delays, | |
| 208 | + on_retry=_on_retry, | |
| 209 | + ) | |
| 210 | + if retries["count"] > 0: | |
| 211 | + status = "retried" | |
| 212 | + backup_service(self.service, date_key) | |
| 213 | + except Exception as exc: | |
| 214 | + status = "failed" | |
| 215 | + error_message = str(exc) | |
| 216 | + alert( | |
| 217 | + f"[{self.service}] collecte échouée pour {date_key.isoformat()} " | |
| 218 | + f"après {self.attempts} relances : {exc}" | |
| 219 | + ) | |
| 220 | + | |
| 221 | + finished_at = datetime.datetime.now(tz=datetime.UTC) | |
| 222 | + duration = (finished_at - started_at).total_seconds() | |
| 223 | + with session_scope() as session: | |
| 224 | + session.add( | |
| 225 | + CollectionRun( | |
| 226 | + service=self.service, | |
| 227 | + date_key=date_key, | |
| 228 | + status=status, | |
| 229 | + records_count=records_count, | |
| 230 | + duration_seconds=duration, | |
| 231 | + error_message=error_message, | |
| 232 | + node=settings.node_name, | |
| 233 | + started_at=started_at, | |
| 234 | + finished_at=finished_at, | |
| 235 | + ) | |
| 236 | + ) | |
| 237 | + | |
| 238 | + summary = { | |
| 239 | + "service": self.service, | |
| 240 | + "date_key": date_key.isoformat(), | |
| 241 | + "status": status, | |
| 242 | + "records_count": records_count, | |
| 243 | + "duration_seconds": duration, | |
| 244 | + "error": error_message, | |
| 245 | + } | |
| 246 | + self.logger.info("Run de collecte terminé", extra=summary) | |
| 247 | + return summary | |
added
src/collectors/creaka_collector.py
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/creaka_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-17 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service crea-ka (table creaka_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class CreakaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes de crea-ka depuis CREAKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "creaka" | |
| 20 | + items_key = "items" | |
| 21 | + pagination = "offset" | |
| 22 | + page_size = 200 # maximum accepté par l'API crea-ka (/api/creators) | |
added
src/collectors/fabrika_collector.py
+23 −0
@@ -0,0 +1,23 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/fabrika_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service fabri-ka (table fabrika_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class FabrikaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes de fabri-ka depuis FABRIKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "fabrika" | |
| 20 | + items_key = "items" | |
| 21 | + pagination = "page" # l'API fabri-ka pagine par page/per_page | |
| 22 | + page_size = 100 # maximum accepté par l'API fabri-ka (422 au-delà) | |
| 23 | + max_pages = 5000 # ~291k produits à 100/page | |
added
src/collectors/foodka_collector.py
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/foodka_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service food-ka (table foodka_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class FoodkaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes de food-ka depuis FOODKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "foodka" | |
| 20 | + items_key = "products" | |
| 21 | + pagination = "offset" | |
| 22 | + page_size = 500 # maximum accepté par l'API food-ka | |
added
src/collectors/immoka_collector.py
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/immoka_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service immo-ka (table immoka_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class ImmokaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes d'immo-ka depuis IMMOKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "immoka" | |
| 20 | + items_key = "listings" | |
| 21 | + pagination = "offset" | |
| 22 | + page_size = 2000 # maximum accepté par l'API immo-ka | |
added
src/collectors/louka_collector.py
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/louka_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service lou-ka (table louka_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class LoukaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes de lou-ka depuis LOUKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "louka" | |
| 20 | + items_key = "listings" | |
| 21 | + pagination = "offset" | |
| 22 | + page_size = 2000 # maximum accepté par l'API lou-ka | |
added
src/collectors/restoka_collector.py
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/restoka_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-17 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service resto-ka (table restoka_data). | |
| 10 | + | |
| 11 | +Contrairement aux autres collecteurs, la liste ``/api/restaurants`` ne | |
| 12 | +contient que les métadonnées : les plats (menus → sections → items, avec | |
| 13 | +options et prix réels) ne sont exposés que sur la fiche détaillée | |
| 14 | +``/api/restaurants/{uid}``. Le fetch pagine donc la liste, puis récupère | |
| 15 | +la fiche complète de chaque restaurant — chaque enregistrement sauvegardé | |
| 16 | +contient ainsi le menu entier du jour. | |
| 17 | +""" | |
| 18 | + | |
| 19 | +from __future__ import annotations | |
| 20 | + | |
| 21 | +from typing import Any | |
| 22 | + | |
| 23 | +import httpx | |
| 24 | + | |
| 25 | +from src.collectors.base_collector import FETCH_TIMEOUT_SECONDS, BaseCollector | |
| 26 | + | |
| 27 | + | |
| 28 | +class RestokaCollector(BaseCollector): | |
| 29 | + """Collecte les restaurants ET leurs plats depuis RESTOKA_SOURCE_URL.""" | |
| 30 | + | |
| 31 | + service = "restoka" | |
| 32 | + items_key = "restaurants" | |
| 33 | + pagination = "offset" | |
| 34 | + page_size = 500 # maximum accepté par l'API resto-ka (/api/restaurants) | |
| 35 | + | |
| 36 | + def fetch(self) -> Any: | |
| 37 | + """Récupère la liste paginée puis la fiche détaillée (plats inclus). | |
| 38 | + | |
| 39 | + Returns: | |
| 40 | + Liste des restaurants avec leur menu complet ; si une fiche | |
| 41 | + détaillée est indisponible, l'enregistrement sommaire de la | |
| 42 | + liste est conservé pour ne jamais perdre le restaurant. | |
| 43 | + """ | |
| 44 | + summaries: list[dict[str, Any]] = super().fetch() | |
| 45 | + detailed: list[dict[str, Any]] = [] | |
| 46 | + with httpx.Client( | |
| 47 | + timeout=FETCH_TIMEOUT_SECONDS, follow_redirects=True | |
| 48 | + ) as client: | |
| 49 | + for summary in summaries: | |
| 50 | + uid = summary.get("uid") if isinstance(summary, dict) else None | |
| 51 | + if not uid: | |
| 52 | + detailed.append(summary) | |
| 53 | + continue | |
| 54 | + try: | |
| 55 | + response = client.get(f"{self.source_url}/{uid}") | |
| 56 | + response.raise_for_status() | |
| 57 | + detailed.append(response.json()) | |
| 58 | + except httpx.HTTPError as exc: | |
| 59 | + self.logger.warning( | |
| 60 | + "Fiche détaillée indisponible, sommaire conservé", | |
| 61 | + extra={"uid": uid, "error": str(exc)}, | |
| 62 | + ) | |
| 63 | + detailed.append(summary) | |
| 64 | + return detailed | |
added
src/collectors/sortika_collector.py
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/collectors/sortika_collector.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-17 | |
| 8 | +# ============================================ | |
| 9 | +"""Collecteur quotidien du service sorti-ka (table sortika_data).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from src.collectors.base_collector import BaseCollector | |
| 14 | + | |
| 15 | + | |
| 16 | +class SortikaCollector(BaseCollector): | |
| 17 | + """Collecte les données quotidiennes de sorti-ka depuis SORTIKA_SOURCE_URL.""" | |
| 18 | + | |
| 19 | + service = "sortika" | |
| 20 | + items_key = "events" | |
| 21 | + pagination = "offset" | |
| 22 | + page_size = 200 # maximum accepté par l'API sorti-ka (/api/events) | |
added
src/config.py
+113 −0
@@ -0,0 +1,113 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/config.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Configuration centrale d'API-KA : chargement du .env et vérification du node m3u96b.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import os | |
| 14 | +import socket | |
| 15 | +from dataclasses import dataclass | |
| 16 | +from functools import lru_cache | |
| 17 | +from pathlib import Path | |
| 18 | + | |
| 19 | +from dotenv import load_dotenv | |
| 20 | + | |
| 21 | +REQUIRED_NODE = "m3u96b" | |
| 22 | + | |
| 23 | +SERVICES: tuple[str, ...] = ( | |
| 24 | + "louka", | |
| 25 | + "immoka", | |
| 26 | + "foodka", | |
| 27 | + "autoka", | |
| 28 | + "fabrika", | |
| 29 | + "restoka", | |
| 30 | + "sortika", | |
| 31 | + "creaka", | |
| 32 | +) | |
| 33 | + | |
| 34 | +SERVICE_TABLES: dict[str, str] = {service: f"{service}_data" for service in SERVICES} | |
| 35 | + | |
| 36 | +SERVICE_SOURCE_ENV: dict[str, str] = { | |
| 37 | + "louka": "LOUKA_SOURCE_URL", | |
| 38 | + "immoka": "IMMOKA_SOURCE_URL", | |
| 39 | + "foodka": "FOODKA_SOURCE_URL", | |
| 40 | + "autoka": "AUTOKA_SOURCE_URL", | |
| 41 | + "fabrika": "FABRIKA_SOURCE_URL", | |
| 42 | + "restoka": "RESTOKA_SOURCE_URL", | |
| 43 | + "sortika": "SORTIKA_SOURCE_URL", | |
| 44 | + "creaka": "CREAKA_SOURCE_URL", | |
| 45 | +} | |
| 46 | + | |
| 47 | +BASE_DIR = Path(__file__).resolve().parents[1] | |
| 48 | + | |
| 49 | +load_dotenv(BASE_DIR / ".env") | |
| 50 | + | |
| 51 | + | |
| 52 | +def verify_node() -> None: | |
| 53 | + """Refuse de démarrer en production si le hostname n'est pas m3u96b. | |
| 54 | + | |
| 55 | + Le suffixe éventuel (ex. ``.local``) est ignoré et la comparaison est | |
| 56 | + insensible à la casse, mais la règle reste stricte : en production, | |
| 57 | + API-KA ne tourne que sur le node m3u96b. | |
| 58 | + """ | |
| 59 | + hostname = socket.gethostname() | |
| 60 | + if os.getenv("APP_ENV") == "production": | |
| 61 | + short = hostname.split(".")[0].lower() | |
| 62 | + if short != REQUIRED_NODE: | |
| 63 | + raise RuntimeError( | |
| 64 | + f"API-KA doit tourner sur {REQUIRED_NODE}, node actuel : {hostname}" | |
| 65 | + ) | |
| 66 | + | |
| 67 | + | |
| 68 | +@dataclass(frozen=True) | |
| 69 | +class Settings: | |
| 70 | + """Configuration immuable de la plateforme, chargée depuis le .env.""" | |
| 71 | + | |
| 72 | + app_env: str | |
| 73 | + node_name: str | |
| 74 | + database_url: str | |
| 75 | + ngrok_authtoken: str | |
| 76 | + ngrok_domain: str | |
| 77 | + api_port: int | |
| 78 | + daily_run_hour: int | |
| 79 | + backup_retention_days: int | |
| 80 | + rate_limit_per_minute: int | |
| 81 | + source_urls: dict[str, str] | |
| 82 | + base_dir: Path | |
| 83 | + logs_dir: Path | |
| 84 | + backups_dir: Path | |
| 85 | + | |
| 86 | + | |
| 87 | +@lru_cache(maxsize=1) | |
| 88 | +def get_settings() -> Settings: | |
| 89 | + """Charge et met en cache la configuration depuis les variables d'environnement.""" | |
| 90 | + logs_dir = BASE_DIR / "logs" | |
| 91 | + backups_dir = BASE_DIR / "data" / "backups" | |
| 92 | + logs_dir.mkdir(parents=True, exist_ok=True) | |
| 93 | + backups_dir.mkdir(parents=True, exist_ok=True) | |
| 94 | + | |
| 95 | + default_db = f"sqlite:///{BASE_DIR / 'data' / 'apika_dev.db'}" | |
| 96 | + return Settings( | |
| 97 | + app_env=os.getenv("APP_ENV", "development"), | |
| 98 | + node_name=os.getenv("NODE_NAME", REQUIRED_NODE), | |
| 99 | + database_url=os.getenv("DATABASE_URL", default_db), | |
| 100 | + ngrok_authtoken=os.getenv("NGROK_AUTHTOKEN", ""), | |
| 101 | + ngrok_domain=os.getenv("NGROK_DOMAIN", "www.api-ka.com"), | |
| 102 | + api_port=int(os.getenv("API_PORT", "8000")), | |
| 103 | + daily_run_hour=int(os.getenv("DAILY_RUN_HOUR", "02")), | |
| 104 | + backup_retention_days=max(90, int(os.getenv("BACKUP_RETENTION_DAYS", "90"))), | |
| 105 | + rate_limit_per_minute=int(os.getenv("RATE_LIMIT_PER_MINUTE", "120")), | |
| 106 | + source_urls={ | |
| 107 | + service: os.getenv(env_var, "") | |
| 108 | + for service, env_var in SERVICE_SOURCE_ENV.items() | |
| 109 | + }, | |
| 110 | + base_dir=BASE_DIR, | |
| 111 | + logs_dir=logs_dir, | |
| 112 | + backups_dir=backups_dir, | |
| 113 | + ) | |
added
src/database/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/database/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Couche base de données d'API-KA : modèles SQLAlchemy, engine et migrations.""" | |
added
src/database/db.py
+112 −0
@@ -0,0 +1,112 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/database/db.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Engine SQLAlchemy, sessions, initialisation du schéma et healthcheck DB.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import argparse | |
| 14 | +from collections.abc import Iterator | |
| 15 | +from contextlib import contextmanager | |
| 16 | + | |
| 17 | +from sqlalchemy import Engine, create_engine, text | |
| 18 | +from sqlalchemy.orm import Session, sessionmaker | |
| 19 | + | |
| 20 | +from src.config import get_settings, verify_node | |
| 21 | +from src.database.models import Base | |
| 22 | + | |
| 23 | +_engine: Engine | None = None | |
| 24 | +_session_factory: sessionmaker[Session] | None = None | |
| 25 | + | |
| 26 | + | |
| 27 | +def get_engine() -> Engine: | |
| 28 | + """Retourne l'engine SQLAlchemy (créé paresseusement, pool_pre_ping actif).""" | |
| 29 | + global _engine | |
| 30 | + if _engine is None: | |
| 31 | + settings = get_settings() | |
| 32 | + kwargs: dict = {"pool_pre_ping": True} | |
| 33 | + if settings.database_url.startswith("sqlite"): | |
| 34 | + kwargs["connect_args"] = {"check_same_thread": False} | |
| 35 | + _engine = create_engine(settings.database_url, **kwargs) | |
| 36 | + return _engine | |
| 37 | + | |
| 38 | + | |
| 39 | +def get_session_factory() -> sessionmaker[Session]: | |
| 40 | + """Retourne la fabrique de sessions liée à l'engine courant.""" | |
| 41 | + global _session_factory | |
| 42 | + if _session_factory is None: | |
| 43 | + _session_factory = sessionmaker(bind=get_engine(), expire_on_commit=False) | |
| 44 | + return _session_factory | |
| 45 | + | |
| 46 | + | |
| 47 | +def reset_engine() -> None: | |
| 48 | + """Réinitialise l'engine (utilisé par les tests après changement de config).""" | |
| 49 | + global _engine, _session_factory | |
| 50 | + if _engine is not None: | |
| 51 | + _engine.dispose() | |
| 52 | + _engine = None | |
| 53 | + _session_factory = None | |
| 54 | + | |
| 55 | + | |
| 56 | +@contextmanager | |
| 57 | +def session_scope() -> Iterator[Session]: | |
| 58 | + """Session transactionnelle : commit en sortie, rollback sur exception.""" | |
| 59 | + session = get_session_factory()() | |
| 60 | + try: | |
| 61 | + yield session | |
| 62 | + session.commit() | |
| 63 | + except Exception: | |
| 64 | + session.rollback() | |
| 65 | + raise | |
| 66 | + finally: | |
| 67 | + session.close() | |
| 68 | + | |
| 69 | + | |
| 70 | +def get_db() -> Iterator[Session]: | |
| 71 | + """Dépendance FastAPI fournissant une session par requête.""" | |
| 72 | + session = get_session_factory()() | |
| 73 | + try: | |
| 74 | + yield session | |
| 75 | + finally: | |
| 76 | + session.close() | |
| 77 | + | |
| 78 | + | |
| 79 | +def init_db() -> None: | |
| 80 | + """Crée toutes les tables si elles n'existent pas (idempotent).""" | |
| 81 | + Base.metadata.create_all(get_engine()) | |
| 82 | + | |
| 83 | + | |
| 84 | +def healthcheck() -> bool: | |
| 85 | + """Vérifie que la base répond (SELECT 1).""" | |
| 86 | + try: | |
| 87 | + with get_engine().connect() as conn: | |
| 88 | + conn.execute(text("SELECT 1")) | |
| 89 | + return True | |
| 90 | + except Exception: | |
| 91 | + return False | |
| 92 | + | |
| 93 | + | |
| 94 | +def main() -> None: | |
| 95 | + """Point d'entrée CLI : ``python -m src.database.db --init``.""" | |
| 96 | + from src.utils.logger import get_logger | |
| 97 | + | |
| 98 | + parser = argparse.ArgumentParser(description="Gestion de la base API-KA (m3u96b)") | |
| 99 | + parser.add_argument("--init", action="store_true", help="Créer les tables") | |
| 100 | + args = parser.parse_args() | |
| 101 | + | |
| 102 | + verify_node() | |
| 103 | + logger = get_logger("apika.db") | |
| 104 | + if args.init: | |
| 105 | + init_db() | |
| 106 | + logger.info("Base initialisée", extra={"database_ok": healthcheck()}) | |
| 107 | + else: | |
| 108 | + logger.info("Healthcheck base", extra={"database_ok": healthcheck()}) | |
| 109 | + | |
| 110 | + | |
| 111 | +if __name__ == "__main__": | |
| 112 | + main() | |
added
src/database/migrations/env.py
+59 −0
@@ -0,0 +1,59 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/database/migrations/env.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Environnement Alembic : migrations du schéma API-KA (PostgreSQL sur m3u96b).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +from logging.config import fileConfig | |
| 14 | + | |
| 15 | +from alembic import context | |
| 16 | +from sqlalchemy import engine_from_config, pool | |
| 17 | + | |
| 18 | +from src.config import get_settings | |
| 19 | +from src.database.models import Base | |
| 20 | + | |
| 21 | +config = context.config | |
| 22 | + | |
| 23 | +if config.config_file_name is not None: | |
| 24 | + fileConfig(config.config_file_name) | |
| 25 | + | |
| 26 | +config.set_main_option("sqlalchemy.url", get_settings().database_url) | |
| 27 | + | |
| 28 | +target_metadata = Base.metadata | |
| 29 | + | |
| 30 | + | |
| 31 | +def run_migrations_offline() -> None: | |
| 32 | + """Exécute les migrations en mode 'offline' (génération SQL sans connexion).""" | |
| 33 | + context.configure( | |
| 34 | + url=config.get_main_option("sqlalchemy.url"), | |
| 35 | + target_metadata=target_metadata, | |
| 36 | + literal_binds=True, | |
| 37 | + dialect_opts={"paramstyle": "named"}, | |
| 38 | + ) | |
| 39 | + with context.begin_transaction(): | |
| 40 | + context.run_migrations() | |
| 41 | + | |
| 42 | + | |
| 43 | +def run_migrations_online() -> None: | |
| 44 | + """Exécute les migrations en mode 'online' (connexion directe à la base).""" | |
| 45 | + connectable = engine_from_config( | |
| 46 | + config.get_section(config.config_ini_section, {}), | |
| 47 | + prefix="sqlalchemy.", | |
| 48 | + poolclass=pool.NullPool, | |
| 49 | + ) | |
| 50 | + with connectable.connect() as connection: | |
| 51 | + context.configure(connection=connection, target_metadata=target_metadata) | |
| 52 | + with context.begin_transaction(): | |
| 53 | + context.run_migrations() | |
| 54 | + | |
| 55 | + | |
| 56 | +if context.is_offline_mode(): | |
| 57 | + run_migrations_offline() | |
| 58 | +else: | |
| 59 | + run_migrations_online() | |
added
src/database/migrations/script.py.mako
+35 −0
@@ -0,0 +1,35 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/database/migrations/script.py.mako | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""${message} | |
| 10 | + | |
| 11 | +Revision ID: ${up_revision} | |
| 12 | +Revises: ${down_revision | comma,n} | |
| 13 | +Create Date: ${create_date} | |
| 14 | +""" | |
| 15 | + | |
| 16 | +from __future__ import annotations | |
| 17 | + | |
| 18 | +import sqlalchemy as sa | |
| 19 | +from alembic import op | |
| 20 | +${imports if imports else ""} | |
| 21 | + | |
| 22 | +revision = ${repr(up_revision)} | |
| 23 | +down_revision = ${repr(down_revision)} | |
| 24 | +branch_labels = ${repr(branch_labels)} | |
| 25 | +depends_on = ${repr(depends_on)} | |
| 26 | + | |
| 27 | + | |
| 28 | +def upgrade() -> None: | |
| 29 | + """Applique la migration.""" | |
| 30 | + ${upgrades if upgrades else "pass"} | |
| 31 | + | |
| 32 | + | |
| 33 | +def downgrade() -> None: | |
| 34 | + """Annule la migration.""" | |
| 35 | + ${downgrades if downgrades else "pass"} | |
added
src/database/migrations/versions/.gitkeep
+0 −0
added
src/database/models.py
+158 −0
@@ -0,0 +1,158 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/database/models.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Modèles SQLAlchemy : 8 tables de données (schéma identique) + collection_runs. | |
| 10 | + | |
| 11 | +Règle append-only : aucune suppression destructive, l'historique complet est conservé. | |
| 12 | +Déduplication par contrainte unique (source, date_key, checksum). | |
| 13 | +""" | |
| 14 | + | |
| 15 | +from __future__ import annotations | |
| 16 | + | |
| 17 | +import datetime | |
| 18 | +from typing import Any | |
| 19 | + | |
| 20 | +from sqlalchemy import ( | |
| 21 | + BigInteger, | |
| 22 | + Date, | |
| 23 | + DateTime, | |
| 24 | + Float, | |
| 25 | + Index, | |
| 26 | + Integer, | |
| 27 | + Text, | |
| 28 | + UniqueConstraint, | |
| 29 | + func, | |
| 30 | +) | |
| 31 | +from sqlalchemy.dialects.postgresql import JSONB | |
| 32 | +from sqlalchemy.orm import DeclarativeBase, Mapped, declared_attr, mapped_column | |
| 33 | +from sqlalchemy.types import JSON | |
| 34 | + | |
| 35 | +from src.config import REQUIRED_NODE | |
| 36 | + | |
| 37 | +# BIGSERIAL sur PostgreSQL ; INTEGER auto-incrémenté en dev SQLite. | |
| 38 | +BigIntPK = BigInteger().with_variant(Integer(), "sqlite") | |
| 39 | +# JSONB sur PostgreSQL ; JSON générique en dev SQLite. | |
| 40 | +JSONPayload = JSONB().with_variant(JSON(), "sqlite") | |
| 41 | + | |
| 42 | + | |
| 43 | +class Base(DeclarativeBase): | |
| 44 | + """Base déclarative commune à tous les modèles API-KA.""" | |
| 45 | + | |
| 46 | + | |
| 47 | +class DataRecordMixin: | |
| 48 | + """Colonnes communes aux 8 tables de données des services KA.""" | |
| 49 | + | |
| 50 | + id: Mapped[int] = mapped_column(BigIntPK, primary_key=True, autoincrement=True) | |
| 51 | + payload: Mapped[dict[str, Any]] = mapped_column(JSONPayload, nullable=False) | |
| 52 | + source: Mapped[str] = mapped_column(Text, nullable=False) | |
| 53 | + collected_at: Mapped[datetime.datetime] = mapped_column( | |
| 54 | + DateTime(timezone=True), nullable=False, server_default=func.now() | |
| 55 | + ) | |
| 56 | + date_key: Mapped[datetime.date] = mapped_column(Date, nullable=False) | |
| 57 | + checksum: Mapped[str] = mapped_column(Text, nullable=False) | |
| 58 | + | |
| 59 | + @declared_attr.directive | |
| 60 | + def __table_args__(cls) -> tuple: # noqa: N805 — API declared_attr de SQLAlchemy | |
| 61 | + table = cls.__tablename__ | |
| 62 | + return ( | |
| 63 | + Index(f"ix_{table}_date_key", "date_key"), | |
| 64 | + Index(f"ix_{table}_source_date_key", "source", "date_key"), | |
| 65 | + UniqueConstraint( | |
| 66 | + "source", | |
| 67 | + "date_key", | |
| 68 | + "checksum", | |
| 69 | + name=f"uq_{table}_source_date_checksum", | |
| 70 | + ), | |
| 71 | + ) | |
| 72 | + | |
| 73 | + | |
| 74 | +class LoukaData(DataRecordMixin, Base): | |
| 75 | + """Données quotidiennes du service lou-ka.""" | |
| 76 | + | |
| 77 | + __tablename__ = "louka_data" | |
| 78 | + | |
| 79 | + | |
| 80 | +class ImmokaData(DataRecordMixin, Base): | |
| 81 | + """Données quotidiennes du service immo-ka.""" | |
| 82 | + | |
| 83 | + __tablename__ = "immoka_data" | |
| 84 | + | |
| 85 | + | |
| 86 | +class FoodkaData(DataRecordMixin, Base): | |
| 87 | + """Données quotidiennes du service food-ka.""" | |
| 88 | + | |
| 89 | + __tablename__ = "foodka_data" | |
| 90 | + | |
| 91 | + | |
| 92 | +class AutokaData(DataRecordMixin, Base): | |
| 93 | + """Données quotidiennes du service auto-ka.""" | |
| 94 | + | |
| 95 | + __tablename__ = "autoka_data" | |
| 96 | + | |
| 97 | + | |
| 98 | +class FabrikaData(DataRecordMixin, Base): | |
| 99 | + """Données quotidiennes du service fabri-ka.""" | |
| 100 | + | |
| 101 | + __tablename__ = "fabrika_data" | |
| 102 | + | |
| 103 | + | |
| 104 | +class RestokaData(DataRecordMixin, Base): | |
| 105 | + """Données quotidiennes du service resto-ka.""" | |
| 106 | + | |
| 107 | + __tablename__ = "restoka_data" | |
| 108 | + | |
| 109 | + | |
| 110 | +class SortikaData(DataRecordMixin, Base): | |
| 111 | + """Données quotidiennes du service sorti-ka.""" | |
| 112 | + | |
| 113 | + __tablename__ = "sortika_data" | |
| 114 | + | |
| 115 | + | |
| 116 | +class CreakaData(DataRecordMixin, Base): | |
| 117 | + """Données quotidiennes du service crea-ka.""" | |
| 118 | + | |
| 119 | + __tablename__ = "creaka_data" | |
| 120 | + | |
| 121 | + | |
| 122 | +DATA_MODELS: dict[str, type[Base]] = { | |
| 123 | + "louka": LoukaData, | |
| 124 | + "immoka": ImmokaData, | |
| 125 | + "foodka": FoodkaData, | |
| 126 | + "autoka": AutokaData, | |
| 127 | + "fabrika": FabrikaData, | |
| 128 | + "restoka": RestokaData, | |
| 129 | + "sortika": SortikaData, | |
| 130 | + "creaka": CreakaData, | |
| 131 | +} | |
| 132 | + | |
| 133 | + | |
| 134 | +class CollectionRun(Base): | |
| 135 | + """Journal de chaque exécution de collecte (succès, échec ou relance).""" | |
| 136 | + | |
| 137 | + __tablename__ = "collection_runs" | |
| 138 | + __table_args__ = ( | |
| 139 | + Index("ix_collection_runs_service_date_key", "service", "date_key"), | |
| 140 | + Index("ix_collection_runs_status", "status"), | |
| 141 | + ) | |
| 142 | + | |
| 143 | + id: Mapped[int] = mapped_column(BigIntPK, primary_key=True, autoincrement=True) | |
| 144 | + service: Mapped[str] = mapped_column(Text, nullable=False) | |
| 145 | + date_key: Mapped[datetime.date] = mapped_column(Date, nullable=False) | |
| 146 | + status: Mapped[str] = mapped_column( | |
| 147 | + Text, nullable=False | |
| 148 | + ) # success / failed / retried | |
| 149 | + records_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) | |
| 150 | + duration_seconds: Mapped[float] = mapped_column(Float, nullable=False, default=0.0) | |
| 151 | + error_message: Mapped[str | None] = mapped_column(Text, nullable=True) | |
| 152 | + node: Mapped[str] = mapped_column(Text, nullable=False, default=REQUIRED_NODE) | |
| 153 | + started_at: Mapped[datetime.datetime] = mapped_column( | |
| 154 | + DateTime(timezone=True), nullable=False | |
| 155 | + ) | |
| 156 | + finished_at: Mapped[datetime.datetime] = mapped_column( | |
| 157 | + DateTime(timezone=True), nullable=False | |
| 158 | + ) | |
added
src/scheduler/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/scheduler/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Planification des collectes : job quotidien à 02:00 et backfill automatique.""" | |
added
src/scheduler/backfill.py
+127 −0
@@ -0,0 +1,127 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/scheduler/backfill.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Rattrapage automatique des dates manquées (7 derniers jours par défaut). | |
| 10 | + | |
| 11 | +Une date est « manquée » pour un service si aucun run réussi (``success`` ou | |
| 12 | +``retried``) n'existe dans ``collection_runs`` ET qu'aucune donnée n'est | |
| 13 | +présente dans la table du service pour cette date. | |
| 14 | +""" | |
| 15 | + | |
| 16 | +from __future__ import annotations | |
| 17 | + | |
| 18 | +import argparse | |
| 19 | +import datetime | |
| 20 | +from typing import Any | |
| 21 | + | |
| 22 | +from sqlalchemy import select | |
| 23 | + | |
| 24 | +from src.collectors import get_collector | |
| 25 | +from src.config import SERVICES, verify_node | |
| 26 | +from src.database.db import init_db, session_scope | |
| 27 | +from src.database.models import DATA_MODELS, CollectionRun | |
| 28 | +from src.utils.logger import get_logger | |
| 29 | + | |
| 30 | +SUCCESS_STATUSES = ("success", "retried") | |
| 31 | + | |
| 32 | + | |
| 33 | +def missing_dates(service: str, days: int = 7) -> list[datetime.date]: | |
| 34 | + """Liste les dates manquées d'un service sur les ``days`` derniers jours. | |
| 35 | + | |
| 36 | + Le jour courant est exclu : il est couvert par le job quotidien lui-même. | |
| 37 | + Les dates antérieures à la première activité du service (tout premier run, | |
| 38 | + même échoué, ou première donnée) ne sont jamais considérées manquées — | |
| 39 | + on ne fabrique pas d'historique antérieur à la naissance de la plateforme. | |
| 40 | + """ | |
| 41 | + if service not in SERVICES: | |
| 42 | + raise ValueError(f"Service inconnu : {service}") | |
| 43 | + today = datetime.date.today() | |
| 44 | + candidates = [today - datetime.timedelta(days=i) for i in range(1, days + 1)] | |
| 45 | + model = DATA_MODELS[service] | |
| 46 | + with session_scope() as session: | |
| 47 | + all_run_dates = set( | |
| 48 | + session.execute( | |
| 49 | + select(CollectionRun.date_key).where(CollectionRun.service == service) | |
| 50 | + ).scalars() | |
| 51 | + ) | |
| 52 | + ok_run_dates = set( | |
| 53 | + session.execute( | |
| 54 | + select(CollectionRun.date_key).where( | |
| 55 | + CollectionRun.service == service, | |
| 56 | + CollectionRun.status.in_(SUCCESS_STATUSES), | |
| 57 | + ) | |
| 58 | + ).scalars() | |
| 59 | + ) | |
| 60 | + data_dates = set( | |
| 61 | + session.execute( | |
| 62 | + select(model.date_key).where(model.source == service).distinct() | |
| 63 | + ).scalars() | |
| 64 | + ) | |
| 65 | + activity_dates = all_run_dates | data_dates | |
| 66 | + if not activity_dates: | |
| 67 | + return [] | |
| 68 | + first_activity = min(activity_dates) | |
| 69 | + return sorted( | |
| 70 | + d | |
| 71 | + for d in candidates | |
| 72 | + if d >= first_activity and d not in ok_run_dates and d not in data_dates | |
| 73 | + ) | |
| 74 | + | |
| 75 | + | |
| 76 | +def backfill_date( | |
| 77 | + date_key: datetime.date, services: list[str] | None = None | |
| 78 | +) -> list[dict[str, Any]]: | |
| 79 | + """Relance la collecte d'une date précise pour les services donnés (défaut : tous).""" | |
| 80 | + results: list[dict[str, Any]] = [] | |
| 81 | + for service in services or list(SERVICES): | |
| 82 | + results.append(get_collector(service).run(date_key=date_key)) | |
| 83 | + return results | |
| 84 | + | |
| 85 | + | |
| 86 | +def run_backfill(days: int = 7) -> list[dict[str, Any]]: | |
| 87 | + """Rattrape toutes les dates manquées des ``days`` derniers jours, tous services. | |
| 88 | + | |
| 89 | + Returns: | |
| 90 | + Résumés des runs de rattrapage exécutés. | |
| 91 | + """ | |
| 92 | + logger = get_logger("apika.backfill") | |
| 93 | + results: list[dict[str, Any]] = [] | |
| 94 | + for service in SERVICES: | |
| 95 | + for date_key in missing_dates(service, days=days): | |
| 96 | + logger.info( | |
| 97 | + "Backfill d'une date manquée", | |
| 98 | + extra={"service": service, "date_key": date_key.isoformat()}, | |
| 99 | + ) | |
| 100 | + results.append(get_collector(service).run(date_key=date_key)) | |
| 101 | + if not results: | |
| 102 | + logger.info("Backfill : aucune date manquée", extra={"days": days}) | |
| 103 | + return results | |
| 104 | + | |
| 105 | + | |
| 106 | +def main() -> None: | |
| 107 | + """Point d'entrée CLI : ``python -m src.scheduler.backfill --date 2026-08-15``.""" | |
| 108 | + parser = argparse.ArgumentParser(description="Backfill API-KA (m3u96b)") | |
| 109 | + parser.add_argument( | |
| 110 | + "--date", type=datetime.date.fromisoformat, help="Date à rattraper" | |
| 111 | + ) | |
| 112 | + parser.add_argument( | |
| 113 | + "--days", type=int, default=7, help="Fenêtre de rattrapage (jours)" | |
| 114 | + ) | |
| 115 | + parser.add_argument("--service", choices=SERVICES, help="Limiter à un service") | |
| 116 | + args = parser.parse_args() | |
| 117 | + | |
| 118 | + verify_node() | |
| 119 | + init_db() | |
| 120 | + if args.date: | |
| 121 | + backfill_date(args.date, [args.service] if args.service else None) | |
| 122 | + else: | |
| 123 | + run_backfill(days=args.days) | |
| 124 | + | |
| 125 | + | |
| 126 | +if __name__ == "__main__": | |
| 127 | + main() | |
added
src/scheduler/daily_job.py
+140 −0
@@ -0,0 +1,140 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/scheduler/daily_job.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Job quotidien (02:00, heure du node m3u96b) orchestrant les 8 collecteurs KA. | |
| 10 | + | |
| 11 | +Les collecteurs s'exécutent en parallèle mais de façon indépendante : | |
| 12 | +l'échec d'un service ne bloque jamais les autres. Le backfill des 7 derniers | |
| 13 | +jours s'exécute juste après le run quotidien. | |
| 14 | +""" | |
| 15 | + | |
| 16 | +from __future__ import annotations | |
| 17 | + | |
| 18 | +import argparse | |
| 19 | +import concurrent.futures | |
| 20 | +import datetime | |
| 21 | +import json | |
| 22 | +from typing import Any | |
| 23 | + | |
| 24 | +from apscheduler.schedulers.blocking import BlockingScheduler | |
| 25 | +from apscheduler.triggers.cron import CronTrigger | |
| 26 | + | |
| 27 | +from src.collectors import build_collectors | |
| 28 | +from src.config import get_settings, verify_node | |
| 29 | +from src.database.db import init_db | |
| 30 | +from src.scheduler.backfill import run_backfill | |
| 31 | +from src.utils.logger import alert, get_logger | |
| 32 | + | |
| 33 | + | |
| 34 | +def run_all(date_key: datetime.date | None = None) -> list[dict[str, Any]]: | |
| 35 | + """Exécute les 8 collecteurs en parallèle puis le backfill des 7 derniers jours. | |
| 36 | + | |
| 37 | + Returns: | |
| 38 | + Résumés des runs du jour (un par service). | |
| 39 | + """ | |
| 40 | + settings = get_settings() | |
| 41 | + logger = get_logger("apika.scheduler") | |
| 42 | + date_key = date_key or datetime.date.today() | |
| 43 | + collectors = build_collectors() | |
| 44 | + | |
| 45 | + logger.info( | |
| 46 | + "Démarrage du run quotidien", | |
| 47 | + extra={ | |
| 48 | + "date_key": date_key.isoformat(), | |
| 49 | + "services": [c.service for c in collectors], | |
| 50 | + }, | |
| 51 | + ) | |
| 52 | + | |
| 53 | + results: list[dict[str, Any]] = [] | |
| 54 | + with concurrent.futures.ThreadPoolExecutor(max_workers=len(collectors)) as pool: | |
| 55 | + futures = { | |
| 56 | + pool.submit(collector.run, date_key): collector.service | |
| 57 | + for collector in collectors | |
| 58 | + } | |
| 59 | + for future in concurrent.futures.as_completed(futures): | |
| 60 | + service = futures[future] | |
| 61 | + try: | |
| 62 | + results.append(future.result()) | |
| 63 | + except Exception as exc: | |
| 64 | + # BaseCollector.run capture déjà ses erreurs ; ceci est le filet | |
| 65 | + # de sécurité garantissant l'indépendance des services. | |
| 66 | + alert(f"[{service}] erreur inattendue du run quotidien : {exc}") | |
| 67 | + results.append( | |
| 68 | + { | |
| 69 | + "service": service, | |
| 70 | + "date_key": date_key.isoformat(), | |
| 71 | + "status": "failed", | |
| 72 | + "records_count": 0, | |
| 73 | + "error": str(exc), | |
| 74 | + } | |
| 75 | + ) | |
| 76 | + | |
| 77 | + failed = [r["service"] for r in results if r["status"] == "failed"] | |
| 78 | + summary = { | |
| 79 | + "date_key": date_key.isoformat(), | |
| 80 | + "node": settings.node_name, | |
| 81 | + "services_total": len(results), | |
| 82 | + "services_failed": failed, | |
| 83 | + "results": results, | |
| 84 | + } | |
| 85 | + | |
| 86 | + daily_log = settings.logs_dir / f"daily_{date_key.isoformat()}.log" | |
| 87 | + with open(daily_log, "a", encoding="utf-8") as fh: | |
| 88 | + fh.write(json.dumps(summary, ensure_ascii=False, default=str) + "\n") | |
| 89 | + | |
| 90 | + logger.info("Run quotidien terminé", extra=summary) | |
| 91 | + | |
| 92 | + backfill_results = run_backfill(days=7) | |
| 93 | + if backfill_results: | |
| 94 | + logger.info( | |
| 95 | + "Backfill post-run terminé", | |
| 96 | + extra={"backfilled_runs": len(backfill_results)}, | |
| 97 | + ) | |
| 98 | + return results | |
| 99 | + | |
| 100 | + | |
| 101 | +def main() -> None: | |
| 102 | + """Point d'entrée CLI : ``--now`` pour une collecte immédiate, sinon planifie 02:00.""" | |
| 103 | + parser = argparse.ArgumentParser(description="Scheduler quotidien API-KA (m3u96b)") | |
| 104 | + parser.add_argument( | |
| 105 | + "--now", action="store_true", help="Collecte immédiate puis sortie" | |
| 106 | + ) | |
| 107 | + parser.add_argument( | |
| 108 | + "--date", | |
| 109 | + type=datetime.date.fromisoformat, | |
| 110 | + help="Date logique (défaut : aujourd'hui)", | |
| 111 | + ) | |
| 112 | + args = parser.parse_args() | |
| 113 | + | |
| 114 | + verify_node() | |
| 115 | + init_db() | |
| 116 | + settings = get_settings() | |
| 117 | + logger = get_logger("apika.scheduler") | |
| 118 | + | |
| 119 | + if args.now: | |
| 120 | + run_all(args.date) | |
| 121 | + return | |
| 122 | + | |
| 123 | + scheduler = BlockingScheduler(timezone=None) | |
| 124 | + scheduler.add_job( | |
| 125 | + run_all, | |
| 126 | + CronTrigger(hour=settings.daily_run_hour, minute=0), | |
| 127 | + id="apika_daily_collection", | |
| 128 | + max_instances=1, | |
| 129 | + coalesce=True, | |
| 130 | + misfire_grace_time=3600, | |
| 131 | + ) | |
| 132 | + logger.info( | |
| 133 | + "Scheduler démarré", | |
| 134 | + extra={"daily_run_hour": settings.daily_run_hour, "node": settings.node_name}, | |
| 135 | + ) | |
| 136 | + scheduler.start() | |
| 137 | + | |
| 138 | + | |
| 139 | +if __name__ == "__main__": | |
| 140 | + main() | |
added
src/utils/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/utils/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Utilitaires transverses d'API-KA : logging JSON, retry, backups.""" | |
added
src/utils/backup.py
+171 −0
@@ -0,0 +1,171 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/utils/backup.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Dump quotidien horodaté par service dans data/backups/YYYY-MM-DD/ (rétention 90 j). | |
| 10 | + | |
| 11 | +PostgreSQL : ``pg_dump`` de la table du service, compressé en .sql.gz. | |
| 12 | +Dev SQLite : export JSON compressé (.json.gz) des lignes du jour. | |
| 13 | +""" | |
| 14 | + | |
| 15 | +from __future__ import annotations | |
| 16 | + | |
| 17 | +import argparse | |
| 18 | +import datetime | |
| 19 | +import gzip | |
| 20 | +import json | |
| 21 | +import shutil | |
| 22 | +import subprocess | |
| 23 | +from pathlib import Path | |
| 24 | + | |
| 25 | +from sqlalchemy import select | |
| 26 | + | |
| 27 | +from src.config import SERVICE_TABLES, SERVICES, get_settings, verify_node | |
| 28 | +from src.database.db import session_scope | |
| 29 | +from src.database.models import DATA_MODELS | |
| 30 | +from src.utils.logger import alert, get_logger | |
| 31 | + | |
| 32 | + | |
| 33 | +def _day_dir(date_key: datetime.date) -> Path: | |
| 34 | + day_dir = get_settings().backups_dir / date_key.isoformat() | |
| 35 | + day_dir.mkdir(parents=True, exist_ok=True) | |
| 36 | + return day_dir | |
| 37 | + | |
| 38 | + | |
| 39 | +def _libpq_url(database_url: str) -> str: | |
| 40 | + """Convertit une URL SQLAlchemy en URL libpq pour pg_dump. | |
| 41 | + | |
| 42 | + ``postgresql+psycopg://...`` → ``postgresql://...`` (pg_dump ne connaît | |
| 43 | + pas les suffixes de driver SQLAlchemy). | |
| 44 | + """ | |
| 45 | + scheme, _, rest = database_url.partition("://") | |
| 46 | + return f"{scheme.split('+', 1)[0]}://{rest}" | |
| 47 | + | |
| 48 | + | |
| 49 | +def backup_service(service: str, date_key: datetime.date | None = None) -> Path: | |
| 50 | + """Sauvegarde la table d'un service dans data/backups/YYYY-MM-DD/. | |
| 51 | + | |
| 52 | + Args: | |
| 53 | + service: Nom du service (``louka``, ``immoka``, …). | |
| 54 | + date_key: Date logique du backup (défaut : aujourd'hui). | |
| 55 | + | |
| 56 | + Returns: | |
| 57 | + Chemin du fichier de backup compressé créé. | |
| 58 | + """ | |
| 59 | + if service not in SERVICES: | |
| 60 | + raise ValueError(f"Service inconnu : {service}") | |
| 61 | + settings = get_settings() | |
| 62 | + logger = get_logger("apika.backup") | |
| 63 | + date_key = date_key or datetime.date.today() | |
| 64 | + table = SERVICE_TABLES[service] | |
| 65 | + timestamp = datetime.datetime.now(tz=datetime.UTC).strftime("%Y%m%dT%H%M%SZ") | |
| 66 | + day_dir = _day_dir(date_key) | |
| 67 | + | |
| 68 | + if settings.database_url.startswith("postgresql"): | |
| 69 | + out_path = day_dir / f"{table}_{timestamp}.sql.gz" | |
| 70 | + result = subprocess.run( | |
| 71 | + [ | |
| 72 | + "pg_dump", | |
| 73 | + "--dbname", | |
| 74 | + _libpq_url(settings.database_url), | |
| 75 | + "--table", | |
| 76 | + table, | |
| 77 | + ], | |
| 78 | + capture_output=True, | |
| 79 | + check=True, | |
| 80 | + ) | |
| 81 | + with gzip.open(out_path, "wb") as fh: | |
| 82 | + fh.write(result.stdout) | |
| 83 | + else: | |
| 84 | + out_path = day_dir / f"{table}_{timestamp}.json.gz" | |
| 85 | + model = DATA_MODELS[service] | |
| 86 | + with session_scope() as session: | |
| 87 | + rows = ( | |
| 88 | + session.execute(select(model).where(model.date_key == date_key)) | |
| 89 | + .scalars() | |
| 90 | + .all() | |
| 91 | + ) | |
| 92 | + payload = [ | |
| 93 | + { | |
| 94 | + "id": row.id, | |
| 95 | + "payload": row.payload, | |
| 96 | + "source": row.source, | |
| 97 | + "collected_at": row.collected_at.isoformat(), | |
| 98 | + "date_key": row.date_key.isoformat(), | |
| 99 | + "checksum": row.checksum, | |
| 100 | + } | |
| 101 | + for row in rows | |
| 102 | + ] | |
| 103 | + with gzip.open(out_path, "wt", encoding="utf-8") as fh: | |
| 104 | + json.dump(payload, fh, ensure_ascii=False) | |
| 105 | + | |
| 106 | + logger.info( | |
| 107 | + "Backup effectué", | |
| 108 | + extra={ | |
| 109 | + "service": service, | |
| 110 | + "date_key": date_key.isoformat(), | |
| 111 | + "file": str(out_path), | |
| 112 | + }, | |
| 113 | + ) | |
| 114 | + return out_path | |
| 115 | + | |
| 116 | + | |
| 117 | +def backup_all(date_key: datetime.date | None = None) -> list[Path]: | |
| 118 | + """Sauvegarde les 8 services ; un échec n'interrompt pas les autres.""" | |
| 119 | + paths: list[Path] = [] | |
| 120 | + for service in SERVICES: | |
| 121 | + try: | |
| 122 | + paths.append(backup_service(service, date_key)) | |
| 123 | + except Exception as exc: | |
| 124 | + alert(f"Backup échoué pour {service} : {exc}") | |
| 125 | + return paths | |
| 126 | + | |
| 127 | + | |
| 128 | +def cleanup_old_backups(retention_days: int | None = None) -> list[Path]: | |
| 129 | + """Supprime les répertoires de backup plus vieux que la rétention (min 90 jours).""" | |
| 130 | + settings = get_settings() | |
| 131 | + retention = max(90, retention_days or settings.backup_retention_days) | |
| 132 | + cutoff = datetime.date.today() - datetime.timedelta(days=retention) | |
| 133 | + removed: list[Path] = [] | |
| 134 | + for day_dir in sorted(settings.backups_dir.iterdir()): | |
| 135 | + if not day_dir.is_dir(): | |
| 136 | + continue | |
| 137 | + try: | |
| 138 | + day = datetime.date.fromisoformat(day_dir.name) | |
| 139 | + except ValueError: | |
| 140 | + continue | |
| 141 | + if day < cutoff: | |
| 142 | + shutil.rmtree(day_dir) | |
| 143 | + removed.append(day_dir) | |
| 144 | + if removed: | |
| 145 | + get_logger("apika.backup").info( | |
| 146 | + "Backups expirés supprimés", | |
| 147 | + extra={"removed": [str(p) for p in removed], "retention_days": retention}, | |
| 148 | + ) | |
| 149 | + return removed | |
| 150 | + | |
| 151 | + | |
| 152 | +def main() -> None: | |
| 153 | + """Point d'entrée CLI : ``python -m src.utils.backup [--service X] [--cleanup]``.""" | |
| 154 | + parser = argparse.ArgumentParser(description="Backups API-KA (m3u96b)") | |
| 155 | + parser.add_argument("--service", choices=SERVICES, help="Un seul service") | |
| 156 | + parser.add_argument( | |
| 157 | + "--cleanup", action="store_true", help="Purger les backups expirés" | |
| 158 | + ) | |
| 159 | + args = parser.parse_args() | |
| 160 | + | |
| 161 | + verify_node() | |
| 162 | + if args.service: | |
| 163 | + backup_service(args.service) | |
| 164 | + else: | |
| 165 | + backup_all() | |
| 166 | + if args.cleanup: | |
| 167 | + cleanup_old_backups() | |
| 168 | + | |
| 169 | + | |
| 170 | +if __name__ == "__main__": | |
| 171 | + main() | |
added
src/utils/logger.py
+103 −0
@@ -0,0 +1,103 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/utils/logger.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Logging structuré JSON avec rotation quotidienne + alertes (logs/alerts.log).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import datetime | |
| 14 | +import json | |
| 15 | +import logging | |
| 16 | +import socket | |
| 17 | +from logging.handlers import TimedRotatingFileHandler | |
| 18 | + | |
| 19 | +from src.config import get_settings | |
| 20 | + | |
| 21 | +_RESERVED_ATTRS = { | |
| 22 | + "name", | |
| 23 | + "msg", | |
| 24 | + "args", | |
| 25 | + "levelname", | |
| 26 | + "levelno", | |
| 27 | + "pathname", | |
| 28 | + "filename", | |
| 29 | + "module", | |
| 30 | + "exc_info", | |
| 31 | + "exc_text", | |
| 32 | + "stack_info", | |
| 33 | + "lineno", | |
| 34 | + "funcName", | |
| 35 | + "created", | |
| 36 | + "msecs", | |
| 37 | + "relativeCreated", | |
| 38 | + "thread", | |
| 39 | + "threadName", | |
| 40 | + "processName", | |
| 41 | + "process", | |
| 42 | + "taskName", | |
| 43 | + "message", | |
| 44 | + "asctime", | |
| 45 | +} | |
| 46 | + | |
| 47 | + | |
| 48 | +class JsonFormatter(logging.Formatter): | |
| 49 | + """Formate chaque enregistrement de log en une ligne JSON.""" | |
| 50 | + | |
| 51 | + def format(self, record: logging.LogRecord) -> str: | |
| 52 | + entry: dict[str, object] = { | |
| 53 | + "timestamp": datetime.datetime.fromtimestamp( | |
| 54 | + record.created, tz=datetime.UTC | |
| 55 | + ).isoformat(), | |
| 56 | + "level": record.levelname, | |
| 57 | + "logger": record.name, | |
| 58 | + "message": record.getMessage(), | |
| 59 | + "node": socket.gethostname(), | |
| 60 | + } | |
| 61 | + for key, value in record.__dict__.items(): | |
| 62 | + if key not in _RESERVED_ATTRS and not key.startswith("_"): | |
| 63 | + entry[key] = value | |
| 64 | + if record.exc_info: | |
| 65 | + entry["exception"] = self.formatException(record.exc_info) | |
| 66 | + return json.dumps(entry, ensure_ascii=False, default=str) | |
| 67 | + | |
| 68 | + | |
| 69 | +def get_logger(name: str = "apika") -> logging.Logger: | |
| 70 | + """Retourne un logger JSON avec rotation quotidienne (logs/apika.log).""" | |
| 71 | + logger = logging.getLogger(name) | |
| 72 | + if logger.handlers: | |
| 73 | + return logger | |
| 74 | + | |
| 75 | + settings = get_settings() | |
| 76 | + logger.setLevel(logging.INFO) | |
| 77 | + logger.propagate = False | |
| 78 | + | |
| 79 | + formatter = JsonFormatter() | |
| 80 | + | |
| 81 | + file_handler = TimedRotatingFileHandler( | |
| 82 | + settings.logs_dir / "apika.log", | |
| 83 | + when="midnight", | |
| 84 | + backupCount=90, | |
| 85 | + encoding="utf-8", | |
| 86 | + ) | |
| 87 | + file_handler.setFormatter(formatter) | |
| 88 | + logger.addHandler(file_handler) | |
| 89 | + | |
| 90 | + stream_handler = logging.StreamHandler() | |
| 91 | + stream_handler.setFormatter(formatter) | |
| 92 | + logger.addHandler(stream_handler) | |
| 93 | + | |
| 94 | + return logger | |
| 95 | + | |
| 96 | + | |
| 97 | +def alert(message: str) -> None: | |
| 98 | + """Écrit une alerte horodatée dans logs/alerts.log et la logge en ERROR.""" | |
| 99 | + settings = get_settings() | |
| 100 | + timestamp = datetime.datetime.now(tz=datetime.UTC).isoformat() | |
| 101 | + with open(settings.logs_dir / "alerts.log", "a", encoding="utf-8") as fh: | |
| 102 | + fh.write(f"{timestamp} [ALERT] {message}\n") | |
| 103 | + get_logger().error(message, extra={"alert": True}) | |
added
src/utils/retry.py
+87 −0
@@ -0,0 +1,87 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : src/utils/retry.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Relance avec backoff exponentiel : 3 tentatives, délais 30s → 2min → 10min.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import functools | |
| 14 | +import time | |
| 15 | +from collections.abc import Callable, Sequence | |
| 16 | +from typing import Any, TypeVar | |
| 17 | + | |
| 18 | +from src.utils.logger import get_logger | |
| 19 | + | |
| 20 | +T = TypeVar("T") | |
| 21 | + | |
| 22 | +DEFAULT_ATTEMPTS = 3 | |
| 23 | +DEFAULT_DELAYS: tuple[float, ...] = (30.0, 120.0, 600.0) | |
| 24 | + | |
| 25 | + | |
| 26 | +def retry_call( | |
| 27 | + func: Callable[[], T], | |
| 28 | + *, | |
| 29 | + attempts: int = DEFAULT_ATTEMPTS, | |
| 30 | + delays: Sequence[float] = DEFAULT_DELAYS, | |
| 31 | + on_retry: Callable[[int, BaseException, float], None] | None = None, | |
| 32 | +) -> T: | |
| 33 | + """Exécute ``func`` avec jusqu'à ``attempts`` relances après l'essai initial. | |
| 34 | + | |
| 35 | + Args: | |
| 36 | + func: Callable sans argument à exécuter. | |
| 37 | + attempts: Nombre maximal de relances après le premier échec. | |
| 38 | + delays: Délais (secondes) avant chaque relance ; le dernier est réutilisé | |
| 39 | + si ``attempts`` dépasse la longueur de la séquence. | |
| 40 | + on_retry: Callback ``(numéro_de_relance, exception, délai)`` appelé | |
| 41 | + avant chaque relance. | |
| 42 | + | |
| 43 | + Returns: | |
| 44 | + Le résultat de ``func``. | |
| 45 | + | |
| 46 | + Raises: | |
| 47 | + BaseException: La dernière exception si toutes les tentatives échouent. | |
| 48 | + """ | |
| 49 | + logger = get_logger() | |
| 50 | + for attempt in range(attempts + 1): | |
| 51 | + try: | |
| 52 | + return func() | |
| 53 | + except Exception as exc: | |
| 54 | + if attempt >= attempts: | |
| 55 | + raise | |
| 56 | + delay = float(delays[min(attempt, len(delays) - 1)]) if delays else 0.0 | |
| 57 | + logger.warning( | |
| 58 | + "Échec, relance planifiée", | |
| 59 | + extra={ | |
| 60 | + "attempt": attempt + 1, | |
| 61 | + "delay_seconds": delay, | |
| 62 | + "error": str(exc), | |
| 63 | + }, | |
| 64 | + ) | |
| 65 | + if on_retry is not None: | |
| 66 | + on_retry(attempt + 1, exc, delay) | |
| 67 | + if delay > 0: | |
| 68 | + time.sleep(delay) | |
| 69 | + raise RuntimeError("retry_call: état inatteignable") # pragma: no cover | |
| 70 | + | |
| 71 | + | |
| 72 | +def retry( | |
| 73 | + attempts: int = DEFAULT_ATTEMPTS, | |
| 74 | + delays: Sequence[float] = DEFAULT_DELAYS, | |
| 75 | +) -> Callable[[Callable[..., T]], Callable[..., T]]: | |
| 76 | + """Décorateur appliquant :func:`retry_call` à la fonction décorée.""" | |
| 77 | + | |
| 78 | + def decorator(func: Callable[..., T]) -> Callable[..., T]: | |
| 79 | + @functools.wraps(func) | |
| 80 | + def wrapper(*args: Any, **kwargs: Any) -> T: | |
| 81 | + return retry_call( | |
| 82 | + lambda: func(*args, **kwargs), attempts=attempts, delays=delays | |
| 83 | + ) | |
| 84 | + | |
| 85 | + return wrapper | |
| 86 | + | |
| 87 | + return decorator | |
added
systemd/apika-api.service
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : systemd/apika-api.service | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +[Unit] | |
| 11 | +Description=API-KA — API publique FastAPI (uvicorn 127.0.0.1:8000) sur m3u96b | |
| 12 | +After=network-online.target postgresql.service | |
| 13 | +Wants=network-online.target | |
| 14 | + | |
| 15 | +[Service] | |
| 16 | +Type=simple | |
| 17 | +User=simon-pierreboucher | |
| 18 | +WorkingDirectory=/opt/api-ka | |
| 19 | +EnvironmentFile=/opt/api-ka/.env | |
| 20 | +ExecStart=/opt/api-ka/venv/bin/uvicorn src.api.main:app --host 127.0.0.1 --port 8000 | |
| 21 | +Restart=always | |
| 22 | +RestartSec=5 | |
| 23 | +StandardOutput=append:/opt/api-ka/logs/apika-api.service.log | |
| 24 | +StandardError=append:/opt/api-ka/logs/apika-api.service.log | |
| 25 | + | |
| 26 | +[Install] | |
| 27 | +WantedBy=multi-user.target | |
added
systemd/apika-ngrok.service
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : systemd/apika-ngrok.service | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +[Unit] | |
| 11 | +Description=API-KA — Tunnel ngrok www.api-ka.com → 127.0.0.1:8000 sur m3u96b | |
| 12 | +After=network-online.target apika-api.service | |
| 13 | +Wants=network-online.target | |
| 14 | + | |
| 15 | +[Service] | |
| 16 | +Type=simple | |
| 17 | +User=simon-pierreboucher | |
| 18 | +WorkingDirectory=/opt/api-ka | |
| 19 | +EnvironmentFile=/opt/api-ka/.env | |
| 20 | +ExecStart=/bin/bash /opt/api-ka/scripts/start_ngrok.sh | |
| 21 | +Restart=always | |
| 22 | +RestartSec=5 | |
| 23 | +StandardOutput=append:/opt/api-ka/logs/apika-ngrok.service.log | |
| 24 | +StandardError=append:/opt/api-ka/logs/apika-ngrok.service.log | |
| 25 | + | |
| 26 | +[Install] | |
| 27 | +WantedBy=multi-user.target | |
added
systemd/apika-scheduler.service
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : systemd/apika-scheduler.service | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | + | |
| 10 | +[Unit] | |
| 11 | +Description=API-KA — Job quotidien 02:00 + backfill (APScheduler) sur m3u96b | |
| 12 | +After=network-online.target postgresql.service | |
| 13 | +Wants=network-online.target | |
| 14 | + | |
| 15 | +[Service] | |
| 16 | +Type=simple | |
| 17 | +User=simon-pierreboucher | |
| 18 | +WorkingDirectory=/opt/api-ka | |
| 19 | +EnvironmentFile=/opt/api-ka/.env | |
| 20 | +ExecStart=/opt/api-ka/venv/bin/python -m src.scheduler.daily_job | |
| 21 | +Restart=always | |
| 22 | +RestartSec=10 | |
| 23 | +StandardOutput=append:/opt/api-ka/logs/apika-scheduler.service.log | |
| 24 | +StandardError=append:/opt/api-ka/logs/apika-scheduler.service.log | |
| 25 | + | |
| 26 | +[Install] | |
| 27 | +WantedBy=multi-user.target | |
added
tests/__init__.py
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : tests/__init__.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Suite de tests pytest d'API-KA : collecteurs, API, retry, backfill.""" | |
added
tests/conftest.py
+51 −0
@@ -0,0 +1,51 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : tests/conftest.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Fixtures pytest : environnement de test (SQLite temporaire) et base initialisée. | |
| 10 | + | |
| 11 | +L'environnement est configuré au chargement du module, AVANT tout import de | |
| 12 | +``src.*`` par les modules de test, afin que ``get_settings()`` lise les bonnes | |
| 13 | +valeurs dès le premier appel. | |
| 14 | +""" | |
| 15 | + | |
| 16 | +from __future__ import annotations | |
| 17 | + | |
| 18 | +import os | |
| 19 | +import tempfile | |
| 20 | + | |
| 21 | +_TMP_DIR = tempfile.mkdtemp(prefix="apika-tests-") | |
| 22 | +os.environ["APP_ENV"] = "test" | |
| 23 | +os.environ["DATABASE_URL"] = f"sqlite:///{_TMP_DIR}/apika_test.db" | |
| 24 | +for _svc in ("LOUKA", "IMMOKA", "FOODKA", "AUTOKA", "FABRIKA"): | |
| 25 | + os.environ[f"{_svc}_SOURCE_URL"] = f"https://example.test/{_svc.lower()}" | |
| 26 | + | |
| 27 | +import pytest # noqa: E402 | |
| 28 | + | |
| 29 | +from src.config import get_settings # noqa: E402 | |
| 30 | + | |
| 31 | +get_settings.cache_clear() | |
| 32 | + | |
| 33 | +from src.database import db as db_module # noqa: E402 | |
| 34 | + | |
| 35 | +db_module.reset_engine() | |
| 36 | +db_module.init_db() | |
| 37 | + | |
| 38 | + | |
| 39 | +@pytest.fixture() | |
| 40 | +def db_session(): | |
| 41 | + """Session SQLAlchemy transactionnelle sur la base de test.""" | |
| 42 | + with db_module.session_scope() as session: | |
| 43 | + yield session | |
| 44 | + | |
| 45 | + | |
| 46 | +@pytest.fixture(autouse=True) | |
| 47 | +def no_backup(monkeypatch: pytest.MonkeyPatch): | |
| 48 | + """Neutralise les backups disque pendant les tests de collecteurs.""" | |
| 49 | + monkeypatch.setattr( | |
| 50 | + "src.collectors.base_collector.backup_service", lambda *args, **kwargs: None | |
| 51 | + ) | |
added
tests/test_api.py
+172 −0
@@ -0,0 +1,172 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : tests/test_api.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Tests des routes de l'API publique : format uniforme, pagination, 404, /health.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import datetime | |
| 14 | +import hashlib | |
| 15 | +import json | |
| 16 | + | |
| 17 | +import pytest | |
| 18 | +from fastapi.testclient import TestClient | |
| 19 | + | |
| 20 | +from src.api.main import app | |
| 21 | +from src.database.db import session_scope | |
| 22 | +from src.database.models import DATA_MODELS, CollectionRun | |
| 23 | + | |
| 24 | +SEED_DATE = datetime.date(2026, 8, 10) | |
| 25 | + | |
| 26 | + | |
| 27 | +def _seed_louka() -> None: | |
| 28 | + model = DATA_MODELS["louka"] | |
| 29 | + now = datetime.datetime.now(tz=datetime.UTC) | |
| 30 | + with session_scope() as session: | |
| 31 | + for i in range(3): | |
| 32 | + payload = {"id": i, "nom": f"louka-{i}"} | |
| 33 | + checksum = hashlib.sha256( | |
| 34 | + json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() | |
| 35 | + ).hexdigest() | |
| 36 | + exists = ( | |
| 37 | + session.query(model) | |
| 38 | + .filter_by(source="louka", date_key=SEED_DATE, checksum=checksum) | |
| 39 | + .first() | |
| 40 | + ) | |
| 41 | + if not exists: | |
| 42 | + session.add( | |
| 43 | + model( | |
| 44 | + payload=payload, | |
| 45 | + source="louka", | |
| 46 | + collected_at=now, | |
| 47 | + date_key=SEED_DATE, | |
| 48 | + checksum=checksum, | |
| 49 | + ) | |
| 50 | + ) | |
| 51 | + session.add( | |
| 52 | + CollectionRun( | |
| 53 | + service="louka", | |
| 54 | + date_key=SEED_DATE, | |
| 55 | + status="success", | |
| 56 | + records_count=3, | |
| 57 | + duration_seconds=1.2, | |
| 58 | + error_message=None, | |
| 59 | + node="m3u96b", | |
| 60 | + started_at=now, | |
| 61 | + finished_at=now, | |
| 62 | + ) | |
| 63 | + ) | |
| 64 | + | |
| 65 | + | |
| 66 | +@pytest.fixture(scope="module") | |
| 67 | +def client(): | |
| 68 | + """Client de test FastAPI avec cycle de vie (lifespan) actif.""" | |
| 69 | + _seed_louka() | |
| 70 | + with TestClient(app) as test_client: | |
| 71 | + yield test_client | |
| 72 | + | |
| 73 | + | |
| 74 | +def test_root(client: TestClient) -> None: | |
| 75 | + """GET / retourne le statut de la plateforme et la version.""" | |
| 76 | + response = client.get("/") | |
| 77 | + assert response.status_code == 200 | |
| 78 | + body = response.json() | |
| 79 | + assert body["success"] is True | |
| 80 | + assert body["data"]["platform"] == "API-KA" | |
| 81 | + assert body["data"]["version"] | |
| 82 | + assert "node" in body["meta"] | |
| 83 | + | |
| 84 | + | |
| 85 | +def test_root_serves_web_platform_to_browsers(client: TestClient) -> None: | |
| 86 | + """GET / avec Accept: text/html sert la plateforme web (docs + playground).""" | |
| 87 | + response = client.get("/", headers={"Accept": "text/html"}) | |
| 88 | + assert response.status_code == 200 | |
| 89 | + assert response.headers["content-type"].startswith("text/html") | |
| 90 | + assert "Playground" in response.text | |
| 91 | + assert "API-KA" in response.text | |
| 92 | + | |
| 93 | + | |
| 94 | +def test_health(client: TestClient) -> None: | |
| 95 | + """GET /health expose le node, l'état DB et les dernières collectes.""" | |
| 96 | + response = client.get("/health") | |
| 97 | + assert response.status_code == 200 | |
| 98 | + data = response.json()["data"] | |
| 99 | + assert data["database"] == "ok" | |
| 100 | + assert data["required_node"] == "m3u96b" | |
| 101 | + assert data["last_collections"]["louka"] is not None | |
| 102 | + assert data["last_collections"]["louka"]["records_count"] == 3 | |
| 103 | + | |
| 104 | + | |
| 105 | +def test_list_service_data(client: TestClient) -> None: | |
| 106 | + """GET /api/v1/louka pagine les données au format uniforme.""" | |
| 107 | + response = client.get("/api/v1/louka", params={"page": 1, "limit": 2}) | |
| 108 | + assert response.status_code == 200 | |
| 109 | + body = response.json() | |
| 110 | + assert body["success"] is True | |
| 111 | + assert len(body["data"]) == 2 | |
| 112 | + assert body["meta"]["page"] == 1 | |
| 113 | + assert body["meta"]["limit"] == 2 | |
| 114 | + assert body["meta"]["total"] >= 3 | |
| 115 | + | |
| 116 | + | |
| 117 | +def test_latest(client: TestClient) -> None: | |
| 118 | + """GET /api/v1/louka/latest retourne la collecte la plus récente.""" | |
| 119 | + response = client.get("/api/v1/louka/latest") | |
| 120 | + assert response.status_code == 200 | |
| 121 | + body = response.json() | |
| 122 | + assert body["meta"]["date_key"] == SEED_DATE.isoformat() | |
| 123 | + assert len(body["data"]) == 3 | |
| 124 | + | |
| 125 | + | |
| 126 | +def test_by_date(client: TestClient) -> None: | |
| 127 | + """GET /api/v1/louka/date/{date} filtre par date logique.""" | |
| 128 | + response = client.get(f"/api/v1/louka/date/{SEED_DATE.isoformat()}") | |
| 129 | + assert response.status_code == 200 | |
| 130 | + assert response.json()["meta"]["total"] == 3 | |
| 131 | + | |
| 132 | + empty = client.get("/api/v1/louka/date/2020-01-01") | |
| 133 | + assert empty.status_code == 200 | |
| 134 | + assert empty.json()["meta"]["total"] == 0 | |
| 135 | + | |
| 136 | + invalid = client.get("/api/v1/louka/date/pas-une-date") | |
| 137 | + assert invalid.status_code == 422 | |
| 138 | + | |
| 139 | + | |
| 140 | +def test_stats(client: TestClient) -> None: | |
| 141 | + """GET /api/v1/louka/stats agrège par jour et expose la dernière réussite.""" | |
| 142 | + response = client.get("/api/v1/louka/stats") | |
| 143 | + assert response.status_code == 200 | |
| 144 | + data = response.json()["data"] | |
| 145 | + assert data["total_records"] >= 3 | |
| 146 | + assert any(day["date_key"] == SEED_DATE.isoformat() for day in data["days"]) | |
| 147 | + assert data["last_success"]["status"] == "success" | |
| 148 | + | |
| 149 | + | |
| 150 | +def test_unknown_service_is_404(client: TestClient) -> None: | |
| 151 | + """Tout service hors de la liste officielle → 404.""" | |
| 152 | + assert client.get("/api/v1/inconnu").status_code == 404 | |
| 153 | + assert client.get("/api/v1/inconnu/latest").status_code == 404 | |
| 154 | + assert client.get("/api/v1/inconnu/stats").status_code == 404 | |
| 155 | + | |
| 156 | + | |
| 157 | +def test_limit_is_capped_at_500(client: TestClient) -> None: | |
| 158 | + """limit > 500 est rejeté (422).""" | |
| 159 | + response = client.get("/api/v1/louka", params={"limit": 501}) | |
| 160 | + assert response.status_code == 422 | |
| 161 | + | |
| 162 | + | |
| 163 | +def test_runs_endpoint(client: TestClient) -> None: | |
| 164 | + """GET /api/v1/runs liste l'historique, filtrable par service et statut.""" | |
| 165 | + response = client.get("/api/v1/runs", params={"service": "louka"}) | |
| 166 | + assert response.status_code == 200 | |
| 167 | + body = response.json() | |
| 168 | + assert body["meta"]["total"] >= 1 | |
| 169 | + assert all(run["service"] == "louka" for run in body["data"]) | |
| 170 | + | |
| 171 | + assert client.get("/api/v1/runs", params={"service": "inconnu"}).status_code == 404 | |
| 172 | + assert client.get("/api/v1/runs", params={"status": "invalide"}).status_code == 422 | |
added
tests/test_backfill.py
+114 −0
@@ -0,0 +1,114 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : tests/test_backfill.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Tests du rattrapage des dates manquées (src/scheduler/backfill.py).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import datetime | |
| 14 | + | |
| 15 | +import pytest | |
| 16 | + | |
| 17 | +from src.database.db import session_scope | |
| 18 | +from src.database.models import CollectionRun | |
| 19 | +from src.scheduler.backfill import backfill_date, missing_dates | |
| 20 | + | |
| 21 | + | |
| 22 | +def _insert_success_run(service: str, date_key: datetime.date) -> None: | |
| 23 | + now = datetime.datetime.now(tz=datetime.UTC) | |
| 24 | + with session_scope() as session: | |
| 25 | + session.add( | |
| 26 | + CollectionRun( | |
| 27 | + service=service, | |
| 28 | + date_key=date_key, | |
| 29 | + status="success", | |
| 30 | + records_count=1, | |
| 31 | + duration_seconds=0.1, | |
| 32 | + error_message=None, | |
| 33 | + node="m3u96b", | |
| 34 | + started_at=now, | |
| 35 | + finished_at=now, | |
| 36 | + ) | |
| 37 | + ) | |
| 38 | + | |
| 39 | + | |
| 40 | +def _insert_failed_run(service: str, date_key: datetime.date) -> None: | |
| 41 | + now = datetime.datetime.now(tz=datetime.UTC) | |
| 42 | + with session_scope() as session: | |
| 43 | + session.add( | |
| 44 | + CollectionRun( | |
| 45 | + service=service, | |
| 46 | + date_key=date_key, | |
| 47 | + status="failed", | |
| 48 | + records_count=0, | |
| 49 | + duration_seconds=0.1, | |
| 50 | + error_message="boom", | |
| 51 | + node="m3u96b", | |
| 52 | + started_at=now, | |
| 53 | + finished_at=now, | |
| 54 | + ) | |
| 55 | + ) | |
| 56 | + | |
| 57 | + | |
| 58 | +def test_missing_dates_empty_for_brand_new_service() -> None: | |
| 59 | + """Aucune activité connue → aucun backfill (on ne fabrique pas d'historique).""" | |
| 60 | + assert missing_dates("foodka", days=7) == [] | |
| 61 | + | |
| 62 | + | |
| 63 | +def test_missing_dates_detects_gaps_after_first_activity() -> None: | |
| 64 | + """Les jours sans collecte APRÈS la première activité sont manqués.""" | |
| 65 | + today = datetime.date.today() | |
| 66 | + _insert_success_run("immoka", today - datetime.timedelta(days=3)) | |
| 67 | + missing = missing_dates("immoka", days=5) | |
| 68 | + assert missing == [ | |
| 69 | + today - datetime.timedelta(days=2), | |
| 70 | + today - datetime.timedelta(days=1), | |
| 71 | + ] | |
| 72 | + | |
| 73 | + | |
| 74 | +def test_missing_dates_includes_failed_only_dates() -> None: | |
| 75 | + """Une date avec seulement un run failed doit être rattrapée.""" | |
| 76 | + yesterday = datetime.date.today() - datetime.timedelta(days=1) | |
| 77 | + _insert_failed_run("fabrika", yesterday) | |
| 78 | + assert yesterday in missing_dates("fabrika", days=3) | |
| 79 | + | |
| 80 | + | |
| 81 | +def test_missing_dates_excludes_successful_runs() -> None: | |
| 82 | + """Une date couverte par un run success n'est plus considérée manquée.""" | |
| 83 | + yesterday = datetime.date.today() - datetime.timedelta(days=1) | |
| 84 | + _insert_success_run("autoka", yesterday) | |
| 85 | + assert yesterday not in missing_dates("autoka", days=2) | |
| 86 | + | |
| 87 | + | |
| 88 | +def test_missing_dates_rejects_unknown_service() -> None: | |
| 89 | + """Un service inconnu lève une ValueError.""" | |
| 90 | + with pytest.raises(ValueError): | |
| 91 | + missing_dates("inconnu") | |
| 92 | + | |
| 93 | + | |
| 94 | +def test_backfill_date_runs_collectors(monkeypatch: pytest.MonkeyPatch) -> None: | |
| 95 | + """backfill_date relance le collecteur de chaque service demandé.""" | |
| 96 | + called: list[tuple[str, datetime.date]] = [] | |
| 97 | + | |
| 98 | + class FakeCollector: | |
| 99 | + def __init__(self, service: str) -> None: | |
| 100 | + self.service = service | |
| 101 | + | |
| 102 | + def run(self, date_key: datetime.date) -> dict: | |
| 103 | + called.append((self.service, date_key)) | |
| 104 | + return {"service": self.service, "status": "success", "records_count": 0} | |
| 105 | + | |
| 106 | + monkeypatch.setattr( | |
| 107 | + "src.scheduler.backfill.get_collector", lambda service: FakeCollector(service) | |
| 108 | + ) | |
| 109 | + | |
| 110 | + target = datetime.date(2026, 8, 15) | |
| 111 | + results = backfill_date(target, services=["fabrika"]) | |
| 112 | + | |
| 113 | + assert called == [("fabrika", target)] | |
| 114 | + assert results[0]["status"] == "success" | |
added
tests/test_collectors.py
+160 −0
@@ -0,0 +1,160 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : tests/test_collectors.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Tests des collecteurs : pipeline complet, déduplication, échecs et statut retried.""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import datetime | |
| 14 | + | |
| 15 | +import pytest | |
| 16 | +from sqlalchemy import func, select | |
| 17 | + | |
| 18 | +from src.collectors import COLLECTORS, build_collectors, get_collector | |
| 19 | +from src.collectors.louka_collector import LoukaCollector | |
| 20 | +from src.config import SERVICES | |
| 21 | +from src.database.db import session_scope | |
| 22 | +from src.database.models import DATA_MODELS, CollectionRun | |
| 23 | + | |
| 24 | +SAMPLE_PAYLOAD = [ | |
| 25 | + {"id": 1, "nom": "enregistrement-a"}, | |
| 26 | + {"id": 2, "nom": "enregistrement-b"}, | |
| 27 | +] | |
| 28 | + | |
| 29 | + | |
| 30 | +def _count_rows(service: str, date_key: datetime.date) -> int: | |
| 31 | + model = DATA_MODELS[service] | |
| 32 | + with session_scope() as session: | |
| 33 | + return ( | |
| 34 | + session.execute( | |
| 35 | + select(func.count()) | |
| 36 | + .select_from(model) | |
| 37 | + .where(model.source == service, model.date_key == date_key) | |
| 38 | + ).scalar() | |
| 39 | + or 0 | |
| 40 | + ) | |
| 41 | + | |
| 42 | + | |
| 43 | +def _last_run(service: str, date_key: datetime.date) -> CollectionRun | None: | |
| 44 | + with session_scope() as session: | |
| 45 | + return ( | |
| 46 | + session.execute( | |
| 47 | + select(CollectionRun) | |
| 48 | + .where( | |
| 49 | + CollectionRun.service == service, CollectionRun.date_key == date_key | |
| 50 | + ) | |
| 51 | + .order_by(CollectionRun.id.desc()) | |
| 52 | + .limit(1) | |
| 53 | + ) | |
| 54 | + .scalars() | |
| 55 | + .first() | |
| 56 | + ) | |
| 57 | + | |
| 58 | + | |
| 59 | +def test_registry_covers_all_services() -> None: | |
| 60 | + """Chaque service KA a un collecteur enregistré.""" | |
| 61 | + assert set(COLLECTORS) == set(SERVICES) | |
| 62 | + assert len(build_collectors()) == 8 | |
| 63 | + with pytest.raises(ValueError): | |
| 64 | + get_collector("inconnu") | |
| 65 | + | |
| 66 | + | |
| 67 | +def test_run_success(monkeypatch: pytest.MonkeyPatch) -> None: | |
| 68 | + """Le pipeline complet insère les données et journalise un run success.""" | |
| 69 | + date_key = datetime.date(2026, 8, 1) | |
| 70 | + collector = LoukaCollector(attempts=1, delays=(0,)) | |
| 71 | + monkeypatch.setattr(collector, "fetch", lambda: SAMPLE_PAYLOAD) | |
| 72 | + | |
| 73 | + result = collector.run(date_key=date_key) | |
| 74 | + | |
| 75 | + assert result["status"] == "success" | |
| 76 | + assert result["records_count"] == 2 | |
| 77 | + assert _count_rows("louka", date_key) == 2 | |
| 78 | + run = _last_run("louka", date_key) | |
| 79 | + assert run is not None | |
| 80 | + assert run.status == "success" | |
| 81 | + assert run.records_count == 2 | |
| 82 | + assert run.node | |
| 83 | + | |
| 84 | + | |
| 85 | +def test_run_deduplicates_by_checksum(monkeypatch: pytest.MonkeyPatch) -> None: | |
| 86 | + """Une seconde collecte du même payload n'insère aucun doublon.""" | |
| 87 | + date_key = datetime.date(2026, 8, 2) | |
| 88 | + collector = LoukaCollector(attempts=1, delays=(0,)) | |
| 89 | + monkeypatch.setattr(collector, "fetch", lambda: SAMPLE_PAYLOAD) | |
| 90 | + | |
| 91 | + first = collector.run(date_key=date_key) | |
| 92 | + second = collector.run(date_key=date_key) | |
| 93 | + | |
| 94 | + assert first["records_count"] == 2 | |
| 95 | + assert second["status"] == "success" | |
| 96 | + assert second["records_count"] == 0 | |
| 97 | + assert _count_rows("louka", date_key) == 2 | |
| 98 | + | |
| 99 | + | |
| 100 | +def test_run_failure_after_retries(monkeypatch: pytest.MonkeyPatch) -> None: | |
| 101 | + """Après épuisement des relances : run failed + message d'erreur journalisé.""" | |
| 102 | + date_key = datetime.date(2026, 8, 3) | |
| 103 | + collector = LoukaCollector(attempts=2, delays=(0, 0)) | |
| 104 | + | |
| 105 | + def broken() -> None: | |
| 106 | + raise RuntimeError("source indisponible") | |
| 107 | + | |
| 108 | + monkeypatch.setattr(collector, "fetch", broken) | |
| 109 | + | |
| 110 | + result = collector.run(date_key=date_key) | |
| 111 | + | |
| 112 | + assert result["status"] == "failed" | |
| 113 | + assert "source indisponible" in (result["error"] or "") | |
| 114 | + run = _last_run("louka", date_key) | |
| 115 | + assert run is not None | |
| 116 | + assert run.status == "failed" | |
| 117 | + assert run.records_count == 0 | |
| 118 | + assert "source indisponible" in (run.error_message or "") | |
| 119 | + | |
| 120 | + | |
| 121 | +def test_run_retried_status(monkeypatch: pytest.MonkeyPatch) -> None: | |
| 122 | + """Un succès après relance est journalisé avec le statut retried.""" | |
| 123 | + date_key = datetime.date(2026, 8, 4) | |
| 124 | + collector = LoukaCollector(attempts=2, delays=(0, 0)) | |
| 125 | + calls = {"count": 0} | |
| 126 | + | |
| 127 | + def flaky() -> list[dict]: | |
| 128 | + calls["count"] += 1 | |
| 129 | + if calls["count"] == 1: | |
| 130 | + raise RuntimeError("transitoire") | |
| 131 | + return SAMPLE_PAYLOAD | |
| 132 | + | |
| 133 | + monkeypatch.setattr(collector, "fetch", flaky) | |
| 134 | + | |
| 135 | + result = collector.run(date_key=date_key) | |
| 136 | + | |
| 137 | + assert result["status"] == "retried" | |
| 138 | + assert result["records_count"] == 2 | |
| 139 | + run = _last_run("louka", date_key) | |
| 140 | + assert run is not None | |
| 141 | + assert run.status == "retried" | |
| 142 | + | |
| 143 | + | |
| 144 | +def test_validate_rejects_empty_payload() -> None: | |
| 145 | + """Un payload vide ou None est rejeté par la validation.""" | |
| 146 | + collector = LoukaCollector(attempts=1, delays=(0,)) | |
| 147 | + with pytest.raises(ValueError): | |
| 148 | + collector.validate(None) | |
| 149 | + with pytest.raises(ValueError): | |
| 150 | + collector.validate([]) | |
| 151 | + | |
| 152 | + | |
| 153 | +def test_checksum_is_stable_and_order_insensitive() -> None: | |
| 154 | + """Le checksum SHA-256 est canonique (indépendant de l'ordre des clés).""" | |
| 155 | + a = LoukaCollector.checksum({"x": 1, "y": 2}) | |
| 156 | + b = LoukaCollector.checksum({"y": 2, "x": 1}) | |
| 157 | + c = LoukaCollector.checksum({"x": 1, "y": 3}) | |
| 158 | + assert a == b | |
| 159 | + assert a != c | |
| 160 | + assert len(a) == 64 | |
added
tests/test_retry.py
+77 −0
@@ -0,0 +1,77 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : API-KA | |
| 3 | +# Fichier : tests/test_retry.py | |
| 4 | +# Node : m3u96b | |
| 5 | +# Author : Simon-Pierre Boucher | |
| 6 | +# Contact : contact@spboucher.ai | |
| 7 | +# Date : 2026-08-16 | |
| 8 | +# ============================================ | |
| 9 | +"""Tests de la logique de retry avec backoff (src/utils/retry.py).""" | |
| 10 | + | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import pytest | |
| 14 | + | |
| 15 | +from src.utils.retry import retry, retry_call | |
| 16 | + | |
| 17 | + | |
| 18 | +def test_retry_call_succeeds_first_try() -> None: | |
| 19 | + """Aucune relance si le premier essai réussit.""" | |
| 20 | + calls = {"count": 0} | |
| 21 | + | |
| 22 | + def ok() -> str: | |
| 23 | + calls["count"] += 1 | |
| 24 | + return "ok" | |
| 25 | + | |
| 26 | + assert retry_call(ok, attempts=3, delays=(0, 0, 0)) == "ok" | |
| 27 | + assert calls["count"] == 1 | |
| 28 | + | |
| 29 | + | |
| 30 | +def test_retry_call_recovers_after_failures() -> None: | |
| 31 | + """Les échecs transitoires sont relancés jusqu'au succès.""" | |
| 32 | + calls = {"count": 0} | |
| 33 | + retries: list[int] = [] | |
| 34 | + | |
| 35 | + def flaky() -> str: | |
| 36 | + calls["count"] += 1 | |
| 37 | + if calls["count"] < 3: | |
| 38 | + raise RuntimeError("transitoire") | |
| 39 | + return "ok" | |
| 40 | + | |
| 41 | + result = retry_call( | |
| 42 | + flaky, | |
| 43 | + attempts=3, | |
| 44 | + delays=(0, 0, 0), | |
| 45 | + on_retry=lambda attempt, exc, delay: retries.append(attempt), | |
| 46 | + ) | |
| 47 | + assert result == "ok" | |
| 48 | + assert calls["count"] == 3 | |
| 49 | + assert retries == [1, 2] | |
| 50 | + | |
| 51 | + | |
| 52 | +def test_retry_call_raises_after_exhaustion() -> None: | |
| 53 | + """Après épuisement des tentatives, la dernière exception est propagée.""" | |
| 54 | + calls = {"count": 0} | |
| 55 | + | |
| 56 | + def always_fails() -> None: | |
| 57 | + calls["count"] += 1 | |
| 58 | + raise ValueError("permanent") | |
| 59 | + | |
| 60 | + with pytest.raises(ValueError, match="permanent"): | |
| 61 | + retry_call(always_fails, attempts=3, delays=(0, 0, 0)) | |
| 62 | + assert calls["count"] == 4 # essai initial + 3 relances | |
| 63 | + | |
| 64 | + | |
| 65 | +def test_retry_decorator() -> None: | |
| 66 | + """Le décorateur applique la même logique que retry_call.""" | |
| 67 | + calls = {"count": 0} | |
| 68 | + | |
| 69 | + @retry(attempts=2, delays=(0, 0)) | |
| 70 | + def flaky(value: int) -> int: | |
| 71 | + calls["count"] += 1 | |
| 72 | + if calls["count"] == 1: | |
| 73 | + raise RuntimeError("transitoire") | |
| 74 | + return value * 2 | |
| 75 | + | |
| 76 | + assert flaky(21) == 42 | |
| 77 | + assert calls["count"] == 2 | |
| 78 | ||