Robustesse: réconciliation des missions fantômes (runner /mission-result + final persisté), retries courrier 3x, deploy refuse un runner occupé, transfert tar-over-ssh
Cause racine du blocage 2026-08-23 14h: kickstart des runners (rotation jeton) a tué la mission claude en vol; l'événement final n'existait pas et la file restait bloquée jusqu'au délai zombie. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
35 changed files +2,915 −385
added
M4M36luster-projects/ka-guardian/.gitignore
+4 −0
@@ -0,0 +1,4 @@ | ||
| 1 | +__pycache__/ | |
| 2 | +data/ | |
| 3 | +.venv/ | |
| 4 | +*.log | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/README.md
+72 −0
@@ -0,0 +1,72 @@ | ||
| 1 | +# KA Guardian — ka2 · ka4 · ka6 | |
| 2 | + | |
| 3 | +Les trois agents gardiens autonomes du Groupe KA. Depuis le 2026-08-23, ka2/ka4/ka6 | |
| 4 | +ne sont plus des bots de cartographie web : ce sont des **agents de maintenance | |
| 5 | +autonomes des connecteurs** des plateformes ·Ka, bâtis sur Claude (CLI headless). | |
| 6 | + | |
| 7 | +## Ce qu'ils font | |
| 8 | + | |
| 9 | +1. **Détection** — toutes les 5 min, chaque agent lit la supervision centralisée | |
| 10 | + d'api-ka (`/api/v1/monitoring/connectors`, ~2 600 connecteurs classés | |
| 11 | + ok/degraded/broken/stale toutes les 2 h). Un connecteur `broken` ou `stale` | |
| 12 | + de son périmètre → incident. | |
| 13 | +2. **Mission** — l'agent dépêche une mission au **runner** du nœud qui héberge | |
| 14 | + l'app : `claude -p` headless, lancé dans le repo de l'app, avec un prompt de | |
| 15 | + mission strict (diagnostiquer, reproduire, corriger minimalement, re-tester, | |
| 16 | + `pm2 restart` du process de sync, commit `[kaX] fix connecteur …`, jamais de push). | |
| 17 | +3. **Surveillance** — après réparation déclarée, l'incident passe en `watching` | |
| 18 | + pendant 8 h. Connecteur de retour à `ok` → résolu. | |
| 19 | +4. **Rollback automatique** — si l'app tombe (healthcheck) ou si le connecteur | |
| 20 | + est toujours cassé après la fenêtre : retour au commit d'avant mission | |
| 21 | + (`git reset --hard`/revert) + `pm2 restart`. 3 tentatives max, puis `abandoned` | |
| 22 | + (intervention humaine). | |
| 23 | +5. **Vitrine live** — chaque site (www.ka2.bot / www.ka4.bot / www.ka6.bot) est | |
| 24 | + une salle de contrôle : flux SSE en direct de chaque action de l'agent | |
| 25 | + (outils, réflexions, verdicts), board d'incidents, couverture par service, | |
| 26 | + historique des missions avec transcript et coût. | |
| 27 | + | |
| 28 | +## Répartition | |
| 29 | + | |
| 30 | +| Agent | Domaine | Services surveillés | | |
| 31 | +|---|---|---| | |
| 32 | +| ka2 | www.ka2.bot :8799 | louka, immoka, restoka | | |
| 33 | +| ka4 | www.ka4.bot :8899 | autoka, foodka, sortika | | |
| 34 | +| ka6 | www.ka6.bot :8999 | fabrika, jobka, creaka + tout nouveau service (défaut) | | |
| 35 | + | |
| 36 | +## Architecture | |
| 37 | + | |
| 38 | +- `orchestrator/` — un process FastAPI par agent (M4M36, launchd | |
| 39 | + `com.kaX.guardian`), SQLite `data/kaX.db`, dashboard servi sur le même port, | |
| 40 | + tunnels ngrok existants conservés. | |
| 41 | +- `runner/` — un service par nœud d'app (M3U96a/b, M4M64a/b, port 7791, launchd | |
| 42 | + `com.ka.guardian-runner`), 1 mission à la fois par nœud, transcripts dans | |
| 43 | + `~/ka-guardian-runner/transcripts/`. Exécute `claude -p --output-format | |
| 44 | + stream-json` et relaie chaque événement à l'orchestrateur (`/api/ingest`). | |
| 45 | +- `topology.json` — source de vérité : agents, services (nœud/dir/pm2/port), | |
| 46 | + IP LAN, politique (fenêtres, cooldowns, modèle). | |
| 47 | +- Auth interne : token partagé `~/.ka-guardian.env` (orchestrateurs + runners), | |
| 48 | + header `X-KA-Token`. Clé Anthropic + clés anti-bot : `~/.claude/.env` des nœuds. | |
| 49 | +- Communication **par le LAN 192.168.2.x** (SSH/Tailscale inter-nœuds bloqué). | |
| 50 | + | |
| 51 | +## Déploiement | |
| 52 | + | |
| 53 | +```bash | |
| 54 | +deploy/deploy.sh all # runners + orchestrateurs | |
| 55 | +deploy/deploy.sh runners # seulement les 4 runners | |
| 56 | +deploy/deploy.sh orchestrators | |
| 57 | +``` | |
| 58 | + | |
| 59 | +## Admin (token requis) | |
| 60 | + | |
| 61 | +```bash | |
| 62 | +# mission manuelle sur un connecteur | |
| 63 | +curl -X POST http://M4M36.maclustr.io:8799/api/admin/mission \ | |
| 64 | + -H "X-KA-Token: $TOKEN" -d '{"service":"louka","source":"kijiji"}' | |
| 65 | +# pause / reprise d'un agent | |
| 66 | +curl -X POST http://M4M36.maclustr.io:8799/api/admin/pause -H "X-KA-Token: $TOKEN" -d '{"paused":true}' | |
| 67 | +``` | |
| 68 | + | |
| 69 | +## Historique | |
| 70 | + | |
| 71 | +Les anciens bots (cartographie du web québécois / influenceurs) sont archivés : | |
| 72 | +`~/ka-bots-backup-20260823.tar.gz` sur M4M36 (code + bases SQLite complètes). | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/deploy.sh
+105 −0
@@ -0,0 +1,105 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# ============================================ | |
| 3 | +# KA Guardian — déploiement complet depuis le laptop | |
| 4 | +# ./deploy.sh runners → runners sur M3U96a M3U96b M4M64a M4M64b | |
| 5 | +# ./deploy.sh orchestrators → ka2/ka4/ka6 sur M4M36 (remplace les anciens bots) | |
| 6 | +# ./deploy.sh all | |
| 7 | +# Idempotent. Le token partagé vit dans ~/.ka-guardian.env (laptop) et est | |
| 8 | +# poussé sur chaque nœud. Les tunnels ngrok existants (www.kaX.bot) sont gardés. | |
| 9 | +# ============================================ | |
| 10 | +set -e | |
| 11 | +ROOT="$(cd "$(dirname "$0")/.." && pwd)" | |
| 12 | +RUNNER_NODES=(M3U96a M3U96b M4M64a M4M64b) | |
| 13 | +ORCH_NODE=M4M36 | |
| 14 | +PY=/opt/homebrew/bin/python3 | |
| 15 | + | |
| 16 | +# --- token partagé | |
| 17 | +TOKEN_FILE="$HOME/.ka-guardian.env" | |
| 18 | +if [[ ! -f $TOKEN_FILE ]]; then | |
| 19 | + echo "KA_GUARDIAN_TOKEN=$(openssl rand -hex 24)" > "$TOKEN_FILE" | |
| 20 | + echo "token généré → $TOKEN_FILE" | |
| 21 | +fi | |
| 22 | +TOKEN_LINE=$(grep KA_GUARDIAN_TOKEN "$TOKEN_FILE") | |
| 23 | + | |
| 24 | +deploy_runners() { | |
| 25 | + for n in $RUNNER_NODES; do | |
| 26 | + echo "=== runner → $n" | |
| 27 | + ssh $n "mkdir -p ~/ka-guardian-runner/transcripts" | |
| 28 | + scp -q "$ROOT/runner/runner.py" $n:ka-guardian-runner/runner.py | |
| 29 | + ssh $n "printf '%s\nKA_GUARDIAN_NODE=%s\n' '$TOKEN_LINE' '$n' > ~/.ka-guardian.env | |
| 30 | + cd ~/ka-guardian-runner | |
| 31 | + [[ -d .venv ]] || $PY -m venv .venv | |
| 32 | + ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' 'pydantic>=2' >/dev/null | |
| 33 | + cat > ~/Library/LaunchAgents/com.ka.guardian-runner.plist <<'PLIST' | |
| 34 | +<?xml version=\"1.0\" encoding=\"UTF-8\"?> | |
| 35 | +<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\"> | |
| 36 | +<plist version=\"1.0\"><dict> | |
| 37 | + <key>Label</key><string>com.ka.guardian-runner</string> | |
| 38 | + <key>ProgramArguments</key><array> | |
| 39 | + <string>/Users/simon-pierreboucher/ka-guardian-runner/.venv/bin/python</string> | |
| 40 | + <string>/Users/simon-pierreboucher/ka-guardian-runner/runner.py</string> | |
| 41 | + </array> | |
| 42 | + <key>EnvironmentVariables</key><dict> | |
| 43 | + <key>PATH</key><string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string> | |
| 44 | + </dict> | |
| 45 | + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/> | |
| 46 | + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/ka-guardian-runner/runner.log</string> | |
| 47 | + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/ka-guardian-runner/runner.log</string> | |
| 48 | +</dict></plist> | |
| 49 | +PLIST | |
| 50 | + launchctl unload ~/Library/LaunchAgents/com.ka.guardian-runner.plist 2>/dev/null || true | |
| 51 | + launchctl load ~/Library/LaunchAgents/com.ka.guardian-runner.plist" | |
| 52 | + sleep 2 | |
| 53 | + ssh $n "curl -sf localhost:7791/health" && echo " ✓ $n runner ok" || echo " ✗ $n runner KO" | |
| 54 | + done | |
| 55 | +} | |
| 56 | + | |
| 57 | +deploy_orchestrators() { | |
| 58 | + echo "=== orchestrateurs → $ORCH_NODE" | |
| 59 | + ssh $ORCH_NODE "printf '%s\n' '$TOKEN_LINE' > ~/.ka-guardian.env; mkdir -p ~/cluster-projects/ka-guardian" | |
| 60 | + rsync -az --delete --exclude data --exclude .venv --exclude .git \ | |
| 61 | + "$ROOT/" $ORCH_NODE:cluster-projects/ka-guardian/ | |
| 62 | + ssh $ORCH_NODE "cd ~/cluster-projects/ka-guardian | |
| 63 | + [[ -d .venv ]] || $PY -m venv .venv | |
| 64 | + ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' >/dev/null | |
| 65 | + mkdir -p data logs | |
| 66 | + # retirer les anciens bots (web+bot), garder les tunnels ngrok | |
| 67 | + for a in ka2 ka4 ka6; do | |
| 68 | + launchctl unload ~/Library/LaunchAgents/com.\$a.web.plist 2>/dev/null || true | |
| 69 | + launchctl unload ~/Library/LaunchAgents/com.\$a.bot.plist 2>/dev/null || true | |
| 70 | + rm -f ~/Library/LaunchAgents/com.\$a.web.plist ~/Library/LaunchAgents/com.\$a.bot.plist | |
| 71 | + done" | |
| 72 | + for a in ka2:8799 ka4:8899 ka6:8999; do | |
| 73 | + agent=${a%%:*}; port=${a##*:} | |
| 74 | + ssh $ORCH_NODE "cat > ~/Library/LaunchAgents/com.$agent.guardian.plist <<PLIST | |
| 75 | +<?xml version=\"1.0\" encoding=\"UTF-8\"?> | |
| 76 | +<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\"> | |
| 77 | +<plist version=\"1.0\"><dict> | |
| 78 | + <key>Label</key><string>com.$agent.guardian</string> | |
| 79 | + <key>ProgramArguments</key><array> | |
| 80 | + <string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/.venv/bin/python</string> | |
| 81 | + <string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/orchestrator/main.py</string> | |
| 82 | + </array> | |
| 83 | + <key>EnvironmentVariables</key><dict> | |
| 84 | + <key>AGENT</key><string>$agent</string> | |
| 85 | + <key>KA_GUARDIAN_SELF_IP</key><string>192.168.2.69</string> | |
| 86 | + <key>PATH</key><string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string> | |
| 87 | + </dict> | |
| 88 | + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/> | |
| 89 | + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/logs/$agent.log</string> | |
| 90 | + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/logs/$agent.log</string> | |
| 91 | +</dict></plist> | |
| 92 | +PLIST | |
| 93 | + launchctl unload ~/Library/LaunchAgents/com.$agent.guardian.plist 2>/dev/null || true | |
| 94 | + launchctl load ~/Library/LaunchAgents/com.$agent.guardian.plist" | |
| 95 | + sleep 2 | |
| 96 | + ssh $ORCH_NODE "curl -sf localhost:$port/health" && echo " ✓ $agent ok (:$port)" || echo " ✗ $agent KO" | |
| 97 | + done | |
| 98 | +} | |
| 99 | + | |
| 100 | +case "${1:-all}" in | |
| 101 | + runners) deploy_runners ;; | |
| 102 | + orchestrators) deploy_orchestrators ;; | |
| 103 | + all) deploy_runners; deploy_orchestrators ;; | |
| 104 | +esac | |
| 105 | +echo "terminé." | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/main.py
+560 −0
@@ -0,0 +1,560 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : KA Guardian | |
| 3 | +# Fichier : orchestrator/main.py | |
| 4 | +# Rôle : Cerveau d'un agent gardien (ka2/ka4/ka6) — surveille la santé | |
| 5 | +# des connecteurs via api-ka, ouvre des incidents, dépêche des | |
| 6 | +# missions claude aux runners des nœuds, surveille la guérison, | |
| 7 | +# rollback automatique si ça empire. Sert aussi le dashboard live. | |
| 8 | +# Author : Simon-Pierre Boucher | |
| 9 | +# Date : 2026-08-23 | |
| 10 | +# ============================================ | |
| 11 | +"""Instance unique par agent: AGENT=ka2|ka4|ka6 (env), port depuis topology.json.""" | |
| 12 | +from __future__ import annotations | |
| 13 | + | |
| 14 | +import asyncio | |
| 15 | +import json | |
| 16 | +import os | |
| 17 | +import pathlib | |
| 18 | +import sqlite3 | |
| 19 | +import time | |
| 20 | +import uuid | |
| 21 | +from typing import Any | |
| 22 | + | |
| 23 | +import httpx | |
| 24 | +from fastapi import FastAPI, Header, HTTPException, Request | |
| 25 | +from fastapi.responses import FileResponse, StreamingResponse | |
| 26 | +from fastapi.staticfiles import StaticFiles | |
| 27 | + | |
| 28 | +BASE = pathlib.Path(__file__).resolve().parent | |
| 29 | +ROOT = BASE.parent | |
| 30 | +HOME = pathlib.Path.home() | |
| 31 | + | |
| 32 | +AGENT = os.environ.get("AGENT", "ka2") | |
| 33 | +TOPO = json.loads((ROOT / "topology.json").read_text()) | |
| 34 | +ME = TOPO["agents"][AGENT] | |
| 35 | +POLICY = TOPO["policy"] | |
| 36 | +SERVICES: dict[str, Any] = TOPO["services"] | |
| 37 | +NODES: dict[str, Any] = TOPO["nodes"] | |
| 38 | +MY_SERVICES: set[str] = set(ME["services"]) | |
| 39 | +RUNNER_PORT = TOPO["runner_port"] | |
| 40 | + | |
| 41 | +# ka6 (ou autre) ramasse les services non assignés qui apparaîtraient dans api-ka. | |
| 42 | +DEFAULT_AGENT = ME.get("default_for_unknown_services", False) | |
| 43 | +ASSIGNED = {s for a in TOPO["agents"].values() for s in a["services"]} | |
| 44 | + | |
| 45 | + | |
| 46 | +def load_env_file(path: pathlib.Path) -> dict[str, str]: | |
| 47 | + out: dict[str, str] = {} | |
| 48 | + if path.exists(): | |
| 49 | + for line in path.read_text().splitlines(): | |
| 50 | + line = line.strip() | |
| 51 | + if line and not line.startswith("#") and "=" in line: | |
| 52 | + k, v = line.split("=", 1) | |
| 53 | + out[k.strip()] = v.strip().strip('"').strip("'") | |
| 54 | + return out | |
| 55 | + | |
| 56 | + | |
| 57 | +TOKEN = load_env_file(HOME / ".ka-guardian.env").get("KA_GUARDIAN_TOKEN", "") | |
| 58 | +DATA = ROOT / "data" | |
| 59 | +DATA.mkdir(exist_ok=True) | |
| 60 | +DB_PATH = DATA / f"{AGENT}.db" | |
| 61 | + | |
| 62 | +# ---------------------------------------------------------------- SQLite --- | |
| 63 | + | |
| 64 | +def db() -> sqlite3.Connection: | |
| 65 | + conn = sqlite3.connect(DB_PATH) | |
| 66 | + conn.row_factory = sqlite3.Row | |
| 67 | + conn.execute("PRAGMA journal_mode=WAL") | |
| 68 | + return conn | |
| 69 | + | |
| 70 | + | |
| 71 | +def init_db() -> None: | |
| 72 | + with db() as c: | |
| 73 | + c.executescript(""" | |
| 74 | + CREATE TABLE IF NOT EXISTS incidents( | |
| 75 | + id TEXT PRIMARY KEY, service TEXT, source TEXT, status_detected TEXT, | |
| 76 | + state TEXT, attempts INTEGER DEFAULT 0, created REAL, updated REAL, | |
| 77 | + resolved REAL, detail TEXT); | |
| 78 | + CREATE TABLE IF NOT EXISTS missions( | |
| 79 | + id TEXT PRIMARY KEY, incident_id TEXT, service TEXT, source TEXT, | |
| 80 | + node TEXT, state TEXT, base_commit TEXT, commits TEXT, verdict TEXT, | |
| 81 | + cost_usd REAL, num_turns INTEGER, health TEXT, started REAL, ended REAL); | |
| 82 | + CREATE TABLE IF NOT EXISTS events( | |
| 83 | + id INTEGER PRIMARY KEY AUTOINCREMENT, mission_id TEXT, ts REAL, | |
| 84 | + type TEXT, data TEXT); | |
| 85 | + CREATE TABLE IF NOT EXISTS snapshots( | |
| 86 | + ts REAL PRIMARY KEY, mine TEXT, ecosystem TEXT); | |
| 87 | + CREATE INDEX IF NOT EXISTS ev_mission ON events(mission_id); | |
| 88 | + """) | |
| 89 | + | |
| 90 | + | |
| 91 | +def now() -> float: | |
| 92 | + return time.time() | |
| 93 | + | |
| 94 | + | |
| 95 | +def jdump(x: Any) -> str: | |
| 96 | + return json.dumps(x, ensure_ascii=False) | |
| 97 | + | |
| 98 | + | |
| 99 | +# ------------------------------------------------------------------- SSE --- | |
| 100 | + | |
| 101 | +class Hub: | |
| 102 | + def __init__(self) -> None: | |
| 103 | + self.clients: set[asyncio.Queue] = set() | |
| 104 | + | |
| 105 | + async def publish(self, msg: dict[str, Any]) -> None: | |
| 106 | + for q in list(self.clients): | |
| 107 | + if q.qsize() < 500: | |
| 108 | + q.put_nowait(msg) | |
| 109 | + | |
| 110 | + def publish_sync(self, msg: dict[str, Any]) -> None: | |
| 111 | + if LOOP: | |
| 112 | + asyncio.run_coroutine_threadsafe(self.publish(msg), LOOP) | |
| 113 | + | |
| 114 | + | |
| 115 | +hub = Hub() | |
| 116 | +LOOP: asyncio.AbstractEventLoop | None = None | |
| 117 | +LATEST: dict[str, Any] = {"mine": {}, "ecosystem": {}, "polled": 0, "paused": False} | |
| 118 | + | |
| 119 | +# ------------------------------------------------------------- missions ---- | |
| 120 | + | |
| 121 | +def active_incident(c: sqlite3.Connection, service: str, source: str) -> sqlite3.Row | None: | |
| 122 | + return c.execute( | |
| 123 | + "SELECT * FROM incidents WHERE service=? AND source=? AND state IN " | |
| 124 | + "('open','dispatched','fixing','watching','cooldown') ORDER BY created DESC LIMIT 1", | |
| 125 | + (service, source)).fetchone() | |
| 126 | + | |
| 127 | + | |
| 128 | +def set_incident(c: sqlite3.Connection, iid: str, **kw: Any) -> None: | |
| 129 | + kw["updated"] = now() | |
| 130 | + keys = ",".join(f"{k}=?" for k in kw) | |
| 131 | + c.execute(f"UPDATE incidents SET {keys} WHERE id=?", (*kw.values(), iid)) | |
| 132 | + | |
| 133 | + | |
| 134 | +def incident_event(iid: str, service: str, source: str, state: str, note: str = "") -> None: | |
| 135 | + hub.publish_sync({"kind": "incident", "incident_id": iid, "service": service, | |
| 136 | + "source": source, "state": state, "note": note, "ts": now()}) | |
| 137 | + | |
| 138 | + | |
| 139 | +def build_prompt(service: str, source: str, health: dict[str, Any]) -> str: | |
| 140 | + svc = SERVICES[service] | |
| 141 | + sync_proc = next((p for p in svc["pm2"] if "sync" in p or "etl" in p), svc["pm2"][-1]) | |
| 142 | + node_alias = NODES[svc["node"]]["alias"] | |
| 143 | + return f"""Tu es {AGENT}, agent autonome de maintenance des connecteurs du Groupe KA. Mission: réparer le connecteur « {source} » de l'app {svc['app']} (service {service}). | |
| 144 | + | |
| 145 | +CONTEXTE SANTÉ (supervision api-ka): | |
| 146 | +- statut détecté: {health.get('status')} | échecs consécutifs: {health.get('consecutive_failures')} | |
| 147 | +- dernier succès: {health.get('last_success')} | volume dernier sync: {health.get('found_last')} (médiane: {health.get('median_found')}) | |
| 148 | +- message: {health.get('message')} | |
| 149 | + | |
| 150 | +TU ES SUR LE NŒUD {node_alias}, ton répertoire courant est le repo de l'app: {svc['dir']}. | |
| 151 | +L'app tourne via pm2 ({', '.join(svc['pm2'])}), site web local sur le port {svc['web_port']}. | |
| 152 | + | |
| 153 | +DÉMARCHE IMPOSÉE: | |
| 154 | +1. Localise le code du connecteur « {source} » (grep dans le repo). Lis ses logs récents (dossier logs/ du repo, `pm2 logs {sync_proc} --nostream --lines 200`). | |
| 155 | +2. Reproduis le problème: exécute le connecteur ou son fetch directement (utilise le venv/node_modules du repo, jamais d'installation globale). | |
| 156 | +3. Diagnostique la cause (HTML/sélecteurs changés? 403/429 anti-bot? API modifiée? URL morte? timeout?) et corrige de façon MINIMALE et ciblée. Si le site bloque, les clés anti-bot (SCRAPFLY, BRIGHTDATA, OXYLABS, SERPER) sont dans ~/.claude/.env — respecte la stratégie d'escalade déjà utilisée dans le repo. | |
| 157 | +4. Re-teste le connecteur réellement: il doit rapporter un volume plausible (ordre de la médiane). INTERDIT d'inventer ou de stubber des données. | |
| 158 | +5. Redémarre UNIQUEMENT le process concerné: `pm2 restart {sync_proc}`. Vérifie ensuite que le site répond: `curl -s -o /dev/null -w '%{{http_code}}' http://localhost:{svc['web_port']}/`. | |
| 159 | +6. Committe ton travail: `git add -A && git commit -m "[{AGENT}] fix connecteur {source}: <résumé court>"`. NE PUSH JAMAIS. | |
| 160 | + | |
| 161 | +INTERDITS ABSOLUS: toucher aux autres connecteurs ou aux autres apps du nœud, modifier la config pm2/ngrok/launchd, installer des paquets globaux, supprimer des données, git push, toucher à ~/.ssh ou aux clés API. Si la source est définitivement morte (site fermé, 404 permanent), ne force rien: documente et conclus. | |
| 162 | + | |
| 163 | +FIN DE MISSION — termine ta toute dernière réponse par un bloc JSON exactement de cette forme: | |
| 164 | +{{"verdict": "repare|echec|site_source_mort|rien_a_faire", "diagnostic": "cause racine en 1-2 phrases", "actions": "ce que tu as fait", "test": "résultat du test final (volume obtenu)", "fichiers": ["fichiers modifiés"]}}""" | |
| 165 | + | |
| 166 | + | |
| 167 | +async def dispatch(incident: sqlite3.Row, health: dict[str, Any]) -> None: | |
| 168 | + service, source, iid = incident["service"], incident["source"], incident["id"] | |
| 169 | + svc = SERVICES[service] | |
| 170 | + node = svc["node"] | |
| 171 | + ip = NODES[node]["lan_ip"] | |
| 172 | + mid = uuid.uuid4().hex[:12] | |
| 173 | + my_ip = os.environ.get("KA_GUARDIAN_SELF_IP", "192.168.2.69") | |
| 174 | + payload = { | |
| 175 | + "mission_id": mid, "agent": AGENT, "service": service, "source": source, | |
| 176 | + "dir": svc["dir"], "pm2": svc["pm2"], "web_port": svc["web_port"], | |
| 177 | + "model": ME.get("model", "sonnet"), | |
| 178 | + "max_turns": POLICY["mission_max_turns"], | |
| 179 | + "timeout_seconds": POLICY["mission_timeout_seconds"], | |
| 180 | + "prompt": build_prompt(service, source, health), | |
| 181 | + "callback_url": f"http://{my_ip}:{ME['port']}/api/ingest/{mid}", | |
| 182 | + } | |
| 183 | + try: | |
| 184 | + async with httpx.AsyncClient() as cl: | |
| 185 | + r = await cl.post(f"http://{ip}:{RUNNER_PORT}/missions", json=payload, | |
| 186 | + headers={"X-KA-Token": TOKEN}, timeout=20) | |
| 187 | + if r.status_code == 409: | |
| 188 | + return # runner occupé, on retentera au prochain tick | |
| 189 | + r.raise_for_status() | |
| 190 | + except Exception as exc: | |
| 191 | + with db() as c: | |
| 192 | + set_incident(c, iid, detail=jdump({"erreur_dispatch": str(exc)})) | |
| 193 | + return | |
| 194 | + with db() as c: | |
| 195 | + c.execute("INSERT INTO missions(id,incident_id,service,source,node,state,started) VALUES(?,?,?,?,?,?,?)", | |
| 196 | + (mid, iid, service, source, node, "running", now())) | |
| 197 | + set_incident(c, iid, state="fixing", attempts=incident["attempts"] + 1) | |
| 198 | + incident_event(iid, service, source, "fixing", f"mission {mid} dépêchée sur {NODES[node]['alias']}") | |
| 199 | + hub.publish_sync({"kind": "mission", "mission_id": mid, "service": service, | |
| 200 | + "source": source, "state": "running", "ts": now()}) | |
| 201 | + | |
| 202 | + | |
| 203 | +async def rollback_mission(mission: sqlite3.Row, reason: str) -> dict[str, Any]: | |
| 204 | + svc = SERVICES[mission["service"]] | |
| 205 | + ip = NODES[mission["node"]]["lan_ip"] | |
| 206 | + try: | |
| 207 | + async with httpx.AsyncClient() as cl: | |
| 208 | + r = await cl.post(f"http://{ip}:{RUNNER_PORT}/rollback", | |
| 209 | + json={"dir": svc["dir"], "base_commit": mission["base_commit"], | |
| 210 | + "pm2": svc["pm2"], "web_port": svc["web_port"]}, | |
| 211 | + headers={"X-KA-Token": TOKEN}, timeout=300) | |
| 212 | + out = r.json() | |
| 213 | + except Exception as exc: | |
| 214 | + out = {"ok": False, "erreur": str(exc)} | |
| 215 | + hub.publish_sync({"kind": "rollback", "mission_id": mission["id"], "service": mission["service"], | |
| 216 | + "source": mission["source"], "reason": reason, "result": out, "ts": now()}) | |
| 217 | + with db() as c: | |
| 218 | + c.execute("INSERT INTO events(mission_id,ts,type,data) VALUES(?,?,?,?)", | |
| 219 | + (mission["id"], now(), "rollback", jdump({"reason": reason, "result": out}))) | |
| 220 | + return out | |
| 221 | + | |
| 222 | + | |
| 223 | +# --------------------------------------------------------------- moteur ---- | |
| 224 | + | |
| 225 | +async def poll_once() -> None: | |
| 226 | + async with httpx.AsyncClient() as cl: | |
| 227 | + r = await cl.get(TOPO["apika_monitoring_url"], timeout=15) | |
| 228 | + data = r.json()["data"] | |
| 229 | + mine: dict[str, Any] = {} | |
| 230 | + for service, block in data["services"].items(): | |
| 231 | + assigned_to_me = service in MY_SERVICES or (DEFAULT_AGENT and service not in ASSIGNED) | |
| 232 | + if assigned_to_me: | |
| 233 | + mine[service] = block | |
| 234 | + LATEST.update({"mine": mine, "ecosystem": data["summary"], "polled": now()}) | |
| 235 | + with db() as c: | |
| 236 | + summary_mine = {s: b["summary"] for s, b in mine.items()} | |
| 237 | + c.execute("INSERT OR REPLACE INTO snapshots(ts,mine,ecosystem) VALUES(?,?,?)", | |
| 238 | + (now(), jdump(summary_mine), jdump(data["summary"]))) | |
| 239 | + c.execute("DELETE FROM snapshots WHERE ts < ?", (now() - 30 * 86400,)) | |
| 240 | + hub.publish_sync({"kind": "snapshot", "mine": {s: b["summary"] for s, b in mine.items()}, | |
| 241 | + "ecosystem": data["summary"], "ts": now()}) | |
| 242 | + | |
| 243 | + trigger = set(POLICY["trigger_statuses"]) | |
| 244 | + with db() as c: | |
| 245 | + for service, block in mine.items(): | |
| 246 | + if service not in SERVICES: | |
| 247 | + continue # service inconnu de la topologie: visible au dashboard, pas d'action | |
| 248 | + for conn in block.get("connectors", []): | |
| 249 | + source, status = conn["source"], conn["status"] | |
| 250 | + inc = active_incident(c, service, source) | |
| 251 | + if status in trigger and inc is None: | |
| 252 | + iid = uuid.uuid4().hex[:10] | |
| 253 | + c.execute("INSERT INTO incidents(id,service,source,status_detected,state,created,updated,detail) " | |
| 254 | + "VALUES(?,?,?,?,?,?,?,?)", | |
| 255 | + (iid, service, source, status, "open", now(), now(), jdump(conn))) | |
| 256 | + incident_event(iid, service, source, "open", f"détecté {status}") | |
| 257 | + elif status in trigger and inc is not None and inc["state"] in ("open", "cooldown"): | |
| 258 | + # Ne pas rafraîchir watching/fixing: `updated` sert de chrono | |
| 259 | + # à la fenêtre de surveillance et au cooldown. | |
| 260 | + c.execute("UPDATE incidents SET status_detected=?, detail=? WHERE id=?", | |
| 261 | + (status, jdump(conn), inc["id"])) | |
| 262 | + elif status == "ok" and inc is not None: | |
| 263 | + if inc["state"] == "watching": | |
| 264 | + set_incident(c, inc["id"], state="resolved", resolved=now()) | |
| 265 | + incident_event(inc["id"], service, source, "resolved", "connecteur de retour à ok — réparation confirmée") | |
| 266 | + elif inc["state"] in ("open", "cooldown"): | |
| 267 | + set_incident(c, inc["id"], state="self_healed", resolved=now()) | |
| 268 | + incident_event(inc["id"], service, source, "self_healed", "revenu à ok sans intervention") | |
| 269 | + | |
| 270 | + | |
| 271 | +async def tick() -> None: | |
| 272 | + if LATEST["paused"]: | |
| 273 | + return | |
| 274 | + try: | |
| 275 | + await poll_once() | |
| 276 | + except Exception as exc: | |
| 277 | + hub.publish_sync({"kind": "log", "level": "warn", "msg": f"poll api-ka échoué: {exc}", "ts": now()}) | |
| 278 | + | |
| 279 | + with db() as c: | |
| 280 | + # 1. Missions zombies (runner mort / callback perdu) | |
| 281 | + for m in c.execute("SELECT * FROM missions WHERE state='running' AND started < ?", | |
| 282 | + (now() - POLICY["mission_timeout_seconds"] - 900,)).fetchall(): | |
| 283 | + c.execute("UPDATE missions SET state='error', ended=? WHERE id=?", (now(), m["id"])) | |
| 284 | + inc = c.execute("SELECT * FROM incidents WHERE id=?", (m["incident_id"],)).fetchone() | |
| 285 | + if inc: | |
| 286 | + set_incident(c, inc["id"], state="cooldown") | |
| 287 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", "mission sans réponse (zombie)") | |
| 288 | + | |
| 289 | + # 2. Watching → rollback si la fenêtre est passée et toujours cassé | |
| 290 | + for inc in c.execute("SELECT * FROM incidents WHERE state='watching'").fetchall(): | |
| 291 | + if now() - inc["updated"] < POLICY["watch_window_hours"] * 3600: | |
| 292 | + continue | |
| 293 | + status = current_status(inc["service"], inc["source"]) | |
| 294 | + m = c.execute("SELECT * FROM missions WHERE incident_id=? AND base_commit IS NOT NULL " | |
| 295 | + "ORDER BY started DESC LIMIT 1", (inc["id"],)).fetchone() | |
| 296 | + if status in POLICY["trigger_statuses"] or status == "degraded": | |
| 297 | + if m and (json.loads(m["commits"] or "[]")): | |
| 298 | + asyncio.create_task(rollback_mission(m, "toujours cassé après la fenêtre de surveillance")) | |
| 299 | + set_incident(c, inc["id"], state="cooldown") | |
| 300 | + incident_event(inc["id"], inc["service"], inc["source"], "cooldown", | |
| 301 | + f"non guéri après {POLICY['watch_window_hours']}h → rollback + cooldown") | |
| 302 | + elif status == "ok": | |
| 303 | + set_incident(c, inc["id"], state="resolved", resolved=now()) | |
| 304 | + incident_event(inc["id"], inc["service"], inc["source"], "resolved", "confirmé ok") | |
| 305 | + | |
| 306 | + # 3. Cooldown expiré → réouverture ou abandon | |
| 307 | + for inc in c.execute("SELECT * FROM incidents WHERE state='cooldown'").fetchall(): | |
| 308 | + if now() - inc["updated"] < POLICY["attempt_cooldown_hours"] * 3600: | |
| 309 | + continue | |
| 310 | + status = current_status(inc["service"], inc["source"]) | |
| 311 | + if status == "ok": | |
| 312 | + set_incident(c, inc["id"], state="resolved", resolved=now()) | |
| 313 | + incident_event(inc["id"], inc["service"], inc["source"], "resolved", "guéri pendant le cooldown") | |
| 314 | + elif inc["attempts"] >= POLICY["max_attempts_per_incident"]: | |
| 315 | + set_incident(c, inc["id"], state="abandoned") | |
| 316 | + incident_event(inc["id"], inc["service"], inc["source"], "abandoned", | |
| 317 | + f"{inc['attempts']} tentatives épuisées — intervention humaine requise") | |
| 318 | + else: | |
| 319 | + set_incident(c, inc["id"], state="open") | |
| 320 | + incident_event(inc["id"], inc["service"], inc["source"], "open", "cooldown terminé, nouvelle tentative") | |
| 321 | + | |
| 322 | + # 4. Dispatch (1 mission à la fois par agent, broken avant stale, plus vieux d'abord) | |
| 323 | + running = c.execute("SELECT COUNT(*) n FROM missions WHERE state='running'").fetchone()["n"] | |
| 324 | + if running < POLICY["max_concurrent_missions"]: | |
| 325 | + busy_nodes = {m["node"] for m in c.execute("SELECT node FROM missions WHERE state='running'").fetchall()} | |
| 326 | + nxt = c.execute( | |
| 327 | + "SELECT * FROM incidents WHERE state='open' " | |
| 328 | + "ORDER BY CASE status_detected WHEN 'broken' THEN 0 WHEN 'manual' THEN 0 ELSE 1 END, created " | |
| 329 | + ).fetchall() | |
| 330 | + for inc in nxt: | |
| 331 | + if inc["service"] not in SERVICES: | |
| 332 | + continue | |
| 333 | + if SERVICES[inc["service"]]["node"] in busy_nodes: | |
| 334 | + continue | |
| 335 | + set_incident(c, inc["id"], state="dispatched") | |
| 336 | + asyncio.create_task(dispatch_row(inc["id"])) | |
| 337 | + break | |
| 338 | + | |
| 339 | + | |
| 340 | +def current_status(service: str, source: str) -> str: | |
| 341 | + block = LATEST["mine"].get(service) or {} | |
| 342 | + for conn in block.get("connectors", []): | |
| 343 | + if conn["source"] == source: | |
| 344 | + return conn["status"] | |
| 345 | + return "inconnu" | |
| 346 | + | |
| 347 | + | |
| 348 | +async def dispatch_row(iid: str) -> None: | |
| 349 | + with db() as c: | |
| 350 | + inc = c.execute("SELECT * FROM incidents WHERE id=?", (iid,)).fetchone() | |
| 351 | + if inc: | |
| 352 | + await dispatch(inc, json.loads(inc["detail"] or "{}")) | |
| 353 | + # dispatch() remet fixing; si l'appel a échoué/409, on relâche | |
| 354 | + with db() as c: | |
| 355 | + cur = c.execute("SELECT state FROM incidents WHERE id=?", (iid,)).fetchone() | |
| 356 | + if cur and cur["state"] == "dispatched": | |
| 357 | + set_incident(c, iid, state="open") | |
| 358 | + | |
| 359 | + | |
| 360 | +async def engine() -> None: | |
| 361 | + global LOOP | |
| 362 | + LOOP = asyncio.get_running_loop() | |
| 363 | + await asyncio.sleep(3) | |
| 364 | + while True: | |
| 365 | + try: | |
| 366 | + await tick() | |
| 367 | + except Exception as exc: | |
| 368 | + hub.publish_sync({"kind": "log", "level": "error", "msg": f"tick: {exc}", "ts": now()}) | |
| 369 | + await asyncio.sleep(POLICY["poll_interval_seconds"]) | |
| 370 | + | |
| 371 | + | |
| 372 | +# ------------------------------------------------------------------- app --- | |
| 373 | + | |
| 374 | +app = FastAPI(title=f"KA Guardian — {AGENT}", docs_url=None, redoc_url=None) | |
| 375 | + | |
| 376 | + | |
| 377 | +@app.on_event("startup") | |
| 378 | +async def startup() -> None: | |
| 379 | + init_db() | |
| 380 | + asyncio.create_task(engine()) | |
| 381 | + | |
| 382 | + | |
| 383 | +def check_token(tok: str | None) -> None: | |
| 384 | + if not TOKEN or tok != TOKEN: | |
| 385 | + raise HTTPException(status_code=401) | |
| 386 | + | |
| 387 | + | |
| 388 | +@app.post("/api/ingest/{mission_id}") | |
| 389 | +async def ingest(mission_id: str, req: Request, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | |
| 390 | + check_token(x_ka_token) | |
| 391 | + ev = await req.json() | |
| 392 | + etype, data = ev.get("type"), ev.get("data", {}) | |
| 393 | + with db() as c: | |
| 394 | + c.execute("INSERT INTO events(mission_id,ts,type,data) VALUES(?,?,?,?)", | |
| 395 | + (mission_id, now(), etype, jdump(data))) | |
| 396 | + m = c.execute("SELECT * FROM missions WHERE id=?", (mission_id,)).fetchone() | |
| 397 | + if m: | |
| 398 | + if etype == "start": | |
| 399 | + c.execute("UPDATE missions SET base_commit=? WHERE id=?", (data.get("base_commit"), mission_id)) | |
| 400 | + elif etype in ("final", "error"): | |
| 401 | + await finalize(c, m, etype, data) | |
| 402 | + hub.publish_sync({"kind": "mission_event", "mission_id": mission_id, "type": etype, | |
| 403 | + "data": data, "ts": now()}) | |
| 404 | + return {"ok": True} | |
| 405 | + | |
| 406 | + | |
| 407 | +async def finalize(c: sqlite3.Connection, m: sqlite3.Row, etype: str, data: dict[str, Any]) -> None: | |
| 408 | + inc = c.execute("SELECT * FROM incidents WHERE id=?", (m["incident_id"],)).fetchone() | |
| 409 | + if etype == "error": | |
| 410 | + c.execute("UPDATE missions SET state='error', ended=? WHERE id=?", (now(), m["id"])) | |
| 411 | + if inc: | |
| 412 | + set_incident(c, inc["id"], state="cooldown") | |
| 413 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", f"erreur mission: {data.get('error')}") | |
| 414 | + return | |
| 415 | + verdict = data.get("verdict", {}) | |
| 416 | + commits = data.get("commits", []) | |
| 417 | + health = data.get("health", {}) | |
| 418 | + c.execute("UPDATE missions SET state='done', ended=?, commits=?, verdict=?, cost_usd=?, num_turns=?, health=?, base_commit=COALESCE(base_commit,?) WHERE id=?", | |
| 419 | + (now(), jdump(commits), jdump(verdict), data.get("cost_usd"), | |
| 420 | + data.get("num_turns"), jdump(health), data.get("base_commit"), m["id"])) | |
| 421 | + m2 = c.execute("SELECT * FROM missions WHERE id=?", (m["id"],)).fetchone() | |
| 422 | + v = verdict.get("verdict", "inconnu") | |
| 423 | + if not inc: | |
| 424 | + return | |
| 425 | + if not health.get("ok", True): | |
| 426 | + # L'app est tombée → rollback immédiat, quoi qu'ait dit l'agent. | |
| 427 | + asyncio.create_task(rollback_mission(m2, "healthcheck app en échec post-mission")) | |
| 428 | + set_incident(c, inc["id"], state="cooldown") | |
| 429 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", "app en mauvaise santé → ROLLBACK immédiat") | |
| 430 | + elif v == "repare" and commits: | |
| 431 | + set_incident(c, inc["id"], state="watching") | |
| 432 | + incident_event(inc["id"], m["service"], m["source"], "watching", | |
| 433 | + f"réparation déclarée ({len(commits)} commit) — surveillance {POLICY['watch_window_hours']}h") | |
| 434 | + elif v in ("echec", "inconnu") and commits: | |
| 435 | + asyncio.create_task(rollback_mission(m2, f"verdict {v} avec commits → annulation par prudence")) | |
| 436 | + set_incident(c, inc["id"], state="cooldown") | |
| 437 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", f"verdict {v} → rollback préventif") | |
| 438 | + elif v == "site_source_mort": | |
| 439 | + set_incident(c, inc["id"], state="abandoned") | |
| 440 | + incident_event(inc["id"], m["service"], m["source"], "abandoned", "source définitivement morte (diagnostic agent)") | |
| 441 | + elif v in ("rien_a_faire", "repare"): | |
| 442 | + set_incident(c, inc["id"], state="watching") | |
| 443 | + incident_event(inc["id"], m["service"], m["source"], "watching", f"verdict {v} — surveillance") | |
| 444 | + else: | |
| 445 | + set_incident(c, inc["id"], state="cooldown") | |
| 446 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", f"verdict {v}") | |
| 447 | + | |
| 448 | + | |
| 449 | +@app.get("/api/state") | |
| 450 | +def state() -> dict[str, Any]: | |
| 451 | + with db() as c: | |
| 452 | + incidents = [dict(r) for r in c.execute( | |
| 453 | + "SELECT * FROM incidents ORDER BY created DESC LIMIT 200").fetchall()] | |
| 454 | + missions = [dict(r) for r in c.execute( | |
| 455 | + "SELECT * FROM missions ORDER BY started DESC LIMIT 100").fetchall()] | |
| 456 | + stats = c.execute("""SELECT | |
| 457 | + (SELECT COUNT(*) FROM missions) n_missions, | |
| 458 | + (SELECT COUNT(*) FROM incidents WHERE state='resolved') n_resolved, | |
| 459 | + (SELECT COUNT(*) FROM incidents WHERE state IN ('open','dispatched','fixing','watching','cooldown')) n_active, | |
| 460 | + (SELECT COUNT(*) FROM incidents WHERE state='abandoned') n_abandoned, | |
| 461 | + (SELECT COUNT(*) FROM events WHERE type='rollback') n_rollbacks, | |
| 462 | + (SELECT ROUND(SUM(cost_usd),2) FROM missions) cost_total, | |
| 463 | + (SELECT ROUND(AVG(resolved-created)/3600.0,1) FROM incidents WHERE state='resolved' AND resolved IS NOT NULL) mttr_h | |
| 464 | + """).fetchone() | |
| 465 | + history = [dict(r) for r in c.execute( | |
| 466 | + "SELECT ts, mine FROM snapshots WHERE ts > ? ORDER BY ts", (now() - 7 * 86400,)).fetchall()] | |
| 467 | + return { | |
| 468 | + "agent": AGENT, "identity": {k: ME[k] for k in ("domain", "accent", "accent2", "tagline", "model")}, | |
| 469 | + "services": {s: {**SERVICES[s], "node_alias": NODES[SERVICES[s]["node"]]["alias"]} for s in ME["services"]}, | |
| 470 | + "siblings": {a: {"domain": v["domain"], "accent": v["accent"], "services": v["services"]} | |
| 471 | + for a, v in TOPO["agents"].items() if a != AGENT}, | |
| 472 | + "latest": LATEST, "incidents": incidents, "missions": missions, | |
| 473 | + "stats": dict(stats), "history": history, "policy": POLICY, "now": now(), | |
| 474 | + } | |
| 475 | + | |
| 476 | + | |
| 477 | +@app.get("/api/missions/{mission_id}") | |
| 478 | +def mission_detail(mission_id: str) -> dict[str, Any]: | |
| 479 | + with db() as c: | |
| 480 | + m = c.execute("SELECT * FROM missions WHERE id=?", (mission_id,)).fetchone() | |
| 481 | + if not m: | |
| 482 | + raise HTTPException(status_code=404) | |
| 483 | + evs = [dict(r) for r in c.execute( | |
| 484 | + "SELECT ts,type,data FROM events WHERE mission_id=? ORDER BY id", (mission_id,)).fetchall()] | |
| 485 | + return {"mission": dict(m), "events": evs} | |
| 486 | + | |
| 487 | + | |
| 488 | +@app.get("/events") | |
| 489 | +async def sse(request: Request) -> StreamingResponse: | |
| 490 | + q: asyncio.Queue = asyncio.Queue() | |
| 491 | + hub.clients.add(q) | |
| 492 | + | |
| 493 | + async def gen(): | |
| 494 | + try: | |
| 495 | + yield f"data: {jdump({'kind': 'hello', 'agent': AGENT, 'ts': now()})}\n\n" | |
| 496 | + while True: | |
| 497 | + if await request.is_disconnected(): | |
| 498 | + break | |
| 499 | + try: | |
| 500 | + msg = await asyncio.wait_for(q.get(), timeout=25) | |
| 501 | + yield f"data: {jdump(msg)}\n\n" | |
| 502 | + except asyncio.TimeoutError: | |
| 503 | + yield ": keepalive\n\n" | |
| 504 | + finally: | |
| 505 | + hub.clients.discard(q) | |
| 506 | + | |
| 507 | + return StreamingResponse(gen(), media_type="text/event-stream", | |
| 508 | + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}) | |
| 509 | + | |
| 510 | + | |
| 511 | +@app.post("/api/admin/mission") | |
| 512 | +async def admin_mission(req: Request, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | |
| 513 | + """Déclenche manuellement une mission sur un connecteur (même sain).""" | |
| 514 | + check_token(x_ka_token) | |
| 515 | + body = await req.json() | |
| 516 | + service, source = body["service"], body["source"] | |
| 517 | + if service not in SERVICES: | |
| 518 | + raise HTTPException(status_code=400, detail="service inconnu") | |
| 519 | + detail = {"source": source, "status": "manual", "message": body.get("note", "mission manuelle")} | |
| 520 | + for conn in (LATEST["mine"].get(service) or {}).get("connectors", []): | |
| 521 | + if conn["source"] == source: | |
| 522 | + detail = {**conn, "status": "manual"} | |
| 523 | + with db() as c: | |
| 524 | + inc = active_incident(c, service, source) | |
| 525 | + if inc is None: | |
| 526 | + iid = uuid.uuid4().hex[:10] | |
| 527 | + c.execute("INSERT INTO incidents(id,service,source,status_detected,state,created,updated,detail) " | |
| 528 | + "VALUES(?,?,?,?,?,?,?,?)", | |
| 529 | + (iid, service, source, "manual", "open", now(), now(), jdump(detail))) | |
| 530 | + else: | |
| 531 | + iid = inc["id"] | |
| 532 | + set_incident(c, iid, state="open", status_detected="manual") | |
| 533 | + incident_event(iid, service, source, "open", "mission manuelle demandée") | |
| 534 | + return {"ok": True, "incident_id": iid} | |
| 535 | + | |
| 536 | + | |
| 537 | +@app.post("/api/admin/pause") | |
| 538 | +async def admin_pause(req: Request, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | |
| 539 | + check_token(x_ka_token) | |
| 540 | + body = await req.json() | |
| 541 | + LATEST["paused"] = bool(body.get("paused", True)) | |
| 542 | + return {"ok": True, "paused": LATEST["paused"]} | |
| 543 | + | |
| 544 | + | |
| 545 | +@app.get("/health") | |
| 546 | +def health() -> dict[str, Any]: | |
| 547 | + return {"ok": True, "agent": AGENT, "polled": LATEST["polled"], "paused": LATEST["paused"]} | |
| 548 | + | |
| 549 | + | |
| 550 | +@app.get("/") | |
| 551 | +def index() -> FileResponse: | |
| 552 | + return FileResponse(BASE / "web" / "index.html") | |
| 553 | + | |
| 554 | + | |
| 555 | +app.mount("/static", StaticFiles(directory=BASE / "web"), name="static") | |
| 556 | + | |
| 557 | + | |
| 558 | +if __name__ == "__main__": | |
| 559 | + import uvicorn | |
| 560 | + uvicorn.run(app, host="0.0.0.0", port=ME["port"]) | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/app.js
+163 −0
@@ -0,0 +1,163 @@ | ||
| 1 | +/* KA Guardian — dashboard live */ | |
| 2 | +const $ = (s) => document.querySelector(s); | |
| 3 | +const esc = (s) => String(s ?? "").replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c])); | |
| 4 | +const fmtT = (ts) => new Date(ts * 1000).toLocaleTimeString("fr-CA", { hour12: false }); | |
| 5 | +const fmtD = (ts) => new Date(ts * 1000).toLocaleString("fr-CA", { dateStyle: "short", timeStyle: "short" }); | |
| 6 | +const STATE_FR = { open: "à réparer", dispatched: "envoi…", fixing: "réparation", watching: "surveillance", | |
| 7 | + cooldown: "attente", resolved: "résolu", self_healed: "auto-guéri", abandoned: "abandonné" }; | |
| 8 | +const STATUSES = ["ok", "degraded", "broken", "stale"]; | |
| 9 | +const STATUS_FR = { ok: "ok", degraded: "dégradé", broken: "cassé", stale: "endormi" }; | |
| 10 | + | |
| 11 | +let STATE = null; | |
| 12 | + | |
| 13 | +async function load() { | |
| 14 | + const r = await fetch("/api/state"); | |
| 15 | + STATE = await r.json(); | |
| 16 | + const id = STATE.identity; | |
| 17 | + document.documentElement.style.setProperty("--accent", id.accent); | |
| 18 | + document.documentElement.style.setProperty("--accent2", id.accent2); | |
| 19 | + document.title = `${STATE.agent.toUpperCase()} — Guardian du Groupe KA`; | |
| 20 | + $("#agent-name").textContent = STATE.agent.toUpperCase() + " · GUARDIAN"; | |
| 21 | + $("#sigil").textContent = "·K" + STATE.agent.replace("ka", ""); | |
| 22 | + $("#foot-name").textContent = STATE.agent.toUpperCase(); | |
| 23 | + const apps = Object.values(STATE.services).map((s) => s.app).join(" · "); | |
| 24 | + $("#tagline").textContent = `${id.tagline} — veille sur ${apps}`; | |
| 25 | + $("#siblings").innerHTML = Object.entries(STATE.siblings) | |
| 26 | + .map(([a, v]) => `<a href="https://${v.domain}" style="color:${v.accent}"><b>${a.toUpperCase()}</b>${v.services.join(", ")}</a>`).join(""); | |
| 27 | + renderTiles(); renderIncidents(); renderCoverage(); renderMissions(); | |
| 28 | +} | |
| 29 | + | |
| 30 | +function renderTiles() { | |
| 31 | + const s = STATE.stats, mine = STATE.latest.mine || {}; | |
| 32 | + let total = 0, sick = 0; | |
| 33 | + for (const b of Object.values(mine)) for (const st of STATUSES) { | |
| 34 | + total += b.summary[st] || 0; | |
| 35 | + if (st !== "ok") sick += b.summary[st] || 0; | |
| 36 | + } | |
| 37 | + const tiles = [ | |
| 38 | + [total || "—", "connecteurs surveillés", true], | |
| 39 | + [s.n_active ?? 0, "incidents actifs"], | |
| 40 | + [s.n_missions ?? 0, "missions lancées"], | |
| 41 | + [s.n_resolved ?? 0, "réparations confirmées"], | |
| 42 | + [s.n_rollbacks ?? 0, "rollbacks"], | |
| 43 | + [s.mttr_h != null ? s.mttr_h + " h" : "—", "temps moyen de guérison"], | |
| 44 | + [s.cost_total != null ? s.cost_total + " $" : "0 $", "coût API total"], | |
| 45 | + ]; | |
| 46 | + $("#tiles").innerHTML = tiles.map(([v, l, hot]) => | |
| 47 | + `<div class="tile${hot ? " hot" : ""}"><div class="v">${esc(v)}</div><div class="l">${l}</div></div>`).join(""); | |
| 48 | +} | |
| 49 | + | |
| 50 | +function renderIncidents() { | |
| 51 | + const active = STATE.incidents.filter((i) => !["resolved", "self_healed"].includes(i.state)).slice(0, 12); | |
| 52 | + const recent = STATE.incidents.filter((i) => ["resolved", "self_healed"].includes(i.state)).slice(0, 5); | |
| 53 | + const row = (i) => `<div class="inc"> | |
| 54 | + <div><div class="src">${esc(i.source)}</div><div class="svc">${esc(i.service)} · ${esc(i.status_detected)} · tentatives ${i.attempts}</div></div> | |
| 55 | + <span class="badge b-${esc(i.state)}">${STATE_FR[i.state] || esc(i.state)}</span></div>`; | |
| 56 | + $("#incidents").innerHTML = (active.length || recent.length) | |
| 57 | + ? active.map(row).join("") + recent.map(row).join("") | |
| 58 | + : `<div class="empty">Aucun incident — tous les connecteurs assignés sont sains. L'agent veille.</div>`; | |
| 59 | +} | |
| 60 | + | |
| 61 | +function renderCoverage() { | |
| 62 | + const mine = STATE.latest.mine || {}; | |
| 63 | + $("#coverage").innerHTML = Object.keys(STATE.services).map((svc) => { | |
| 64 | + const meta = STATE.services[svc], sum = (mine[svc] || {}).summary || {}; | |
| 65 | + const total = STATUSES.reduce((a, st) => a + (sum[st] || 0), 0) || 1; | |
| 66 | + const strip = STATUSES.filter((st) => sum[st] > 0) | |
| 67 | + .map((st) => `<span class="s-${st}" style="flex:${sum[st]}" title="${STATUS_FR[st]}: ${sum[st]}"></span>`).join(""); | |
| 68 | + const counts = STATUSES.map((st) => | |
| 69 | + `<span><i class="s-${st}"></i>${sum[st] || 0} ${STATUS_FR[st]}</span>`).join(""); | |
| 70 | + return `<div class="cov"><div class="head"><span class="app">${esc(meta.app)}</span> | |
| 71 | + <span class="node">${esc(meta.node_alias)} · ${total} connecteurs</span></div> | |
| 72 | + <div class="strip">${strip || "<span style='flex:1;background:var(--line)'></span>"}</div> | |
| 73 | + <div class="counts">${counts}</div></div>`; | |
| 74 | + }).join(""); | |
| 75 | +} | |
| 76 | + | |
| 77 | +function renderMissions() { | |
| 78 | + const vd = (m) => { | |
| 79 | + if (m.state === "running") return `<span class="v-running">en cours…</span>`; | |
| 80 | + if (m.state === "error") return `<span class="v-echec">erreur</span>`; | |
| 81 | + const v = (JSON.parse(m.verdict || "{}").verdict) || "—"; | |
| 82 | + return `<span class="v-${esc(v)}">${esc(v)}</span>`; | |
| 83 | + }; | |
| 84 | + $("#missions tbody").innerHTML = STATE.missions.map((m) => { | |
| 85 | + const commits = JSON.parse(m.commits || "[]").length; | |
| 86 | + const dur = m.ended ? Math.round((m.ended - m.started) / 60) + " min" : ""; | |
| 87 | + return `<tr data-id="${esc(m.id)}"><td class="mono">${fmtD(m.started)}</td> | |
| 88 | + <td class="mono">${esc(m.source)}</td><td>${esc(m.service)}</td><td class="mono">${esc(m.node)}</td> | |
| 89 | + <td>${vd(m)}</td><td>${commits || "—"}</td><td>${m.num_turns ?? ""} ${dur ? "· " + dur : ""}</td> | |
| 90 | + <td class="mono">${m.cost_usd != null ? m.cost_usd.toFixed(2) + " $" : ""}</td></tr>`; | |
| 91 | + }).join("") || `<tr><td colspan="8" class="empty">Aucune mission encore — la première panne détectée lancera l'agent.</td></tr>`; | |
| 92 | +} | |
| 93 | + | |
| 94 | +/* ---- feed en direct ---- */ | |
| 95 | +const feed = $("#feed"); | |
| 96 | +function feedLine(cls, tag, msg, ts) { | |
| 97 | + if (feed.querySelector(".feed-empty")) feed.innerHTML = ""; | |
| 98 | + const el = document.createElement("div"); | |
| 99 | + el.className = "fe " + cls; | |
| 100 | + el.innerHTML = `<span class="t">${fmtT(ts || Date.now() / 1000)}</span><span class="tag">${esc(tag)}</span><span class="m">${esc(msg)}</span>`; | |
| 101 | + feed.appendChild(el); | |
| 102 | + while (feed.children.length > 400) feed.removeChild(feed.firstChild); | |
| 103 | + feed.scrollTop = feed.scrollHeight; | |
| 104 | +} | |
| 105 | + | |
| 106 | +let refetchTimer = null; | |
| 107 | +const refetch = () => { clearTimeout(refetchTimer); refetchTimer = setTimeout(load, 800); }; | |
| 108 | + | |
| 109 | +function connectSSE() { | |
| 110 | + const es = new EventSource("/events"); | |
| 111 | + es.onopen = () => $("#live-dot").classList.add("on"); | |
| 112 | + es.onerror = () => $("#live-dot").classList.remove("on"); | |
| 113 | + es.onmessage = (e) => { | |
| 114 | + const m = JSON.parse(e.data); | |
| 115 | + if (m.kind === "mission_event") { | |
| 116 | + const d = m.data || {}; | |
| 117 | + if (m.type === "start") feedLine("tool", "mission", `démarrage — commit de base ${(d.base_commit || "").slice(0, 8)} sur ${d.node || ""}`, m.ts); | |
| 118 | + else if (m.type === "tool") feedLine("tool", d.name || "outil", d.input || "", m.ts); | |
| 119 | + else if (m.type === "text") feedLine("text", "agent", d.text || "", m.ts); | |
| 120 | + else if (m.type === "result") feedLine("text", "résultat", (d.result || "").slice(0, 600), m.ts); | |
| 121 | + else if (m.type === "final") { feedLine("text", "verdict", JSON.stringify(d.verdict || {}), m.ts); refetch(); } | |
| 122 | + else if (m.type === "error") { feedLine("rollback", "erreur", d.error || "", m.ts); refetch(); } | |
| 123 | + else if (m.type === "rollback") feedLine("rollback", "rollback", JSON.stringify(d), m.ts); | |
| 124 | + } else if (m.kind === "incident") { | |
| 125 | + feedLine("incident", "incident", `${m.service}/${m.source} → ${STATE_FR[m.state] || m.state}${m.note ? " — " + m.note : ""}`, m.ts); | |
| 126 | + refetch(); | |
| 127 | + } else if (m.kind === "rollback") { | |
| 128 | + feedLine("rollback", "rollback", `${m.service}/${m.source} — ${m.reason} (${(m.result || {}).mode || "?"})`, m.ts); | |
| 129 | + refetch(); | |
| 130 | + } else if (m.kind === "mission") { | |
| 131 | + feedLine("tool", "mission", `nouvelle mission sur ${m.service}/${m.source}`, m.ts); | |
| 132 | + refetch(); | |
| 133 | + } else if (m.kind === "snapshot") { | |
| 134 | + if (STATE) { STATE.latest.mine = Object.fromEntries(Object.entries(m.mine).map(([s, sum]) => [s, { summary: sum }])); renderCoverage(); } | |
| 135 | + } else if (m.kind === "log") { | |
| 136 | + feedLine("incident", m.level || "log", m.msg, m.ts); | |
| 137 | + } | |
| 138 | + }; | |
| 139 | +} | |
| 140 | + | |
| 141 | +/* ---- modal mission ---- */ | |
| 142 | +document.addEventListener("click", async (e) => { | |
| 143 | + const tr = e.target.closest("#missions tbody tr[data-id]"); | |
| 144 | + if (tr) { | |
| 145 | + const r = await fetch("/api/missions/" + tr.dataset.id); | |
| 146 | + const { mission, events } = await r.json(); | |
| 147 | + const v = JSON.parse(mission.verdict || "{}"); | |
| 148 | + $("#modal-title").textContent = `Mission ${mission.id} — ${mission.source} (${mission.service})`; | |
| 149 | + $("#modal-body").innerHTML = | |
| 150 | + `<div class="meta">verdict: ${esc(v.verdict || mission.state)}\ndiagnostic: ${esc(v.diagnostic || "")}\nactions: ${esc(v.actions || "")}\ntest: ${esc(v.test || "")}\ncommit de base: ${esc(mission.base_commit || "")}\ncommits: ${esc((JSON.parse(mission.commits || "[]")).join(" | ") || "aucun")}</div>` + | |
| 151 | + events.map((ev) => { | |
| 152 | + const d = JSON.parse(ev.data || "{}"); | |
| 153 | + const txt = ev.type === "tool" ? `${d.name}: ${d.input}` : ev.type === "text" ? d.text | |
| 154 | + : ev.type === "result" ? d.result : JSON.stringify(d); | |
| 155 | + return `<div class="meta"><b>${fmtT(ev.ts)} · ${esc(ev.type)}</b>\n${esc((txt || "").slice(0, 2500))}</div>`; | |
| 156 | + }).join(""); | |
| 157 | + $("#modal").hidden = false; | |
| 158 | + } | |
| 159 | + if (e.target.id === "modal-close" || e.target.id === "modal") $("#modal").hidden = true; | |
| 160 | +}); | |
| 161 | + | |
| 162 | +load().then(connectSSE); | |
| 163 | +setInterval(load, 60000); | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/index.html
+69 −0
@@ -0,0 +1,69 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="fr"> | |
| 3 | +<head> | |
| 4 | +<meta charset="utf-8"> | |
| 5 | +<meta name="viewport" content="width=device-width, initial-scale=1"> | |
| 6 | +<title>KA Guardian</title> | |
| 7 | +<meta name="description" content="Agent autonome de maintenance des connecteurs du Groupe KA — incidents, missions et réparations en direct."> | |
| 8 | +<link rel="preconnect" href="https://fonts.googleapis.com"> | |
| 9 | +<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> | |
| 10 | +<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;700&family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet"> | |
| 11 | +<link rel="stylesheet" href="/static/style.css"> | |
| 12 | +</head> | |
| 13 | +<body> | |
| 14 | +<header class="top"> | |
| 15 | + <div class="brand"> | |
| 16 | + <div class="sigil" id="sigil">·K</div> | |
| 17 | + <div> | |
| 18 | + <h1 id="agent-name">KA·—</h1> | |
| 19 | + <p class="tagline"><span class="live-dot" id="live-dot"></span><span id="tagline">connexion…</span></p> | |
| 20 | + </div> | |
| 21 | + </div> | |
| 22 | + <nav class="siblings" id="siblings"></nav> | |
| 23 | +</header> | |
| 24 | + | |
| 25 | +<main> | |
| 26 | + <section class="tiles" id="tiles"></section> | |
| 27 | + | |
| 28 | + <section class="cols"> | |
| 29 | + <div class="panel feed-panel"> | |
| 30 | + <h2>⚡ Flux en direct <span class="hint">tout ce que l'agent fait, en temps réel</span></h2> | |
| 31 | + <div class="feed" id="feed"><div class="feed-empty">En attente d'activité…</div></div> | |
| 32 | + </div> | |
| 33 | + <div class="side"> | |
| 34 | + <div class="panel"> | |
| 35 | + <h2>🚨 Incidents</h2> | |
| 36 | + <div id="incidents" class="incidents"></div> | |
| 37 | + </div> | |
| 38 | + <div class="panel"> | |
| 39 | + <h2>🛰 Couverture</h2> | |
| 40 | + <div id="coverage" class="coverage"></div> | |
| 41 | + </div> | |
| 42 | + </div> | |
| 43 | + </section> | |
| 44 | + | |
| 45 | + <section class="panel"> | |
| 46 | + <h2>📜 Missions <span class="hint">cliquer une ligne pour le déroulé complet</span></h2> | |
| 47 | + <div class="table-wrap"> | |
| 48 | + <table id="missions"> | |
| 49 | + <thead><tr><th>Quand</th><th>Connecteur</th><th>Service</th><th>Nœud</th><th>Verdict</th><th>Commits</th><th>Tours</th><th>Coût</th></tr></thead> | |
| 50 | + <tbody></tbody> | |
| 51 | + </table> | |
| 52 | + </div> | |
| 53 | + </section> | |
| 54 | +</main> | |
| 55 | + | |
| 56 | +<footer> | |
| 57 | + <p><strong id="foot-name">KA</strong> — agent gardien autonome du <a href="https://www.groupe-ka.com">Groupe KA</a>. Il détecte les connecteurs en panne, les répare avec Claude, surveille la guérison et annule ses changements si ça empire.</p> | |
| 58 | +</footer> | |
| 59 | + | |
| 60 | +<div class="modal" id="modal" hidden> | |
| 61 | + <div class="modal-box"> | |
| 62 | + <div class="modal-head"><h3 id="modal-title">Mission</h3><button id="modal-close">✕</button></div> | |
| 63 | + <div class="modal-body" id="modal-body"></div> | |
| 64 | + </div> | |
| 65 | +</div> | |
| 66 | + | |
| 67 | +<script src="/static/app.js"></script> | |
| 68 | +</body> | |
| 69 | +</html> | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/style.css
+118 −0
@@ -0,0 +1,118 @@ | ||
| 1 | +/* KA Guardian — salle de contrôle */ | |
| 2 | +:root { | |
| 3 | + --bg: #0d1017; --bg2: #12161f; --panel: #151a24; --panel2: #1a2030; | |
| 4 | + --line: #232a3a; --ink: #e8ecf4; --ink2: #9aa5b8; --muted: #667089; | |
| 5 | + --accent: #22d3ee; --accent2: #0891b2; | |
| 6 | + --ok: #4ade80; --degraded: #fbbf24; --broken: #f87171; --stale: #60a5fa; | |
| 7 | + --mono: "JetBrains Mono", ui-monospace, SFMono-Regular, monospace; | |
| 8 | + --sans: "Space Grotesk", -apple-system, sans-serif; | |
| 9 | +} | |
| 10 | +* { box-sizing: border-box; margin: 0; } | |
| 11 | +body { background: var(--bg); color: var(--ink); font-family: var(--sans); min-height: 100vh; } | |
| 12 | +a { color: var(--accent); text-decoration: none; } | |
| 13 | + | |
| 14 | +.top { display: flex; justify-content: space-between; align-items: center; gap: 16px; flex-wrap: wrap; | |
| 15 | + padding: 22px clamp(16px, 4vw, 48px); border-bottom: 1px solid var(--line); | |
| 16 | + background: radial-gradient(1200px 300px at 20% -50%, color-mix(in oklab, var(--accent) 18%, transparent), transparent), var(--bg2); } | |
| 17 | +.brand { display: flex; gap: 16px; align-items: center; } | |
| 18 | +.sigil { width: 54px; height: 54px; border-radius: 14px; display: grid; place-items: center; | |
| 19 | + font: 700 22px var(--mono); color: #0b0e14; | |
| 20 | + background: linear-gradient(135deg, var(--accent), var(--accent2)); | |
| 21 | + box-shadow: 0 0 24px color-mix(in oklab, var(--accent) 45%, transparent); } | |
| 22 | +h1 { font-size: 26px; letter-spacing: 0.04em; } | |
| 23 | +.tagline { color: var(--ink2); font-size: 14px; display: flex; align-items: center; gap: 8px; } | |
| 24 | +.live-dot { width: 9px; height: 9px; border-radius: 50%; background: var(--muted); flex: none; } | |
| 25 | +.live-dot.on { background: var(--ok); animation: pulse 1.8s infinite; } | |
| 26 | +@keyframes pulse { 0%,100% { box-shadow: 0 0 0 0 color-mix(in oklab, var(--ok) 55%, transparent); } | |
| 27 | + 60% { box-shadow: 0 0 0 8px transparent; } } | |
| 28 | +.siblings { display: flex; gap: 10px; } | |
| 29 | +.siblings a { border: 1px solid var(--line); border-radius: 10px; padding: 7px 13px; font: 600 13px var(--mono); | |
| 30 | + color: var(--ink2); background: var(--panel); } | |
| 31 | +.siblings a b { margin-right: 6px; } | |
| 32 | + | |
| 33 | +main { padding: 22px clamp(16px, 4vw, 48px); display: grid; gap: 20px; max-width: 1500px; margin: 0 auto; } | |
| 34 | + | |
| 35 | +.tiles { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 12px; } | |
| 36 | +.tile { background: var(--panel); border: 1px solid var(--line); border-radius: 14px; padding: 14px 16px; } | |
| 37 | +.tile .v { font: 600 30px/1.15 var(--mono); } | |
| 38 | +.tile .l { color: var(--ink2); font-size: 12.5px; margin-top: 4px; letter-spacing: 0.03em; text-transform: uppercase; } | |
| 39 | +.tile.hot .v { color: var(--accent); } | |
| 40 | + | |
| 41 | +.cols { display: grid; grid-template-columns: minmax(0, 1.5fr) minmax(0, 1fr); gap: 20px; } | |
| 42 | +@media (max-width: 980px) { .cols { grid-template-columns: 1fr; } } | |
| 43 | +.side { display: grid; gap: 20px; align-content: start; } | |
| 44 | + | |
| 45 | +.panel { background: var(--panel); border: 1px solid var(--line); border-radius: 16px; padding: 18px; min-width: 0; } | |
| 46 | +.panel h2 { font-size: 16px; margin-bottom: 14px; letter-spacing: 0.02em; } | |
| 47 | +.hint { color: var(--muted); font-size: 12px; font-weight: 400; margin-left: 8px; } | |
| 48 | + | |
| 49 | +/* Flux en direct */ | |
| 50 | +.feed { font: 13px/1.55 var(--mono); background: #0a0d13; border: 1px solid var(--line); border-radius: 12px; | |
| 51 | + padding: 14px; height: 560px; overflow-y: auto; display: flex; flex-direction: column; gap: 8px; } | |
| 52 | +.feed-empty { color: var(--muted); } | |
| 53 | +.fe { display: flex; gap: 10px; align-items: baseline; animation: fadein 0.3s; } | |
| 54 | +@keyframes fadein { from { opacity: 0; transform: translateY(4px); } } | |
| 55 | +.fe .t { color: var(--muted); flex: none; font-size: 11px; } | |
| 56 | +.fe .tag { flex: none; font-size: 11px; font-weight: 600; padding: 1px 8px; border-radius: 999px; | |
| 57 | + border: 1px solid var(--line); color: var(--ink2); } | |
| 58 | +.fe.tool .tag { color: var(--accent); border-color: color-mix(in oklab, var(--accent) 40%, var(--line)); } | |
| 59 | +.fe.text .tag { color: var(--ok); } | |
| 60 | +.fe.incident .tag { color: var(--degraded); } | |
| 61 | +.fe.rollback .tag { color: var(--broken); } | |
| 62 | +.fe .m { color: var(--ink2); overflow-wrap: anywhere; white-space: pre-wrap; } | |
| 63 | +.fe.text .m { color: var(--ink); } | |
| 64 | + | |
| 65 | +/* Incidents */ | |
| 66 | +.incidents { display: grid; gap: 9px; max-height: 330px; overflow-y: auto; } | |
| 67 | +.inc { display: flex; align-items: center; gap: 10px; background: var(--panel2); border: 1px solid var(--line); | |
| 68 | + border-radius: 11px; padding: 10px 12px; font-size: 13px; } | |
| 69 | +.inc .src { font: 600 13px var(--mono); overflow-wrap: anywhere; } | |
| 70 | +.inc .svc { color: var(--muted); font-size: 11.5px; } | |
| 71 | +.badge { margin-left: auto; flex: none; font: 600 11px var(--mono); padding: 3px 9px; border-radius: 999px; } | |
| 72 | +.b-open { background: color-mix(in oklab, var(--broken) 18%, transparent); color: var(--broken); } | |
| 73 | +.b-fixing { background: color-mix(in oklab, var(--accent) 18%, transparent); color: var(--accent); } | |
| 74 | +.b-dispatched{ background: color-mix(in oklab, var(--accent) 18%, transparent); color: var(--accent); } | |
| 75 | +.b-watching { background: color-mix(in oklab, var(--stale) 18%, transparent); color: var(--stale); } | |
| 76 | +.b-cooldown { background: color-mix(in oklab, var(--degraded) 18%, transparent);color: var(--degraded); } | |
| 77 | +.b-resolved, .b-self_healed { background: color-mix(in oklab, var(--ok) 16%, transparent); color: var(--ok); } | |
| 78 | +.b-abandoned { background: #2a2f3d; color: var(--muted); } | |
| 79 | +.empty { color: var(--muted); font-size: 13px; } | |
| 80 | + | |
| 81 | +/* Couverture */ | |
| 82 | +.coverage { display: grid; gap: 12px; } | |
| 83 | +.cov { background: var(--panel2); border: 1px solid var(--line); border-radius: 12px; padding: 12px 14px; } | |
| 84 | +.cov .head { display: flex; justify-content: space-between; align-items: baseline; margin-bottom: 8px; } | |
| 85 | +.cov .app { font-weight: 700; } | |
| 86 | +.cov .node { color: var(--muted); font: 11px var(--mono); } | |
| 87 | +.strip { display: flex; height: 10px; border-radius: 6px; overflow: hidden; gap: 2px; background: var(--bg); } | |
| 88 | +.strip span { min-width: 3px; } | |
| 89 | +.s-ok { background: var(--ok); } .s-degraded { background: var(--degraded); } | |
| 90 | +.s-broken { background: var(--broken); } .s-stale { background: var(--stale); } | |
| 91 | +.counts { display: flex; gap: 12px; margin-top: 8px; flex-wrap: wrap; font: 11.5px var(--mono); color: var(--ink2); } | |
| 92 | +.counts i { display: inline-block; width: 8px; height: 8px; border-radius: 2px; margin-right: 5px; } | |
| 93 | + | |
| 94 | +/* Missions */ | |
| 95 | +.table-wrap { overflow-x: auto; } | |
| 96 | +table { width: 100%; border-collapse: collapse; font-size: 13px; } | |
| 97 | +th { text-align: left; color: var(--muted); font-size: 11.5px; text-transform: uppercase; letter-spacing: 0.05em; | |
| 98 | + padding: 8px 10px; border-bottom: 1px solid var(--line); } | |
| 99 | +td { padding: 9px 10px; border-bottom: 1px solid color-mix(in oklab, var(--line) 55%, transparent); } | |
| 100 | +tbody tr { cursor: pointer; } tbody tr:hover { background: var(--panel2); } | |
| 101 | +.mono { font-family: var(--mono); font-size: 12.5px; } | |
| 102 | +.v-repare { color: var(--ok); } .v-echec, .v-inconnu { color: var(--broken); } | |
| 103 | +.v-site_source_mort { color: var(--muted); } .v-rien_a_faire { color: var(--stale); } | |
| 104 | +.v-running { color: var(--accent); } | |
| 105 | + | |
| 106 | +footer { padding: 26px clamp(16px, 4vw, 48px); border-top: 1px solid var(--line); color: var(--ink2); | |
| 107 | + font-size: 13.5px; max-width: 1500px; margin: 0 auto; } | |
| 108 | + | |
| 109 | +/* Modal */ | |
| 110 | +.modal { position: fixed; inset: 0; background: rgba(5, 7, 11, 0.75); display: grid; place-items: center; z-index: 50; } | |
| 111 | +.modal-box { background: var(--panel); border: 1px solid var(--line); border-radius: 16px; width: min(880px, 94vw); | |
| 112 | + max-height: 86vh; display: flex; flex-direction: column; } | |
| 113 | +.modal-head { display: flex; justify-content: space-between; align-items: center; padding: 14px 18px; | |
| 114 | + border-bottom: 1px solid var(--line); } | |
| 115 | +.modal-head button { background: none; border: none; color: var(--ink2); font-size: 18px; cursor: pointer; } | |
| 116 | +.modal-body { padding: 16px 18px; overflow-y: auto; font: 12.5px/1.6 var(--mono); display: grid; gap: 8px; } | |
| 117 | +.modal-body .meta { background: var(--panel2); border-radius: 10px; padding: 10px 12px; color: var(--ink2); | |
| 118 | + white-space: pre-wrap; overflow-wrap: anywhere; } | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/runner/runner.py
+287 −0
@@ -0,0 +1,287 @@ | ||
| 1 | +# ============================================ | |
| 2 | +# Projet : KA Guardian | |
| 3 | +# Fichier : runner/runner.py | |
| 4 | +# Rôle : Bras d'exécution sur chaque nœud d'app — lance les missions | |
| 5 | +# claude -p headless dans le repo de l'app, streame les événements | |
| 6 | +# vers l'orchestrateur (ka2/ka4/ka6), et sait rollback + restart. | |
| 7 | +# Author : Simon-Pierre Boucher | |
| 8 | +# Date : 2026-08-23 | |
| 9 | +# ============================================ | |
| 10 | +"""KA Guardian Runner. | |
| 11 | + | |
| 12 | +Un seul runner par nœud (port 7791), une seule mission à la fois par nœud. | |
| 13 | +Auth: header X-KA-Token == $KA_GUARDIAN_TOKEN (fichier ~/.ka-guardian.env). | |
| 14 | +""" | |
| 15 | +from __future__ import annotations | |
| 16 | + | |
| 17 | +import json | |
| 18 | +import os | |
| 19 | +import pathlib | |
| 20 | +import shlex | |
| 21 | +import socket | |
| 22 | +import subprocess | |
| 23 | +import threading | |
| 24 | +import time | |
| 25 | +from typing import Any | |
| 26 | + | |
| 27 | +import httpx | |
| 28 | +from fastapi import FastAPI, Header, HTTPException, Request | |
| 29 | +from pydantic import BaseModel | |
| 30 | + | |
| 31 | +HOME = pathlib.Path.home() | |
| 32 | +ENV_FILE = HOME / ".ka-guardian.env" | |
| 33 | +CLAUDE_ENV_FILE = HOME / ".claude" / ".env" | |
| 34 | +TRANSCRIPTS = HOME / "ka-guardian-runner" / "transcripts" | |
| 35 | +TRANSCRIPTS.mkdir(parents=True, exist_ok=True) | |
| 36 | +CLAUDE_BIN = "/opt/homebrew/bin/claude" | |
| 37 | + | |
| 38 | + | |
| 39 | +def load_env_file(path: pathlib.Path) -> dict[str, str]: | |
| 40 | + out: dict[str, str] = {} | |
| 41 | + if path.exists(): | |
| 42 | + for line in path.read_text().splitlines(): | |
| 43 | + line = line.strip() | |
| 44 | + if line and not line.startswith("#") and "=" in line: | |
| 45 | + k, v = line.split("=", 1) | |
| 46 | + out[k.strip()] = v.strip().strip('"').strip("'") | |
| 47 | + return out | |
| 48 | + | |
| 49 | + | |
| 50 | +LOCAL_ENV = load_env_file(ENV_FILE) | |
| 51 | +TOKEN = LOCAL_ENV.get("KA_GUARDIAN_TOKEN", "") | |
| 52 | +NODE = LOCAL_ENV.get("KA_GUARDIAN_NODE", socket.gethostname()) | |
| 53 | + | |
| 54 | +app = FastAPI(title="KA Guardian Runner", docs_url=None, redoc_url=None) | |
| 55 | + | |
| 56 | +_lock = threading.Lock() | |
| 57 | +_current: dict[str, Any] = {} # mission en cours (métadonnées) | |
| 58 | + | |
| 59 | + | |
| 60 | +def check_token(x_ka_token: str | None) -> None: | |
| 61 | + if not TOKEN or x_ka_token != TOKEN: | |
| 62 | + raise HTTPException(status_code=401, detail="token invalide") | |
| 63 | + | |
| 64 | + | |
| 65 | +def expand(dir_: str) -> str: | |
| 66 | + return os.path.expanduser(dir_) | |
| 67 | + | |
| 68 | + | |
| 69 | +def git(dir_: str, *args: str) -> subprocess.CompletedProcess: | |
| 70 | + return subprocess.run( | |
| 71 | + ["git", "-C", expand(dir_), *args], | |
| 72 | + capture_output=True, text=True, timeout=120, | |
| 73 | + ) | |
| 74 | + | |
| 75 | + | |
| 76 | +def sh(cmd: str, timeout: int = 120) -> subprocess.CompletedProcess: | |
| 77 | + env = {**os.environ, "PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"} | |
| 78 | + return subprocess.run(["/bin/zsh", "-c", cmd], capture_output=True, text=True, timeout=timeout, env=env) | |
| 79 | + | |
| 80 | + | |
| 81 | +def healthcheck(web_port: int, pm2_names: list[str]) -> dict[str, Any]: | |
| 82 | + """Vérifie que l'app répond en HTTP et que ses process pm2 sont online.""" | |
| 83 | + http_ok = False | |
| 84 | + try: | |
| 85 | + r = httpx.get(f"http://127.0.0.1:{web_port}/", timeout=10, follow_redirects=True) | |
| 86 | + http_ok = r.status_code < 500 | |
| 87 | + except Exception: | |
| 88 | + http_ok = False | |
| 89 | + pm2_status: dict[str, str] = {} | |
| 90 | + try: | |
| 91 | + out = sh("pm2 jlist").stdout | |
| 92 | + procs = {p["name"]: p.get("pm2_env", {}).get("status", "?") for p in json.loads(out)} | |
| 93 | + for name in pm2_names: | |
| 94 | + pm2_status[name] = procs.get(name, "absent") | |
| 95 | + except Exception as exc: # pm2 absent ou jlist illisible | |
| 96 | + pm2_status = {n: f"inconnu ({exc})" for n in pm2_names} | |
| 97 | + ok = http_ok and all(s in ("online", "launching") for s in pm2_status.values()) | |
| 98 | + return {"ok": ok, "http_ok": http_ok, "pm2": pm2_status} | |
| 99 | + | |
| 100 | + | |
| 101 | +class MissionIn(BaseModel): | |
| 102 | + mission_id: str | |
| 103 | + agent: str | |
| 104 | + service: str | |
| 105 | + source: str | |
| 106 | + dir: str | |
| 107 | + pm2: list[str] | |
| 108 | + web_port: int | |
| 109 | + model: str = "sonnet" | |
| 110 | + max_turns: int = 70 | |
| 111 | + timeout_seconds: int = 3600 | |
| 112 | + prompt: str | |
| 113 | + callback_url: str # http://<orchestrateur>/api/ingest/<mission_id> | |
| 114 | + | |
| 115 | + | |
| 116 | +def post_event(url: str, payload: dict[str, Any]) -> None: | |
| 117 | + try: | |
| 118 | + httpx.post(url, json=payload, headers={"X-KA-Token": TOKEN}, timeout=15) | |
| 119 | + except Exception: | |
| 120 | + pass # l'orchestrateur relira le transcript au besoin | |
| 121 | + | |
| 122 | + | |
| 123 | +def condense_stream_line(obj: dict[str, Any]) -> list[dict[str, Any]]: | |
| 124 | + """Transforme une ligne stream-json de claude en événements courts pour le feed.""" | |
| 125 | + events: list[dict[str, Any]] = [] | |
| 126 | + t = obj.get("type") | |
| 127 | + if t == "system" and obj.get("subtype") == "init": | |
| 128 | + events.append({"type": "init", "data": {"model": obj.get("model", "?")}}) | |
| 129 | + elif t == "assistant": | |
| 130 | + for block in obj.get("message", {}).get("content", []): | |
| 131 | + if block.get("type") == "text" and block.get("text", "").strip(): | |
| 132 | + events.append({"type": "text", "data": {"text": block["text"][:1500]}}) | |
| 133 | + elif block.get("type") == "tool_use": | |
| 134 | + inp = json.dumps(block.get("input", {}), ensure_ascii=False) | |
| 135 | + events.append({"type": "tool", "data": {"name": block.get("name", "?"), "input": inp[:400]}}) | |
| 136 | + elif t == "result": | |
| 137 | + events.append({"type": "result", "data": { | |
| 138 | + "subtype": obj.get("subtype"), | |
| 139 | + "result": (obj.get("result") or "")[:4000], | |
| 140 | + "cost_usd": obj.get("total_cost_usd"), | |
| 141 | + "num_turns": obj.get("num_turns"), | |
| 142 | + "duration_ms": obj.get("duration_ms"), | |
| 143 | + }}) | |
| 144 | + return events | |
| 145 | + | |
| 146 | + | |
| 147 | +def extract_verdict(result_text: str) -> dict[str, Any]: | |
| 148 | + """Extrait le dernier bloc JSON {verdict: ...} de la réponse finale.""" | |
| 149 | + import re | |
| 150 | + for m in reversed(re.findall(r"\{[^{}]*\"verdict\"[\s\S]*?\}", result_text)): | |
| 151 | + try: | |
| 152 | + v = json.loads(m) | |
| 153 | + if "verdict" in v: | |
| 154 | + return v | |
| 155 | + except Exception: | |
| 156 | + continue | |
| 157 | + return {"verdict": "inconnu", "diagnostic": result_text[-500:] if result_text else ""} | |
| 158 | + | |
| 159 | + | |
| 160 | +def run_mission(m: MissionIn) -> None: | |
| 161 | + global _current | |
| 162 | + workdir = expand(m.dir) | |
| 163 | + transcript = TRANSCRIPTS / f"{m.mission_id}.jsonl" | |
| 164 | + base_commit = "" | |
| 165 | + try: | |
| 166 | + # Snapshot pré-mission : tree sale → commit de sûreté pour un rollback net. | |
| 167 | + if git(m.dir, "status", "--porcelain").stdout.strip(): | |
| 168 | + git(m.dir, "add", "-A") | |
| 169 | + git(m.dir, "commit", "-m", f"[{m.agent}] snapshot pré-mission {m.source}") | |
| 170 | + base_commit = git(m.dir, "rev-parse", "HEAD").stdout.strip() | |
| 171 | + post_event(m.callback_url, {"type": "start", "data": {"base_commit": base_commit, "node": NODE}}) | |
| 172 | + | |
| 173 | + env = { | |
| 174 | + **os.environ, | |
| 175 | + **load_env_file(CLAUDE_ENV_FILE), | |
| 176 | + "PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", | |
| 177 | + "HOME": str(HOME), | |
| 178 | + } | |
| 179 | + cmd = [ | |
| 180 | + CLAUDE_BIN, "-p", m.prompt, | |
| 181 | + "--output-format", "stream-json", "--verbose", | |
| 182 | + "--model", m.model, | |
| 183 | + "--max-turns", str(m.max_turns), | |
| 184 | + "--dangerously-skip-permissions", | |
| 185 | + ] | |
| 186 | + proc = subprocess.Popen( | |
| 187 | + cmd, cwd=workdir, env=env, | |
| 188 | + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, bufsize=1, | |
| 189 | + ) | |
| 190 | + _current["pid"] = proc.pid | |
| 191 | + result_text, cost, turns = "", None, None | |
| 192 | + deadline = time.time() + m.timeout_seconds | |
| 193 | + with transcript.open("w") as tf: | |
| 194 | + for line in proc.stdout: # type: ignore[union-attr] | |
| 195 | + tf.write(line) | |
| 196 | + if time.time() > deadline: | |
| 197 | + proc.kill() | |
| 198 | + post_event(m.callback_url, {"type": "error", "data": {"error": "timeout mission"}}) | |
| 199 | + break | |
| 200 | + line = line.strip() | |
| 201 | + if not line: | |
| 202 | + continue | |
| 203 | + try: | |
| 204 | + obj = json.loads(line) | |
| 205 | + except Exception: | |
| 206 | + continue | |
| 207 | + for ev in condense_stream_line(obj): | |
| 208 | + if ev["type"] == "result": | |
| 209 | + result_text = ev["data"].get("result", "") | |
| 210 | + cost = ev["data"].get("cost_usd") | |
| 211 | + turns = ev["data"].get("num_turns") | |
| 212 | + post_event(m.callback_url, ev) | |
| 213 | + proc.wait(timeout=60) | |
| 214 | + | |
| 215 | + commits_raw = git(m.dir, "rev-list", "--oneline", f"{base_commit}..HEAD").stdout.strip() | |
| 216 | + commits = commits_raw.splitlines() if commits_raw else [] | |
| 217 | + verdict = extract_verdict(result_text) | |
| 218 | + health = healthcheck(m.web_port, m.pm2) | |
| 219 | + post_event(m.callback_url, {"type": "final", "data": { | |
| 220 | + "verdict": verdict, "commits": commits, "base_commit": base_commit, | |
| 221 | + "cost_usd": cost, "num_turns": turns, "health": health, | |
| 222 | + "exit_code": proc.returncode, | |
| 223 | + }}) | |
| 224 | + except Exception as exc: | |
| 225 | + post_event(m.callback_url, {"type": "error", "data": {"error": str(exc), "base_commit": base_commit}}) | |
| 226 | + finally: | |
| 227 | + with _lock: | |
| 228 | + _current.clear() | |
| 229 | + | |
| 230 | + | |
| 231 | +@app.get("/health") | |
| 232 | +def health() -> dict[str, Any]: | |
| 233 | + return {"ok": True, "node": NODE, "busy": bool(_current), "current": _current.get("mission_id")} | |
| 234 | + | |
| 235 | + | |
| 236 | +@app.post("/missions") | |
| 237 | +def missions(m: MissionIn, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | |
| 238 | + check_token(x_ka_token) | |
| 239 | + workdir = expand(m.dir) | |
| 240 | + if not os.path.isdir(workdir): | |
| 241 | + raise HTTPException(status_code=400, detail=f"dir introuvable: {workdir}") | |
| 242 | + with _lock: | |
| 243 | + if _current: | |
| 244 | + raise HTTPException(status_code=409, detail=f"mission déjà en cours: {_current.get('mission_id')}") | |
| 245 | + _current.update({"mission_id": m.mission_id, "service": m.service, "source": m.source, "started": time.time()}) | |
| 246 | + threading.Thread(target=run_mission, args=(m,), daemon=True).start() | |
| 247 | + return {"accepted": True, "node": NODE} | |
| 248 | + | |
| 249 | + | |
| 250 | +class RollbackIn(BaseModel): | |
| 251 | + dir: str | |
| 252 | + base_commit: str | |
| 253 | + pm2: list[str] | |
| 254 | + web_port: int | |
| 255 | + | |
| 256 | + | |
| 257 | +@app.post("/rollback") | |
| 258 | +def rollback(r: RollbackIn, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | |
| 259 | + """Revient au commit d'avant mission (reset --hard si tous les commits sont de l'agent, sinon revert).""" | |
| 260 | + check_token(x_ka_token) | |
| 261 | + log = git(r.dir, "log", "--format=%s", f"{r.base_commit}..HEAD").stdout.strip() | |
| 262 | + msgs = log.splitlines() if log else [] | |
| 263 | + if not msgs: | |
| 264 | + mode = "aucun_commit" | |
| 265 | + elif all(s.startswith("[ka") for s in msgs): | |
| 266 | + git(r.dir, "reset", "--hard", r.base_commit) | |
| 267 | + mode = "reset_hard" | |
| 268 | + else: | |
| 269 | + rc = git(r.dir, "revert", "--no-edit", f"{r.base_commit}..HEAD") | |
| 270 | + if rc.returncode != 0: | |
| 271 | + git(r.dir, "revert", "--abort") | |
| 272 | + git(r.dir, "reset", "--hard", r.base_commit) | |
| 273 | + mode = "reset_hard(fallback)" | |
| 274 | + else: | |
| 275 | + mode = "revert" | |
| 276 | + # Pas de git clean : les process de sync de l'app écrivent en continu, | |
| 277 | + # on ne supprime jamais de fichiers non trackés apparus pendant la mission. | |
| 278 | + for name in r.pm2: | |
| 279 | + sh(f"pm2 restart {shlex.quote(name)} --update-env", timeout=180) | |
| 280 | + time.sleep(8) | |
| 281 | + h = healthcheck(r.web_port, r.pm2) | |
| 282 | + return {"ok": True, "mode": mode, "health": h, "head": git(r.dir, "rev-parse", "HEAD").stdout.strip()} | |
| 283 | + | |
| 284 | + | |
| 285 | +if __name__ == "__main__": | |
| 286 | + import uvicorn | |
| 287 | + uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("KA_GUARDIAN_RUNNER_PORT", "7791"))) | |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/topology.json
+62 −0
@@ -0,0 +1,62 @@ | ||
| 1 | +{ | |
| 2 | + "comment": "KA Guardian — topologie des agents, services et nœuds. Source de vérité partagée orchestrateurs/runners.", | |
| 3 | + "apika_monitoring_url": "http://192.168.2.82:8000/api/v1/monitoring/connectors", | |
| 4 | + "runner_port": 7791, | |
| 5 | + "agents": { | |
| 6 | + "ka2": { | |
| 7 | + "port": 8799, | |
| 8 | + "domain": "www.ka2.bot", | |
| 9 | + "accent": "#22d3ee", | |
| 10 | + "accent2": "#0891b2", | |
| 11 | + "tagline": "Gardien immobilier & local", | |
| 12 | + "model": "sonnet", | |
| 13 | + "services": ["louka", "immoka", "restoka"] | |
| 14 | + }, | |
| 15 | + "ka4": { | |
| 16 | + "port": 8899, | |
| 17 | + "domain": "www.ka4.bot", | |
| 18 | + "accent": "#fbbf24", | |
| 19 | + "accent2": "#d97706", | |
| 20 | + "tagline": "Gardien mobilité & quotidien", | |
| 21 | + "model": "sonnet", | |
| 22 | + "services": ["autoka", "foodka", "sortika"] | |
| 23 | + }, | |
| 24 | + "ka6": { | |
| 25 | + "port": 8999, | |
| 26 | + "domain": "www.ka6.bot", | |
| 27 | + "accent": "#a78bfa", | |
| 28 | + "accent2": "#7c3aed", | |
| 29 | + "tagline": "Gardien flagship — gros volumes", | |
| 30 | + "model": "sonnet", | |
| 31 | + "services": ["fabrika", "jobka", "creaka"], | |
| 32 | + "default_for_unknown_services": true | |
| 33 | + } | |
| 34 | + }, | |
| 35 | + "nodes": { | |
| 36 | + "m3u96a": { "lan_ip": "192.168.2.87", "alias": "M3U96a" }, | |
| 37 | + "m3u96b": { "lan_ip": "192.168.2.82", "alias": "M3U96b" }, | |
| 38 | + "m4m64a": { "lan_ip": "192.168.2.83", "alias": "M4M64a" }, | |
| 39 | + "m4m64b": { "lan_ip": "192.168.2.78", "alias": "M4M64b" } | |
| 40 | + }, | |
| 41 | + "services": { | |
| 42 | + "louka": { "app": "Lou·Ka", "node": "m3u96b", "dir": "~/apps/lou-ka", "pm2": ["lou-ka-web", "lou-ka-sync"], "web_port": 8095, "site": "https://www.lou-ka.com" }, | |
| 43 | + "restoka": { "app": "Resto·Ka", "node": "m3u96b", "dir": "~/apps/resto-ka", "pm2": ["resto-ka", "resto-ka-sync"], "web_port": 8115, "site": "https://www.resto-ka.com" }, | |
| 44 | + "creaka": { "app": "Créa·Ka", "node": "m3u96b", "dir": "~/apps/crea-ka", "pm2": ["crea-ka-web", "crea-ka-sync"], "web_port": 8160, "site": "https://www.crea-ka.com" }, | |
| 45 | + "immoka": { "app": "Immo·Ka", "node": "m4m64a", "dir": "~/apps/immo-ka", "pm2": ["immo-ka-web", "immo-ka-sync"], "web_port": 8096, "site": "https://www.immo-ka.com" }, | |
| 46 | + "fabrika": { "app": "Fabri·Ka", "node": "m4m64a", "dir": "~/fabri-ka", "pm2": ["fabri-ka-web", "fabri-ka-sync"], "web_port": 8097, "site": "https://www.fabri-ka.com" }, | |
| 47 | + "autoka": { "app": "Auto·Ka", "node": "m4m64b", "dir": "~/auto-ka", "pm2": ["auto-ka-web", "auto-ka-sync"], "web_port": 8095, "site": "https://www.auto-ka.com" }, | |
| 48 | + "foodka": { "app": "Food·Ka", "node": "m4m64b", "dir": "~/apps/food-ka", "pm2": ["food-ka-web", "food-ka-sync"], "web_port": 8097, "site": "https://www.food-ka.com" }, | |
| 49 | + "sortika": { "app": "Sorti·Ka", "node": "m3u96a", "dir": "~/apps/sorti-ka", "pm2": ["sorti-ka-web", "sorti-ka-sync"], "web_port": 8120, "site": "https://www.sorti-ka.com" }, | |
| 50 | + "jobka": { "app": "Job·Ka", "node": "m3u96a", "dir": "~/apps/job-ka", "pm2": ["job-ka-web", "job-ka-sync"], "web_port": 8096, "site": "https://www.job-ka.com" } | |
| 51 | + }, | |
| 52 | + "policy": { | |
| 53 | + "poll_interval_seconds": 300, | |
| 54 | + "trigger_statuses": ["broken", "stale"], | |
| 55 | + "max_concurrent_missions": 1, | |
| 56 | + "max_attempts_per_incident": 3, | |
| 57 | + "attempt_cooldown_hours": 6, | |
| 58 | + "watch_window_hours": 8, | |
| 59 | + "mission_max_turns": 70, | |
| 60 | + "mission_timeout_seconds": 3600 | |
| 61 | + } | |
| 62 | +} | |
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/deploy.sh
+2 −1
@@ -60,7 +60,8 @@ deploy_orchestrators() { | ||
| 60 | 60 | rsync -az --delete --exclude data --exclude .venv --exclude .git \ |
| 61 | 61 | "$ROOT/" $ORCH_NODE:cluster-projects/ka-guardian/ |
| 62 | 62 | ssh $ORCH_NODE "cd ~/cluster-projects/ka-guardian |
| 63 | − [[ -d .venv ]] || $PY -m venv .venv | |
| 63 | + PY=\$(command -v /opt/homebrew/bin/python3 || command -v /opt/homebrew/bin/python3.13) | |
| 64 | + [[ -d .venv ]] || \$PY -m venv .venv | |
| 64 | 65 | ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' >/dev/null |
| 65 | 66 | mkdir -p data logs |
| 66 | 67 | # retirer les anciens bots (web+bot), garder les tunnels ngrok |
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/courier.sh
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# ============================================ | |
| 3 | +# KA Guardian — courrier inter-nœuds (100 % zsh) | |
| 4 | +# macOS 26 « Local Network Privacy » refuse le trafic LAN dès que python | |
| 5 | +# (homebrew) est dans la chaîne de processus — même ses enfants ssh/curl. | |
| 6 | +# Ce service launchd (binaire responsable: zsh) expédie donc les requêtes: | |
| 7 | +# spool/outbox/<id>.job → ssh <nœud> curl http://127.0.0.1:<port><path> | |
| 8 | +# spool/done/<id>.resp ← corps de réponse + dernière ligne = code HTTP | |
| 9 | +# Format .job: ligne 1 = "<ip> <port> <path> <timeout>", reste = payload JSON. | |
| 10 | +# ============================================ | |
| 11 | +SPOOL=$HOME/ka-guardian-spool | |
| 12 | +KEY=$HOME/.ssh/ka_guardian_ed25519 | |
| 13 | +mkdir -p $SPOOL/outbox $SPOOL/done $SPOOL/tmp | |
| 14 | +source $HOME/.ka-guardian.env 2>/dev/null || true | |
| 15 | + | |
| 16 | +while true; do | |
| 17 | + for f in $SPOOL/outbox/*.job(N); do | |
| 18 | + id=${f:t:r} | |
| 19 | + hdr=$(head -1 $f) | |
| 20 | + parts=(${(z)hdr}) | |
| 21 | + ip=$parts[1]; port=$parts[2]; path=$parts[3]; tmo=${parts[4]:-30} | |
| 22 | + tail -n +2 $f | /usr/bin/ssh -i $KEY \ | |
| 23 | + -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ConnectTimeout=8 \ | |
| 24 | + -o ControlMaster=auto -o ControlPath=/tmp/kg-cm-%h -o ControlPersist=120 \ | |
| 25 | + simon-pierreboucher@$ip \ | |
| 26 | + "/usr/bin/curl -s -m $tmo -X POST http://127.0.0.1:$port$path -H 'Content-Type: application/json' -H 'X-KA-Token: $KA_GUARDIAN_TOKEN' -d @- -w '\n%{http_code}'" \ | |
| 27 | + > $SPOOL/tmp/$id.resp 2>>$SPOOL/courier.log | |
| 28 | + rc=$? | |
| 29 | + [[ $rc -ne 0 ]] && print "\ncourier_ssh_rc_$rc" >> $SPOOL/tmp/$id.resp | |
| 30 | + mv $SPOOL/tmp/$id.resp $SPOOL/done/$id.resp | |
| 31 | + rm -f $f | |
| 32 | + done | |
| 33 | + # ménage: réponses jamais réclamées depuis > 1 h | |
| 34 | + for old in $SPOOL/done/*.resp(N.mh+1); do rm -f $old; done | |
| 35 | + sleep 1 | |
| 36 | +done | |
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/deploy.sh
+30 −1
@@ -21,6 +21,34 @@ if [[ ! -f $TOKEN_FILE ]]; then | ||
| 21 | 21 | fi |
| 22 | 22 | TOKEN_LINE=$(grep KA_GUARDIAN_TOKEN "$TOKEN_FILE") |
| 23 | 23 | |
| 24 | +deploy_courier() { | |
| 25 | + # Courrier zsh sur TOUS les nœuds (orchestrateur + runners): expédie le | |
| 26 | + # trafic LAN inter-nœuds via ssh (macOS 26 Local Network Privacy). | |
| 27 | + for n in $RUNNER_NODES $ORCH_NODE; do | |
| 28 | + echo "=== courrier → $n" | |
| 29 | + scp -q "$ROOT/deploy/courier.sh" $n:ka-guardian-courier.sh | |
| 30 | + ssh $n 'chmod +x ~/ka-guardian-courier.sh | |
| 31 | + cat > ~/Library/LaunchAgents/com.ka.guardian-courier.plist <<PLIST | |
| 32 | +<?xml version="1.0" encoding="UTF-8"?> | |
| 33 | +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> | |
| 34 | +<plist version="1.0"><dict> | |
| 35 | + <key>Label</key><string>com.ka.guardian-courier</string> | |
| 36 | + <key>ProgramArguments</key><array> | |
| 37 | + <string>/bin/zsh</string> | |
| 38 | + <string>/Users/simon-pierreboucher/ka-guardian-courier.sh</string> | |
| 39 | + </array> | |
| 40 | + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/> | |
| 41 | + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/courier.log</string> | |
| 42 | + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/courier.log</string> | |
| 43 | +</dict></plist> | |
| 44 | +PLIST | |
| 45 | + mkdir -p ~/ka-guardian-spool | |
| 46 | + launchctl unload ~/Library/LaunchAgents/com.ka.guardian-courier.plist 2>/dev/null || true | |
| 47 | + launchctl load ~/Library/LaunchAgents/com.ka.guardian-courier.plist' \ | |
| 48 | + && echo " ✓ $n courrier ok" || echo " ✗ $n courrier KO" | |
| 49 | + done | |
| 50 | +} | |
| 51 | + | |
| 24 | 52 | deploy_runners() { |
| 25 | 53 | for n in $RUNNER_NODES; do |
| 26 | 54 | echo "=== runner → $n" |
@@ -99,8 +127,9 @@ PLIST | ||
| 99 | 127 | } |
| 100 | 128 | |
| 101 | 129 | case "${1:-all}" in |
| 130 | + courier) deploy_courier ;; | |
| 102 | 131 | runners) deploy_runners ;; |
| 103 | 132 | orchestrators) deploy_orchestrators ;; |
| 104 | − all) deploy_runners; deploy_orchestrators ;; | |
| 133 | + all) deploy_courier; deploy_runners; deploy_orchestrators ;; | |
| 105 | 134 | esac |
| 106 | 135 | echo "terminé." |
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/main.py
+245 −33
Large diff (352 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/app.js
+97 −38
Large diff (201 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/index.html
+108 −30
Large diff (173 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/style.css
+200 −99
Large diff (318 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/runner/runner.py
+21 −1
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/topology.json
+4 −2
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/deploy/courier.sh
+17 −8
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/deploy/deploy.sh
+13 −2
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/main.py
+127 −18
Large diff (225 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/web/app.js
+10 −60
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/commander.html
+109 −0
Large diff (109 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/commander.js
+101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/favicon-ka2.svg
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/favicon-ka4.svg
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/favicon-ka6.svg
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/web/index.html
+6 −40
Large diff (64 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/web/style.css
+21 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/runner/runner.py
+58 −3
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/topology.json
+148 −28
Large diff (205 lines) is not loaded by default to keep the page fast.Load diff
modified
deploy/courier.sh
+13 −7
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
modified
deploy/deploy.sh
+12 −4
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified
orchestrator/main.py
+49 −8
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
modified
runner/runner.py
+27 −2
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff