Robustesse: réconciliation des missions fantômes (runner /mission-result + final persisté), retries courrier 3x, deploy refuse un runner occupé, transfert tar-over-ssh
Cause racine du blocage 2026-08-23 14h: kickstart des runners (rotation jeton) a tué la mission claude en vol; l'événement final n'existait pas et la file restait bloquée jusqu'au délai zombie. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
35 changed files +2,915 −385
added
M4M36luster-projects/ka-guardian/.gitignore
+4 −0
@@ -0,0 +1,4 @@ | |||||
| 1 | +__pycache__/ | ||||
| 2 | +data/ | ||||
| 3 | +.venv/ | ||||
| 4 | +*.log | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/README.md
+72 −0
@@ -0,0 +1,72 @@ | |||||
| 1 | +# KA Guardian — ka2 · ka4 · ka6 | ||||
| 2 | + | ||||
| 3 | +Les trois agents gardiens autonomes du Groupe KA. Depuis le 2026-08-23, ka2/ka4/ka6 | ||||
| 4 | +ne sont plus des bots de cartographie web : ce sont des **agents de maintenance | ||||
| 5 | +autonomes des connecteurs** des plateformes ·Ka, bâtis sur Claude (CLI headless). | ||||
| 6 | + | ||||
| 7 | +## Ce qu'ils font | ||||
| 8 | + | ||||
| 9 | +1. **Détection** — toutes les 5 min, chaque agent lit la supervision centralisée | ||||
| 10 | + d'api-ka (`/api/v1/monitoring/connectors`, ~2 600 connecteurs classés | ||||
| 11 | + ok/degraded/broken/stale toutes les 2 h). Un connecteur `broken` ou `stale` | ||||
| 12 | + de son périmètre → incident. | ||||
| 13 | +2. **Mission** — l'agent dépêche une mission au **runner** du nœud qui héberge | ||||
| 14 | + l'app : `claude -p` headless, lancé dans le repo de l'app, avec un prompt de | ||||
| 15 | + mission strict (diagnostiquer, reproduire, corriger minimalement, re-tester, | ||||
| 16 | + `pm2 restart` du process de sync, commit `[kaX] fix connecteur …`, jamais de push). | ||||
| 17 | +3. **Surveillance** — après réparation déclarée, l'incident passe en `watching` | ||||
| 18 | + pendant 8 h. Connecteur de retour à `ok` → résolu. | ||||
| 19 | +4. **Rollback automatique** — si l'app tombe (healthcheck) ou si le connecteur | ||||
| 20 | + est toujours cassé après la fenêtre : retour au commit d'avant mission | ||||
| 21 | + (`git reset --hard`/revert) + `pm2 restart`. 3 tentatives max, puis `abandoned` | ||||
| 22 | + (intervention humaine). | ||||
| 23 | +5. **Vitrine live** — chaque site (www.ka2.bot / www.ka4.bot / www.ka6.bot) est | ||||
| 24 | + une salle de contrôle : flux SSE en direct de chaque action de l'agent | ||||
| 25 | + (outils, réflexions, verdicts), board d'incidents, couverture par service, | ||||
| 26 | + historique des missions avec transcript et coût. | ||||
| 27 | + | ||||
| 28 | +## Répartition | ||||
| 29 | + | ||||
| 30 | +| Agent | Domaine | Services surveillés | | ||||
| 31 | +|---|---|---| | ||||
| 32 | +| ka2 | www.ka2.bot :8799 | louka, immoka, restoka | | ||||
| 33 | +| ka4 | www.ka4.bot :8899 | autoka, foodka, sortika | | ||||
| 34 | +| ka6 | www.ka6.bot :8999 | fabrika, jobka, creaka + tout nouveau service (défaut) | | ||||
| 35 | + | ||||
| 36 | +## Architecture | ||||
| 37 | + | ||||
| 38 | +- `orchestrator/` — un process FastAPI par agent (M4M36, launchd | ||||
| 39 | + `com.kaX.guardian`), SQLite `data/kaX.db`, dashboard servi sur le même port, | ||||
| 40 | + tunnels ngrok existants conservés. | ||||
| 41 | +- `runner/` — un service par nœud d'app (M3U96a/b, M4M64a/b, port 7791, launchd | ||||
| 42 | + `com.ka.guardian-runner`), 1 mission à la fois par nœud, transcripts dans | ||||
| 43 | + `~/ka-guardian-runner/transcripts/`. Exécute `claude -p --output-format | ||||
| 44 | + stream-json` et relaie chaque événement à l'orchestrateur (`/api/ingest`). | ||||
| 45 | +- `topology.json` — source de vérité : agents, services (nœud/dir/pm2/port), | ||||
| 46 | + IP LAN, politique (fenêtres, cooldowns, modèle). | ||||
| 47 | +- Auth interne : token partagé `~/.ka-guardian.env` (orchestrateurs + runners), | ||||
| 48 | + header `X-KA-Token`. Clé Anthropic + clés anti-bot : `~/.claude/.env` des nœuds. | ||||
| 49 | +- Communication **par le LAN 192.168.2.x** (SSH/Tailscale inter-nœuds bloqué). | ||||
| 50 | + | ||||
| 51 | +## Déploiement | ||||
| 52 | + | ||||
| 53 | +```bash | ||||
| 54 | +deploy/deploy.sh all # runners + orchestrateurs | ||||
| 55 | +deploy/deploy.sh runners # seulement les 4 runners | ||||
| 56 | +deploy/deploy.sh orchestrators | ||||
| 57 | +``` | ||||
| 58 | + | ||||
| 59 | +## Admin (token requis) | ||||
| 60 | + | ||||
| 61 | +```bash | ||||
| 62 | +# mission manuelle sur un connecteur | ||||
| 63 | +curl -X POST http://M4M36.maclustr.io:8799/api/admin/mission \ | ||||
| 64 | + -H "X-KA-Token: $TOKEN" -d '{"service":"louka","source":"kijiji"}' | ||||
| 65 | +# pause / reprise d'un agent | ||||
| 66 | +curl -X POST http://M4M36.maclustr.io:8799/api/admin/pause -H "X-KA-Token: $TOKEN" -d '{"paused":true}' | ||||
| 67 | +``` | ||||
| 68 | + | ||||
| 69 | +## Historique | ||||
| 70 | + | ||||
| 71 | +Les anciens bots (cartographie du web québécois / influenceurs) sont archivés : | ||||
| 72 | +`~/ka-bots-backup-20260823.tar.gz` sur M4M36 (code + bases SQLite complètes). | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/deploy.sh
+105 −0
@@ -0,0 +1,105 @@ | |||||
| 1 | +#!/bin/zsh | ||||
| 2 | +# ============================================ | ||||
| 3 | +# KA Guardian — déploiement complet depuis le laptop | ||||
| 4 | +# ./deploy.sh runners → runners sur M3U96a M3U96b M4M64a M4M64b | ||||
| 5 | +# ./deploy.sh orchestrators → ka2/ka4/ka6 sur M4M36 (remplace les anciens bots) | ||||
| 6 | +# ./deploy.sh all | ||||
| 7 | +# Idempotent. Le token partagé vit dans ~/.ka-guardian.env (laptop) et est | ||||
| 8 | +# poussé sur chaque nœud. Les tunnels ngrok existants (www.kaX.bot) sont gardés. | ||||
| 9 | +# ============================================ | ||||
| 10 | +set -e | ||||
| 11 | +ROOT="$(cd "$(dirname "$0")/.." && pwd)" | ||||
| 12 | +RUNNER_NODES=(M3U96a M3U96b M4M64a M4M64b) | ||||
| 13 | +ORCH_NODE=M4M36 | ||||
| 14 | +PY=/opt/homebrew/bin/python3 | ||||
| 15 | + | ||||
| 16 | +# --- token partagé | ||||
| 17 | +TOKEN_FILE="$HOME/.ka-guardian.env" | ||||
| 18 | +if [[ ! -f $TOKEN_FILE ]]; then | ||||
| 19 | + echo "KA_GUARDIAN_TOKEN=$(openssl rand -hex 24)" > "$TOKEN_FILE" | ||||
| 20 | + echo "token généré → $TOKEN_FILE" | ||||
| 21 | +fi | ||||
| 22 | +TOKEN_LINE=$(grep KA_GUARDIAN_TOKEN "$TOKEN_FILE") | ||||
| 23 | + | ||||
| 24 | +deploy_runners() { | ||||
| 25 | + for n in $RUNNER_NODES; do | ||||
| 26 | + echo "=== runner → $n" | ||||
| 27 | + ssh $n "mkdir -p ~/ka-guardian-runner/transcripts" | ||||
| 28 | + scp -q "$ROOT/runner/runner.py" $n:ka-guardian-runner/runner.py | ||||
| 29 | + ssh $n "printf '%s\nKA_GUARDIAN_NODE=%s\n' '$TOKEN_LINE' '$n' > ~/.ka-guardian.env | ||||
| 30 | + cd ~/ka-guardian-runner | ||||
| 31 | + [[ -d .venv ]] || $PY -m venv .venv | ||||
| 32 | + ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' 'pydantic>=2' >/dev/null | ||||
| 33 | + cat > ~/Library/LaunchAgents/com.ka.guardian-runner.plist <<'PLIST' | ||||
| 34 | +<?xml version=\"1.0\" encoding=\"UTF-8\"?> | ||||
| 35 | +<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\"> | ||||
| 36 | +<plist version=\"1.0\"><dict> | ||||
| 37 | + <key>Label</key><string>com.ka.guardian-runner</string> | ||||
| 38 | + <key>ProgramArguments</key><array> | ||||
| 39 | + <string>/Users/simon-pierreboucher/ka-guardian-runner/.venv/bin/python</string> | ||||
| 40 | + <string>/Users/simon-pierreboucher/ka-guardian-runner/runner.py</string> | ||||
| 41 | + </array> | ||||
| 42 | + <key>EnvironmentVariables</key><dict> | ||||
| 43 | + <key>PATH</key><string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string> | ||||
| 44 | + </dict> | ||||
| 45 | + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/> | ||||
| 46 | + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/ka-guardian-runner/runner.log</string> | ||||
| 47 | + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/ka-guardian-runner/runner.log</string> | ||||
| 48 | +</dict></plist> | ||||
| 49 | +PLIST | ||||
| 50 | + launchctl unload ~/Library/LaunchAgents/com.ka.guardian-runner.plist 2>/dev/null || true | ||||
| 51 | + launchctl load ~/Library/LaunchAgents/com.ka.guardian-runner.plist" | ||||
| 52 | + sleep 2 | ||||
| 53 | + ssh $n "curl -sf localhost:7791/health" && echo " ✓ $n runner ok" || echo " ✗ $n runner KO" | ||||
| 54 | + done | ||||
| 55 | +} | ||||
| 56 | + | ||||
| 57 | +deploy_orchestrators() { | ||||
| 58 | + echo "=== orchestrateurs → $ORCH_NODE" | ||||
| 59 | + ssh $ORCH_NODE "printf '%s\n' '$TOKEN_LINE' > ~/.ka-guardian.env; mkdir -p ~/cluster-projects/ka-guardian" | ||||
| 60 | + rsync -az --delete --exclude data --exclude .venv --exclude .git \ | ||||
| 61 | + "$ROOT/" $ORCH_NODE:cluster-projects/ka-guardian/ | ||||
| 62 | + ssh $ORCH_NODE "cd ~/cluster-projects/ka-guardian | ||||
| 63 | + [[ -d .venv ]] || $PY -m venv .venv | ||||
| 64 | + ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' >/dev/null | ||||
| 65 | + mkdir -p data logs | ||||
| 66 | + # retirer les anciens bots (web+bot), garder les tunnels ngrok | ||||
| 67 | + for a in ka2 ka4 ka6; do | ||||
| 68 | + launchctl unload ~/Library/LaunchAgents/com.\$a.web.plist 2>/dev/null || true | ||||
| 69 | + launchctl unload ~/Library/LaunchAgents/com.\$a.bot.plist 2>/dev/null || true | ||||
| 70 | + rm -f ~/Library/LaunchAgents/com.\$a.web.plist ~/Library/LaunchAgents/com.\$a.bot.plist | ||||
| 71 | + done" | ||||
| 72 | + for a in ka2:8799 ka4:8899 ka6:8999; do | ||||
| 73 | + agent=${a%%:*}; port=${a##*:} | ||||
| 74 | + ssh $ORCH_NODE "cat > ~/Library/LaunchAgents/com.$agent.guardian.plist <<PLIST | ||||
| 75 | +<?xml version=\"1.0\" encoding=\"UTF-8\"?> | ||||
| 76 | +<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\"> | ||||
| 77 | +<plist version=\"1.0\"><dict> | ||||
| 78 | + <key>Label</key><string>com.$agent.guardian</string> | ||||
| 79 | + <key>ProgramArguments</key><array> | ||||
| 80 | + <string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/.venv/bin/python</string> | ||||
| 81 | + <string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/orchestrator/main.py</string> | ||||
| 82 | + </array> | ||||
| 83 | + <key>EnvironmentVariables</key><dict> | ||||
| 84 | + <key>AGENT</key><string>$agent</string> | ||||
| 85 | + <key>KA_GUARDIAN_SELF_IP</key><string>192.168.2.69</string> | ||||
| 86 | + <key>PATH</key><string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string> | ||||
| 87 | + </dict> | ||||
| 88 | + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/> | ||||
| 89 | + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/logs/$agent.log</string> | ||||
| 90 | + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/cluster-projects/ka-guardian/logs/$agent.log</string> | ||||
| 91 | +</dict></plist> | ||||
| 92 | +PLIST | ||||
| 93 | + launchctl unload ~/Library/LaunchAgents/com.$agent.guardian.plist 2>/dev/null || true | ||||
| 94 | + launchctl load ~/Library/LaunchAgents/com.$agent.guardian.plist" | ||||
| 95 | + sleep 2 | ||||
| 96 | + ssh $ORCH_NODE "curl -sf localhost:$port/health" && echo " ✓ $agent ok (:$port)" || echo " ✗ $agent KO" | ||||
| 97 | + done | ||||
| 98 | +} | ||||
| 99 | + | ||||
| 100 | +case "${1:-all}" in | ||||
| 101 | + runners) deploy_runners ;; | ||||
| 102 | + orchestrators) deploy_orchestrators ;; | ||||
| 103 | + all) deploy_runners; deploy_orchestrators ;; | ||||
| 104 | +esac | ||||
| 105 | +echo "terminé." | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/main.py
+560 −0
@@ -0,0 +1,560 @@ | |||||
| 1 | +# ============================================ | ||||
| 2 | +# Projet : KA Guardian | ||||
| 3 | +# Fichier : orchestrator/main.py | ||||
| 4 | +# Rôle : Cerveau d'un agent gardien (ka2/ka4/ka6) — surveille la santé | ||||
| 5 | +# des connecteurs via api-ka, ouvre des incidents, dépêche des | ||||
| 6 | +# missions claude aux runners des nœuds, surveille la guérison, | ||||
| 7 | +# rollback automatique si ça empire. Sert aussi le dashboard live. | ||||
| 8 | +# Author : Simon-Pierre Boucher | ||||
| 9 | +# Date : 2026-08-23 | ||||
| 10 | +# ============================================ | ||||
| 11 | +"""Instance unique par agent: AGENT=ka2|ka4|ka6 (env), port depuis topology.json.""" | ||||
| 12 | +from __future__ import annotations | ||||
| 13 | + | ||||
| 14 | +import asyncio | ||||
| 15 | +import json | ||||
| 16 | +import os | ||||
| 17 | +import pathlib | ||||
| 18 | +import sqlite3 | ||||
| 19 | +import time | ||||
| 20 | +import uuid | ||||
| 21 | +from typing import Any | ||||
| 22 | + | ||||
| 23 | +import httpx | ||||
| 24 | +from fastapi import FastAPI, Header, HTTPException, Request | ||||
| 25 | +from fastapi.responses import FileResponse, StreamingResponse | ||||
| 26 | +from fastapi.staticfiles import StaticFiles | ||||
| 27 | + | ||||
| 28 | +BASE = pathlib.Path(__file__).resolve().parent | ||||
| 29 | +ROOT = BASE.parent | ||||
| 30 | +HOME = pathlib.Path.home() | ||||
| 31 | + | ||||
| 32 | +AGENT = os.environ.get("AGENT", "ka2") | ||||
| 33 | +TOPO = json.loads((ROOT / "topology.json").read_text()) | ||||
| 34 | +ME = TOPO["agents"][AGENT] | ||||
| 35 | +POLICY = TOPO["policy"] | ||||
| 36 | +SERVICES: dict[str, Any] = TOPO["services"] | ||||
| 37 | +NODES: dict[str, Any] = TOPO["nodes"] | ||||
| 38 | +MY_SERVICES: set[str] = set(ME["services"]) | ||||
| 39 | +RUNNER_PORT = TOPO["runner_port"] | ||||
| 40 | + | ||||
| 41 | +# ka6 (ou autre) ramasse les services non assignés qui apparaîtraient dans api-ka. | ||||
| 42 | +DEFAULT_AGENT = ME.get("default_for_unknown_services", False) | ||||
| 43 | +ASSIGNED = {s for a in TOPO["agents"].values() for s in a["services"]} | ||||
| 44 | + | ||||
| 45 | + | ||||
| 46 | +def load_env_file(path: pathlib.Path) -> dict[str, str]: | ||||
| 47 | + out: dict[str, str] = {} | ||||
| 48 | + if path.exists(): | ||||
| 49 | + for line in path.read_text().splitlines(): | ||||
| 50 | + line = line.strip() | ||||
| 51 | + if line and not line.startswith("#") and "=" in line: | ||||
| 52 | + k, v = line.split("=", 1) | ||||
| 53 | + out[k.strip()] = v.strip().strip('"').strip("'") | ||||
| 54 | + return out | ||||
| 55 | + | ||||
| 56 | + | ||||
| 57 | +TOKEN = load_env_file(HOME / ".ka-guardian.env").get("KA_GUARDIAN_TOKEN", "") | ||||
| 58 | +DATA = ROOT / "data" | ||||
| 59 | +DATA.mkdir(exist_ok=True) | ||||
| 60 | +DB_PATH = DATA / f"{AGENT}.db" | ||||
| 61 | + | ||||
| 62 | +# ---------------------------------------------------------------- SQLite --- | ||||
| 63 | + | ||||
| 64 | +def db() -> sqlite3.Connection: | ||||
| 65 | + conn = sqlite3.connect(DB_PATH) | ||||
| 66 | + conn.row_factory = sqlite3.Row | ||||
| 67 | + conn.execute("PRAGMA journal_mode=WAL") | ||||
| 68 | + return conn | ||||
| 69 | + | ||||
| 70 | + | ||||
| 71 | +def init_db() -> None: | ||||
| 72 | + with db() as c: | ||||
| 73 | + c.executescript(""" | ||||
| 74 | + CREATE TABLE IF NOT EXISTS incidents( | ||||
| 75 | + id TEXT PRIMARY KEY, service TEXT, source TEXT, status_detected TEXT, | ||||
| 76 | + state TEXT, attempts INTEGER DEFAULT 0, created REAL, updated REAL, | ||||
| 77 | + resolved REAL, detail TEXT); | ||||
| 78 | + CREATE TABLE IF NOT EXISTS missions( | ||||
| 79 | + id TEXT PRIMARY KEY, incident_id TEXT, service TEXT, source TEXT, | ||||
| 80 | + node TEXT, state TEXT, base_commit TEXT, commits TEXT, verdict TEXT, | ||||
| 81 | + cost_usd REAL, num_turns INTEGER, health TEXT, started REAL, ended REAL); | ||||
| 82 | + CREATE TABLE IF NOT EXISTS events( | ||||
| 83 | + id INTEGER PRIMARY KEY AUTOINCREMENT, mission_id TEXT, ts REAL, | ||||
| 84 | + type TEXT, data TEXT); | ||||
| 85 | + CREATE TABLE IF NOT EXISTS snapshots( | ||||
| 86 | + ts REAL PRIMARY KEY, mine TEXT, ecosystem TEXT); | ||||
| 87 | + CREATE INDEX IF NOT EXISTS ev_mission ON events(mission_id); | ||||
| 88 | + """) | ||||
| 89 | + | ||||
| 90 | + | ||||
| 91 | +def now() -> float: | ||||
| 92 | + return time.time() | ||||
| 93 | + | ||||
| 94 | + | ||||
| 95 | +def jdump(x: Any) -> str: | ||||
| 96 | + return json.dumps(x, ensure_ascii=False) | ||||
| 97 | + | ||||
| 98 | + | ||||
| 99 | +# ------------------------------------------------------------------- SSE --- | ||||
| 100 | + | ||||
| 101 | +class Hub: | ||||
| 102 | + def __init__(self) -> None: | ||||
| 103 | + self.clients: set[asyncio.Queue] = set() | ||||
| 104 | + | ||||
| 105 | + async def publish(self, msg: dict[str, Any]) -> None: | ||||
| 106 | + for q in list(self.clients): | ||||
| 107 | + if q.qsize() < 500: | ||||
| 108 | + q.put_nowait(msg) | ||||
| 109 | + | ||||
| 110 | + def publish_sync(self, msg: dict[str, Any]) -> None: | ||||
| 111 | + if LOOP: | ||||
| 112 | + asyncio.run_coroutine_threadsafe(self.publish(msg), LOOP) | ||||
| 113 | + | ||||
| 114 | + | ||||
| 115 | +hub = Hub() | ||||
| 116 | +LOOP: asyncio.AbstractEventLoop | None = None | ||||
| 117 | +LATEST: dict[str, Any] = {"mine": {}, "ecosystem": {}, "polled": 0, "paused": False} | ||||
| 118 | + | ||||
| 119 | +# ------------------------------------------------------------- missions ---- | ||||
| 120 | + | ||||
| 121 | +def active_incident(c: sqlite3.Connection, service: str, source: str) -> sqlite3.Row | None: | ||||
| 122 | + return c.execute( | ||||
| 123 | + "SELECT * FROM incidents WHERE service=? AND source=? AND state IN " | ||||
| 124 | + "('open','dispatched','fixing','watching','cooldown') ORDER BY created DESC LIMIT 1", | ||||
| 125 | + (service, source)).fetchone() | ||||
| 126 | + | ||||
| 127 | + | ||||
| 128 | +def set_incident(c: sqlite3.Connection, iid: str, **kw: Any) -> None: | ||||
| 129 | + kw["updated"] = now() | ||||
| 130 | + keys = ",".join(f"{k}=?" for k in kw) | ||||
| 131 | + c.execute(f"UPDATE incidents SET {keys} WHERE id=?", (*kw.values(), iid)) | ||||
| 132 | + | ||||
| 133 | + | ||||
| 134 | +def incident_event(iid: str, service: str, source: str, state: str, note: str = "") -> None: | ||||
| 135 | + hub.publish_sync({"kind": "incident", "incident_id": iid, "service": service, | ||||
| 136 | + "source": source, "state": state, "note": note, "ts": now()}) | ||||
| 137 | + | ||||
| 138 | + | ||||
| 139 | +def build_prompt(service: str, source: str, health: dict[str, Any]) -> str: | ||||
| 140 | + svc = SERVICES[service] | ||||
| 141 | + sync_proc = next((p for p in svc["pm2"] if "sync" in p or "etl" in p), svc["pm2"][-1]) | ||||
| 142 | + node_alias = NODES[svc["node"]]["alias"] | ||||
| 143 | + return f"""Tu es {AGENT}, agent autonome de maintenance des connecteurs du Groupe KA. Mission: réparer le connecteur « {source} » de l'app {svc['app']} (service {service}). | ||||
| 144 | + | ||||
| 145 | +CONTEXTE SANTÉ (supervision api-ka): | ||||
| 146 | +- statut détecté: {health.get('status')} | échecs consécutifs: {health.get('consecutive_failures')} | ||||
| 147 | +- dernier succès: {health.get('last_success')} | volume dernier sync: {health.get('found_last')} (médiane: {health.get('median_found')}) | ||||
| 148 | +- message: {health.get('message')} | ||||
| 149 | + | ||||
| 150 | +TU ES SUR LE NŒUD {node_alias}, ton répertoire courant est le repo de l'app: {svc['dir']}. | ||||
| 151 | +L'app tourne via pm2 ({', '.join(svc['pm2'])}), site web local sur le port {svc['web_port']}. | ||||
| 152 | + | ||||
| 153 | +DÉMARCHE IMPOSÉE: | ||||
| 154 | +1. Localise le code du connecteur « {source} » (grep dans le repo). Lis ses logs récents (dossier logs/ du repo, `pm2 logs {sync_proc} --nostream --lines 200`). | ||||
| 155 | +2. Reproduis le problème: exécute le connecteur ou son fetch directement (utilise le venv/node_modules du repo, jamais d'installation globale). | ||||
| 156 | +3. Diagnostique la cause (HTML/sélecteurs changés? 403/429 anti-bot? API modifiée? URL morte? timeout?) et corrige de façon MINIMALE et ciblée. Si le site bloque, les clés anti-bot (SCRAPFLY, BRIGHTDATA, OXYLABS, SERPER) sont dans ~/.claude/.env — respecte la stratégie d'escalade déjà utilisée dans le repo. | ||||
| 157 | +4. Re-teste le connecteur réellement: il doit rapporter un volume plausible (ordre de la médiane). INTERDIT d'inventer ou de stubber des données. | ||||
| 158 | +5. Redémarre UNIQUEMENT le process concerné: `pm2 restart {sync_proc}`. Vérifie ensuite que le site répond: `curl -s -o /dev/null -w '%{{http_code}}' http://localhost:{svc['web_port']}/`. | ||||
| 159 | +6. Committe ton travail: `git add -A && git commit -m "[{AGENT}] fix connecteur {source}: <résumé court>"`. NE PUSH JAMAIS. | ||||
| 160 | + | ||||
| 161 | +INTERDITS ABSOLUS: toucher aux autres connecteurs ou aux autres apps du nœud, modifier la config pm2/ngrok/launchd, installer des paquets globaux, supprimer des données, git push, toucher à ~/.ssh ou aux clés API. Si la source est définitivement morte (site fermé, 404 permanent), ne force rien: documente et conclus. | ||||
| 162 | + | ||||
| 163 | +FIN DE MISSION — termine ta toute dernière réponse par un bloc JSON exactement de cette forme: | ||||
| 164 | +{{"verdict": "repare|echec|site_source_mort|rien_a_faire", "diagnostic": "cause racine en 1-2 phrases", "actions": "ce que tu as fait", "test": "résultat du test final (volume obtenu)", "fichiers": ["fichiers modifiés"]}}""" | ||||
| 165 | + | ||||
| 166 | + | ||||
| 167 | +async def dispatch(incident: sqlite3.Row, health: dict[str, Any]) -> None: | ||||
| 168 | + service, source, iid = incident["service"], incident["source"], incident["id"] | ||||
| 169 | + svc = SERVICES[service] | ||||
| 170 | + node = svc["node"] | ||||
| 171 | + ip = NODES[node]["lan_ip"] | ||||
| 172 | + mid = uuid.uuid4().hex[:12] | ||||
| 173 | + my_ip = os.environ.get("KA_GUARDIAN_SELF_IP", "192.168.2.69") | ||||
| 174 | + payload = { | ||||
| 175 | + "mission_id": mid, "agent": AGENT, "service": service, "source": source, | ||||
| 176 | + "dir": svc["dir"], "pm2": svc["pm2"], "web_port": svc["web_port"], | ||||
| 177 | + "model": ME.get("model", "sonnet"), | ||||
| 178 | + "max_turns": POLICY["mission_max_turns"], | ||||
| 179 | + "timeout_seconds": POLICY["mission_timeout_seconds"], | ||||
| 180 | + "prompt": build_prompt(service, source, health), | ||||
| 181 | + "callback_url": f"http://{my_ip}:{ME['port']}/api/ingest/{mid}", | ||||
| 182 | + } | ||||
| 183 | + try: | ||||
| 184 | + async with httpx.AsyncClient() as cl: | ||||
| 185 | + r = await cl.post(f"http://{ip}:{RUNNER_PORT}/missions", json=payload, | ||||
| 186 | + headers={"X-KA-Token": TOKEN}, timeout=20) | ||||
| 187 | + if r.status_code == 409: | ||||
| 188 | + return # runner occupé, on retentera au prochain tick | ||||
| 189 | + r.raise_for_status() | ||||
| 190 | + except Exception as exc: | ||||
| 191 | + with db() as c: | ||||
| 192 | + set_incident(c, iid, detail=jdump({"erreur_dispatch": str(exc)})) | ||||
| 193 | + return | ||||
| 194 | + with db() as c: | ||||
| 195 | + c.execute("INSERT INTO missions(id,incident_id,service,source,node,state,started) VALUES(?,?,?,?,?,?,?)", | ||||
| 196 | + (mid, iid, service, source, node, "running", now())) | ||||
| 197 | + set_incident(c, iid, state="fixing", attempts=incident["attempts"] + 1) | ||||
| 198 | + incident_event(iid, service, source, "fixing", f"mission {mid} dépêchée sur {NODES[node]['alias']}") | ||||
| 199 | + hub.publish_sync({"kind": "mission", "mission_id": mid, "service": service, | ||||
| 200 | + "source": source, "state": "running", "ts": now()}) | ||||
| 201 | + | ||||
| 202 | + | ||||
| 203 | +async def rollback_mission(mission: sqlite3.Row, reason: str) -> dict[str, Any]: | ||||
| 204 | + svc = SERVICES[mission["service"]] | ||||
| 205 | + ip = NODES[mission["node"]]["lan_ip"] | ||||
| 206 | + try: | ||||
| 207 | + async with httpx.AsyncClient() as cl: | ||||
| 208 | + r = await cl.post(f"http://{ip}:{RUNNER_PORT}/rollback", | ||||
| 209 | + json={"dir": svc["dir"], "base_commit": mission["base_commit"], | ||||
| 210 | + "pm2": svc["pm2"], "web_port": svc["web_port"]}, | ||||
| 211 | + headers={"X-KA-Token": TOKEN}, timeout=300) | ||||
| 212 | + out = r.json() | ||||
| 213 | + except Exception as exc: | ||||
| 214 | + out = {"ok": False, "erreur": str(exc)} | ||||
| 215 | + hub.publish_sync({"kind": "rollback", "mission_id": mission["id"], "service": mission["service"], | ||||
| 216 | + "source": mission["source"], "reason": reason, "result": out, "ts": now()}) | ||||
| 217 | + with db() as c: | ||||
| 218 | + c.execute("INSERT INTO events(mission_id,ts,type,data) VALUES(?,?,?,?)", | ||||
| 219 | + (mission["id"], now(), "rollback", jdump({"reason": reason, "result": out}))) | ||||
| 220 | + return out | ||||
| 221 | + | ||||
| 222 | + | ||||
| 223 | +# --------------------------------------------------------------- moteur ---- | ||||
| 224 | + | ||||
| 225 | +async def poll_once() -> None: | ||||
| 226 | + async with httpx.AsyncClient() as cl: | ||||
| 227 | + r = await cl.get(TOPO["apika_monitoring_url"], timeout=15) | ||||
| 228 | + data = r.json()["data"] | ||||
| 229 | + mine: dict[str, Any] = {} | ||||
| 230 | + for service, block in data["services"].items(): | ||||
| 231 | + assigned_to_me = service in MY_SERVICES or (DEFAULT_AGENT and service not in ASSIGNED) | ||||
| 232 | + if assigned_to_me: | ||||
| 233 | + mine[service] = block | ||||
| 234 | + LATEST.update({"mine": mine, "ecosystem": data["summary"], "polled": now()}) | ||||
| 235 | + with db() as c: | ||||
| 236 | + summary_mine = {s: b["summary"] for s, b in mine.items()} | ||||
| 237 | + c.execute("INSERT OR REPLACE INTO snapshots(ts,mine,ecosystem) VALUES(?,?,?)", | ||||
| 238 | + (now(), jdump(summary_mine), jdump(data["summary"]))) | ||||
| 239 | + c.execute("DELETE FROM snapshots WHERE ts < ?", (now() - 30 * 86400,)) | ||||
| 240 | + hub.publish_sync({"kind": "snapshot", "mine": {s: b["summary"] for s, b in mine.items()}, | ||||
| 241 | + "ecosystem": data["summary"], "ts": now()}) | ||||
| 242 | + | ||||
| 243 | + trigger = set(POLICY["trigger_statuses"]) | ||||
| 244 | + with db() as c: | ||||
| 245 | + for service, block in mine.items(): | ||||
| 246 | + if service not in SERVICES: | ||||
| 247 | + continue # service inconnu de la topologie: visible au dashboard, pas d'action | ||||
| 248 | + for conn in block.get("connectors", []): | ||||
| 249 | + source, status = conn["source"], conn["status"] | ||||
| 250 | + inc = active_incident(c, service, source) | ||||
| 251 | + if status in trigger and inc is None: | ||||
| 252 | + iid = uuid.uuid4().hex[:10] | ||||
| 253 | + c.execute("INSERT INTO incidents(id,service,source,status_detected,state,created,updated,detail) " | ||||
| 254 | + "VALUES(?,?,?,?,?,?,?,?)", | ||||
| 255 | + (iid, service, source, status, "open", now(), now(), jdump(conn))) | ||||
| 256 | + incident_event(iid, service, source, "open", f"détecté {status}") | ||||
| 257 | + elif status in trigger and inc is not None and inc["state"] in ("open", "cooldown"): | ||||
| 258 | + # Ne pas rafraîchir watching/fixing: `updated` sert de chrono | ||||
| 259 | + # à la fenêtre de surveillance et au cooldown. | ||||
| 260 | + c.execute("UPDATE incidents SET status_detected=?, detail=? WHERE id=?", | ||||
| 261 | + (status, jdump(conn), inc["id"])) | ||||
| 262 | + elif status == "ok" and inc is not None: | ||||
| 263 | + if inc["state"] == "watching": | ||||
| 264 | + set_incident(c, inc["id"], state="resolved", resolved=now()) | ||||
| 265 | + incident_event(inc["id"], service, source, "resolved", "connecteur de retour à ok — réparation confirmée") | ||||
| 266 | + elif inc["state"] in ("open", "cooldown"): | ||||
| 267 | + set_incident(c, inc["id"], state="self_healed", resolved=now()) | ||||
| 268 | + incident_event(inc["id"], service, source, "self_healed", "revenu à ok sans intervention") | ||||
| 269 | + | ||||
| 270 | + | ||||
| 271 | +async def tick() -> None: | ||||
| 272 | + if LATEST["paused"]: | ||||
| 273 | + return | ||||
| 274 | + try: | ||||
| 275 | + await poll_once() | ||||
| 276 | + except Exception as exc: | ||||
| 277 | + hub.publish_sync({"kind": "log", "level": "warn", "msg": f"poll api-ka échoué: {exc}", "ts": now()}) | ||||
| 278 | + | ||||
| 279 | + with db() as c: | ||||
| 280 | + # 1. Missions zombies (runner mort / callback perdu) | ||||
| 281 | + for m in c.execute("SELECT * FROM missions WHERE state='running' AND started < ?", | ||||
| 282 | + (now() - POLICY["mission_timeout_seconds"] - 900,)).fetchall(): | ||||
| 283 | + c.execute("UPDATE missions SET state='error', ended=? WHERE id=?", (now(), m["id"])) | ||||
| 284 | + inc = c.execute("SELECT * FROM incidents WHERE id=?", (m["incident_id"],)).fetchone() | ||||
| 285 | + if inc: | ||||
| 286 | + set_incident(c, inc["id"], state="cooldown") | ||||
| 287 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", "mission sans réponse (zombie)") | ||||
| 288 | + | ||||
| 289 | + # 2. Watching → rollback si la fenêtre est passée et toujours cassé | ||||
| 290 | + for inc in c.execute("SELECT * FROM incidents WHERE state='watching'").fetchall(): | ||||
| 291 | + if now() - inc["updated"] < POLICY["watch_window_hours"] * 3600: | ||||
| 292 | + continue | ||||
| 293 | + status = current_status(inc["service"], inc["source"]) | ||||
| 294 | + m = c.execute("SELECT * FROM missions WHERE incident_id=? AND base_commit IS NOT NULL " | ||||
| 295 | + "ORDER BY started DESC LIMIT 1", (inc["id"],)).fetchone() | ||||
| 296 | + if status in POLICY["trigger_statuses"] or status == "degraded": | ||||
| 297 | + if m and (json.loads(m["commits"] or "[]")): | ||||
| 298 | + asyncio.create_task(rollback_mission(m, "toujours cassé après la fenêtre de surveillance")) | ||||
| 299 | + set_incident(c, inc["id"], state="cooldown") | ||||
| 300 | + incident_event(inc["id"], inc["service"], inc["source"], "cooldown", | ||||
| 301 | + f"non guéri après {POLICY['watch_window_hours']}h → rollback + cooldown") | ||||
| 302 | + elif status == "ok": | ||||
| 303 | + set_incident(c, inc["id"], state="resolved", resolved=now()) | ||||
| 304 | + incident_event(inc["id"], inc["service"], inc["source"], "resolved", "confirmé ok") | ||||
| 305 | + | ||||
| 306 | + # 3. Cooldown expiré → réouverture ou abandon | ||||
| 307 | + for inc in c.execute("SELECT * FROM incidents WHERE state='cooldown'").fetchall(): | ||||
| 308 | + if now() - inc["updated"] < POLICY["attempt_cooldown_hours"] * 3600: | ||||
| 309 | + continue | ||||
| 310 | + status = current_status(inc["service"], inc["source"]) | ||||
| 311 | + if status == "ok": | ||||
| 312 | + set_incident(c, inc["id"], state="resolved", resolved=now()) | ||||
| 313 | + incident_event(inc["id"], inc["service"], inc["source"], "resolved", "guéri pendant le cooldown") | ||||
| 314 | + elif inc["attempts"] >= POLICY["max_attempts_per_incident"]: | ||||
| 315 | + set_incident(c, inc["id"], state="abandoned") | ||||
| 316 | + incident_event(inc["id"], inc["service"], inc["source"], "abandoned", | ||||
| 317 | + f"{inc['attempts']} tentatives épuisées — intervention humaine requise") | ||||
| 318 | + else: | ||||
| 319 | + set_incident(c, inc["id"], state="open") | ||||
| 320 | + incident_event(inc["id"], inc["service"], inc["source"], "open", "cooldown terminé, nouvelle tentative") | ||||
| 321 | + | ||||
| 322 | + # 4. Dispatch (1 mission à la fois par agent, broken avant stale, plus vieux d'abord) | ||||
| 323 | + running = c.execute("SELECT COUNT(*) n FROM missions WHERE state='running'").fetchone()["n"] | ||||
| 324 | + if running < POLICY["max_concurrent_missions"]: | ||||
| 325 | + busy_nodes = {m["node"] for m in c.execute("SELECT node FROM missions WHERE state='running'").fetchall()} | ||||
| 326 | + nxt = c.execute( | ||||
| 327 | + "SELECT * FROM incidents WHERE state='open' " | ||||
| 328 | + "ORDER BY CASE status_detected WHEN 'broken' THEN 0 WHEN 'manual' THEN 0 ELSE 1 END, created " | ||||
| 329 | + ).fetchall() | ||||
| 330 | + for inc in nxt: | ||||
| 331 | + if inc["service"] not in SERVICES: | ||||
| 332 | + continue | ||||
| 333 | + if SERVICES[inc["service"]]["node"] in busy_nodes: | ||||
| 334 | + continue | ||||
| 335 | + set_incident(c, inc["id"], state="dispatched") | ||||
| 336 | + asyncio.create_task(dispatch_row(inc["id"])) | ||||
| 337 | + break | ||||
| 338 | + | ||||
| 339 | + | ||||
| 340 | +def current_status(service: str, source: str) -> str: | ||||
| 341 | + block = LATEST["mine"].get(service) or {} | ||||
| 342 | + for conn in block.get("connectors", []): | ||||
| 343 | + if conn["source"] == source: | ||||
| 344 | + return conn["status"] | ||||
| 345 | + return "inconnu" | ||||
| 346 | + | ||||
| 347 | + | ||||
| 348 | +async def dispatch_row(iid: str) -> None: | ||||
| 349 | + with db() as c: | ||||
| 350 | + inc = c.execute("SELECT * FROM incidents WHERE id=?", (iid,)).fetchone() | ||||
| 351 | + if inc: | ||||
| 352 | + await dispatch(inc, json.loads(inc["detail"] or "{}")) | ||||
| 353 | + # dispatch() remet fixing; si l'appel a échoué/409, on relâche | ||||
| 354 | + with db() as c: | ||||
| 355 | + cur = c.execute("SELECT state FROM incidents WHERE id=?", (iid,)).fetchone() | ||||
| 356 | + if cur and cur["state"] == "dispatched": | ||||
| 357 | + set_incident(c, iid, state="open") | ||||
| 358 | + | ||||
| 359 | + | ||||
| 360 | +async def engine() -> None: | ||||
| 361 | + global LOOP | ||||
| 362 | + LOOP = asyncio.get_running_loop() | ||||
| 363 | + await asyncio.sleep(3) | ||||
| 364 | + while True: | ||||
| 365 | + try: | ||||
| 366 | + await tick() | ||||
| 367 | + except Exception as exc: | ||||
| 368 | + hub.publish_sync({"kind": "log", "level": "error", "msg": f"tick: {exc}", "ts": now()}) | ||||
| 369 | + await asyncio.sleep(POLICY["poll_interval_seconds"]) | ||||
| 370 | + | ||||
| 371 | + | ||||
| 372 | +# ------------------------------------------------------------------- app --- | ||||
| 373 | + | ||||
| 374 | +app = FastAPI(title=f"KA Guardian — {AGENT}", docs_url=None, redoc_url=None) | ||||
| 375 | + | ||||
| 376 | + | ||||
| 377 | +@app.on_event("startup") | ||||
| 378 | +async def startup() -> None: | ||||
| 379 | + init_db() | ||||
| 380 | + asyncio.create_task(engine()) | ||||
| 381 | + | ||||
| 382 | + | ||||
| 383 | +def check_token(tok: str | None) -> None: | ||||
| 384 | + if not TOKEN or tok != TOKEN: | ||||
| 385 | + raise HTTPException(status_code=401) | ||||
| 386 | + | ||||
| 387 | + | ||||
| 388 | +@app.post("/api/ingest/{mission_id}") | ||||
| 389 | +async def ingest(mission_id: str, req: Request, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | ||||
| 390 | + check_token(x_ka_token) | ||||
| 391 | + ev = await req.json() | ||||
| 392 | + etype, data = ev.get("type"), ev.get("data", {}) | ||||
| 393 | + with db() as c: | ||||
| 394 | + c.execute("INSERT INTO events(mission_id,ts,type,data) VALUES(?,?,?,?)", | ||||
| 395 | + (mission_id, now(), etype, jdump(data))) | ||||
| 396 | + m = c.execute("SELECT * FROM missions WHERE id=?", (mission_id,)).fetchone() | ||||
| 397 | + if m: | ||||
| 398 | + if etype == "start": | ||||
| 399 | + c.execute("UPDATE missions SET base_commit=? WHERE id=?", (data.get("base_commit"), mission_id)) | ||||
| 400 | + elif etype in ("final", "error"): | ||||
| 401 | + await finalize(c, m, etype, data) | ||||
| 402 | + hub.publish_sync({"kind": "mission_event", "mission_id": mission_id, "type": etype, | ||||
| 403 | + "data": data, "ts": now()}) | ||||
| 404 | + return {"ok": True} | ||||
| 405 | + | ||||
| 406 | + | ||||
| 407 | +async def finalize(c: sqlite3.Connection, m: sqlite3.Row, etype: str, data: dict[str, Any]) -> None: | ||||
| 408 | + inc = c.execute("SELECT * FROM incidents WHERE id=?", (m["incident_id"],)).fetchone() | ||||
| 409 | + if etype == "error": | ||||
| 410 | + c.execute("UPDATE missions SET state='error', ended=? WHERE id=?", (now(), m["id"])) | ||||
| 411 | + if inc: | ||||
| 412 | + set_incident(c, inc["id"], state="cooldown") | ||||
| 413 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", f"erreur mission: {data.get('error')}") | ||||
| 414 | + return | ||||
| 415 | + verdict = data.get("verdict", {}) | ||||
| 416 | + commits = data.get("commits", []) | ||||
| 417 | + health = data.get("health", {}) | ||||
| 418 | + c.execute("UPDATE missions SET state='done', ended=?, commits=?, verdict=?, cost_usd=?, num_turns=?, health=?, base_commit=COALESCE(base_commit,?) WHERE id=?", | ||||
| 419 | + (now(), jdump(commits), jdump(verdict), data.get("cost_usd"), | ||||
| 420 | + data.get("num_turns"), jdump(health), data.get("base_commit"), m["id"])) | ||||
| 421 | + m2 = c.execute("SELECT * FROM missions WHERE id=?", (m["id"],)).fetchone() | ||||
| 422 | + v = verdict.get("verdict", "inconnu") | ||||
| 423 | + if not inc: | ||||
| 424 | + return | ||||
| 425 | + if not health.get("ok", True): | ||||
| 426 | + # L'app est tombée → rollback immédiat, quoi qu'ait dit l'agent. | ||||
| 427 | + asyncio.create_task(rollback_mission(m2, "healthcheck app en échec post-mission")) | ||||
| 428 | + set_incident(c, inc["id"], state="cooldown") | ||||
| 429 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", "app en mauvaise santé → ROLLBACK immédiat") | ||||
| 430 | + elif v == "repare" and commits: | ||||
| 431 | + set_incident(c, inc["id"], state="watching") | ||||
| 432 | + incident_event(inc["id"], m["service"], m["source"], "watching", | ||||
| 433 | + f"réparation déclarée ({len(commits)} commit) — surveillance {POLICY['watch_window_hours']}h") | ||||
| 434 | + elif v in ("echec", "inconnu") and commits: | ||||
| 435 | + asyncio.create_task(rollback_mission(m2, f"verdict {v} avec commits → annulation par prudence")) | ||||
| 436 | + set_incident(c, inc["id"], state="cooldown") | ||||
| 437 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", f"verdict {v} → rollback préventif") | ||||
| 438 | + elif v == "site_source_mort": | ||||
| 439 | + set_incident(c, inc["id"], state="abandoned") | ||||
| 440 | + incident_event(inc["id"], m["service"], m["source"], "abandoned", "source définitivement morte (diagnostic agent)") | ||||
| 441 | + elif v in ("rien_a_faire", "repare"): | ||||
| 442 | + set_incident(c, inc["id"], state="watching") | ||||
| 443 | + incident_event(inc["id"], m["service"], m["source"], "watching", f"verdict {v} — surveillance") | ||||
| 444 | + else: | ||||
| 445 | + set_incident(c, inc["id"], state="cooldown") | ||||
| 446 | + incident_event(inc["id"], m["service"], m["source"], "cooldown", f"verdict {v}") | ||||
| 447 | + | ||||
| 448 | + | ||||
| 449 | +@app.get("/api/state") | ||||
| 450 | +def state() -> dict[str, Any]: | ||||
| 451 | + with db() as c: | ||||
| 452 | + incidents = [dict(r) for r in c.execute( | ||||
| 453 | + "SELECT * FROM incidents ORDER BY created DESC LIMIT 200").fetchall()] | ||||
| 454 | + missions = [dict(r) for r in c.execute( | ||||
| 455 | + "SELECT * FROM missions ORDER BY started DESC LIMIT 100").fetchall()] | ||||
| 456 | + stats = c.execute("""SELECT | ||||
| 457 | + (SELECT COUNT(*) FROM missions) n_missions, | ||||
| 458 | + (SELECT COUNT(*) FROM incidents WHERE state='resolved') n_resolved, | ||||
| 459 | + (SELECT COUNT(*) FROM incidents WHERE state IN ('open','dispatched','fixing','watching','cooldown')) n_active, | ||||
| 460 | + (SELECT COUNT(*) FROM incidents WHERE state='abandoned') n_abandoned, | ||||
| 461 | + (SELECT COUNT(*) FROM events WHERE type='rollback') n_rollbacks, | ||||
| 462 | + (SELECT ROUND(SUM(cost_usd),2) FROM missions) cost_total, | ||||
| 463 | + (SELECT ROUND(AVG(resolved-created)/3600.0,1) FROM incidents WHERE state='resolved' AND resolved IS NOT NULL) mttr_h | ||||
| 464 | + """).fetchone() | ||||
| 465 | + history = [dict(r) for r in c.execute( | ||||
| 466 | + "SELECT ts, mine FROM snapshots WHERE ts > ? ORDER BY ts", (now() - 7 * 86400,)).fetchall()] | ||||
| 467 | + return { | ||||
| 468 | + "agent": AGENT, "identity": {k: ME[k] for k in ("domain", "accent", "accent2", "tagline", "model")}, | ||||
| 469 | + "services": {s: {**SERVICES[s], "node_alias": NODES[SERVICES[s]["node"]]["alias"]} for s in ME["services"]}, | ||||
| 470 | + "siblings": {a: {"domain": v["domain"], "accent": v["accent"], "services": v["services"]} | ||||
| 471 | + for a, v in TOPO["agents"].items() if a != AGENT}, | ||||
| 472 | + "latest": LATEST, "incidents": incidents, "missions": missions, | ||||
| 473 | + "stats": dict(stats), "history": history, "policy": POLICY, "now": now(), | ||||
| 474 | + } | ||||
| 475 | + | ||||
| 476 | + | ||||
| 477 | +@app.get("/api/missions/{mission_id}") | ||||
| 478 | +def mission_detail(mission_id: str) -> dict[str, Any]: | ||||
| 479 | + with db() as c: | ||||
| 480 | + m = c.execute("SELECT * FROM missions WHERE id=?", (mission_id,)).fetchone() | ||||
| 481 | + if not m: | ||||
| 482 | + raise HTTPException(status_code=404) | ||||
| 483 | + evs = [dict(r) for r in c.execute( | ||||
| 484 | + "SELECT ts,type,data FROM events WHERE mission_id=? ORDER BY id", (mission_id,)).fetchall()] | ||||
| 485 | + return {"mission": dict(m), "events": evs} | ||||
| 486 | + | ||||
| 487 | + | ||||
| 488 | +@app.get("/events") | ||||
| 489 | +async def sse(request: Request) -> StreamingResponse: | ||||
| 490 | + q: asyncio.Queue = asyncio.Queue() | ||||
| 491 | + hub.clients.add(q) | ||||
| 492 | + | ||||
| 493 | + async def gen(): | ||||
| 494 | + try: | ||||
| 495 | + yield f"data: {jdump({'kind': 'hello', 'agent': AGENT, 'ts': now()})}\n\n" | ||||
| 496 | + while True: | ||||
| 497 | + if await request.is_disconnected(): | ||||
| 498 | + break | ||||
| 499 | + try: | ||||
| 500 | + msg = await asyncio.wait_for(q.get(), timeout=25) | ||||
| 501 | + yield f"data: {jdump(msg)}\n\n" | ||||
| 502 | + except asyncio.TimeoutError: | ||||
| 503 | + yield ": keepalive\n\n" | ||||
| 504 | + finally: | ||||
| 505 | + hub.clients.discard(q) | ||||
| 506 | + | ||||
| 507 | + return StreamingResponse(gen(), media_type="text/event-stream", | ||||
| 508 | + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}) | ||||
| 509 | + | ||||
| 510 | + | ||||
| 511 | +@app.post("/api/admin/mission") | ||||
| 512 | +async def admin_mission(req: Request, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | ||||
| 513 | + """Déclenche manuellement une mission sur un connecteur (même sain).""" | ||||
| 514 | + check_token(x_ka_token) | ||||
| 515 | + body = await req.json() | ||||
| 516 | + service, source = body["service"], body["source"] | ||||
| 517 | + if service not in SERVICES: | ||||
| 518 | + raise HTTPException(status_code=400, detail="service inconnu") | ||||
| 519 | + detail = {"source": source, "status": "manual", "message": body.get("note", "mission manuelle")} | ||||
| 520 | + for conn in (LATEST["mine"].get(service) or {}).get("connectors", []): | ||||
| 521 | + if conn["source"] == source: | ||||
| 522 | + detail = {**conn, "status": "manual"} | ||||
| 523 | + with db() as c: | ||||
| 524 | + inc = active_incident(c, service, source) | ||||
| 525 | + if inc is None: | ||||
| 526 | + iid = uuid.uuid4().hex[:10] | ||||
| 527 | + c.execute("INSERT INTO incidents(id,service,source,status_detected,state,created,updated,detail) " | ||||
| 528 | + "VALUES(?,?,?,?,?,?,?,?)", | ||||
| 529 | + (iid, service, source, "manual", "open", now(), now(), jdump(detail))) | ||||
| 530 | + else: | ||||
| 531 | + iid = inc["id"] | ||||
| 532 | + set_incident(c, iid, state="open", status_detected="manual") | ||||
| 533 | + incident_event(iid, service, source, "open", "mission manuelle demandée") | ||||
| 534 | + return {"ok": True, "incident_id": iid} | ||||
| 535 | + | ||||
| 536 | + | ||||
| 537 | +@app.post("/api/admin/pause") | ||||
| 538 | +async def admin_pause(req: Request, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | ||||
| 539 | + check_token(x_ka_token) | ||||
| 540 | + body = await req.json() | ||||
| 541 | + LATEST["paused"] = bool(body.get("paused", True)) | ||||
| 542 | + return {"ok": True, "paused": LATEST["paused"]} | ||||
| 543 | + | ||||
| 544 | + | ||||
| 545 | +@app.get("/health") | ||||
| 546 | +def health() -> dict[str, Any]: | ||||
| 547 | + return {"ok": True, "agent": AGENT, "polled": LATEST["polled"], "paused": LATEST["paused"]} | ||||
| 548 | + | ||||
| 549 | + | ||||
| 550 | +@app.get("/") | ||||
| 551 | +def index() -> FileResponse: | ||||
| 552 | + return FileResponse(BASE / "web" / "index.html") | ||||
| 553 | + | ||||
| 554 | + | ||||
| 555 | +app.mount("/static", StaticFiles(directory=BASE / "web"), name="static") | ||||
| 556 | + | ||||
| 557 | + | ||||
| 558 | +if __name__ == "__main__": | ||||
| 559 | + import uvicorn | ||||
| 560 | + uvicorn.run(app, host="0.0.0.0", port=ME["port"]) | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/app.js
+163 −0
@@ -0,0 +1,163 @@ | |||||
| 1 | +/* KA Guardian — dashboard live */ | ||||
| 2 | +const $ = (s) => document.querySelector(s); | ||||
| 3 | +const esc = (s) => String(s ?? "").replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c])); | ||||
| 4 | +const fmtT = (ts) => new Date(ts * 1000).toLocaleTimeString("fr-CA", { hour12: false }); | ||||
| 5 | +const fmtD = (ts) => new Date(ts * 1000).toLocaleString("fr-CA", { dateStyle: "short", timeStyle: "short" }); | ||||
| 6 | +const STATE_FR = { open: "à réparer", dispatched: "envoi…", fixing: "réparation", watching: "surveillance", | ||||
| 7 | + cooldown: "attente", resolved: "résolu", self_healed: "auto-guéri", abandoned: "abandonné" }; | ||||
| 8 | +const STATUSES = ["ok", "degraded", "broken", "stale"]; | ||||
| 9 | +const STATUS_FR = { ok: "ok", degraded: "dégradé", broken: "cassé", stale: "endormi" }; | ||||
| 10 | + | ||||
| 11 | +let STATE = null; | ||||
| 12 | + | ||||
| 13 | +async function load() { | ||||
| 14 | + const r = await fetch("/api/state"); | ||||
| 15 | + STATE = await r.json(); | ||||
| 16 | + const id = STATE.identity; | ||||
| 17 | + document.documentElement.style.setProperty("--accent", id.accent); | ||||
| 18 | + document.documentElement.style.setProperty("--accent2", id.accent2); | ||||
| 19 | + document.title = `${STATE.agent.toUpperCase()} — Guardian du Groupe KA`; | ||||
| 20 | + $("#agent-name").textContent = STATE.agent.toUpperCase() + " · GUARDIAN"; | ||||
| 21 | + $("#sigil").textContent = "·K" + STATE.agent.replace("ka", ""); | ||||
| 22 | + $("#foot-name").textContent = STATE.agent.toUpperCase(); | ||||
| 23 | + const apps = Object.values(STATE.services).map((s) => s.app).join(" · "); | ||||
| 24 | + $("#tagline").textContent = `${id.tagline} — veille sur ${apps}`; | ||||
| 25 | + $("#siblings").innerHTML = Object.entries(STATE.siblings) | ||||
| 26 | + .map(([a, v]) => `<a href="https://${v.domain}" style="color:${v.accent}"><b>${a.toUpperCase()}</b>${v.services.join(", ")}</a>`).join(""); | ||||
| 27 | + renderTiles(); renderIncidents(); renderCoverage(); renderMissions(); | ||||
| 28 | +} | ||||
| 29 | + | ||||
| 30 | +function renderTiles() { | ||||
| 31 | + const s = STATE.stats, mine = STATE.latest.mine || {}; | ||||
| 32 | + let total = 0, sick = 0; | ||||
| 33 | + for (const b of Object.values(mine)) for (const st of STATUSES) { | ||||
| 34 | + total += b.summary[st] || 0; | ||||
| 35 | + if (st !== "ok") sick += b.summary[st] || 0; | ||||
| 36 | + } | ||||
| 37 | + const tiles = [ | ||||
| 38 | + [total || "—", "connecteurs surveillés", true], | ||||
| 39 | + [s.n_active ?? 0, "incidents actifs"], | ||||
| 40 | + [s.n_missions ?? 0, "missions lancées"], | ||||
| 41 | + [s.n_resolved ?? 0, "réparations confirmées"], | ||||
| 42 | + [s.n_rollbacks ?? 0, "rollbacks"], | ||||
| 43 | + [s.mttr_h != null ? s.mttr_h + " h" : "—", "temps moyen de guérison"], | ||||
| 44 | + [s.cost_total != null ? s.cost_total + " $" : "0 $", "coût API total"], | ||||
| 45 | + ]; | ||||
| 46 | + $("#tiles").innerHTML = tiles.map(([v, l, hot]) => | ||||
| 47 | + `<div class="tile${hot ? " hot" : ""}"><div class="v">${esc(v)}</div><div class="l">${l}</div></div>`).join(""); | ||||
| 48 | +} | ||||
| 49 | + | ||||
| 50 | +function renderIncidents() { | ||||
| 51 | + const active = STATE.incidents.filter((i) => !["resolved", "self_healed"].includes(i.state)).slice(0, 12); | ||||
| 52 | + const recent = STATE.incidents.filter((i) => ["resolved", "self_healed"].includes(i.state)).slice(0, 5); | ||||
| 53 | + const row = (i) => `<div class="inc"> | ||||
| 54 | + <div><div class="src">${esc(i.source)}</div><div class="svc">${esc(i.service)} · ${esc(i.status_detected)} · tentatives ${i.attempts}</div></div> | ||||
| 55 | + <span class="badge b-${esc(i.state)}">${STATE_FR[i.state] || esc(i.state)}</span></div>`; | ||||
| 56 | + $("#incidents").innerHTML = (active.length || recent.length) | ||||
| 57 | + ? active.map(row).join("") + recent.map(row).join("") | ||||
| 58 | + : `<div class="empty">Aucun incident — tous les connecteurs assignés sont sains. L'agent veille.</div>`; | ||||
| 59 | +} | ||||
| 60 | + | ||||
| 61 | +function renderCoverage() { | ||||
| 62 | + const mine = STATE.latest.mine || {}; | ||||
| 63 | + $("#coverage").innerHTML = Object.keys(STATE.services).map((svc) => { | ||||
| 64 | + const meta = STATE.services[svc], sum = (mine[svc] || {}).summary || {}; | ||||
| 65 | + const total = STATUSES.reduce((a, st) => a + (sum[st] || 0), 0) || 1; | ||||
| 66 | + const strip = STATUSES.filter((st) => sum[st] > 0) | ||||
| 67 | + .map((st) => `<span class="s-${st}" style="flex:${sum[st]}" title="${STATUS_FR[st]}: ${sum[st]}"></span>`).join(""); | ||||
| 68 | + const counts = STATUSES.map((st) => | ||||
| 69 | + `<span><i class="s-${st}"></i>${sum[st] || 0} ${STATUS_FR[st]}</span>`).join(""); | ||||
| 70 | + return `<div class="cov"><div class="head"><span class="app">${esc(meta.app)}</span> | ||||
| 71 | + <span class="node">${esc(meta.node_alias)} · ${total} connecteurs</span></div> | ||||
| 72 | + <div class="strip">${strip || "<span style='flex:1;background:var(--line)'></span>"}</div> | ||||
| 73 | + <div class="counts">${counts}</div></div>`; | ||||
| 74 | + }).join(""); | ||||
| 75 | +} | ||||
| 76 | + | ||||
| 77 | +function renderMissions() { | ||||
| 78 | + const vd = (m) => { | ||||
| 79 | + if (m.state === "running") return `<span class="v-running">en cours…</span>`; | ||||
| 80 | + if (m.state === "error") return `<span class="v-echec">erreur</span>`; | ||||
| 81 | + const v = (JSON.parse(m.verdict || "{}").verdict) || "—"; | ||||
| 82 | + return `<span class="v-${esc(v)}">${esc(v)}</span>`; | ||||
| 83 | + }; | ||||
| 84 | + $("#missions tbody").innerHTML = STATE.missions.map((m) => { | ||||
| 85 | + const commits = JSON.parse(m.commits || "[]").length; | ||||
| 86 | + const dur = m.ended ? Math.round((m.ended - m.started) / 60) + " min" : ""; | ||||
| 87 | + return `<tr data-id="${esc(m.id)}"><td class="mono">${fmtD(m.started)}</td> | ||||
| 88 | + <td class="mono">${esc(m.source)}</td><td>${esc(m.service)}</td><td class="mono">${esc(m.node)}</td> | ||||
| 89 | + <td>${vd(m)}</td><td>${commits || "—"}</td><td>${m.num_turns ?? ""} ${dur ? "· " + dur : ""}</td> | ||||
| 90 | + <td class="mono">${m.cost_usd != null ? m.cost_usd.toFixed(2) + " $" : ""}</td></tr>`; | ||||
| 91 | + }).join("") || `<tr><td colspan="8" class="empty">Aucune mission encore — la première panne détectée lancera l'agent.</td></tr>`; | ||||
| 92 | +} | ||||
| 93 | + | ||||
| 94 | +/* ---- feed en direct ---- */ | ||||
| 95 | +const feed = $("#feed"); | ||||
| 96 | +function feedLine(cls, tag, msg, ts) { | ||||
| 97 | + if (feed.querySelector(".feed-empty")) feed.innerHTML = ""; | ||||
| 98 | + const el = document.createElement("div"); | ||||
| 99 | + el.className = "fe " + cls; | ||||
| 100 | + el.innerHTML = `<span class="t">${fmtT(ts || Date.now() / 1000)}</span><span class="tag">${esc(tag)}</span><span class="m">${esc(msg)}</span>`; | ||||
| 101 | + feed.appendChild(el); | ||||
| 102 | + while (feed.children.length > 400) feed.removeChild(feed.firstChild); | ||||
| 103 | + feed.scrollTop = feed.scrollHeight; | ||||
| 104 | +} | ||||
| 105 | + | ||||
| 106 | +let refetchTimer = null; | ||||
| 107 | +const refetch = () => { clearTimeout(refetchTimer); refetchTimer = setTimeout(load, 800); }; | ||||
| 108 | + | ||||
| 109 | +function connectSSE() { | ||||
| 110 | + const es = new EventSource("/events"); | ||||
| 111 | + es.onopen = () => $("#live-dot").classList.add("on"); | ||||
| 112 | + es.onerror = () => $("#live-dot").classList.remove("on"); | ||||
| 113 | + es.onmessage = (e) => { | ||||
| 114 | + const m = JSON.parse(e.data); | ||||
| 115 | + if (m.kind === "mission_event") { | ||||
| 116 | + const d = m.data || {}; | ||||
| 117 | + if (m.type === "start") feedLine("tool", "mission", `démarrage — commit de base ${(d.base_commit || "").slice(0, 8)} sur ${d.node || ""}`, m.ts); | ||||
| 118 | + else if (m.type === "tool") feedLine("tool", d.name || "outil", d.input || "", m.ts); | ||||
| 119 | + else if (m.type === "text") feedLine("text", "agent", d.text || "", m.ts); | ||||
| 120 | + else if (m.type === "result") feedLine("text", "résultat", (d.result || "").slice(0, 600), m.ts); | ||||
| 121 | + else if (m.type === "final") { feedLine("text", "verdict", JSON.stringify(d.verdict || {}), m.ts); refetch(); } | ||||
| 122 | + else if (m.type === "error") { feedLine("rollback", "erreur", d.error || "", m.ts); refetch(); } | ||||
| 123 | + else if (m.type === "rollback") feedLine("rollback", "rollback", JSON.stringify(d), m.ts); | ||||
| 124 | + } else if (m.kind === "incident") { | ||||
| 125 | + feedLine("incident", "incident", `${m.service}/${m.source} → ${STATE_FR[m.state] || m.state}${m.note ? " — " + m.note : ""}`, m.ts); | ||||
| 126 | + refetch(); | ||||
| 127 | + } else if (m.kind === "rollback") { | ||||
| 128 | + feedLine("rollback", "rollback", `${m.service}/${m.source} — ${m.reason} (${(m.result || {}).mode || "?"})`, m.ts); | ||||
| 129 | + refetch(); | ||||
| 130 | + } else if (m.kind === "mission") { | ||||
| 131 | + feedLine("tool", "mission", `nouvelle mission sur ${m.service}/${m.source}`, m.ts); | ||||
| 132 | + refetch(); | ||||
| 133 | + } else if (m.kind === "snapshot") { | ||||
| 134 | + if (STATE) { STATE.latest.mine = Object.fromEntries(Object.entries(m.mine).map(([s, sum]) => [s, { summary: sum }])); renderCoverage(); } | ||||
| 135 | + } else if (m.kind === "log") { | ||||
| 136 | + feedLine("incident", m.level || "log", m.msg, m.ts); | ||||
| 137 | + } | ||||
| 138 | + }; | ||||
| 139 | +} | ||||
| 140 | + | ||||
| 141 | +/* ---- modal mission ---- */ | ||||
| 142 | +document.addEventListener("click", async (e) => { | ||||
| 143 | + const tr = e.target.closest("#missions tbody tr[data-id]"); | ||||
| 144 | + if (tr) { | ||||
| 145 | + const r = await fetch("/api/missions/" + tr.dataset.id); | ||||
| 146 | + const { mission, events } = await r.json(); | ||||
| 147 | + const v = JSON.parse(mission.verdict || "{}"); | ||||
| 148 | + $("#modal-title").textContent = `Mission ${mission.id} — ${mission.source} (${mission.service})`; | ||||
| 149 | + $("#modal-body").innerHTML = | ||||
| 150 | + `<div class="meta">verdict: ${esc(v.verdict || mission.state)}\ndiagnostic: ${esc(v.diagnostic || "")}\nactions: ${esc(v.actions || "")}\ntest: ${esc(v.test || "")}\ncommit de base: ${esc(mission.base_commit || "")}\ncommits: ${esc((JSON.parse(mission.commits || "[]")).join(" | ") || "aucun")}</div>` + | ||||
| 151 | + events.map((ev) => { | ||||
| 152 | + const d = JSON.parse(ev.data || "{}"); | ||||
| 153 | + const txt = ev.type === "tool" ? `${d.name}: ${d.input}` : ev.type === "text" ? d.text | ||||
| 154 | + : ev.type === "result" ? d.result : JSON.stringify(d); | ||||
| 155 | + return `<div class="meta"><b>${fmtT(ev.ts)} · ${esc(ev.type)}</b>\n${esc((txt || "").slice(0, 2500))}</div>`; | ||||
| 156 | + }).join(""); | ||||
| 157 | + $("#modal").hidden = false; | ||||
| 158 | + } | ||||
| 159 | + if (e.target.id === "modal-close" || e.target.id === "modal") $("#modal").hidden = true; | ||||
| 160 | +}); | ||||
| 161 | + | ||||
| 162 | +load().then(connectSSE); | ||||
| 163 | +setInterval(load, 60000); | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/index.html
+69 −0
@@ -0,0 +1,69 @@ | |||||
| 1 | +<!doctype html> | ||||
| 2 | +<html lang="fr"> | ||||
| 3 | +<head> | ||||
| 4 | +<meta charset="utf-8"> | ||||
| 5 | +<meta name="viewport" content="width=device-width, initial-scale=1"> | ||||
| 6 | +<title>KA Guardian</title> | ||||
| 7 | +<meta name="description" content="Agent autonome de maintenance des connecteurs du Groupe KA — incidents, missions et réparations en direct."> | ||||
| 8 | +<link rel="preconnect" href="https://fonts.googleapis.com"> | ||||
| 9 | +<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> | ||||
| 10 | +<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;700&family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet"> | ||||
| 11 | +<link rel="stylesheet" href="/static/style.css"> | ||||
| 12 | +</head> | ||||
| 13 | +<body> | ||||
| 14 | +<header class="top"> | ||||
| 15 | + <div class="brand"> | ||||
| 16 | + <div class="sigil" id="sigil">·K</div> | ||||
| 17 | + <div> | ||||
| 18 | + <h1 id="agent-name">KA·—</h1> | ||||
| 19 | + <p class="tagline"><span class="live-dot" id="live-dot"></span><span id="tagline">connexion…</span></p> | ||||
| 20 | + </div> | ||||
| 21 | + </div> | ||||
| 22 | + <nav class="siblings" id="siblings"></nav> | ||||
| 23 | +</header> | ||||
| 24 | + | ||||
| 25 | +<main> | ||||
| 26 | + <section class="tiles" id="tiles"></section> | ||||
| 27 | + | ||||
| 28 | + <section class="cols"> | ||||
| 29 | + <div class="panel feed-panel"> | ||||
| 30 | + <h2>⚡ Flux en direct <span class="hint">tout ce que l'agent fait, en temps réel</span></h2> | ||||
| 31 | + <div class="feed" id="feed"><div class="feed-empty">En attente d'activité…</div></div> | ||||
| 32 | + </div> | ||||
| 33 | + <div class="side"> | ||||
| 34 | + <div class="panel"> | ||||
| 35 | + <h2>🚨 Incidents</h2> | ||||
| 36 | + <div id="incidents" class="incidents"></div> | ||||
| 37 | + </div> | ||||
| 38 | + <div class="panel"> | ||||
| 39 | + <h2>🛰 Couverture</h2> | ||||
| 40 | + <div id="coverage" class="coverage"></div> | ||||
| 41 | + </div> | ||||
| 42 | + </div> | ||||
| 43 | + </section> | ||||
| 44 | + | ||||
| 45 | + <section class="panel"> | ||||
| 46 | + <h2>📜 Missions <span class="hint">cliquer une ligne pour le déroulé complet</span></h2> | ||||
| 47 | + <div class="table-wrap"> | ||||
| 48 | + <table id="missions"> | ||||
| 49 | + <thead><tr><th>Quand</th><th>Connecteur</th><th>Service</th><th>Nœud</th><th>Verdict</th><th>Commits</th><th>Tours</th><th>Coût</th></tr></thead> | ||||
| 50 | + <tbody></tbody> | ||||
| 51 | + </table> | ||||
| 52 | + </div> | ||||
| 53 | + </section> | ||||
| 54 | +</main> | ||||
| 55 | + | ||||
| 56 | +<footer> | ||||
| 57 | + <p><strong id="foot-name">KA</strong> — agent gardien autonome du <a href="https://www.groupe-ka.com">Groupe KA</a>. Il détecte les connecteurs en panne, les répare avec Claude, surveille la guérison et annule ses changements si ça empire.</p> | ||||
| 58 | +</footer> | ||||
| 59 | + | ||||
| 60 | +<div class="modal" id="modal" hidden> | ||||
| 61 | + <div class="modal-box"> | ||||
| 62 | + <div class="modal-head"><h3 id="modal-title">Mission</h3><button id="modal-close">✕</button></div> | ||||
| 63 | + <div class="modal-body" id="modal-body"></div> | ||||
| 64 | + </div> | ||||
| 65 | +</div> | ||||
| 66 | + | ||||
| 67 | +<script src="/static/app.js"></script> | ||||
| 68 | +</body> | ||||
| 69 | +</html> | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/style.css
+118 −0
@@ -0,0 +1,118 @@ | |||||
| 1 | +/* KA Guardian — salle de contrôle */ | ||||
| 2 | +:root { | ||||
| 3 | + --bg: #0d1017; --bg2: #12161f; --panel: #151a24; --panel2: #1a2030; | ||||
| 4 | + --line: #232a3a; --ink: #e8ecf4; --ink2: #9aa5b8; --muted: #667089; | ||||
| 5 | + --accent: #22d3ee; --accent2: #0891b2; | ||||
| 6 | + --ok: #4ade80; --degraded: #fbbf24; --broken: #f87171; --stale: #60a5fa; | ||||
| 7 | + --mono: "JetBrains Mono", ui-monospace, SFMono-Regular, monospace; | ||||
| 8 | + --sans: "Space Grotesk", -apple-system, sans-serif; | ||||
| 9 | +} | ||||
| 10 | +* { box-sizing: border-box; margin: 0; } | ||||
| 11 | +body { background: var(--bg); color: var(--ink); font-family: var(--sans); min-height: 100vh; } | ||||
| 12 | +a { color: var(--accent); text-decoration: none; } | ||||
| 13 | + | ||||
| 14 | +.top { display: flex; justify-content: space-between; align-items: center; gap: 16px; flex-wrap: wrap; | ||||
| 15 | + padding: 22px clamp(16px, 4vw, 48px); border-bottom: 1px solid var(--line); | ||||
| 16 | + background: radial-gradient(1200px 300px at 20% -50%, color-mix(in oklab, var(--accent) 18%, transparent), transparent), var(--bg2); } | ||||
| 17 | +.brand { display: flex; gap: 16px; align-items: center; } | ||||
| 18 | +.sigil { width: 54px; height: 54px; border-radius: 14px; display: grid; place-items: center; | ||||
| 19 | + font: 700 22px var(--mono); color: #0b0e14; | ||||
| 20 | + background: linear-gradient(135deg, var(--accent), var(--accent2)); | ||||
| 21 | + box-shadow: 0 0 24px color-mix(in oklab, var(--accent) 45%, transparent); } | ||||
| 22 | +h1 { font-size: 26px; letter-spacing: 0.04em; } | ||||
| 23 | +.tagline { color: var(--ink2); font-size: 14px; display: flex; align-items: center; gap: 8px; } | ||||
| 24 | +.live-dot { width: 9px; height: 9px; border-radius: 50%; background: var(--muted); flex: none; } | ||||
| 25 | +.live-dot.on { background: var(--ok); animation: pulse 1.8s infinite; } | ||||
| 26 | +@keyframes pulse { 0%,100% { box-shadow: 0 0 0 0 color-mix(in oklab, var(--ok) 55%, transparent); } | ||||
| 27 | + 60% { box-shadow: 0 0 0 8px transparent; } } | ||||
| 28 | +.siblings { display: flex; gap: 10px; } | ||||
| 29 | +.siblings a { border: 1px solid var(--line); border-radius: 10px; padding: 7px 13px; font: 600 13px var(--mono); | ||||
| 30 | + color: var(--ink2); background: var(--panel); } | ||||
| 31 | +.siblings a b { margin-right: 6px; } | ||||
| 32 | + | ||||
| 33 | +main { padding: 22px clamp(16px, 4vw, 48px); display: grid; gap: 20px; max-width: 1500px; margin: 0 auto; } | ||||
| 34 | + | ||||
| 35 | +.tiles { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 12px; } | ||||
| 36 | +.tile { background: var(--panel); border: 1px solid var(--line); border-radius: 14px; padding: 14px 16px; } | ||||
| 37 | +.tile .v { font: 600 30px/1.15 var(--mono); } | ||||
| 38 | +.tile .l { color: var(--ink2); font-size: 12.5px; margin-top: 4px; letter-spacing: 0.03em; text-transform: uppercase; } | ||||
| 39 | +.tile.hot .v { color: var(--accent); } | ||||
| 40 | + | ||||
| 41 | +.cols { display: grid; grid-template-columns: minmax(0, 1.5fr) minmax(0, 1fr); gap: 20px; } | ||||
| 42 | +@media (max-width: 980px) { .cols { grid-template-columns: 1fr; } } | ||||
| 43 | +.side { display: grid; gap: 20px; align-content: start; } | ||||
| 44 | + | ||||
| 45 | +.panel { background: var(--panel); border: 1px solid var(--line); border-radius: 16px; padding: 18px; min-width: 0; } | ||||
| 46 | +.panel h2 { font-size: 16px; margin-bottom: 14px; letter-spacing: 0.02em; } | ||||
| 47 | +.hint { color: var(--muted); font-size: 12px; font-weight: 400; margin-left: 8px; } | ||||
| 48 | + | ||||
| 49 | +/* Flux en direct */ | ||||
| 50 | +.feed { font: 13px/1.55 var(--mono); background: #0a0d13; border: 1px solid var(--line); border-radius: 12px; | ||||
| 51 | + padding: 14px; height: 560px; overflow-y: auto; display: flex; flex-direction: column; gap: 8px; } | ||||
| 52 | +.feed-empty { color: var(--muted); } | ||||
| 53 | +.fe { display: flex; gap: 10px; align-items: baseline; animation: fadein 0.3s; } | ||||
| 54 | +@keyframes fadein { from { opacity: 0; transform: translateY(4px); } } | ||||
| 55 | +.fe .t { color: var(--muted); flex: none; font-size: 11px; } | ||||
| 56 | +.fe .tag { flex: none; font-size: 11px; font-weight: 600; padding: 1px 8px; border-radius: 999px; | ||||
| 57 | + border: 1px solid var(--line); color: var(--ink2); } | ||||
| 58 | +.fe.tool .tag { color: var(--accent); border-color: color-mix(in oklab, var(--accent) 40%, var(--line)); } | ||||
| 59 | +.fe.text .tag { color: var(--ok); } | ||||
| 60 | +.fe.incident .tag { color: var(--degraded); } | ||||
| 61 | +.fe.rollback .tag { color: var(--broken); } | ||||
| 62 | +.fe .m { color: var(--ink2); overflow-wrap: anywhere; white-space: pre-wrap; } | ||||
| 63 | +.fe.text .m { color: var(--ink); } | ||||
| 64 | + | ||||
| 65 | +/* Incidents */ | ||||
| 66 | +.incidents { display: grid; gap: 9px; max-height: 330px; overflow-y: auto; } | ||||
| 67 | +.inc { display: flex; align-items: center; gap: 10px; background: var(--panel2); border: 1px solid var(--line); | ||||
| 68 | + border-radius: 11px; padding: 10px 12px; font-size: 13px; } | ||||
| 69 | +.inc .src { font: 600 13px var(--mono); overflow-wrap: anywhere; } | ||||
| 70 | +.inc .svc { color: var(--muted); font-size: 11.5px; } | ||||
| 71 | +.badge { margin-left: auto; flex: none; font: 600 11px var(--mono); padding: 3px 9px; border-radius: 999px; } | ||||
| 72 | +.b-open { background: color-mix(in oklab, var(--broken) 18%, transparent); color: var(--broken); } | ||||
| 73 | +.b-fixing { background: color-mix(in oklab, var(--accent) 18%, transparent); color: var(--accent); } | ||||
| 74 | +.b-dispatched{ background: color-mix(in oklab, var(--accent) 18%, transparent); color: var(--accent); } | ||||
| 75 | +.b-watching { background: color-mix(in oklab, var(--stale) 18%, transparent); color: var(--stale); } | ||||
| 76 | +.b-cooldown { background: color-mix(in oklab, var(--degraded) 18%, transparent);color: var(--degraded); } | ||||
| 77 | +.b-resolved, .b-self_healed { background: color-mix(in oklab, var(--ok) 16%, transparent); color: var(--ok); } | ||||
| 78 | +.b-abandoned { background: #2a2f3d; color: var(--muted); } | ||||
| 79 | +.empty { color: var(--muted); font-size: 13px; } | ||||
| 80 | + | ||||
| 81 | +/* Couverture */ | ||||
| 82 | +.coverage { display: grid; gap: 12px; } | ||||
| 83 | +.cov { background: var(--panel2); border: 1px solid var(--line); border-radius: 12px; padding: 12px 14px; } | ||||
| 84 | +.cov .head { display: flex; justify-content: space-between; align-items: baseline; margin-bottom: 8px; } | ||||
| 85 | +.cov .app { font-weight: 700; } | ||||
| 86 | +.cov .node { color: var(--muted); font: 11px var(--mono); } | ||||
| 87 | +.strip { display: flex; height: 10px; border-radius: 6px; overflow: hidden; gap: 2px; background: var(--bg); } | ||||
| 88 | +.strip span { min-width: 3px; } | ||||
| 89 | +.s-ok { background: var(--ok); } .s-degraded { background: var(--degraded); } | ||||
| 90 | +.s-broken { background: var(--broken); } .s-stale { background: var(--stale); } | ||||
| 91 | +.counts { display: flex; gap: 12px; margin-top: 8px; flex-wrap: wrap; font: 11.5px var(--mono); color: var(--ink2); } | ||||
| 92 | +.counts i { display: inline-block; width: 8px; height: 8px; border-radius: 2px; margin-right: 5px; } | ||||
| 93 | + | ||||
| 94 | +/* Missions */ | ||||
| 95 | +.table-wrap { overflow-x: auto; } | ||||
| 96 | +table { width: 100%; border-collapse: collapse; font-size: 13px; } | ||||
| 97 | +th { text-align: left; color: var(--muted); font-size: 11.5px; text-transform: uppercase; letter-spacing: 0.05em; | ||||
| 98 | + padding: 8px 10px; border-bottom: 1px solid var(--line); } | ||||
| 99 | +td { padding: 9px 10px; border-bottom: 1px solid color-mix(in oklab, var(--line) 55%, transparent); } | ||||
| 100 | +tbody tr { cursor: pointer; } tbody tr:hover { background: var(--panel2); } | ||||
| 101 | +.mono { font-family: var(--mono); font-size: 12.5px; } | ||||
| 102 | +.v-repare { color: var(--ok); } .v-echec, .v-inconnu { color: var(--broken); } | ||||
| 103 | +.v-site_source_mort { color: var(--muted); } .v-rien_a_faire { color: var(--stale); } | ||||
| 104 | +.v-running { color: var(--accent); } | ||||
| 105 | + | ||||
| 106 | +footer { padding: 26px clamp(16px, 4vw, 48px); border-top: 1px solid var(--line); color: var(--ink2); | ||||
| 107 | + font-size: 13.5px; max-width: 1500px; margin: 0 auto; } | ||||
| 108 | + | ||||
| 109 | +/* Modal */ | ||||
| 110 | +.modal { position: fixed; inset: 0; background: rgba(5, 7, 11, 0.75); display: grid; place-items: center; z-index: 50; } | ||||
| 111 | +.modal-box { background: var(--panel); border: 1px solid var(--line); border-radius: 16px; width: min(880px, 94vw); | ||||
| 112 | + max-height: 86vh; display: flex; flex-direction: column; } | ||||
| 113 | +.modal-head { display: flex; justify-content: space-between; align-items: center; padding: 14px 18px; | ||||
| 114 | + border-bottom: 1px solid var(--line); } | ||||
| 115 | +.modal-head button { background: none; border: none; color: var(--ink2); font-size: 18px; cursor: pointer; } | ||||
| 116 | +.modal-body { padding: 16px 18px; overflow-y: auto; font: 12.5px/1.6 var(--mono); display: grid; gap: 8px; } | ||||
| 117 | +.modal-body .meta { background: var(--panel2); border-radius: 10px; padding: 10px 12px; color: var(--ink2); | ||||
| 118 | + white-space: pre-wrap; overflow-wrap: anywhere; } | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/runner/runner.py
+287 −0
@@ -0,0 +1,287 @@ | |||||
| 1 | +# ============================================ | ||||
| 2 | +# Projet : KA Guardian | ||||
| 3 | +# Fichier : runner/runner.py | ||||
| 4 | +# Rôle : Bras d'exécution sur chaque nœud d'app — lance les missions | ||||
| 5 | +# claude -p headless dans le repo de l'app, streame les événements | ||||
| 6 | +# vers l'orchestrateur (ka2/ka4/ka6), et sait rollback + restart. | ||||
| 7 | +# Author : Simon-Pierre Boucher | ||||
| 8 | +# Date : 2026-08-23 | ||||
| 9 | +# ============================================ | ||||
| 10 | +"""KA Guardian Runner. | ||||
| 11 | + | ||||
| 12 | +Un seul runner par nœud (port 7791), une seule mission à la fois par nœud. | ||||
| 13 | +Auth: header X-KA-Token == $KA_GUARDIAN_TOKEN (fichier ~/.ka-guardian.env). | ||||
| 14 | +""" | ||||
| 15 | +from __future__ import annotations | ||||
| 16 | + | ||||
| 17 | +import json | ||||
| 18 | +import os | ||||
| 19 | +import pathlib | ||||
| 20 | +import shlex | ||||
| 21 | +import socket | ||||
| 22 | +import subprocess | ||||
| 23 | +import threading | ||||
| 24 | +import time | ||||
| 25 | +from typing import Any | ||||
| 26 | + | ||||
| 27 | +import httpx | ||||
| 28 | +from fastapi import FastAPI, Header, HTTPException, Request | ||||
| 29 | +from pydantic import BaseModel | ||||
| 30 | + | ||||
| 31 | +HOME = pathlib.Path.home() | ||||
| 32 | +ENV_FILE = HOME / ".ka-guardian.env" | ||||
| 33 | +CLAUDE_ENV_FILE = HOME / ".claude" / ".env" | ||||
| 34 | +TRANSCRIPTS = HOME / "ka-guardian-runner" / "transcripts" | ||||
| 35 | +TRANSCRIPTS.mkdir(parents=True, exist_ok=True) | ||||
| 36 | +CLAUDE_BIN = "/opt/homebrew/bin/claude" | ||||
| 37 | + | ||||
| 38 | + | ||||
| 39 | +def load_env_file(path: pathlib.Path) -> dict[str, str]: | ||||
| 40 | + out: dict[str, str] = {} | ||||
| 41 | + if path.exists(): | ||||
| 42 | + for line in path.read_text().splitlines(): | ||||
| 43 | + line = line.strip() | ||||
| 44 | + if line and not line.startswith("#") and "=" in line: | ||||
| 45 | + k, v = line.split("=", 1) | ||||
| 46 | + out[k.strip()] = v.strip().strip('"').strip("'") | ||||
| 47 | + return out | ||||
| 48 | + | ||||
| 49 | + | ||||
| 50 | +LOCAL_ENV = load_env_file(ENV_FILE) | ||||
| 51 | +TOKEN = LOCAL_ENV.get("KA_GUARDIAN_TOKEN", "") | ||||
| 52 | +NODE = LOCAL_ENV.get("KA_GUARDIAN_NODE", socket.gethostname()) | ||||
| 53 | + | ||||
| 54 | +app = FastAPI(title="KA Guardian Runner", docs_url=None, redoc_url=None) | ||||
| 55 | + | ||||
| 56 | +_lock = threading.Lock() | ||||
| 57 | +_current: dict[str, Any] = {} # mission en cours (métadonnées) | ||||
| 58 | + | ||||
| 59 | + | ||||
| 60 | +def check_token(x_ka_token: str | None) -> None: | ||||
| 61 | + if not TOKEN or x_ka_token != TOKEN: | ||||
| 62 | + raise HTTPException(status_code=401, detail="token invalide") | ||||
| 63 | + | ||||
| 64 | + | ||||
| 65 | +def expand(dir_: str) -> str: | ||||
| 66 | + return os.path.expanduser(dir_) | ||||
| 67 | + | ||||
| 68 | + | ||||
| 69 | +def git(dir_: str, *args: str) -> subprocess.CompletedProcess: | ||||
| 70 | + return subprocess.run( | ||||
| 71 | + ["git", "-C", expand(dir_), *args], | ||||
| 72 | + capture_output=True, text=True, timeout=120, | ||||
| 73 | + ) | ||||
| 74 | + | ||||
| 75 | + | ||||
| 76 | +def sh(cmd: str, timeout: int = 120) -> subprocess.CompletedProcess: | ||||
| 77 | + env = {**os.environ, "PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"} | ||||
| 78 | + return subprocess.run(["/bin/zsh", "-c", cmd], capture_output=True, text=True, timeout=timeout, env=env) | ||||
| 79 | + | ||||
| 80 | + | ||||
| 81 | +def healthcheck(web_port: int, pm2_names: list[str]) -> dict[str, Any]: | ||||
| 82 | + """Vérifie que l'app répond en HTTP et que ses process pm2 sont online.""" | ||||
| 83 | + http_ok = False | ||||
| 84 | + try: | ||||
| 85 | + r = httpx.get(f"http://127.0.0.1:{web_port}/", timeout=10, follow_redirects=True) | ||||
| 86 | + http_ok = r.status_code < 500 | ||||
| 87 | + except Exception: | ||||
| 88 | + http_ok = False | ||||
| 89 | + pm2_status: dict[str, str] = {} | ||||
| 90 | + try: | ||||
| 91 | + out = sh("pm2 jlist").stdout | ||||
| 92 | + procs = {p["name"]: p.get("pm2_env", {}).get("status", "?") for p in json.loads(out)} | ||||
| 93 | + for name in pm2_names: | ||||
| 94 | + pm2_status[name] = procs.get(name, "absent") | ||||
| 95 | + except Exception as exc: # pm2 absent ou jlist illisible | ||||
| 96 | + pm2_status = {n: f"inconnu ({exc})" for n in pm2_names} | ||||
| 97 | + ok = http_ok and all(s in ("online", "launching") for s in pm2_status.values()) | ||||
| 98 | + return {"ok": ok, "http_ok": http_ok, "pm2": pm2_status} | ||||
| 99 | + | ||||
| 100 | + | ||||
| 101 | +class MissionIn(BaseModel): | ||||
| 102 | + mission_id: str | ||||
| 103 | + agent: str | ||||
| 104 | + service: str | ||||
| 105 | + source: str | ||||
| 106 | + dir: str | ||||
| 107 | + pm2: list[str] | ||||
| 108 | + web_port: int | ||||
| 109 | + model: str = "sonnet" | ||||
| 110 | + max_turns: int = 70 | ||||
| 111 | + timeout_seconds: int = 3600 | ||||
| 112 | + prompt: str | ||||
| 113 | + callback_url: str # http://<orchestrateur>/api/ingest/<mission_id> | ||||
| 114 | + | ||||
| 115 | + | ||||
| 116 | +def post_event(url: str, payload: dict[str, Any]) -> None: | ||||
| 117 | + try: | ||||
| 118 | + httpx.post(url, json=payload, headers={"X-KA-Token": TOKEN}, timeout=15) | ||||
| 119 | + except Exception: | ||||
| 120 | + pass # l'orchestrateur relira le transcript au besoin | ||||
| 121 | + | ||||
| 122 | + | ||||
| 123 | +def condense_stream_line(obj: dict[str, Any]) -> list[dict[str, Any]]: | ||||
| 124 | + """Transforme une ligne stream-json de claude en événements courts pour le feed.""" | ||||
| 125 | + events: list[dict[str, Any]] = [] | ||||
| 126 | + t = obj.get("type") | ||||
| 127 | + if t == "system" and obj.get("subtype") == "init": | ||||
| 128 | + events.append({"type": "init", "data": {"model": obj.get("model", "?")}}) | ||||
| 129 | + elif t == "assistant": | ||||
| 130 | + for block in obj.get("message", {}).get("content", []): | ||||
| 131 | + if block.get("type") == "text" and block.get("text", "").strip(): | ||||
| 132 | + events.append({"type": "text", "data": {"text": block["text"][:1500]}}) | ||||
| 133 | + elif block.get("type") == "tool_use": | ||||
| 134 | + inp = json.dumps(block.get("input", {}), ensure_ascii=False) | ||||
| 135 | + events.append({"type": "tool", "data": {"name": block.get("name", "?"), "input": inp[:400]}}) | ||||
| 136 | + elif t == "result": | ||||
| 137 | + events.append({"type": "result", "data": { | ||||
| 138 | + "subtype": obj.get("subtype"), | ||||
| 139 | + "result": (obj.get("result") or "")[:4000], | ||||
| 140 | + "cost_usd": obj.get("total_cost_usd"), | ||||
| 141 | + "num_turns": obj.get("num_turns"), | ||||
| 142 | + "duration_ms": obj.get("duration_ms"), | ||||
| 143 | + }}) | ||||
| 144 | + return events | ||||
| 145 | + | ||||
| 146 | + | ||||
| 147 | +def extract_verdict(result_text: str) -> dict[str, Any]: | ||||
| 148 | + """Extrait le dernier bloc JSON {verdict: ...} de la réponse finale.""" | ||||
| 149 | + import re | ||||
| 150 | + for m in reversed(re.findall(r"\{[^{}]*\"verdict\"[\s\S]*?\}", result_text)): | ||||
| 151 | + try: | ||||
| 152 | + v = json.loads(m) | ||||
| 153 | + if "verdict" in v: | ||||
| 154 | + return v | ||||
| 155 | + except Exception: | ||||
| 156 | + continue | ||||
| 157 | + return {"verdict": "inconnu", "diagnostic": result_text[-500:] if result_text else ""} | ||||
| 158 | + | ||||
| 159 | + | ||||
| 160 | +def run_mission(m: MissionIn) -> None: | ||||
| 161 | + global _current | ||||
| 162 | + workdir = expand(m.dir) | ||||
| 163 | + transcript = TRANSCRIPTS / f"{m.mission_id}.jsonl" | ||||
| 164 | + base_commit = "" | ||||
| 165 | + try: | ||||
| 166 | + # Snapshot pré-mission : tree sale → commit de sûreté pour un rollback net. | ||||
| 167 | + if git(m.dir, "status", "--porcelain").stdout.strip(): | ||||
| 168 | + git(m.dir, "add", "-A") | ||||
| 169 | + git(m.dir, "commit", "-m", f"[{m.agent}] snapshot pré-mission {m.source}") | ||||
| 170 | + base_commit = git(m.dir, "rev-parse", "HEAD").stdout.strip() | ||||
| 171 | + post_event(m.callback_url, {"type": "start", "data": {"base_commit": base_commit, "node": NODE}}) | ||||
| 172 | + | ||||
| 173 | + env = { | ||||
| 174 | + **os.environ, | ||||
| 175 | + **load_env_file(CLAUDE_ENV_FILE), | ||||
| 176 | + "PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", | ||||
| 177 | + "HOME": str(HOME), | ||||
| 178 | + } | ||||
| 179 | + cmd = [ | ||||
| 180 | + CLAUDE_BIN, "-p", m.prompt, | ||||
| 181 | + "--output-format", "stream-json", "--verbose", | ||||
| 182 | + "--model", m.model, | ||||
| 183 | + "--max-turns", str(m.max_turns), | ||||
| 184 | + "--dangerously-skip-permissions", | ||||
| 185 | + ] | ||||
| 186 | + proc = subprocess.Popen( | ||||
| 187 | + cmd, cwd=workdir, env=env, | ||||
| 188 | + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, bufsize=1, | ||||
| 189 | + ) | ||||
| 190 | + _current["pid"] = proc.pid | ||||
| 191 | + result_text, cost, turns = "", None, None | ||||
| 192 | + deadline = time.time() + m.timeout_seconds | ||||
| 193 | + with transcript.open("w") as tf: | ||||
| 194 | + for line in proc.stdout: # type: ignore[union-attr] | ||||
| 195 | + tf.write(line) | ||||
| 196 | + if time.time() > deadline: | ||||
| 197 | + proc.kill() | ||||
| 198 | + post_event(m.callback_url, {"type": "error", "data": {"error": "timeout mission"}}) | ||||
| 199 | + break | ||||
| 200 | + line = line.strip() | ||||
| 201 | + if not line: | ||||
| 202 | + continue | ||||
| 203 | + try: | ||||
| 204 | + obj = json.loads(line) | ||||
| 205 | + except Exception: | ||||
| 206 | + continue | ||||
| 207 | + for ev in condense_stream_line(obj): | ||||
| 208 | + if ev["type"] == "result": | ||||
| 209 | + result_text = ev["data"].get("result", "") | ||||
| 210 | + cost = ev["data"].get("cost_usd") | ||||
| 211 | + turns = ev["data"].get("num_turns") | ||||
| 212 | + post_event(m.callback_url, ev) | ||||
| 213 | + proc.wait(timeout=60) | ||||
| 214 | + | ||||
| 215 | + commits_raw = git(m.dir, "rev-list", "--oneline", f"{base_commit}..HEAD").stdout.strip() | ||||
| 216 | + commits = commits_raw.splitlines() if commits_raw else [] | ||||
| 217 | + verdict = extract_verdict(result_text) | ||||
| 218 | + health = healthcheck(m.web_port, m.pm2) | ||||
| 219 | + post_event(m.callback_url, {"type": "final", "data": { | ||||
| 220 | + "verdict": verdict, "commits": commits, "base_commit": base_commit, | ||||
| 221 | + "cost_usd": cost, "num_turns": turns, "health": health, | ||||
| 222 | + "exit_code": proc.returncode, | ||||
| 223 | + }}) | ||||
| 224 | + except Exception as exc: | ||||
| 225 | + post_event(m.callback_url, {"type": "error", "data": {"error": str(exc), "base_commit": base_commit}}) | ||||
| 226 | + finally: | ||||
| 227 | + with _lock: | ||||
| 228 | + _current.clear() | ||||
| 229 | + | ||||
| 230 | + | ||||
| 231 | +@app.get("/health") | ||||
| 232 | +def health() -> dict[str, Any]: | ||||
| 233 | + return {"ok": True, "node": NODE, "busy": bool(_current), "current": _current.get("mission_id")} | ||||
| 234 | + | ||||
| 235 | + | ||||
| 236 | +@app.post("/missions") | ||||
| 237 | +def missions(m: MissionIn, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | ||||
| 238 | + check_token(x_ka_token) | ||||
| 239 | + workdir = expand(m.dir) | ||||
| 240 | + if not os.path.isdir(workdir): | ||||
| 241 | + raise HTTPException(status_code=400, detail=f"dir introuvable: {workdir}") | ||||
| 242 | + with _lock: | ||||
| 243 | + if _current: | ||||
| 244 | + raise HTTPException(status_code=409, detail=f"mission déjà en cours: {_current.get('mission_id')}") | ||||
| 245 | + _current.update({"mission_id": m.mission_id, "service": m.service, "source": m.source, "started": time.time()}) | ||||
| 246 | + threading.Thread(target=run_mission, args=(m,), daemon=True).start() | ||||
| 247 | + return {"accepted": True, "node": NODE} | ||||
| 248 | + | ||||
| 249 | + | ||||
| 250 | +class RollbackIn(BaseModel): | ||||
| 251 | + dir: str | ||||
| 252 | + base_commit: str | ||||
| 253 | + pm2: list[str] | ||||
| 254 | + web_port: int | ||||
| 255 | + | ||||
| 256 | + | ||||
| 257 | +@app.post("/rollback") | ||||
| 258 | +def rollback(r: RollbackIn, x_ka_token: str | None = Header(default=None)) -> dict[str, Any]: | ||||
| 259 | + """Revient au commit d'avant mission (reset --hard si tous les commits sont de l'agent, sinon revert).""" | ||||
| 260 | + check_token(x_ka_token) | ||||
| 261 | + log = git(r.dir, "log", "--format=%s", f"{r.base_commit}..HEAD").stdout.strip() | ||||
| 262 | + msgs = log.splitlines() if log else [] | ||||
| 263 | + if not msgs: | ||||
| 264 | + mode = "aucun_commit" | ||||
| 265 | + elif all(s.startswith("[ka") for s in msgs): | ||||
| 266 | + git(r.dir, "reset", "--hard", r.base_commit) | ||||
| 267 | + mode = "reset_hard" | ||||
| 268 | + else: | ||||
| 269 | + rc = git(r.dir, "revert", "--no-edit", f"{r.base_commit}..HEAD") | ||||
| 270 | + if rc.returncode != 0: | ||||
| 271 | + git(r.dir, "revert", "--abort") | ||||
| 272 | + git(r.dir, "reset", "--hard", r.base_commit) | ||||
| 273 | + mode = "reset_hard(fallback)" | ||||
| 274 | + else: | ||||
| 275 | + mode = "revert" | ||||
| 276 | + # Pas de git clean : les process de sync de l'app écrivent en continu, | ||||
| 277 | + # on ne supprime jamais de fichiers non trackés apparus pendant la mission. | ||||
| 278 | + for name in r.pm2: | ||||
| 279 | + sh(f"pm2 restart {shlex.quote(name)} --update-env", timeout=180) | ||||
| 280 | + time.sleep(8) | ||||
| 281 | + h = healthcheck(r.web_port, r.pm2) | ||||
| 282 | + return {"ok": True, "mode": mode, "health": h, "head": git(r.dir, "rev-parse", "HEAD").stdout.strip()} | ||||
| 283 | + | ||||
| 284 | + | ||||
| 285 | +if __name__ == "__main__": | ||||
| 286 | + import uvicorn | ||||
| 287 | + uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("KA_GUARDIAN_RUNNER_PORT", "7791"))) | ||||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/topology.json
+62 −0
@@ -0,0 +1,62 @@ | |||||
| 1 | +{ | ||||
| 2 | + "comment": "KA Guardian — topologie des agents, services et nœuds. Source de vérité partagée orchestrateurs/runners.", | ||||
| 3 | + "apika_monitoring_url": "http://192.168.2.82:8000/api/v1/monitoring/connectors", | ||||
| 4 | + "runner_port": 7791, | ||||
| 5 | + "agents": { | ||||
| 6 | + "ka2": { | ||||
| 7 | + "port": 8799, | ||||
| 8 | + "domain": "www.ka2.bot", | ||||
| 9 | + "accent": "#22d3ee", | ||||
| 10 | + "accent2": "#0891b2", | ||||
| 11 | + "tagline": "Gardien immobilier & local", | ||||
| 12 | + "model": "sonnet", | ||||
| 13 | + "services": ["louka", "immoka", "restoka"] | ||||
| 14 | + }, | ||||
| 15 | + "ka4": { | ||||
| 16 | + "port": 8899, | ||||
| 17 | + "domain": "www.ka4.bot", | ||||
| 18 | + "accent": "#fbbf24", | ||||
| 19 | + "accent2": "#d97706", | ||||
| 20 | + "tagline": "Gardien mobilité & quotidien", | ||||
| 21 | + "model": "sonnet", | ||||
| 22 | + "services": ["autoka", "foodka", "sortika"] | ||||
| 23 | + }, | ||||
| 24 | + "ka6": { | ||||
| 25 | + "port": 8999, | ||||
| 26 | + "domain": "www.ka6.bot", | ||||
| 27 | + "accent": "#a78bfa", | ||||
| 28 | + "accent2": "#7c3aed", | ||||
| 29 | + "tagline": "Gardien flagship — gros volumes", | ||||
| 30 | + "model": "sonnet", | ||||
| 31 | + "services": ["fabrika", "jobka", "creaka"], | ||||
| 32 | + "default_for_unknown_services": true | ||||
| 33 | + } | ||||
| 34 | + }, | ||||
| 35 | + "nodes": { | ||||
| 36 | + "m3u96a": { "lan_ip": "192.168.2.87", "alias": "M3U96a" }, | ||||
| 37 | + "m3u96b": { "lan_ip": "192.168.2.82", "alias": "M3U96b" }, | ||||
| 38 | + "m4m64a": { "lan_ip": "192.168.2.83", "alias": "M4M64a" }, | ||||
| 39 | + "m4m64b": { "lan_ip": "192.168.2.78", "alias": "M4M64b" } | ||||
| 40 | + }, | ||||
| 41 | + "services": { | ||||
| 42 | + "louka": { "app": "Lou·Ka", "node": "m3u96b", "dir": "~/apps/lou-ka", "pm2": ["lou-ka-web", "lou-ka-sync"], "web_port": 8095, "site": "https://www.lou-ka.com" }, | ||||
| 43 | + "restoka": { "app": "Resto·Ka", "node": "m3u96b", "dir": "~/apps/resto-ka", "pm2": ["resto-ka", "resto-ka-sync"], "web_port": 8115, "site": "https://www.resto-ka.com" }, | ||||
| 44 | + "creaka": { "app": "Créa·Ka", "node": "m3u96b", "dir": "~/apps/crea-ka", "pm2": ["crea-ka-web", "crea-ka-sync"], "web_port": 8160, "site": "https://www.crea-ka.com" }, | ||||
| 45 | + "immoka": { "app": "Immo·Ka", "node": "m4m64a", "dir": "~/apps/immo-ka", "pm2": ["immo-ka-web", "immo-ka-sync"], "web_port": 8096, "site": "https://www.immo-ka.com" }, | ||||
| 46 | + "fabrika": { "app": "Fabri·Ka", "node": "m4m64a", "dir": "~/fabri-ka", "pm2": ["fabri-ka-web", "fabri-ka-sync"], "web_port": 8097, "site": "https://www.fabri-ka.com" }, | ||||
| 47 | + "autoka": { "app": "Auto·Ka", "node": "m4m64b", "dir": "~/auto-ka", "pm2": ["auto-ka-web", "auto-ka-sync"], "web_port": 8095, "site": "https://www.auto-ka.com" }, | ||||
| 48 | + "foodka": { "app": "Food·Ka", "node": "m4m64b", "dir": "~/apps/food-ka", "pm2": ["food-ka-web", "food-ka-sync"], "web_port": 8097, "site": "https://www.food-ka.com" }, | ||||
| 49 | + "sortika": { "app": "Sorti·Ka", "node": "m3u96a", "dir": "~/apps/sorti-ka", "pm2": ["sorti-ka-web", "sorti-ka-sync"], "web_port": 8120, "site": "https://www.sorti-ka.com" }, | ||||
| 50 | + "jobka": { "app": "Job·Ka", "node": "m3u96a", "dir": "~/apps/job-ka", "pm2": ["job-ka-web", "job-ka-sync"], "web_port": 8096, "site": "https://www.job-ka.com" } | ||||
| 51 | + }, | ||||
| 52 | + "policy": { | ||||
| 53 | + "poll_interval_seconds": 300, | ||||
| 54 | + "trigger_statuses": ["broken", "stale"], | ||||
| 55 | + "max_concurrent_missions": 1, | ||||
| 56 | + "max_attempts_per_incident": 3, | ||||
| 57 | + "attempt_cooldown_hours": 6, | ||||
| 58 | + "watch_window_hours": 8, | ||||
| 59 | + "mission_max_turns": 70, | ||||
| 60 | + "mission_timeout_seconds": 3600 | ||||
| 61 | + } | ||||
| 62 | +} | ||||
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/deploy.sh
+2 −1
@@ -60,7 +60,8 @@ deploy_orchestrators() { | |||||
| 60 | rsync -az --delete --exclude data --exclude .venv --exclude .git \ | 60 | rsync -az --delete --exclude data --exclude .venv --exclude .git \ | ||
| 61 | "$ROOT/" $ORCH_NODE:cluster-projects/ka-guardian/ | 61 | "$ROOT/" $ORCH_NODE:cluster-projects/ka-guardian/ | ||
| 62 | ssh $ORCH_NODE "cd ~/cluster-projects/ka-guardian | 62 | ssh $ORCH_NODE "cd ~/cluster-projects/ka-guardian | ||
| 63 | − [[ -d .venv ]] || $PY -m venv .venv | 63 | + PY=\$(command -v /opt/homebrew/bin/python3 || command -v /opt/homebrew/bin/python3.13) | ||
| 64 | + [[ -d .venv ]] || \$PY -m venv .venv | ||||
| 64 | ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' >/dev/null | 65 | ./.venv/bin/pip -q install 'fastapi>=0.110' 'uvicorn>=0.29' 'httpx>=0.27' >/dev/null | ||
| 65 | mkdir -p data logs | 66 | mkdir -p data logs | ||
| 66 | # retirer les anciens bots (web+bot), garder les tunnels ngrok | 67 | # retirer les anciens bots (web+bot), garder les tunnels ngrok | ||
added
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/courier.sh
+36 −0
@@ -0,0 +1,36 @@ | |||||
| 1 | +#!/bin/zsh | ||||
| 2 | +# ============================================ | ||||
| 3 | +# KA Guardian — courrier inter-nœuds (100 % zsh) | ||||
| 4 | +# macOS 26 « Local Network Privacy » refuse le trafic LAN dès que python | ||||
| 5 | +# (homebrew) est dans la chaîne de processus — même ses enfants ssh/curl. | ||||
| 6 | +# Ce service launchd (binaire responsable: zsh) expédie donc les requêtes: | ||||
| 7 | +# spool/outbox/<id>.job → ssh <nœud> curl http://127.0.0.1:<port><path> | ||||
| 8 | +# spool/done/<id>.resp ← corps de réponse + dernière ligne = code HTTP | ||||
| 9 | +# Format .job: ligne 1 = "<ip> <port> <path> <timeout>", reste = payload JSON. | ||||
| 10 | +# ============================================ | ||||
| 11 | +SPOOL=$HOME/ka-guardian-spool | ||||
| 12 | +KEY=$HOME/.ssh/ka_guardian_ed25519 | ||||
| 13 | +mkdir -p $SPOOL/outbox $SPOOL/done $SPOOL/tmp | ||||
| 14 | +source $HOME/.ka-guardian.env 2>/dev/null || true | ||||
| 15 | + | ||||
| 16 | +while true; do | ||||
| 17 | + for f in $SPOOL/outbox/*.job(N); do | ||||
| 18 | + id=${f:t:r} | ||||
| 19 | + hdr=$(head -1 $f) | ||||
| 20 | + parts=(${(z)hdr}) | ||||
| 21 | + ip=$parts[1]; port=$parts[2]; path=$parts[3]; tmo=${parts[4]:-30} | ||||
| 22 | + tail -n +2 $f | /usr/bin/ssh -i $KEY \ | ||||
| 23 | + -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ConnectTimeout=8 \ | ||||
| 24 | + -o ControlMaster=auto -o ControlPath=/tmp/kg-cm-%h -o ControlPersist=120 \ | ||||
| 25 | + simon-pierreboucher@$ip \ | ||||
| 26 | + "/usr/bin/curl -s -m $tmo -X POST http://127.0.0.1:$port$path -H 'Content-Type: application/json' -H 'X-KA-Token: $KA_GUARDIAN_TOKEN' -d @- -w '\n%{http_code}'" \ | ||||
| 27 | + > $SPOOL/tmp/$id.resp 2>>$SPOOL/courier.log | ||||
| 28 | + rc=$? | ||||
| 29 | + [[ $rc -ne 0 ]] && print "\ncourier_ssh_rc_$rc" >> $SPOOL/tmp/$id.resp | ||||
| 30 | + mv $SPOOL/tmp/$id.resp $SPOOL/done/$id.resp | ||||
| 31 | + rm -f $f | ||||
| 32 | + done | ||||
| 33 | + # ménage: réponses jamais réclamées depuis > 1 h | ||||
| 34 | + for old in $SPOOL/done/*.resp(N.mh+1); do rm -f $old; done | ||||
| 35 | + sleep 1 | ||||
| 36 | +done | ||||
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/deploy/deploy.sh
+30 −1
@@ -21,6 +21,34 @@ if [[ ! -f $TOKEN_FILE ]]; then | |||||
| 21 | fi | 21 | fi | ||
| 22 | TOKEN_LINE=$(grep KA_GUARDIAN_TOKEN "$TOKEN_FILE") | 22 | TOKEN_LINE=$(grep KA_GUARDIAN_TOKEN "$TOKEN_FILE") | ||
| 23 | 23 | ||||
| 24 | +deploy_courier() { | ||||
| 25 | + # Courrier zsh sur TOUS les nœuds (orchestrateur + runners): expédie le | ||||
| 26 | + # trafic LAN inter-nœuds via ssh (macOS 26 Local Network Privacy). | ||||
| 27 | + for n in $RUNNER_NODES $ORCH_NODE; do | ||||
| 28 | + echo "=== courrier → $n" | ||||
| 29 | + scp -q "$ROOT/deploy/courier.sh" $n:ka-guardian-courier.sh | ||||
| 30 | + ssh $n 'chmod +x ~/ka-guardian-courier.sh | ||||
| 31 | + cat > ~/Library/LaunchAgents/com.ka.guardian-courier.plist <<PLIST | ||||
| 32 | +<?xml version="1.0" encoding="UTF-8"?> | ||||
| 33 | +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> | ||||
| 34 | +<plist version="1.0"><dict> | ||||
| 35 | + <key>Label</key><string>com.ka.guardian-courier</string> | ||||
| 36 | + <key>ProgramArguments</key><array> | ||||
| 37 | + <string>/bin/zsh</string> | ||||
| 38 | + <string>/Users/simon-pierreboucher/ka-guardian-courier.sh</string> | ||||
| 39 | + </array> | ||||
| 40 | + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/> | ||||
| 41 | + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/courier.log</string> | ||||
| 42 | + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/courier.log</string> | ||||
| 43 | +</dict></plist> | ||||
| 44 | +PLIST | ||||
| 45 | + mkdir -p ~/ka-guardian-spool | ||||
| 46 | + launchctl unload ~/Library/LaunchAgents/com.ka.guardian-courier.plist 2>/dev/null || true | ||||
| 47 | + launchctl load ~/Library/LaunchAgents/com.ka.guardian-courier.plist' \ | ||||
| 48 | + && echo " ✓ $n courrier ok" || echo " ✗ $n courrier KO" | ||||
| 49 | + done | ||||
| 50 | +} | ||||
| 51 | + | ||||
| 24 | deploy_runners() { | 52 | deploy_runners() { | ||
| 25 | for n in $RUNNER_NODES; do | 53 | for n in $RUNNER_NODES; do | ||
| 26 | echo "=== runner → $n" | 54 | echo "=== runner → $n" | ||
@@ -99,8 +127,9 @@ PLIST | |||||
| 99 | } | 127 | } | ||
| 100 | 128 | ||||
| 101 | case "${1:-all}" in | 129 | case "${1:-all}" in | ||
| 130 | + courier) deploy_courier ;; | ||||
| 102 | runners) deploy_runners ;; | 131 | runners) deploy_runners ;; | ||
| 103 | orchestrators) deploy_orchestrators ;; | 132 | orchestrators) deploy_orchestrators ;; | ||
| 104 | − all) deploy_runners; deploy_orchestrators ;; | 133 | + all) deploy_courier; deploy_runners; deploy_orchestrators ;; | ||
| 105 | esac | 134 | esac | ||
| 106 | echo "terminé." | 135 | echo "terminé." | ||
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/main.py
+245 −33
Large diff (352 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/app.js
+97 −38
Large diff (201 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/index.html
+108 −30
Large diff (173 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/orchestrator/web/style.css
+200 −99
Large diff (318 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/runner/runner.py
+21 −1
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/M4M36luster-projects/ka-guardian/topology.json
+4 −2
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/deploy/courier.sh
+17 −8
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/deploy/deploy.sh
+13 −2
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/main.py
+127 −18
Large diff (225 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/web/app.js
+10 −60
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/commander.html
+109 −0
Large diff (109 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/commander.js
+101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/favicon-ka2.svg
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/favicon-ka4.svg
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
M4M36luster-projects/ka-guardian/orchestrator/web/favicon-ka6.svg
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/web/index.html
+6 −40
Large diff (64 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/orchestrator/web/style.css
+21 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/runner/runner.py
+58 −3
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
modified
M4M36luster-projects/ka-guardian/topology.json
+148 −28
Large diff (205 lines) is not loaded by default to keep the page fast.Load diff
modified
deploy/courier.sh
+13 −7
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
modified
deploy/deploy.sh
+12 −4
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified
orchestrator/main.py
+49 −8
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
modified
runner/runner.py
+27 −2
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff