SPB Git
spboucher.ai

spb/maclustr-dispatch

Public

maclustr-dispatch v1.0.0 — mld : découverte LAN, scan, picker, deploy/move/health, prepare/bootstrap, manifestes (2026-09-04)

Simon-Pierre Boucher committed (Sep 4, 2026) Browse files

18 changed files +1,529 −0

18 changed files
Unified Split
added .gitignore +3 −0
@@ -0,0 +1,3 @@
1+__pycache__/
2+*.pyc
3+.DS_Store
added README.md +65 −0
@@ -0,0 +1,65 @@
1+# maclustr-dispatch — `mld`
2+
3+Orchestrateur de déploiement du cluster MacLustr. Il tourne **sur la passerelle M1M32** et
4+déploie chaque application sur le **nœud le plus efficient** du cluster, via le LAN
5+(192.168.2.x, clé SSH `maclustr-m1m32` autorisée sur tous les nœuds).
6+
7+```
8+laptop ──ssh──▶ M1M32 (passerelle : spbgit + mld + staging ~/dispatch/stage)
9+ │ scan live (RAM libre, charge, disque, ports, runtimes)
10+ │ choix du nœud (score) → rsync LAN → PM2/launchd → ngrok → healthcheck → registre
11+ ▼
12+ M3U96a · M3U96b · M2U64 · M4M64a · M4M64b · M4M36 · M2M32 · …
13+```
14+
15+## Depuis le laptop
16+
17+`~/Desktop/cluster-skill/mld <commande>` relaie à la passerelle et rapatrie le registre
18+(`cluster-skill/cluster-registry.json`).
19+
20+| Commande | Effet |
21+|---|---|
22+| `mld scan` / `mld nodes` | sonde (ou relit) ressources + runtimes des nœuds |
23+| `mld apps` / `mld show <app>` | manifestes et emplacement courant |
24+| `mld pick <app>` · `mld pick --ram 4 --runtime node` | classement des nœuds pour une app / un besoin |
25+| `mld plan [apps…]` | plan de placement global (glouton, projection des ressources) |
26+| `mld stage <dir-local> <app>` | pousse un dossier du laptop dans le staging de la passerelle |
27+| `mld deploy <app> [--node X]` | staging → nœud choisi (auto sans `--node`), démarrage, santé, registre |
28+| `mld move <app> [--to X] [--from Y]` | migration : pré-copie à chaud, arrêt, delta final, déploiement, santé, retrait (rollback auto) |
29+| `mld stop/start/restart <app>` · `mld logs <app>` · `mld health <app>` | cycle de vie |
30+| `mld status [--live]` · `mld registry` | registre (`~/dispatch/registry.json` sur M1M32) |
31+| `mld retire <app>` | arrête et efface la copie du nœud |
32+| `mld prepare <nœud> [--ka-helpers]` | installe node/pnpm/pm2/ngrok/python via Homebrew, config ngrok, pm2 startup, marqueur ; `--ka-helpers` = pousseur Ka + alias `gitsrv` + clé autorisée sur la passerelle |
33+| `mld bootstrap <nœud>` | Xcode CLT + Homebrew sur un Mac vierge (sudo NOPASSWD temporaire) |
34+| `mld import <nœud>` | brouillons de manifestes depuis les PM2 vivants d'un nœud (`~/dispatch/apps/_drafts/`) |
35+
36+## Manifestes (`~/dispatch/apps/<app>.json`, mode 600)
37+
38+Un manifeste décrit *comment* tourne une app : `dir` (même chemin absolu sur tous les nœuds),
39+`processes` (PM2 : script, args, interpreter, cwd, env, cron_restart…), `ngrok` (domaine réservé →
40+processus `<app>-ngrok`), `launchd` (plists générés), `requires` (runtimes, RAM, ports),
41+`placement` (`pin`, `prefer`, `avoid` + raison), `sync_excludes`, `extra_paths`, `env_overrides`
42+(réécriture de `.env` au déploiement), `hooks.post_sync` / `post_start`, `ka_repo` (inscription au pousseur).
43+Gabarits : `{{HOME}}`, `{{NODE}}`, `{{LAN_IP}}`, `{{IP:<alias>}}`.
44+
45+## Scoring
46+
47+`0.45·RAM libre après placement + 0.25·cœurs idle + 0.10·disque + 0.15·cœurs − 0.04·apps hébergées
48+(+0.15 prefer, +0.05 si déjà sur ce nœud)`. Inéligible : hors ligne, rôle gateway (sauf épingle),
49+runtime manquant, RAM libre < besoin + 2 Go, disque, port occupé (hors nœud d'origine).
50+
51+## Ajouter un nœud
52+
53+1. Activer *Session à distance* sur le Mac, autoriser la clé de la passerelle
54+ (`~/.ssh/id_ed25519.pub` de M1M32) et celle du laptop dans `~/.ssh/authorized_keys`.
55+2. `echo <alias> > ~/.maclustr-node` sur le nœud ; ajouter l'alias dans `mld/config.py` (`NODES`).
56+3. `mld discover` → `mld bootstrap <alias>` (si Mac vierge) → `mld prepare <alias>`.
57+
58+## Fichiers
59+
60+- `mld/config.py` inventaire statique, rôles, chemins, détection des runtimes
61+- `mld/nodes.py` découverte LAN, sonde · `mld/picker.py` score/plan · `mld/deploy.py` fetch/deploy/move/health
62+- `mld/render.py` ecosystem PM2 + plists · `mld/prepare.py` runtimes/bootstrap/ka-helpers · `mld/importer.py`
63+- `assets/` pousseur Ka (script + plist)
64+
65+Pas de dépendance : Python 3 système (3.9) + ssh/rsync (GNU rsync via Homebrew sur la passerelle).
added assets/com.ka.pousseur.plist +12 −0
@@ -0,0 +1,12 @@
1+<?xml version="1.0" encoding="UTF-8"?>
2+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3+<plist version="1.0"><dict>
4+ <key>Label</key><string>com.ka.pousseur</string>
5+ <key>ProgramArguments</key><array>
6+ <string>/bin/zsh</string>
7+ <string>/Users/simon-pierreboucher/ka-pousseur.sh</string>
8+ </array>
9+ <key>RunAtLoad</key><true/><key>KeepAlive</key><true/>
10+ <key>StandardOutPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/pousseur.log</string>
11+ <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/pousseur.log</string>
12+</dict></plist>
added assets/ka-pousseur.sh +34 −0
@@ -0,0 +1,34 @@
1+#!/bin/zsh
2+# ============================================
3+# KA pousseur — pousse vers spbgit les commits en attente des repos Ka du nœud.
4+# Chaîne 100 % binaires Apple (launchd → zsh → /usr/bin/git → /usr/bin/ssh) :
5+# contourne macOS 26 Local Network Privacy, qui bloque les `git push` lancés
6+# par les missions Claude (descendants du runner python homebrew).
7+# Repos surveillés : un chemin par ligne dans ~/.ka-pousseur-repos.
8+# Push SIMPLE uniquement (jamais de force) : un non-fast-forward est loggé,
9+# jamais résolu d'autorité.
10+# ============================================
11+export PATH=/usr/bin:/bin:/usr/sbin:/sbin
12+SPOOL=~/ka-guardian-spool
13+LOG=$SPOOL/pousseur.log
14+REPOS_FILE=~/.ka-pousseur-repos
15+mkdir -p $SPOOL
16+
17+while true; do
18+ if [[ -f $REPOS_FILE ]]; then
19+ while IFS= read -r dir; do
20+ [[ -z $dir || $dir == \#* ]] && continue
21+ d=${dir/#\~/$HOME}
22+ [[ -d $d/.git ]] || continue
23+ ahead=$(/usr/bin/git -C "$d" rev-list --count origin/main..main 2>/dev/null)
24+ if [[ -n $ahead && $ahead -gt 0 ]]; then
25+ if /usr/bin/git -C "$d" push origin main >>$LOG 2>&1; then
26+ echo "$(date '+%F %T') poussé $ahead commit(s): $d" >>$LOG
27+ else
28+ echo "$(date '+%F %T') ÉCHEC push ($ahead en attente): $d" >>$LOG
29+ fi
30+ fi
31+ done <$REPOS_FILE
32+ fi
33+ sleep 300
34+done
added bin/mld +4 −0
@@ -0,0 +1,4 @@
1+#!/bin/bash
2+# mld — MacLustr Dispatch (à exécuter sur la passerelle M1M32)
3+export PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
4+cd "$(dirname "$0")/.." && exec /usr/bin/python3 -m mld "$@"
added mld/__init__.py +2 −0
@@ -0,0 +1,2 @@
1+"""maclustr-dispatch — orchestrateur de déploiement du cluster MacLustr (passerelle M1M32)."""
2+__version__ = "1.0.0"
added mld/__main__.py +3 −0
@@ -0,0 +1,3 @@
1+from .cli import main
2+
3+main()
added mld/cli.py +208 −0
@@ -0,0 +1,208 @@
1+"""CLI `mld` — MacLustr Dispatch (tourne sur la passerelle M1M32)."""
2+import argparse
3+import json
4+import sys
5+from . import config, deploy, manifest, nodes, picker, registry
6+
7+
8+def cmd_discover(a):
9+ nodes.discover()
10+
11+
12+def cmd_scan(a):
13+ nodes.scan(aliases=a.nodes or None)
14+
15+
16+def cmd_nodes(a):
17+ res = nodes.load_scan() or nodes.scan(verbose=False)
18+ nodes.print_scan(res)
19+
20+
21+def cmd_apps(a):
22+ ms = manifest.load_all()
23+ r = registry.load()["apps"]
24+ print("%-18s %-8s %-6s %-30s %-8s %s" % ("app", "nœud", "port", "domaine", "épingle", "runtimes"))
25+ for name, m in ms.items():
26+ cur = r.get(name, {}).get("node") or "—"
27+ print("%-18s %-8s %-6s %-30s %-8s %s" % (name, cur, m.get("port") or "", m.get("domain") or "", m["placement"].get("pin") or "", ",".join(m["requires"]["runtimes"])))
28+
29+
30+def cmd_show(a):
31+ print(json.dumps(manifest.load(a.app), indent=2, ensure_ascii=False))
32+
33+
34+def cmd_pick(a):
35+ m = manifest.load(a.app) if a.app else {"app": "(ad hoc)", "requires": {"runtimes": a.runtime or ["pm2"], "ram_gb": a.ram or 1, "ports": []}, "ram_mb_observed": int((a.ram or 1) * 1024), "size_mb": 0, "placement": {"pin": None, "prefer": None, "avoid": [], "reason": ""}}
36+ scan = nodes.load_scan() if not a.rescan else nodes.scan(verbose=False)
37+ rows = picker.rank(m, scan, exclude=a.exclude or ())
38+ if a.json:
39+ print(json.dumps(rows, indent=2))
40+ return
41+ print("classement pour %s :" % m["app"])
42+ for r in rows:
43+ if r["ok"]:
44+ print(" %-8s score %.3f" % (r["node"], r["score"]))
45+ elif scan.get(r["node"], {}).get("online") and scan[r["node"]].get("role") == "worker":
46+ print(" %-8s — %s" % (r["node"], r["why"]))
47+
48+
49+def cmd_plan(a):
50+ ms = manifest.load_all()
51+ if a.apps:
52+ ms = {k: v for k, v in ms.items() if k in a.apps}
53+ scan = nodes.scan(verbose=False) if a.rescan else (nodes.load_scan() or nodes.scan(verbose=False))
54+ rows = picker.plan(ms, scan, exclude=a.exclude or ())
55+ if a.json:
56+ print(json.dumps(rows, indent=2, ensure_ascii=False))
57+ return
58+ print("%-19s %-8s %-8s %-6s %7s %s" % ("app", "actuel", "proposé", "score", "taille", "note"))
59+ counts = {}
60+ moving_mb = 0
61+ for r in rows:
62+ counts[r["node"]] = counts.get(r["node"], 0) + 1
63+ mv = r["node"] and r["current"] and r["node"] != r["current"]
64+ if mv:
65+ moving_mb += r["size_mb"]
66+ note = ("épinglé — " + r["reason"]) if r["pin"] else (("alt: " + ", ".join("%s %.2f" % x for x in r["alternatives"])) if r["alternatives"] else "")
67+ if not r["node"]:
68+ note = "AUCUN NŒUD : " + " ; ".join("%s (%s)" % b for b in r["blockers"])
69+ flag = "→" if mv else ("=" if r["node"] else "!")
70+ print("%-19s %-8s %-8s %-6s %6dM %s %s" % (r["app"], r["current"] or "—", r["node"] or "—", ("%.3f" % r["score"]) if r["score"] is not None else "", r["size_mb"], flag, note))
71+ print("\nrépartition : " + ", ".join("%s×%s" % (k, v) for k, v in sorted(counts.items(), key=lambda x: -x[1]) if k))
72+ print("données à déplacer : %.1f Go (→ = migration, = = redéploiement en place)" % (moving_mb / 1024.0))
73+
74+
75+def cmd_fetch(a):
76+ deploy.fetch(a.app, a.from_node, final=a.final)
77+
78+
79+def cmd_deploy(a):
80+ if a.node:
81+ node = a.node
82+ else:
83+ scan = nodes.load_scan() or nodes.scan(verbose=False)
84+ rows = picker.rank(manifest.load(a.app), scan)
85+ if not rows or not rows[0]["ok"]:
86+ raise SystemExit("aucun nœud éligible : " + "; ".join("%s: %s" % (r["node"], r["why"]) for r in rows[:6]))
87+ node = rows[0]["node"]
88+ print("nœud choisi : %s (score %.3f)" % (node, rows[0]["score"]))
89+ try:
90+ deploy.deploy(a.app, node, reinstall=a.reinstall, skip_sync=a.skip_sync)
91+ except deploy.DeployError as e:
92+ raise SystemExit("ÉCHEC : %s" % e)
93+
94+
95+def cmd_move(a):
96+ to = a.to
97+ if not to:
98+ scan = nodes.load_scan() or nodes.scan(verbose=False)
99+ rows = picker.rank(manifest.load(a.app), scan, exclude=[registry.node_of(a.app)] if a.away else ())
100+ if not rows or not rows[0]["ok"]:
101+ raise SystemExit("aucun nœud éligible : " + "; ".join("%s: %s" % (r["node"], r["why"]) for r in rows[:6]))
102+ to = rows[0]["node"]
103+ print("nœud choisi : %s (score %.3f)" % (to, rows[0]["score"]))
104+ try:
105+ deploy.move(a.app, to, from_alias=a.from_node, keep_source=a.keep_source, reinstall=a.reinstall)
106+ except deploy.DeployError as e:
107+ raise SystemExit("ÉCHEC : %s" % e)
108+
109+
110+def cmd_stop(a):
111+ deploy.stop(a.app, a.node or registry.node_of(a.app))
112+ registry.set_app(a.app, status="stopped")
113+
114+
115+def cmd_start(a):
116+ node = a.node or registry.node_of(a.app)
117+ deploy.start(a.app, node)
118+ ok, det = deploy.health(a.app, node)
119+ registry.set_app(a.app, node=node, status="online" if ok else "unhealthy", health=det)
120+ print("santé :", det)
121+
122+
123+def cmd_restart(a):
124+ cmd_stop(a)
125+ cmd_start(a)
126+
127+
128+def cmd_retire(a):
129+ deploy.retire(a.app, a.node or registry.node_of(a.app), remove_dir=not a.keep_dir)
130+ if not a.keep_registry:
131+ registry.remove_app(a.app, note=a.note or "")
132+
133+
134+def cmd_status(a):
135+ if a.live:
136+ for row in deploy.live_status(a.app):
137+ print("%-18s %-8s %-30s %-3s %s" % row)
138+ else:
139+ print("%-18s %-8s %-6s %-30s %-10s %s" % ("app", "nœud", "port", "domaine", "état", "déployée"))
140+ for row in deploy.status(a.app):
141+ print("%-18s %-8s %-6s %-30s %-10s %s" % tuple("" if x is None else x for x in row))
142+
143+
144+def cmd_health(a):
145+ ok, det = deploy.health(a.app, a.node or registry.node_of(a.app))
146+ print("OK" if ok else "KO", det)
147+ sys.exit(0 if ok else 1)
148+
149+
150+def cmd_registry(a):
151+ print(json.dumps(registry.load(), indent=2, ensure_ascii=False))
152+
153+
154+def cmd_import(a):
155+ from . import importer
156+ for p in importer.import_node(a.node, apps_filter=a.apps or None):
157+ print("brouillon :", p)
158+
159+
160+def cmd_prepare(a):
161+ from . import prepare
162+ prepare.prepare(a.node, runtimes=a.runtime or None, ka_helpers=a.ka_helpers)
163+
164+
165+def cmd_bootstrap(a):
166+ from . import prepare
167+ prepare.bootstrap(a.node)
168+
169+
170+def cmd_logs(a):
171+ node = a.node or registry.node_of(a.app)
172+ from . import ssh
173+ ip = nodes.ip_of(node)
174+ names = manifest.pm2_names(manifest.load(a.app))
175+ rc, out, err = ssh.run(ip, "pm2 logs %s --nostream --lines %d 2>/dev/null" % (" ".join(names), a.lines), timeout=60)
176+ print(out)
177+
178+
179+def main(argv=None):
180+ p = argparse.ArgumentParser(prog="mld", description="MacLustr Dispatch — déploie les apps depuis la passerelle %s vers le nœud le plus efficient." % config.GATEWAY)
181+ sp = p.add_subparsers(dest="cmd", required=True)
182+ s = sp.add_parser("discover", help="balaye le LAN, met en cache les IP des nœuds"); s.set_defaults(f=cmd_discover)
183+ s = sp.add_parser("scan", help="sonde ressources + runtimes de chaque nœud"); s.add_argument("nodes", nargs="*"); s.set_defaults(f=cmd_scan)
184+ s = sp.add_parser("nodes", help="dernier scan"); s.set_defaults(f=cmd_nodes)
185+ s = sp.add_parser("apps", help="manifestes + emplacement courant"); s.set_defaults(f=cmd_apps)
186+ s = sp.add_parser("show", help="affiche un manifeste"); s.add_argument("app"); s.set_defaults(f=cmd_show)
187+ s = sp.add_parser("pick", help="classe les nœuds pour une app (ou un besoin ad hoc)"); s.add_argument("app", nargs="?"); s.add_argument("--ram", type=float); s.add_argument("--runtime", action="append"); s.add_argument("--exclude", action="append"); s.add_argument("--rescan", action="store_true"); s.add_argument("--json", action="store_true"); s.set_defaults(f=cmd_pick)
188+ s = sp.add_parser("plan", help="plan de placement pour toutes les apps (ou une liste)"); s.add_argument("apps", nargs="*"); s.add_argument("--exclude", action="append"); s.add_argument("--rescan", action="store_true"); s.add_argument("--json", action="store_true"); s.set_defaults(f=cmd_plan)
189+ s = sp.add_parser("fetch", help="copie l'app d'un nœud vers le staging"); s.add_argument("app"); s.add_argument("--from", dest="from_node", required=True); s.add_argument("--final", action="store_true"); s.set_defaults(f=cmd_fetch)
190+ s = sp.add_parser("deploy", help="staging -> nœud (choisi automatiquement sans --node)"); s.add_argument("app"); s.add_argument("--node"); s.add_argument("--reinstall", action="store_true"); s.add_argument("--skip-sync", action="store_true"); s.set_defaults(f=cmd_deploy)
191+ s = sp.add_parser("move", help="migre une app (fetch, stop, deploy, retire)"); s.add_argument("app"); s.add_argument("--to"); s.add_argument("--from", dest="from_node"); s.add_argument("--away", action="store_true", help="exclut le nœud actuel"); s.add_argument("--keep-source", action="store_true"); s.add_argument("--reinstall", action="store_true"); s.set_defaults(f=cmd_move)
192+ s = sp.add_parser("stop"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_stop)
193+ s = sp.add_parser("start"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_start)
194+ s = sp.add_parser("restart"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_restart)
195+ s = sp.add_parser("retire", help="arrête + efface la copie d'un nœud"); s.add_argument("app"); s.add_argument("--node"); s.add_argument("--keep-dir", action="store_true"); s.add_argument("--keep-registry", action="store_true"); s.add_argument("--note"); s.set_defaults(f=cmd_retire)
196+ s = sp.add_parser("status"); s.add_argument("app", nargs="?"); s.add_argument("--live", action="store_true"); s.set_defaults(f=cmd_status)
197+ s = sp.add_parser("health"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_health)
198+ s = sp.add_parser("logs"); s.add_argument("app"); s.add_argument("--node"); s.add_argument("--lines", type=int, default=40); s.set_defaults(f=cmd_logs)
199+ s = sp.add_parser("registry", help="registre JSON complet"); s.set_defaults(f=cmd_registry)
200+ s = sp.add_parser("import", help="brouillons de manifestes depuis les PM2 d'un nœud"); s.add_argument("node"); s.add_argument("apps", nargs="*"); s.set_defaults(f=cmd_import)
201+ s = sp.add_parser("prepare", help="installe runtimes/ngrok/pm2-startup sur un nœud"); s.add_argument("node"); s.add_argument("--runtime", action="append"); s.add_argument("--ka-helpers", action="store_true"); s.set_defaults(f=cmd_prepare)
202+ s = sp.add_parser("bootstrap", help="Xcode CLT + Homebrew sur un nœud vierge (long)"); s.add_argument("node"); s.set_defaults(f=cmd_bootstrap)
203+ a = p.parse_args(argv)
204+ a.f(a)
205+
206+
207+if __name__ == "__main__":
208+ main()
added mld/config.py +76 −0
@@ -0,0 +1,76 @@
1+"""maclustr-dispatch — configuration centrale.
2+
3+Tout tourne SUR la passerelle (M1M32). Python 3.9 (système macOS) : pas de syntaxe > 3.9.
4+"""
5+import os
6+
7+USER = "simon-pierreboucher"
8+HOME = os.path.expanduser("~")
9+GATEWAY = "M1M32"
10+DOMAIN = "maclustr.io"
11+LAN_PREFIX = "192.168.2."
12+SSH_KEY = os.path.join(HOME, ".ssh", "id_ed25519") # clé « maclustr-m1m32 », autorisée sur tous les nœuds
13+STATE = os.path.join(HOME, "dispatch") # état : apps/, stage/, registry.json, lan.json, logs/
14+APPS_DIR = os.path.join(STATE, "apps")
15+STAGE_DIR = os.path.join(STATE, "stage")
16+LOG_DIR = os.path.join(STATE, "logs")
17+REGISTRY = os.path.join(STATE, "registry.json")
18+LAN_CACHE = os.path.join(STATE, "lan.json")
19+SCAN_CACHE = os.path.join(STATE, "scan.json")
20+NGROK_CFG = os.path.join(HOME, "Library", "Application Support", "ngrok", "ngrok.yml")
21+RUN_DIR = "~/dispatch-run" # sur les nœuds : ecosystem PM2 générés
22+SUDO_PW_FILE = os.path.join(STATE, ".sudo") # mot de passe sudo des nœuds (0600), pour `prepare`
23+
24+# Inventaire statique (alias -> cœurs, RAM Go, modèle). Les IP LAN sont découvertes en live.
25+NODES = {
26+ "M3U96a": {"cores": 32, "ram_gb": 96, "model": "Mac Studio"},
27+ "M3U96b": {"cores": 32, "ram_gb": 96, "model": "Mac Studio"},
28+ "M2U64": {"cores": 24, "ram_gb": 64, "model": "Mac Studio"},
29+ "M4M64a": {"cores": 16, "ram_gb": 64, "model": "Mac Studio"},
30+ "M4M64b": {"cores": 16, "ram_gb": 64, "model": "Mac Studio"},
31+ "M4BP48": {"cores": 16, "ram_gb": 48, "model": "MacBook Pro"},
32+ "M4BP36": {"cores": 14, "ram_gb": 36, "model": "MacBook Pro"},
33+ "M4M36": {"cores": 14, "ram_gb": 36, "model": "Mac Studio"},
34+ "M2M32": {"cores": 12, "ram_gb": 32, "model": "Mac Studio"},
35+ "M2M32b": {"cores": 12, "ram_gb": 32, "model": "Mac Studio"},
36+ "M2M32c": {"cores": 12, "ram_gb": 32, "model": "Mac Studio"},
37+ "m4mc": {"cores": 12, "ram_gb": 24, "model": "Mac mini"},
38+ "M1M32": {"cores": 10, "ram_gb": 32, "model": "Mac Studio"},
39+ "m4ma": {"cores": 10, "ram_gb": 24, "model": "Mac mini"},
40+ "m4mb": {"cores": 10, "ram_gb": 16, "model": "Mac mini"},
41+ "m2m16": {"cores": 10, "ram_gb": 16, "model": "Mac mini"},
42+ "M3BA24": {"cores": 8, "ram_gb": 24, "model": "MacBook Air"},
43+ "M3BA16": {"cores": 8, "ram_gb": 16, "model": "MacBook Air"},
44+ "m2m8a": {"cores": 8, "ram_gb": 8, "model": "Mac mini"},
45+ "m2m8b": {"cores": 8, "ram_gb": 8, "model": "Mac mini"},
46+ "m4md": {"cores": 10, "ram_gb": 16, "model": "Mac mini"}, # ajouté 2026-09-04 (Mac16,10, M4, 100.78.227.79)
47+}
48+
49+# Rôles : gateway = jamais d'app ; worker = éligible. Un nœud absent du scan est « offline ».
50+ROLES = {alias: ("gateway" if alias == GATEWAY else "worker") for alias in NODES}
51+
52+# Runtimes que `prepare` sait installer via Homebrew, et comment les détecter sur un nœud.
53+RUNTIME_CHECKS = {
54+ "node": "test -x /opt/homebrew/bin/node",
55+ "pnpm": "test -x /opt/homebrew/bin/pnpm",
56+ "pm2": "test -x /opt/homebrew/bin/pm2",
57+ "ngrok": "(test -x /opt/homebrew/bin/ngrok || test -x $HOME/bin/ngrok) && test -f \"$HOME/Library/Application Support/ngrok/ngrok.yml\"",
58+ "python@3.14": "test -x /opt/homebrew/opt/python@3.14/bin/python3.14",
59+ "python@3.13": "test -x /opt/homebrew/opt/python@3.13/bin/python3.13",
60+ "python@3.12": "test -x /opt/homebrew/opt/python@3.12/bin/python3.12",
61+ "uv": "test -x /opt/homebrew/bin/uv || test -x $HOME/.local/bin/uv",
62+ "uv-python@3.12": "ls -d $HOME/.local/share/uv/python/cpython-3.12* >/dev/null 2>&1",
63+ "docker": "test -x /opt/homebrew/bin/docker",
64+ "postgresql@17": "test -x /opt/homebrew/opt/postgresql@17/bin/postgres",
65+ "brew": "test -x /opt/homebrew/bin/brew",
66+ "git": "test -x /opt/homebrew/bin/git || test -x /usr/bin/git",
67+}
68+BREW_FORMULAE = {
69+ "node": "node", "pnpm": "pnpm", "pm2": "pm2", "python@3.14": "python@3.14", "python@3.13": "python@3.13",
70+ "python@3.12": "python@3.12", "uv": "uv", "docker": "docker", "postgresql@17": "postgresql@17", "git": "git",
71+}
72+BREW_CASKS = {"ngrok": "ngrok"}
73+
74+os.makedirs(APPS_DIR, exist_ok=True)
75+os.makedirs(STAGE_DIR, exist_ok=True)
76+os.makedirs(LOG_DIR, exist_ok=True)
added mld/deploy.py +334 −0
@@ -0,0 +1,334 @@
1+"""Cycle de vie d'une app : fetch (nœud -> staging), deploy (staging -> nœud), stop, start, retire, move, health."""
2+import json
3+import os
4+import shlex
5+import time
6+from . import config, manifest, nodes, registry, render, ssh
7+
8+SYNC_BASE_EXCLUDES = [".DS_Store", "*.log", "__pycache__/", ".pytest_cache/", ".mypy_cache/"]
9+
10+
11+class DeployError(Exception):
12+ pass
13+
14+
15+def _ctx(alias, ip):
16+ lan = nodes.load_lan()["ips"]
17+ return {"HOME": "/Users/%s" % config.USER, "NODE": alias, "LAN_IP": ip, "ips": lan, "USER": config.USER}
18+
19+
20+def _abs(p):
21+ return p.replace("~", "/Users/%s" % config.USER, 1) if p.startswith("~") else p
22+
23+
24+def _log(app, msg):
25+ line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
26+ print(line, flush=True)
27+ with open(os.path.join(config.LOG_DIR, app + ".log"), "a") as f:
28+ f.write(time.strftime("%Y-%m-%d ") + line + "\n")
29+
30+
31+def _ip(alias):
32+ ip = nodes.ip_of(alias)
33+ if not ip:
34+ raise DeployError("nœud %s injoignable sur le LAN" % alias)
35+ return ip
36+
37+
38+def _ensure_dir(ip, d):
39+ """Crée le répertoire cible ; sous /opt (racine root) passe par sudo -S avec le mot de passe de ~/dispatch/.sudo."""
40+ if ssh.ok(ip, "mkdir -p %s 2>/dev/null && test -w %s" % (shlex.quote(d), shlex.quote(d))):
41+ return
42+ pw = open(config.SUDO_PW_FILE).read().strip() if os.path.exists(config.SUDO_PW_FILE) else None
43+ if not pw:
44+ raise DeployError("impossible de créer %s (sudo requis, %s absent)" % (d, config.SUDO_PW_FILE))
45+ rc, out, err = ssh.run(ip, "echo %s | sudo -S -p '' mkdir -p %s && echo %s | sudo -S -p '' chown %s %s" % (shlex.quote(pw), shlex.quote(d), shlex.quote(pw), config.USER, shlex.quote(d)))
46+ if rc != 0:
47+ raise DeployError("création de %s impossible : %s" % (d, err.strip()[-200:]))
48+
49+
50+def stage_dir(app):
51+ return os.path.join(config.STAGE_DIR, app)
52+
53+
54+# ---------------------------------------------------------------- fetch : nœud -> staging
55+def fetch(app, from_alias, final=False):
56+ m = manifest.load(app)
57+ ip = _ip(from_alias)
58+ _log(app, "fetch %s depuis %s (%s)%s" % (m["dir"], from_alias, ip, " [final]" if final else ""))
59+ excl = SYNC_BASE_EXCLUDES + list(m.get("sync_excludes") or [])
60+ rc, out, err = ssh.rsync_pull(ip, _abs(m["dir"]), os.path.join(stage_dir(app), "dir"), excludes=excl)
61+ if rc not in (0, 24):
62+ raise DeployError("rsync fetch a échoué (%s): %s" % (rc, err.strip()[-300:]))
63+ for extra in m.get("extra_paths") or []:
64+ dest = os.path.join(stage_dir(app), "extra", extra.strip("~/").replace("/", "__"))
65+ rc2, out2, err2 = ssh.run(ip, "test -e %s" % shlex.quote(_abs(extra)))
66+ if rc2 != 0:
67+ _log(app, " extra absent sur la source, ignoré : %s" % extra)
68+ continue
69+ isdir = ssh.ok(ip, "test -d %s" % shlex.quote(_abs(extra)))
70+ if isdir:
71+ rc2, out2, err2 = ssh.rsync_pull(ip, _abs(extra), dest, delete=True)
72+ else:
73+ os.makedirs(dest, exist_ok=True)
74+ import subprocess
75+ subprocess.run(["scp", "-p"] + ssh.SSH_OPTS + ["%s:%s" % (ssh.target(ip), _abs(extra)), os.path.join(dest, os.path.basename(extra))], capture_output=True)
76+ json.dump({"app": app, "from": from_alias, "ip": ip, "ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "final": final},
77+ open(os.path.join(stage_dir(app), "fetch.json"), "w"), indent=2)
78+ stat = [l for l in out.splitlines() if "Total transferred file size" in l or "Number of regular files transferred" in l or "Number of files transferred" in l or "Total bytes sent" in l]
79+ _log(app, " ok — " + " ; ".join(s.strip() for s in stat))
80+ return True
81+
82+
83+# ---------------------------------------------------------------- stop / start / status sur un nœud
84+def stop(app, alias, keep_dir=True):
85+ m = manifest.load(app)
86+ ip = _ip(alias)
87+ _log(app, "stop sur %s" % alias)
88+ names = manifest.pm2_names(m)
89+ script = ""
90+ if names:
91+ script += "for n in %s; do pm2 delete $n >/dev/null 2>&1 && echo \" pm2 delete $n\"; done; pm2 save --force >/dev/null 2>&1\n" % " ".join(names)
92+ for lbl in manifest.launchd_labels(m):
93+ script += "launchctl bootout gui/$(id -u)/%s >/dev/null 2>&1 && echo ' bootout %s'; rm -f ~/Library/LaunchAgents/%s.plist\n" % (lbl, lbl, lbl)
94+ if m.get("ngrok"):
95+ script += "pkill -f -- 'url=%s' 2>/dev/null; pkill -f -- 'url=https://%s' 2>/dev/null; pkill -f -- 'domain=%s' 2>/dev/null; true\n" % (m["ngrok"]["url"], m["ngrok"]["url"], m["ngrok"]["url"])
96+ for hook in (m.get("hooks") or {}).get("pre_stop") or []:
97+ script += hook + "\n"
98+ # tue ce qui écoute encore sur les ports de l'app (orphelins)
99+ for p in m["requires"].get("ports") or []:
100+ script += "sleep 1; for pid in $(lsof -nP -iTCP:%s -sTCP:LISTEN -t 2>/dev/null); do kill $pid 2>/dev/null && echo ' kill port %s pid '$pid; done\n" % (p, p)
101+ if m.get("ngrok"):
102+ script += "sleep 4\n" # laisse ngrok libérer le domaine réservé côté service
103+ rc, out, err = ssh.run(ip, script, timeout=180)
104+ for l in out.splitlines():
105+ _log(app, l)
106+ registry.log(app, "stopped", node=alias)
107+ return rc == 0
108+
109+
110+def _install_hooks(m, ip, ctx):
111+ """Commandes post-sync : réécritures .env, hooks, inscription pousseur."""
112+ script = ""
113+ d = _abs(m["dir"])
114+ for fname, kv in (m.get("env_overrides") or {}).items():
115+ target = os.path.join(d, fname)
116+ kv = manifest.render(kv, ctx)
117+ py = "import re,sys,os\np=%r\nkv=%r\ns=open(p).read() if os.path.exists(p) else ''\nfor k,v in kv.items():\n line=k+'='+v\n if re.search(r'^'+re.escape(k)+r'=.*$', s, re.M): s=re.sub(r'^'+re.escape(k)+r'=.*$', line.replace('\\\\','\\\\\\\\'), s, flags=re.M)\n else: s=s.rstrip('\\n')+'\\n'+line+'\\n'\nopen(p,'w').write(s)\nprint(' .env réécrit: '+', '.join(kv))" % (target, kv)
118+ script += "python3 - <<'PYEOF'\n%s\nPYEOF\n" % py
119+ if m.get("ka_repo"):
120+ script += "grep -qx %s ~/.ka-pousseur-repos 2>/dev/null || echo %s >> ~/.ka-pousseur-repos; echo ' pousseur: %s inscrit'\n" % (shlex.quote(m["dir"]), shlex.quote(m["dir"]), m["dir"])
121+ for hook in (m.get("hooks") or {}).get("post_sync") or []:
122+ script += "cd %s && (%s)\n" % (shlex.quote(d), manifest.render(hook, ctx))
123+ return script
124+
125+
126+def deploy(app, alias, reinstall=False, skip_sync=False):
127+ """staging -> nœud, puis démarrage (PM2 + launchd) et healthcheck."""
128+ m = manifest.load(app)
129+ ip = _ip(alias)
130+ ctx = _ctx(alias, ip)
131+ sd = os.path.join(stage_dir(app), "dir")
132+ if not skip_sync and not os.path.isdir(sd):
133+ raise DeployError("staging vide pour %s — faire `mld fetch %s --from <nœud>` ou `mld stage`" % (app, app))
134+ _log(app, "deploy → %s (%s)" % (alias, ip))
135+ # 1. runtimes
136+ scan = nodes.load_scan().get(alias) or nodes.probe(alias, ip)
137+ missing = [r for r in m["requires"]["runtimes"] if not scan.get("runtimes", {}).get(r)]
138+ if missing:
139+ raise DeployError("runtimes manquants sur %s : %s (→ `mld prepare %s`)" % (alias, ", ".join(missing), alias))
140+ # 2. sync
141+ d = _abs(m["dir"])
142+ if not skip_sync:
143+ _ensure_dir(ip, d)
144+ excl = SYNC_BASE_EXCLUDES + list(m.get("sync_excludes") or [])
145+ rc, out, err = ssh.rsync_push(sd, ip, d, excludes=excl, delete=True)
146+ if rc not in (0, 24):
147+ raise DeployError("rsync deploy a échoué (%s): %s" % (rc, err.strip()[-300:]))
148+ stat = [l.strip() for l in out.splitlines() if "Total transferred file size" in l or "Total bytes sent" in l]
149+ _log(app, " sync ok — %s" % (stat[0] if stat else ""))
150+ for extra in m.get("extra_paths") or []:
151+ src = os.path.join(stage_dir(app), "extra", extra.strip("~/").replace("/", "__"))
152+ if not os.path.exists(src):
153+ continue
154+ if os.path.isdir(src) and not os.path.exists(os.path.join(src, os.path.basename(extra))):
155+ ssh.rsync_push(src, ip, _abs(extra), delete=False)
156+ else:
157+ import subprocess
158+ ssh.run(ip, "mkdir -p %s" % shlex.quote(os.path.dirname(_abs(extra))))
159+ subprocess.run(["scp", "-p"] + ssh.SSH_OPTS + [os.path.join(src, os.path.basename(extra)), "%s:%s" % (ssh.target(ip), _abs(extra))], capture_output=True)
160+ _log(app, " extra: %s" % extra)
161+ # 3. install (optionnel) + hooks
162+ script = "cd %s || exit 3\n" % shlex.quote(d)
163+ if reinstall:
164+ script += "if [ -f pnpm-lock.yaml ]; then pnpm install --frozen-lockfile; elif [ -f package-lock.json ]; then npm ci; elif [ -f package.json ]; then npm install; fi\n"
165+ script += "if [ -f requirements.txt ] && [ -x .venv/bin/pip ]; then .venv/bin/pip install -q -r requirements.txt; fi\n"
166+ elif os.path.exists(os.path.join(sd, "package.json")) and os.path.isdir(os.path.join(sd, "node_modules")):
167+ script += "npm rebuild >/dev/null 2>&1 && echo ' npm rebuild ok' || echo ' npm rebuild: avertissement'\n"
168+ script += _install_hooks(m, ip, ctx)
169+ rc, out, err = ssh.run(ip, script, timeout=1800)
170+ for l in out.splitlines()[-15:]:
171+ _log(app, l)
172+ if rc != 0:
173+ raise DeployError("post-sync a échoué (%s): %s" % (rc, (err or out).strip()[-400:]))
174+ # 4. lancement
175+ start(app, alias, m=m, ctx=ctx, ip=ip)
176+ # 5. santé
177+ healthy, detail = health(app, alias, ip=ip, m=m)
178+ registry.set_app(app, node=alias, ip=ip, port=m.get("port"), domain=m.get("domain"), dir=m["dir"], label=m.get("label") or app,
179+ processes=manifest.pm2_names(m), launchd=manifest.launchd_labels(m), deployed=time.strftime("%Y-%m-%d %H:%M"),
180+ status="online" if healthy else "unhealthy", health=detail)
181+ registry.log(app, "deployed", node=alias, healthy=healthy)
182+ _log(app, " santé : %s" % detail)
183+ if not healthy:
184+ names = manifest.pm2_names(m)
185+ rc, out, _ = ssh.run(ip, "pm2 logs %s --nostream --lines 6 2>/dev/null | grep -v '^\\[TAILING' | tail -12" % " ".join(names), timeout=40)
186+ for l in out.splitlines():
187+ _log(app, " log: " + l[:160])
188+ raise DeployError("healthcheck KO sur %s : %s" % (alias, detail))
189+ return True
190+
191+
192+def start(app, alias, m=None, ctx=None, ip=None):
193+ m = m or manifest.load(app)
194+ ip = ip or _ip(alias)
195+ ctx = ctx or _ctx(alias, ip)
196+ _log(app, "start sur %s" % alias)
197+ run_dir = _abs(config.RUN_DIR)
198+ eco = render.ecosystem(m, ctx)
199+ eco_path = "%s/%s.config.cjs" % (run_dir, app)
200+ ssh.write_remote_file(ip, eco_path, eco, mode="600")
201+ script = "mkdir -p %s\n" % run_dir
202+ names = manifest.pm2_names(m)
203+ if names:
204+ script += "for n in %s; do pm2 delete $n >/dev/null 2>&1; done\n" % " ".join(names)
205+ script += "pm2 start %s --update-env 2>&1 | grep -E 'error|Error|\\[PM2\\]' | head -5; pm2 save --force >/dev/null 2>&1; pm2 jlist | python3 -c \"import json,sys; L=json.load(sys.stdin); print(' pm2: '+', '.join(p['name']+'='+p['pm2_env']['status'] for p in L if p['name'] in %s))\" 2>/dev/null || pm2 ls | grep -E '%s'\n" % (shlex.quote(eco_path), json.dumps(names), "|".join(names))
206+ for item in m.get("launchd") or []:
207+ pl = render.plist(item, ctx)
208+ ppath = "%s/Library/LaunchAgents/%s.plist" % (ctx["HOME"], item["label"])
209+ ssh.write_remote_file(ip, ppath, pl, mode="644")
210+ script += "launchctl bootout gui/$(id -u)/%s >/dev/null 2>&1; launchctl bootstrap gui/$(id -u) %s && echo ' launchd: %s chargé' || echo ' launchd: %s ÉCHEC'\n" % (item["label"], shlex.quote(ppath), item["label"], item["label"])
211+ for hook in (m.get("hooks") or {}).get("post_start") or []:
212+ script += "cd %s && (%s)\n" % (shlex.quote(_abs(m["dir"])), manifest.render(hook, ctx))
213+ rc, out, err = ssh.run(ip, script, timeout=300)
214+ for l in out.splitlines():
215+ _log(app, l)
216+ if err.strip():
217+ _log(app, " stderr: " + err.strip()[-200:])
218+ return rc == 0
219+
220+
221+def health(app, alias, ip=None, m=None, timeout=90):
222+ m = m or manifest.load(app)
223+ ip = ip or _ip(alias)
224+ if not m.get("port"):
225+ # pas de port HTTP : vérifie que les processus PM2 sont online
226+ names = manifest.pm2_names(m)
227+ if not names:
228+ return True, "aucun port/processus à vérifier"
229+ rc, out, _ = ssh.run(ip, "pm2 jlist | python3 -c \"import json,sys; L=json.load(sys.stdin); print(sum(1 for p in L if p['name'] in %s and p['pm2_env']['status']=='online'))\"" % json.dumps(names))
230+ n = int((out.strip() or "0").splitlines()[-1] or 0)
231+ return n == len(names), "%d/%d processus PM2 online" % (n, len(names))
232+ url = "http://127.0.0.1:%s%s" % (m["port"], m.get("health_path") or "/")
233+ deadline = time.time() + timeout
234+ code = "000"
235+ while time.time() < deadline:
236+ rc, out, _ = ssh.run(ip, "curl -s -o /dev/null -m 8 -w '%%{http_code}' %s" % shlex.quote(url), timeout=20)
237+ code = out.strip()[-3:] if out.strip() else "000"
238+ if code.isdigit() and code not in ("000",) and int(code) < 500:
239+ break
240+ time.sleep(4)
241+ local_ok = code.isdigit() and code != "000" and int(code) < 500
242+ pub = ""
243+ if m.get("domain") and local_ok and m.get("ngrok"):
244+ import subprocess
245+ for _ in range(20):
246+ p = subprocess.run(["curl", "-s", "-o", "/dev/null", "-m", "12", "-w", "%{http_code}", "https://%s%s" % (m["domain"], m.get("health_path") or "/")], capture_output=True, text=True)
247+ pub = p.stdout.strip()
248+ if pub.isdigit() and int(pub) < 500 and pub != "404":
249+ break
250+ time.sleep(5)
251+ detail = "local %s" % code + (" / public https://%s %s" % (m["domain"], pub) if pub else "")
252+ ok = local_ok and (not pub or (pub.isdigit() and int(pub) < 500 and pub != "404"))
253+ return ok, detail
254+
255+
256+def retire(app, alias, remove_dir=True):
257+ """Arrête et efface la copie d'un nœud (après migration réussie)."""
258+ m = manifest.load(app)
259+ ip = _ip(alias)
260+ stop(app, alias)
261+ if remove_dir:
262+ d = _abs(m["dir"])
263+ rc, out, err = ssh.run(ip, "rm -rf %s && echo ' rm -rf %s'; rm -f %s/%s.config.cjs; grep -v -x %s ~/.ka-pousseur-repos > ~/.ka-pousseur-repos.tmp 2>/dev/null && mv ~/.ka-pousseur-repos.tmp ~/.ka-pousseur-repos; true" % (shlex.quote(d), d, _abs(config.RUN_DIR), app, shlex.quote(m["dir"])), timeout=600)
264+ for l in out.splitlines():
265+ _log(app, l)
266+ registry.log(app, "retired", node=alias)
267+ return True
268+
269+
270+def move(app, to_alias, from_alias=None, keep_source=False, reinstall=False):
271+ """Migration complète : pré-sync à chaud, arrêt, sync final, déploiement, santé, retrait de la source (ou rollback)."""
272+ from_alias = from_alias or registry.node_of(app) or manifest.load(app).get("source_node")
273+ if not from_alias:
274+ raise DeployError("nœud source inconnu pour %s (préciser --from)" % app)
275+ if from_alias == to_alias:
276+ _log(app, "déjà sur %s → redéploiement en place" % to_alias)
277+ fetch(app, from_alias)
278+ stop(app, from_alias)
279+ fetch(app, from_alias, final=True)
280+ return deploy(app, to_alias, reinstall=reinstall)
281+ _log(app, "MOVE %s : %s → %s" % (app, from_alias, to_alias))
282+ fetch(app, from_alias) # 1. copie à chaud (gros volume)
283+ stop(app, from_alias) # 2. arrêt source
284+ fetch(app, from_alias, final=True) # 3. delta final (cohérence des bases)
285+ try:
286+ deploy(app, to_alias, reinstall=reinstall) # 4. cible
287+ except DeployError as e:
288+ _log(app, "ÉCHEC sur %s : %s — rollback sur %s" % (to_alias, e, from_alias))
289+ try:
290+ stop(app, to_alias)
291+ except Exception:
292+ pass
293+ start(app, from_alias)
294+ ok, det = health(app, from_alias)
295+ registry.set_app(app, node=from_alias, status="online" if ok else "unhealthy", health=det)
296+ raise
297+ if not keep_source:
298+ retire(app, from_alias) # 5. nettoyage source
299+ registry.log(app, "moved", **{"from": from_alias, "to": to_alias})
300+ _log(app, "MOVE terminé : %s tourne sur %s" % (app, to_alias))
301+ return True
302+
303+
304+def status(app=None):
305+ r = registry.load()
306+ apps = [app] if app else sorted(r["apps"])
307+ rows = []
308+ for a in apps:
309+ v = r["apps"].get(a)
310+ if not v:
311+ rows.append((a, "—", "", "", "non déployée", ""))
312+ continue
313+ rows.append((a, v.get("node"), v.get("port"), v.get("domain") or "", v.get("status"), v.get("deployed")))
314+ return rows
315+
316+
317+def live_status(app=None, public=True):
318+ """Vérifie en live (PM2 + port + domaine) chaque app du registre."""
319+ r = registry.load()
320+ apps = [app] if app else sorted(r["apps"])
321+ out = []
322+ for a in apps:
323+ v = r["apps"].get(a)
324+ if not v:
325+ continue
326+ m = manifest.load(a)
327+ try:
328+ ip = _ip(v["node"])
329+ ok, det = health(a, v["node"], ip=ip, m=m, timeout=10) if public else (None, "")
330+ except DeployError as e:
331+ ok, det = False, str(e)
332+ registry.set_app(a, status="online" if ok else "unhealthy", health=det)
333+ out.append((a, v["node"], v.get("domain") or "", "OK" if ok else "KO", det))
334+ return out
added mld/importer.py +81 −0
@@ -0,0 +1,81 @@
1+"""`mld import <nœud>` : brouillons de manifestes à partir des processus PM2 / launchd vivants d'un nœud.
2+
3+Regroupe par préfixe de nom (lou-ka-web, lou-ka-sync, lou-ka-ngrok → lou-ka). Les brouillons vont dans
4+~/dispatch/apps/_drafts/<app>.json ; on les relit, on ajuste, puis on les déplace dans ~/dispatch/apps/.
5+"""
6+import json
7+import os
8+import re
9+from . import config, manifest, nodes, ssh
10+
11+SUFFIXES = ("-web", "-sync", "-ngrok", "-api", "-tunnel", "-server", "-etl", "-scheduler", "-ingest", "-market",
12+ "-worker", "-proxy", "-crawler", "-enrichment", "-sitemaps", "-tun-search", "-tun-embed")
13+ENV_DROP_PREFIX = ("PM2_", "npm_", "__", "TERM", "SSH_", "LC_", "HOMEBREW", "ITERM", "XPC_", "COLOR", "LS_", "SHELL", "PWD", "OLDPWD",
14+ "PATH", "HOME", "USER", "LOGNAME", "LANG", "TMPDIR", "SECURITYSESSIONID", "COMMAND_MODE", "unique_id", "FPATH",
15+ "CLAUDE", "OTEL", "DISPLAY", "MallocNanoZone", "ZSH", "EDITOR", "PAGER", "LESS", "NoDefaultCurrentDirectoryInExePath",
16+ "COREPACK", "AI_AGENT", "OSLogRateLimit", "GIT_", "ANTHROPIC_MODEL", "DISABLE_", "MANPATH", "INFOPATH", "STARSHIP",
17+ "TERMINFO", "WINDOWID", "NVM", "P9K", "TMUX", "SSH", "_")
18+
19+
20+def app_of(name):
21+ for s in SUFFIXES:
22+ if name.endswith(s):
23+ return name[: -len(s)]
24+ return name
25+
26+
27+def clean_env(env, name):
28+ out = {}
29+ for k, v in (env or {}).items():
30+ if k == name or k.startswith(ENV_DROP_PREFIX) or not isinstance(v, str):
31+ continue
32+ out[k] = v
33+ return out
34+
35+
36+def import_node(alias, apps_filter=None):
37+ ip = nodes.ip_of(alias)
38+ rc, out, err = ssh.run(ip, "pm2 jlist 2>/dev/null; echo; echo __MEM__; pm2 jlist 2>/dev/null | python3 -c \"import json,sys; print(json.dumps({p['name']: p.get('monit',{}).get('memory',0) for p in json.load(sys.stdin)}))\"", timeout=60)
39+ raw, _, mem = out.partition("__MEM__")
40+ L = json.loads(raw.strip() or "[]")
41+ mems = json.loads(mem.strip() or "{}")
42+ groups = {}
43+ for p in L:
44+ e = p.get("pm2_env", {})
45+ app = app_of(p["name"])
46+ if apps_filter and app not in apps_filter:
47+ continue
48+ g = groups.setdefault(app, {"app": app, "processes": [], "ngrok": None, "cwd": None, "ram": 0})
49+ g["ram"] += int(mems.get(p["name"], 0)) // (1024 * 1024)
50+ args = e.get("args") or []
51+ script = e.get("pm_exec_path") or ""
52+ if "ngrok" in script or (args and "ngrok" in " ".join(map(str, args))):
53+ blob = " ".join([script] + [str(a) for a in args])
54+ mo = re.search(r"--(?:url|domain)=(?:https?://)?([^\s]+)", blob)
55+ port = re.search(r"\s(\d{2,5})(?:\s|$)", blob)
56+ g["ngrok"] = {"name": p["name"], "url": mo.group(1) if mo else None, "port": int(port.group(1)) if port else None}
57+ continue
58+ g["processes"].append({
59+ "name": p["name"], "manager": "pm2", "script": script, "args": args,
60+ "interpreter": e.get("exec_interpreter") if e.get("exec_interpreter") not in (None, "none", "N/A") else None,
61+ "cwd": e.get("pm_cwd"), "env": clean_env(e.get("env"), p["name"]), "cron_restart": e.get("cron_restart"),
62+ "autorestart": e.get("autorestart", True), "max_memory_restart": e.get("max_memory_restart"),
63+ })
64+ g["cwd"] = g["cwd"] or e.get("pm_cwd")
65+ drafts = os.path.join(config.APPS_DIR, "_drafts")
66+ os.makedirs(drafts, exist_ok=True)
67+ written = []
68+ for app, g in groups.items():
69+ cwd = g["cwd"] or ""
70+ m = json.loads(json.dumps(manifest.DEFAULTS))
71+ mo = re.match(r"(/Users/[^/]+/(?:apps/)?[^/]+|/opt/[^/]+)", cwd)
72+ d = mo.group(1) if mo else cwd
73+ m.update({"app": app, "label": app, "dir": d.replace("/Users/%s" % config.USER, "~"), "processes": g["processes"], "ngrok": g["ngrok"],
74+ "domain": (g["ngrok"] or {}).get("url"), "port": (g["ngrok"] or {}).get("port"), "ram_mb_observed": max(g["ram"], 128)})
75+ m["requires"]["ports"] = [m["port"]] if m["port"] else []
76+ m["placement"]["reason"] = "importé de %s" % alias
77+ manifest.detect_requirements(m)
78+ p = os.path.join(drafts, app + ".json")
79+ json.dump(m, open(p, "w"), indent=2, ensure_ascii=False)
80+ written.append(p)
81+ return written
added mld/manifest.py +112 −0
@@ -0,0 +1,112 @@
1+"""Manifestes d'apps : ~/dispatch/apps/<app>.json (source de vérité de « comment » tourne une app).
2+
3+Schéma :
4+{
5+ "app": "lou-ka", "label": "Lou-Ka", "domain": "www.lou-ka.com", "port": 8095, "health_path": "/",
6+ "dir": "~/apps/lou-ka", # répertoire de l'app (même chemin absolu sur tous les nœuds)
7+ "extra_paths": ["~/.ssh/trouveka_tunnel"], # autres fichiers/dossiers à copier avec l'app
8+ "sync_excludes": ["data/backups/"], # exclusions rsync (relatives à dir)
9+ "requires": {"runtimes": ["python@3.14", "pm2", "ngrok"], "ram_gb": 2, "ports": [8095]},
10+ "ram_mb_observed": 400, "size_mb": 4359,
11+ "placement": {"pin": null, "prefer": null, "avoid": [], "reason": ""},
12+ "processes": [{"name": "lou-ka-web", "manager": "pm2", "script": "...", "args": [...], "interpreter": null,
13+ "cwd": "...", "env": {...}, "cron_restart": null, "autorestart": true, "max_memory_restart": null}],
14+ "ngrok": {"name": "lou-ka-ngrok", "url": "www.lou-ka.com", "port": 8095} | null,
15+ "launchd": [{"label": "...", "program_arguments": [...], "working_directory": "...", "env": {...}, "keep_alive": true}],
16+ "env_overrides": {".env": {"DATABASE_URL": "postgresql://...@{{IP:M2M32}}:5432/x"}}, # réécritures de fichiers .env
17+ "hooks": {"post_sync": ["bash ..."], "post_start": []},
18+ "ka_repo": true # inscrire dir dans ~/.ka-pousseur-repos du nœud hôte
19+}
20+Gabarits acceptés dans env/args/env_overrides : {{HOME}}, {{NODE}}, {{LAN_IP}}, {{IP:<alias>}}.
21+"""
22+import glob
23+import json
24+import os
25+import re
26+from . import config
27+
28+DEFAULTS = {
29+ "label": None, "domain": None, "port": None, "health_path": "/", "extra_paths": [], "sync_excludes": [],
30+ "requires": {"runtimes": ["pm2"], "ram_gb": 1, "ports": []}, "ram_mb_observed": 512, "size_mb": 0,
31+ "placement": {"pin": None, "prefer": None, "avoid": [], "reason": ""}, "processes": [], "ngrok": None,
32+ "launchd": [], "env_overrides": {}, "hooks": {"post_sync": [], "post_start": []}, "ka_repo": False,
33+}
34+
35+
36+def path(app):
37+ return os.path.join(config.APPS_DIR, app + ".json")
38+
39+
40+def load(app):
41+ p = path(app)
42+ if not os.path.exists(p):
43+ raise SystemExit("manifeste introuvable : %s" % p)
44+ m = json.load(open(p))
45+ for k, v in DEFAULTS.items():
46+ m.setdefault(k, json.loads(json.dumps(v)))
47+ m["requires"].setdefault("runtimes", ["pm2"])
48+ m["requires"].setdefault("ram_gb", 1)
49+ m["requires"].setdefault("ports", [m["port"]] if m.get("port") else [])
50+ return m
51+
52+
53+def save(m):
54+ json.dump(m, open(path(m["app"]), "w"), indent=2, ensure_ascii=False)
55+ os.chmod(path(m["app"]), 0o600)
56+
57+
58+def all_apps():
59+ return sorted(os.path.basename(p)[:-5] for p in glob.glob(os.path.join(config.APPS_DIR, "*.json")))
60+
61+
62+def load_all():
63+ return {a: load(a) for a in all_apps()}
64+
65+
66+def render(value, ctx):
67+ """Remplace {{HOME}}, {{NODE}}, {{LAN_IP}}, {{IP:alias}} dans une chaîne / liste / dict."""
68+ if isinstance(value, str):
69+ def sub(mo):
70+ key = mo.group(1)
71+ if key.startswith("IP:"):
72+ return ctx.get("ips", {}).get(key[3:], mo.group(0))
73+ return str(ctx.get(key, mo.group(0)))
74+ return re.sub(r"\{\{([A-Za-z0-9_:@.-]+)\}\}", sub, value)
75+ if isinstance(value, list):
76+ return [render(v, ctx) for v in value]
77+ if isinstance(value, dict):
78+ return {k: render(v, ctx) for k, v in value.items()}
79+ return value
80+
81+
82+def pm2_names(m):
83+ names = [p["name"] for p in m["processes"] if p.get("manager", "pm2") == "pm2"]
84+ if m.get("ngrok"):
85+ names.append(m["ngrok"]["name"])
86+ return names
87+
88+
89+def launchd_labels(m):
90+ return [l["label"] for l in m.get("launchd", [])]
91+
92+
93+def detect_requirements(m):
94+ """Déduit les runtimes requis à partir des processus (complète requires.runtimes)."""
95+ req = set(m["requires"].get("runtimes") or [])
96+ req.add("pm2")
97+ if m.get("ngrok"):
98+ req.add("ngrok")
99+ blob = json.dumps(m["processes"]) + json.dumps(m.get("launchd", []))
100+ if "/opt/homebrew/bin/node" in blob or "next" in blob or "npm" in blob or ".mjs" in blob or ".js" in blob:
101+ req.add("node")
102+ if "pnpm" in blob:
103+ req.add("pnpm")
104+ vp = m.get("venv_python") or ""
105+ for ver in ("3.14", "3.13", "3.12"):
106+ if "python@%s" % ver in vp or "/%s/" % ver in vp or "python%s" % ver in vp:
107+ req.add("python@%s" % ver)
108+ if "uv/python/cpython-3.12" in vp:
109+ req.discard("python@3.12")
110+ req.add("uv-python@3.12")
111+ m["requires"]["runtimes"] = sorted(req)
112+ return m
added mld/nodes.py +155 −0
@@ -0,0 +1,155 @@
1+"""Découverte LAN des nœuds + sonde de ressources/runtimes (scan)."""
2+import json
3+import os
4+import time
5+from concurrent.futures import ThreadPoolExecutor
6+from . import config, ssh
7+
8+RUNTIME_PROBE = "\n".join('if %s; then echo "RT %s 1"; else echo "RT %s 0"; fi' % (cmd, name, name) for name, cmd in config.RUNTIME_CHECKS.items())
9+
10+
11+def _probe_script():
12+ # sonde 100 % shell (les minis sans Xcode CLT n'ont pas de python3 fonctionnel) : lignes "KV clé valeur"
13+ sh = r'''
14+export LC_ALL=C
15+echo "KV host $(hostname -s)"
16+echo "KV marker $(cat ~/.maclustr-node 2>/dev/null)"
17+echo "KV ip $(ifconfig | awk '/inet 192\.168\.2\./{print $2; exit}')"
18+echo "KV os $(sw_vers -productVersion)"
19+echo "KV cores $(sysctl -n hw.ncpu)"
20+echo "KV ram_gb $(( $(sysctl -n hw.memsize) / 1073741824 ))"
21+echo "KV load1 $(sysctl -n vm.loadavg | tr -d '{}' | awk '{print $1}' | tr ',' '.')"
22+echo "KV free_gb $(vm_stat | awk '/Pages (free|inactive|speculative|purgeable)/{gsub("\\.","",$NF); s+=$NF} END{printf "%.1f", s*16384/1073741824}')"
23+echo "KV disk_free_gb $(df -g / | awk 'NR==2{print $4}')"
24+echo "KV boot $(sysctl -n kern.boottime | sed -E 's/.*sec = ([0-9]+).*/\1/')"
25+echo "KV ports $(lsof -nP -iTCP -sTCP:LISTEN 2>/dev/null | awk 'NR>1{n=split($9,a,":"); print a[n]}' | sort -un | tr '\n' ',')"
26+echo "KV pm2 $(pm2 jlist 2>/dev/null | tr -d '\n' | grep -o '"name":"[^"]*"' | cut -d'"' -f4 | sort -u | tr '\n' ',')"
27+'''
28+ return sh + "\n" + RUNTIME_PROBE + "\n"
29+
30+
31+def load_lan():
32+ if os.path.exists(config.LAN_CACHE):
33+ return json.load(open(config.LAN_CACHE))
34+ return {"updated": None, "ips": {}}
35+
36+
37+def save_lan(d):
38+ d["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
39+ json.dump(d, open(config.LAN_CACHE, "w"), indent=2)
40+
41+
42+def _identify(ip):
43+ rc, out, _ = ssh.run(ip, "cat ~/.maclustr-node 2>/dev/null || hostname -s", timeout=12, path=False)
44+ if rc == 0 and out.strip():
45+ return ip, out.strip().splitlines()[0].strip()
46+ return ip, None
47+
48+
49+def discover(rng=range(1, 255), verbose=True):
50+ """Balaye 192.168.2.0/24 en SSH (clé passerelle) et lit le marqueur ~/.maclustr-node."""
51+ ips = [config.LAN_PREFIX + str(i) for i in rng]
52+ found = {}
53+ with ThreadPoolExecutor(max_workers=48) as ex:
54+ for ip, alias in ex.map(_identify, ips):
55+ if alias:
56+ # normalise l'alias (le marqueur est la source de vérité, sinon hostname)
57+ for a in config.NODES:
58+ if a.lower() == alias.lower():
59+ alias = a
60+ break
61+ found[alias] = ip
62+ lan = load_lan()
63+ lan["ips"] = found
64+ save_lan(lan)
65+ if verbose:
66+ for a in sorted(found, key=lambda x: list(config.NODES).index(x) if x in config.NODES else 99):
67+ print(" %-8s %s" % (a, found[a]))
68+ missing = [a for a in config.NODES if a not in found]
69+ if missing:
70+ print(" hors LAN / éteints : %s" % ", ".join(missing))
71+ return found
72+
73+
74+def ip_of(alias, rediscover=True):
75+ lan = load_lan()
76+ ip = lan["ips"].get(alias)
77+ if ip and ssh.ok(ip, "true", timeout=15):
78+ return ip
79+ if rediscover:
80+ found = discover(verbose=False)
81+ return found.get(alias)
82+ return ip
83+
84+
85+def probe(alias, ip):
86+ rc, out, err = ssh.run(ip, _probe_script(), timeout=90)
87+ info = {"alias": alias, "ip": ip, "online": False}
88+ if "KV cores" not in out:
89+ info["error"] = (err or out).strip()[:200]
90+ return info
91+ rts = {}
92+ kv = {}
93+ for line in out.splitlines():
94+ if line.startswith("KV "):
95+ parts = line.split(" ", 2)
96+ kv[parts[1]] = parts[2].strip() if len(parts) > 2 else ""
97+ elif line.startswith("RT "):
98+ _, name, val = line.split(" ", 2)
99+ rts[name] = val.strip() == "1"
100+ def num(k, f=float, d=0):
101+ try:
102+ return f(kv.get(k, "") or d)
103+ except Exception:
104+ return d
105+ info.update({"host": kv.get("host", ""), "marker": kv.get("marker", ""), "os": kv.get("os", ""), "cores": num("cores", int),
106+ "ram_gb": num("ram_gb", int), "load1": num("load1"), "free_gb": num("free_gb"), "disk_free_gb": num("disk_free_gb", int)})
107+ if kv.get("ip"):
108+ info["ip"] = kv["ip"]
109+ try:
110+ info["uptime_h"] = round((time.time() - int(kv.get("boot") or 0)) / 3600, 1) if kv.get("boot") else None
111+ except Exception:
112+ info["uptime_h"] = None
113+ info["ports"] = sorted(set(int(p) for p in kv.get("ports", "").split(",") if p.isdigit()))
114+ info["pm2"] = [p for p in kv.get("pm2", "").split(",") if p]
115+ info["runtimes"] = rts
116+ info["online"] = info["cores"] > 0
117+ info.update({k: v for k, v in config.NODES.get(alias, {}).items() if k not in info or not info.get(k)})
118+ info["role"] = config.ROLES.get(alias, "worker")
119+ return info
120+
121+
122+def scan(aliases=None, verbose=True):
123+ lan = load_lan()
124+ if not lan["ips"]:
125+ discover(verbose=False)
126+ lan = load_lan()
127+ targets = [(a, ip) for a, ip in lan["ips"].items() if not aliases or a in aliases]
128+ res = {}
129+ with ThreadPoolExecutor(max_workers=24) as ex:
130+ for info in ex.map(lambda t: probe(*t), targets):
131+ res[info["alias"]] = info
132+ for a in config.NODES:
133+ if a not in res:
134+ res[a] = {"alias": a, "online": False, "role": config.ROLES[a], "runtimes": {}, **config.NODES[a]}
135+ json.dump({"updated": time.strftime("%Y-%m-%dT%H:%M:%S"), "nodes": res}, open(config.SCAN_CACHE, "w"), indent=2)
136+ if verbose:
137+ print_scan(res)
138+ return res
139+
140+
141+def load_scan():
142+ if os.path.exists(config.SCAN_CACHE):
143+ return json.load(open(config.SCAN_CACHE))["nodes"]
144+ return {}
145+
146+
147+def print_scan(res):
148+ print("%-8s %-6s %-16s %5s %6s %6s %6s %7s %4s %s" % ("nœud", "rôle", "ip", "cœurs", "RAM", "libre", "load", "disque", "pm2", "runtimes"))
149+ for a in config.NODES:
150+ n = res.get(a, {})
151+ if not n.get("online"):
152+ print("%-8s %-6s %-16s — hors ligne" % (a, n.get("role", ""), n.get("ip", "")))
153+ continue
154+ rt = ",".join(k.replace("python@", "py").replace("uv-python@", "uvpy") for k, v in sorted(n["runtimes"].items()) if v)
155+ print("%-8s %-6s %-16s %5s %5sG %5sG %6.2f %6sG %4s %s" % (a, n["role"], n["ip"], n["cores"], n["ram_gb"], n["free_gb"], n["load1"], n["disk_free_gb"], len(n.get("pm2", [])), rt))
added mld/picker.py +96 −0
@@ -0,0 +1,96 @@
1+"""Sélection du nœud le plus efficient pour une app (ou un lot d'apps : plan)."""
2+from . import config, registry
3+
4+
5+def eligible(node, m, projected, exclude=()):
6+ """Retourne (ok, raison)."""
7+ a = node["alias"]
8+ pin = m["placement"].get("pin")
9+ if node.get("role") != "worker" and not (pin == a):
10+ return False, "rôle %s" % node.get("role")
11+ if not node.get("online"):
12+ return False, "hors ligne"
13+ if a in exclude or a in (m["placement"].get("avoid") or []):
14+ return False, "exclu"
15+ if pin and pin != a:
16+ return False, "épinglé sur %s" % pin
17+ missing = [r for r in m["requires"]["runtimes"] if not node["runtimes"].get(r)]
18+ if missing:
19+ return False, "runtimes manquants : %s" % ",".join(missing)
20+ need = max(float(m["requires"].get("ram_gb") or 1), m.get("ram_mb_observed", 512) * 1.5 / 1024)
21+ free = node["free_gb"] - projected[a]["ram_gb"]
22+ if free - need < 2.0:
23+ return False, "RAM insuffisante (%.1f G libres, besoin %.1f G)" % (free, need)
24+ if node["disk_free_gb"] < (m.get("size_mb", 0) / 1024) * 1.5 + 20:
25+ return False, "disque insuffisant"
26+ busy = set(node.get("ports", [])) | projected[a]["ports"]
27+ if not _is_home(a, m):
28+ for p in m["requires"].get("ports") or []:
29+ if p in busy:
30+ return False, "port %s occupé" % p
31+ return True, ""
32+
33+
34+def _is_home(alias, m):
35+ """Le nœud héberge déjà cette app (source d'import ou registre) : ses propres ports ne la bloquent pas."""
36+ return alias == m.get("source_node") or alias == registry.node_of(m["app"])
37+
38+
39+STAY_BONUS = 0.05 # évite de déplacer des Go de données pour un gain de score marginal
40+
41+
42+def score(node, m, projected):
43+ a = node["alias"]
44+ need = max(float(m["requires"].get("ram_gb") or 1), m.get("ram_mb_observed", 512) * 1.5 / 1024)
45+ free_after = node["free_gb"] - projected[a]["ram_gb"] - need
46+ ram_ratio = max(0.0, free_after / max(node["ram_gb"], 1))
47+ idle = max(0.0, 1.0 - (node["load1"] + 0.5 * projected[a]["apps"]) / max(node["cores"], 1))
48+ disk = min(1.0, node["disk_free_gb"] / 500.0)
49+ cores = min(1.0, node["cores"] / 32.0)
50+ hosted = len(registry.apps_on(a)) + projected[a]["apps"]
51+ s = 0.45 * ram_ratio + 0.25 * idle + 0.10 * disk + 0.15 * cores - 0.04 * hosted
52+ if m["placement"].get("prefer") == a:
53+ s += 0.15
54+ if _is_home(a, m):
55+ s += STAY_BONUS
56+ return round(s, 4)
57+
58+
59+def rank(m, scan, projected=None, exclude=()):
60+ projected = projected or empty_projection(scan)
61+ rows = []
62+ for a, node in scan.items():
63+ ok, why = eligible(node, m, projected, exclude)
64+ rows.append({"node": a, "ok": ok, "why": why, "score": score(node, m, projected) if ok else -1})
65+ rows.sort(key=lambda r: (-r["ok"], -r["score"]))
66+ return rows
67+
68+
69+def empty_projection(scan):
70+ return {a: {"ram_gb": 0.0, "apps": 0, "ports": set()} for a in scan}
71+
72+
73+def plan(manifests, scan, exclude=()):
74+ """Placement glouton : apps les plus lourdes d'abord, projection des ressources consommées."""
75+ projected = empty_projection(scan)
76+ order = sorted(manifests.values(), key=lambda m: -(m.get("ram_mb_observed", 512) + m.get("size_mb", 0) / 20.0))
77+ out = []
78+ for m in order:
79+ rows = rank(m, scan, projected, exclude)
80+ best = rows[0] if rows and rows[0]["ok"] else None
81+ if best:
82+ a = best["node"]
83+ need = max(float(m["requires"].get("ram_gb") or 1), m.get("ram_mb_observed", 512) * 1.5 / 1024)
84+ projected[a]["ram_gb"] += need
85+ projected[a]["apps"] += 1
86+ projected[a]["ports"] |= set(m["requires"].get("ports") or [])
87+ pin = m["placement"].get("pin")
88+ blockers = []
89+ if not best:
90+ cand = [r for r in rows if not r["ok"] and (r["node"] == pin if pin else (scan[r["node"]].get("online") and scan[r["node"]].get("role") == "worker"))]
91+ cand.sort(key=lambda r: -(scan[r["node"]].get("ram_gb") or 0))
92+ blockers = [(r["node"], r["why"]) for r in cand][:8]
93+ out.append({"app": m["app"], "node": best["node"] if best else None, "score": best["score"] if best else None,
94+ "current": registry.node_of(m["app"]) or m.get("source_node"), "pin": pin, "reason": m["placement"].get("reason", ""),
95+ "size_mb": m.get("size_mb", 0), "alternatives": [(r["node"], r["score"]) for r in rows[1:4] if r["ok"]], "blockers": blockers})
96+ return out
added mld/prepare.py +129 −0
@@ -0,0 +1,129 @@
1+"""`mld prepare <nœud>` : installe les runtimes manquants (Homebrew), copie la config ngrok, active pm2 au démarrage, pose le marqueur."""
2+import os
3+import shlex
4+from . import config, nodes, ssh
5+
6+
7+def _sudo_pw():
8+ if os.path.exists(config.SUDO_PW_FILE):
9+ return open(config.SUDO_PW_FILE).read().strip()
10+ return None
11+
12+
13+def prepare(alias, runtimes=None, ka_helpers=False):
14+ ip = nodes.ip_of(alias)
15+ if not ip:
16+ raise SystemExit("nœud %s injoignable" % alias)
17+ info = nodes.probe(alias, ip)
18+ have = info.get("runtimes", {})
19+ want = runtimes or ["node", "pnpm", "pm2", "ngrok", "python@3.14", "python@3.13", "uv", "git"]
20+ print("prepare %s (%s) — présents : %s" % (alias, ip, ", ".join(k for k, v in have.items() if v)))
21+ if not have.get("brew"):
22+ print(" ✗ Homebrew absent : bootstrap requis (Xcode CLT). Lancer d'abord : mld bootstrap %s" % alias)
23+ return False
24+ missing = [r for r in want if not have.get(r)]
25+ script = ""
26+ for r in missing:
27+ if r in config.BREW_FORMULAE:
28+ script += "echo ' brew install %s'; brew install -q %s >/dev/null 2>&1 || echo ' ! échec %s'\n" % (r, config.BREW_FORMULAE[r], r)
29+ elif r in config.BREW_CASKS:
30+ script += "echo ' brew install --cask ngrok'; brew install -q --cask ngrok >/dev/null 2>&1 || brew install -q ngrok/ngrok/ngrok >/dev/null 2>&1 || echo ' ! échec ngrok'\n"
31+ # config ngrok (authtoken) depuis la passerelle
32+ if "ngrok" in missing or not have.get("ngrok"):
33+ if os.path.exists(config.NGROK_CFG):
34+ ssh.run(ip, "mkdir -p \"$HOME/Library/Application Support/ngrok\"")
35+ ssh.scp_to(ip, config.NGROK_CFG, "'/Users/%s/Library/Application Support/ngrok/ngrok.yml'" % config.USER)
36+ script += "echo ' ngrok.yml copié'\n"
37+ # marqueur
38+ script += "[ -f ~/.maclustr-node ] || echo %s > ~/.maclustr-node\n" % alias
39+ # pm2 startup (launchd) — nécessite sudo
40+ pw = _sudo_pw()
41+ script += "if ! ls ~/Library/LaunchAgents 2>/dev/null | grep -qi pm2; then cmd=$(pm2 startup launchd -u %s --hp /Users/%s 2>/dev/null | grep '^sudo' | head -1); if [ -n \"$cmd\" ]; then %s echo \" pm2 startup: $( ${cmd#sudo } >/dev/null 2>&1 && echo ok || echo 'à faire manuellement')\"; fi; fi\n" % (
42+ config.USER, config.USER, ("echo %s | sudo -S -v 2>/dev/null;" % shlex.quote(pw)) if pw else "")
43+ rc, out, err = ssh.run(ip, script, timeout=1800)
44+ if ka_helpers:
45+ out += ka_helpers_install(alias, ip)
46+ print(out)
47+ if err.strip():
48+ print(" stderr:", err.strip()[-300:])
49+ info = nodes.probe(alias, ip)
50+ still = [r for r in want if not info.get("runtimes", {}).get(r)]
51+ print(" résultat : %s" % ("tout est là" if not still else "manquent encore " + ", ".join(still)))
52+ return not still
53+
54+
55+def bootstrap(alias):
56+ """Installe Xcode CLT + Homebrew sans GUI. Active un sudo NOPASSWD temporaire (retiré à la fin). Long (plusieurs Go)."""
57+ ip = nodes.ip_of(alias)
58+ pw = _sudo_pw()
59+ if not pw:
60+ raise SystemExit("mot de passe sudo requis dans %s" % config.SUDO_PW_FILE)
61+ script = r'''
62+export LC_ALL=C
63+echo %s | sudo -S -p '' sh -c 'echo "%s ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/mld-bootstrap && chmod 440 /etc/sudoers.d/mld-bootstrap'
64+trap 'sudo rm -f /etc/sudoers.d/mld-bootstrap' EXIT
65+if ! xcode-select -p >/dev/null 2>&1; then
66+ echo ' installation Command Line Tools...'
67+ touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
68+ PROD=$(softwareupdate -l 2>&1 | grep -oE 'Label: Command Line Tools for Xcode[^,]*' | sed 's/^Label: //' | sort -V | tail -1)
69+ echo " produit: ${PROD:-introuvable}"
70+ if [ -n "$PROD" ]; then sudo softwareupdate -i "$PROD" --verbose 2>&1 | tail -3; fi
71+ rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
72+ sudo xcode-select --switch /Library/Developer/CommandLineTools 2>/dev/null || true
73+fi
74+if [ ! -x /opt/homebrew/bin/brew ]; then
75+ echo ' installation Homebrew...'
76+ NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" 2>&1 | tail -4
77+ grep -q 'opt/homebrew/bin/brew shellenv' ~/.zprofile 2>/dev/null || echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile
78+fi
79+echo " CLT: $(xcode-select -p 2>/dev/null || echo absent) brew: $(/opt/homebrew/bin/brew --version 2>/dev/null | head -1 || echo absent)"
80+''' % (shlex.quote(pw), config.USER)
81+ rc, out, err = ssh.run(ip, script, timeout=5400)
82+ print(out)
83+ if err.strip():
84+ print(" stderr:", err.strip()[-400:])
85+ return rc == 0
86+
87+
88+def gateway_mdns():
89+ import subprocess
90+ name = subprocess.run(["scutil", "--get", "LocalHostName"], capture_output=True, text=True).stdout.strip()
91+ return (name + ".local") if name else nodes.load_lan()["ips"].get(config.GATEWAY, "")
92+
93+
94+def ka_helpers_install(alias, ip):
95+ """Pousseur Ka (launchd zsh → git push vers spbgit) + alias SSH `gitsrv` → passerelle + clé autorisée sur la passerelle."""
96+ assets = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "assets")
97+ out = [" ka-helpers sur %s :" % alias]
98+ ssh.write_remote_file(ip, "/Users/%s/ka-pousseur.sh" % config.USER, open(os.path.join(assets, "ka-pousseur.sh")).read(), mode="755")
99+ ssh.write_remote_file(ip, "/Users/%s/Library/LaunchAgents/com.ka.pousseur.plist" % config.USER, open(os.path.join(assets, "com.ka.pousseur.plist")).read(), mode="644")
100+ mdns = gateway_mdns()
101+ block = "Host gitsrv\n HostName %s\n User %s\n IdentityFile ~/.ssh/ka_guardian_ed25519\n IdentitiesOnly yes\n ControlMaster no\n ControlPath none\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n LogLevel ERROR\n\n" % (mdns, config.USER)
102+ script = r"""
103+mkdir -p ~/.ssh ~/ka-guardian-spool; touch ~/.ka-pousseur-repos; chmod 700 ~/.ssh
104+[ -f ~/.ssh/ka_guardian_ed25519 ] || ssh-keygen -q -t ed25519 -N '' -C "ka-guardian@$(cat ~/.maclustr-node 2>/dev/null || hostname -s)" -f ~/.ssh/ka_guardian_ed25519
105+# alias gitsrv en TÊTE du config (avant Host *) — remplace un bloc existant
106+python3 - <<'PYEOF'
107+import os,re
108+p=os.path.expanduser('~/.ssh/config'); s=open(p).read() if os.path.exists(p) else ''
109+s=re.sub(r'(?ms)^Host gitsrv\n(?:[ \t]+.*\n?)*\n?', '', s)
110+open(p,'w').write(%r + s); os.chmod(p, 0o600); print(' gitsrv -> %s')
111+PYEOF
112+cat ~/.ssh/ka_guardian_ed25519.pub
113+launchctl bootout gui/$(id -u)/com.ka.pousseur >/dev/null 2>&1; launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.ka.pousseur.plist && echo ' pousseur launchd chargé'
114+""" % (block, mdns)
115+ rc, o, e = ssh.run(ip, script, timeout=120)
116+ pub = [l for l in o.splitlines() if l.startswith("ssh-ed25519")]
117+ out += [" " + l for l in o.splitlines() if not l.startswith("ssh-ed25519")]
118+ if pub:
119+ ak = os.path.expanduser("~/.ssh/authorized_keys")
120+ cur = open(ak).read() if os.path.exists(ak) else ""
121+ if pub[0].split()[1] not in cur:
122+ with open(ak, "a") as f:
123+ f.write(pub[0].strip() + "\n")
124+ out.append(" clé %s autorisée sur la passerelle" % (pub[0].split()[-1] if len(pub[0].split()) > 2 else alias))
125+ else:
126+ out.append(" clé déjà autorisée sur la passerelle")
127+ rc, o, e = ssh.run(ip, "ssh -o BatchMode=yes -o ConnectTimeout=6 gitsrv 'echo ok' 2>&1 | tail -1", timeout=30)
128+ out.append(" test ssh gitsrv : %s" % (o.strip() or e.strip()[-80:]))
129+ return "\n".join(out) + "\n"
added mld/registry.py +51 −0
@@ -0,0 +1,51 @@
1+"""Registre des déploiements (source de vérité) : ~/dispatch/registry.json."""
2+import json
3+import os
4+import time
5+from . import config
6+
7+
8+def load():
9+ if os.path.exists(config.REGISTRY):
10+ return json.load(open(config.REGISTRY))
11+ return {"updated": None, "gateway": config.GATEWAY, "apps": {}, "history": []}
12+
13+
14+def save(r):
15+ r["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
16+ r["gateway"] = config.GATEWAY
17+ tmp = config.REGISTRY + ".tmp"
18+ json.dump(r, open(tmp, "w"), indent=2, ensure_ascii=False)
19+ os.replace(tmp, config.REGISTRY)
20+
21+
22+def node_of(app):
23+ return load()["apps"].get(app, {}).get("node")
24+
25+
26+def apps_on(alias):
27+ return [a for a, v in load()["apps"].items() if v.get("node") == alias]
28+
29+
30+def set_app(app, **fields):
31+ r = load()
32+ cur = r["apps"].get(app, {})
33+ cur.update(fields)
34+ cur["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
35+ r["apps"][app] = cur
36+ save(r)
37+
38+
39+def remove_app(app, note=""):
40+ r = load()
41+ if app in r["apps"]:
42+ r["history"].append({"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "app": app, "event": "removed", "was": r["apps"][app], "note": note})
43+ del r["apps"][app]
44+ save(r)
45+
46+
47+def log(app, event, **kw):
48+ r = load()
49+ r["history"].append(dict(ts=time.strftime("%Y-%m-%dT%H:%M:%S"), app=app, event=event, **kw))
50+ r["history"] = r["history"][-500:]
51+ save(r)
added mld/render.py +89 −0
@@ -0,0 +1,89 @@
1+"""Génération des fichiers de lancement : ecosystem PM2 (.cjs) et plists launchd."""
2+import json
3+import plistlib
4+import shlex
5+
6+
7+def _mem(v):
8+ if not v:
9+ return None
10+ if isinstance(v, str):
11+ return v
12+ return "%dM" % (int(v) // (1024 * 1024))
13+
14+
15+def _home(path, ctx):
16+ """PM2/launchd n'expandent pas « ~ » : chemin absolu obligatoire."""
17+ if isinstance(path, str) and path.startswith("~"):
18+ return ctx["HOME"] + path[1:]
19+ return path
20+
21+
22+def ecosystem(m, ctx):
23+ """Retourne le contenu JS d'un ecosystem.config.cjs pour les processus PM2 + tunnel ngrok."""
24+ from .manifest import render
25+ apps = []
26+ for p in m["processes"]:
27+ if p.get("manager", "pm2") != "pm2":
28+ continue
29+ a = {
30+ "name": p["name"],
31+ "script": _home(render(p["script"], ctx), ctx),
32+ "cwd": _home(render(p.get("cwd") or m["dir"], ctx), ctx),
33+ "exec_mode": "fork",
34+ "instances": 1,
35+ "autorestart": p.get("autorestart", True),
36+ "merge_logs": True,
37+ "time": True,
38+ "env": render(p.get("env") or {}, ctx),
39+ }
40+ if p.get("args"):
41+ args = p["args"]
42+ a["args"] = " ".join(shlex.quote(str(x)) for x in render(args, ctx)) if isinstance(args, list) else render(args, ctx)
43+ interp = p.get("interpreter")
44+ if interp and interp not in ("none", "N/A"):
45+ a["interpreter"] = render(interp, ctx)
46+ elif interp in ("none",) or (p["script"].endswith((".sh", ".mjs", ".js", ".py", ".cjs")) is False):
47+ a["interpreter"] = "none"
48+ if p.get("cron_restart"):
49+ a["cron_restart"] = p["cron_restart"]
50+ if p.get("max_memory_restart"):
51+ a["max_memory_restart"] = _mem(p["max_memory_restart"])
52+ if p.get("restart_delay"):
53+ a["restart_delay"] = p["restart_delay"]
54+ if p.get("node_args"):
55+ a["node_args"] = p["node_args"]
56+ apps.append(a)
57+ if m.get("ngrok"):
58+ n = m["ngrok"]
59+ args = ["http", "--url=%s" % n["url"], str(n["port"]), "--log=stdout"] + list(n.get("extra_args") or [])
60+ apps.append({
61+ "name": n["name"], "script": n.get("binary") or "/opt/homebrew/bin/ngrok", "args": " ".join(args),
62+ "interpreter": "none", "cwd": _home(render(m["dir"], ctx), ctx), "exec_mode": "fork", "autorestart": True,
63+ "restart_delay": 5000, "merge_logs": True, "time": True,
64+ })
65+ head = "// généré par maclustr-dispatch — ne pas éditer à la main (app: %s, nœud: %s)\n" % (m["app"], ctx.get("NODE"))
66+ return head + "module.exports = { apps: %s };\n" % json.dumps(apps, indent=2, ensure_ascii=False)
67+
68+
69+def plist(item, ctx):
70+ from .manifest import render
71+ d = {
72+ "Label": item["label"],
73+ "ProgramArguments": render(item["program_arguments"], ctx),
74+ "RunAtLoad": item.get("run_at_load", True),
75+ "KeepAlive": item.get("keep_alive", True),
76+ "StandardOutPath": render(item.get("stdout") or "{{HOME}}/Library/Logs/%s.log" % item["label"], ctx),
77+ "StandardErrorPath": render(item.get("stderr") or "{{HOME}}/Library/Logs/%s.err.log" % item["label"], ctx),
78+ }
79+ if item.get("working_directory"):
80+ d["WorkingDirectory"] = _home(render(item["working_directory"], ctx), ctx)
81+ env = dict(item.get("env") or {})
82+ env.setdefault("HOME", ctx["HOME"])
83+ env.setdefault("PATH", "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin")
84+ d["EnvironmentVariables"] = render(env, ctx)
85+ if item.get("start_interval"):
86+ d["StartInterval"] = int(item["start_interval"])
87+ if item.get("throttle_interval"):
88+ d["ThrottleInterval"] = int(item["throttle_interval"])
89+ return plistlib.dumps(d).decode()
added mld/ssh.py +75 −0
@@ -0,0 +1,75 @@
1+"""SSH / rsync vers les nœuds via le LAN 192.168.2.x (le SSH inter-nœuds Tailscale est bloqué par ACL)."""
2+import os
3+import shlex
4+import subprocess
5+from . import config
6+
7+SSH_OPTS = [
8+ "-o", "BatchMode=yes", "-o", "ConnectTimeout=8", "-o", "StrictHostKeyChecking=no",
9+ "-o", "UserKnownHostsFile=/dev/null", "-o", "LogLevel=ERROR", "-o", "ServerAliveInterval=15",
10+ "-i", config.SSH_KEY,
11+]
12+# PATH complet sur le nœud (les shells non interactifs n'ont pas Homebrew)
13+REMOTE_PATH = "export PATH=/opt/homebrew/bin:/opt/homebrew/sbin:$HOME/bin:$HOME/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin; "
14+
15+
16+def target(ip):
17+ return "%s@%s" % (config.USER, ip)
18+
19+
20+def run(ip, cmd, timeout=120, stdin=None, path=True):
21+ """Exécute `cmd` (bash) sur le nœud. Retourne (rc, stdout, stderr)."""
22+ full = (REMOTE_PATH if path else "") + cmd
23+ try:
24+ p = subprocess.run(["ssh"] + SSH_OPTS + [target(ip), "bash -s"], input=full if stdin is None else stdin,
25+ capture_output=True, text=True, timeout=timeout)
26+ return p.returncode, p.stdout, p.stderr
27+ except subprocess.TimeoutExpired:
28+ return 124, "", "timeout after %ss" % timeout
29+
30+
31+def run_script(ip, script, timeout=300):
32+ """Envoie un script complet sur stdin de `bash -s`."""
33+ return run(ip, "", timeout=timeout, stdin=REMOTE_PATH + "\n" + script)
34+
35+
36+def ok(ip, cmd, timeout=60):
37+ return run(ip, cmd, timeout)[0] == 0
38+
39+
40+def rsync_push(local_dir, ip, remote_dir, excludes=(), delete=True, timeout=7200, extra=()):
41+ """Passerelle -> nœud. remote_dir peut contenir ~ (résolu côté nœud)."""
42+ args = ["rsync", "-a", "--stats", "-e", "ssh " + " ".join(shlex.quote(o) for o in SSH_OPTS)]
43+ if delete:
44+ args.append("--delete")
45+ for e in excludes:
46+ args += ["--exclude", e]
47+ args += list(extra)
48+ args += ["--rsync-path", "mkdir -p %s && rsync" % shlex.quote(remote_dir), local_dir.rstrip("/") + "/", "%s:%s/" % (target(ip), remote_dir)]
49+ p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
50+ return p.returncode, p.stdout, p.stderr
51+
52+
53+def rsync_pull(ip, remote_dir, local_dir, excludes=(), delete=True, timeout=7200):
54+ """Nœud -> passerelle (staging)."""
55+ os.makedirs(local_dir, exist_ok=True)
56+ args = ["rsync", "-a", "--stats", "-e", "ssh " + " ".join(shlex.quote(o) for o in SSH_OPTS)]
57+ if delete:
58+ args.append("--delete")
59+ for e in excludes:
60+ args += ["--exclude", e]
61+ args += ["%s:%s/" % (target(ip), remote_dir), local_dir.rstrip("/") + "/"]
62+ p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
63+ return p.returncode, p.stdout, p.stderr
64+
65+
66+def scp_to(ip, local_file, remote_path):
67+ p = subprocess.run(["scp"] + SSH_OPTS + [local_file, "%s:%s" % (target(ip), remote_path)], capture_output=True, text=True)
68+ return p.returncode == 0
69+
70+
71+def write_remote_file(ip, remote_path, content, mode="644"):
72+ """Écrit un fichier sur le nœud via heredoc (contenu texte)."""
73+ q = shlex.quote(remote_path)
74+ script = "mkdir -p $(dirname %s) && cat > %s <<'__MLD_EOF__'\n%s\n__MLD_EOF__\nchmod %s %s" % (q, q, content, mode, q)
75+ return run(ip, script, timeout=60)
76
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere