SPB Git forge
29commits 1branches 0releases
684.0 KBsize
maindefault branch
2 days agolast push
Python 95.8% Shell 4.2%

maclustr-dispatch v1.0.0 — mld : découverte LAN, scan, picker, deploy/move/health, prepare/bootstrap, manifestes (2026-09-04)

Simon-Pierre Boucher committed 19 days ago (Sep 4, 2026)

18 changed files +1,529 −0

added .gitignore +3 −0
@@ -0,0 +1,3 @@
1 +__pycache__/
2 +*.pyc
3 +.DS_Store
added README.md +65 −0
@@ -0,0 +1,65 @@
1 +# maclustr-dispatch — `mld`
2 +
3 +Orchestrateur de déploiement du cluster MacLustr. Il tourne **sur la passerelle M1M32** et
4 +déploie chaque application sur le **nœud le plus efficient** du cluster, via le LAN
5 +(192.168.2.x, clé SSH `maclustr-m1m32` autorisée sur tous les nœuds).
6 +
7 +```
8 +laptop ──ssh──▶ M1M32 (passerelle : spbgit + mld + staging ~/dispatch/stage)
9 + │ scan live (RAM libre, charge, disque, ports, runtimes)
10 + │ choix du nœud (score) → rsync LAN → PM2/launchd → ngrok → healthcheck → registre
11 +
12 + M3U96a · M3U96b · M2U64 · M4M64a · M4M64b · M4M36 · M2M32 · …
13 +```
14 +
15 +## Depuis le laptop
16 +
17 +`~/Desktop/cluster-skill/mld <commande>` relaie à la passerelle et rapatrie le registre
18 +(`cluster-skill/cluster-registry.json`).
19 +
20 +| Commande | Effet |
21 +|---|---|
22 +| `mld scan` / `mld nodes` | sonde (ou relit) ressources + runtimes des nœuds |
23 +| `mld apps` / `mld show <app>` | manifestes et emplacement courant |
24 +| `mld pick <app>` · `mld pick --ram 4 --runtime node` | classement des nœuds pour une app / un besoin |
25 +| `mld plan [apps…]` | plan de placement global (glouton, projection des ressources) |
26 +| `mld stage <dir-local> <app>` | pousse un dossier du laptop dans le staging de la passerelle |
27 +| `mld deploy <app> [--node X]` | staging → nœud choisi (auto sans `--node`), démarrage, santé, registre |
28 +| `mld move <app> [--to X] [--from Y]` | migration : pré-copie à chaud, arrêt, delta final, déploiement, santé, retrait (rollback auto) |
29 +| `mld stop/start/restart <app>` · `mld logs <app>` · `mld health <app>` | cycle de vie |
30 +| `mld status [--live]` · `mld registry` | registre (`~/dispatch/registry.json` sur M1M32) |
31 +| `mld retire <app>` | arrête et efface la copie du nœud |
32 +| `mld prepare <nœud> [--ka-helpers]` | installe node/pnpm/pm2/ngrok/python via Homebrew, config ngrok, pm2 startup, marqueur ; `--ka-helpers` = pousseur Ka + alias `gitsrv` + clé autorisée sur la passerelle |
33 +| `mld bootstrap <nœud>` | Xcode CLT + Homebrew sur un Mac vierge (sudo NOPASSWD temporaire) |
34 +| `mld import <nœud>` | brouillons de manifestes depuis les PM2 vivants d'un nœud (`~/dispatch/apps/_drafts/`) |
35 +
36 +## Manifestes (`~/dispatch/apps/<app>.json`, mode 600)
37 +
38 +Un manifeste décrit *comment* tourne une app : `dir` (même chemin absolu sur tous les nœuds),
39 +`processes` (PM2 : script, args, interpreter, cwd, env, cron_restart…), `ngrok` (domaine réservé →
40 +processus `<app>-ngrok`), `launchd` (plists générés), `requires` (runtimes, RAM, ports),
41 +`placement` (`pin`, `prefer`, `avoid` + raison), `sync_excludes`, `extra_paths`, `env_overrides`
42 +(réécriture de `.env` au déploiement), `hooks.post_sync` / `post_start`, `ka_repo` (inscription au pousseur).
43 +Gabarits : `{{HOME}}`, `{{NODE}}`, `{{LAN_IP}}`, `{{IP:<alias>}}`.
44 +
45 +## Scoring
46 +
47 +`0.45·RAM libre après placement + 0.25·cœurs idle + 0.10·disque + 0.15·cœurs − 0.04·apps hébergées
48 +(+0.15 prefer, +0.05 si déjà sur ce nœud)`. Inéligible : hors ligne, rôle gateway (sauf épingle),
49 +runtime manquant, RAM libre < besoin + 2 Go, disque, port occupé (hors nœud d'origine).
50 +
51 +## Ajouter un nœud
52 +
53 +1. Activer *Session à distance* sur le Mac, autoriser la clé de la passerelle
54 + (`~/.ssh/id_ed25519.pub` de M1M32) et celle du laptop dans `~/.ssh/authorized_keys`.
55 +2. `echo <alias> > ~/.maclustr-node` sur le nœud ; ajouter l'alias dans `mld/config.py` (`NODES`).
56 +3. `mld discover``mld bootstrap <alias>` (si Mac vierge) → `mld prepare <alias>`.
57 +
58 +## Fichiers
59 +
60 +- `mld/config.py` inventaire statique, rôles, chemins, détection des runtimes
61 +- `mld/nodes.py` découverte LAN, sonde · `mld/picker.py` score/plan · `mld/deploy.py` fetch/deploy/move/health
62 +- `mld/render.py` ecosystem PM2 + plists · `mld/prepare.py` runtimes/bootstrap/ka-helpers · `mld/importer.py`
63 +- `assets/` pousseur Ka (script + plist)
64 +
65 +Pas de dépendance : Python 3 système (3.9) + ssh/rsync (GNU rsync via Homebrew sur la passerelle).
added assets/com.ka.pousseur.plist +12 −0
@@ -0,0 +1,12 @@
1 +<?xml version="1.0" encoding="UTF-8"?>
2 +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3 +<plist version="1.0"><dict>
4 + <key>Label</key><string>com.ka.pousseur</string>
5 + <key>ProgramArguments</key><array>
6 + <string>/bin/zsh</string>
7 + <string>/Users/simon-pierreboucher/ka-pousseur.sh</string>
8 + </array>
9 + <key>RunAtLoad</key><true/><key>KeepAlive</key><true/>
10 + <key>StandardOutPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/pousseur.log</string>
11 + <key>StandardErrorPath</key><string>/Users/simon-pierreboucher/ka-guardian-spool/pousseur.log</string>
12 +</dict></plist>
added assets/ka-pousseur.sh +34 −0
@@ -0,0 +1,34 @@
1 +#!/bin/zsh
2 +# ============================================
3 +# KA pousseur — pousse vers spbgit les commits en attente des repos Ka du nœud.
4 +# Chaîne 100 % binaires Apple (launchd → zsh → /usr/bin/git → /usr/bin/ssh) :
5 +# contourne macOS 26 Local Network Privacy, qui bloque les `git push` lancés
6 +# par les missions Claude (descendants du runner python homebrew).
7 +# Repos surveillés : un chemin par ligne dans ~/.ka-pousseur-repos.
8 +# Push SIMPLE uniquement (jamais de force) : un non-fast-forward est loggé,
9 +# jamais résolu d'autorité.
10 +# ============================================
11 +export PATH=/usr/bin:/bin:/usr/sbin:/sbin
12 +SPOOL=~/ka-guardian-spool
13 +LOG=$SPOOL/pousseur.log
14 +REPOS_FILE=~/.ka-pousseur-repos
15 +mkdir -p $SPOOL
16 +
17 +while true; do
18 + if [[ -f $REPOS_FILE ]]; then
19 + while IFS= read -r dir; do
20 + [[ -z $dir || $dir == \#* ]] && continue
21 + d=${dir/#\~/$HOME}
22 + [[ -d $d/.git ]] || continue
23 + ahead=$(/usr/bin/git -C "$d" rev-list --count origin/main..main 2>/dev/null)
24 + if [[ -n $ahead && $ahead -gt 0 ]]; then
25 + if /usr/bin/git -C "$d" push origin main >>$LOG 2>&1; then
26 + echo "$(date '+%F %T') poussé $ahead commit(s): $d" >>$LOG
27 + else
28 + echo "$(date '+%F %T') ÉCHEC push ($ahead en attente): $d" >>$LOG
29 + fi
30 + fi
31 + done <$REPOS_FILE
32 + fi
33 + sleep 300
34 +done
added bin/mld +4 −0
@@ -0,0 +1,4 @@
1 +#!/bin/bash
2 +# mld — MacLustr Dispatch (à exécuter sur la passerelle M1M32)
3 +export PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
4 +cd "$(dirname "$0")/.." && exec /usr/bin/python3 -m mld "$@"
added mld/__init__.py +2 −0
@@ -0,0 +1,2 @@
1 +"""maclustr-dispatch — orchestrateur de déploiement du cluster MacLustr (passerelle M1M32)."""
2 +__version__ = "1.0.0"
added mld/__main__.py +3 −0
@@ -0,0 +1,3 @@
1 +from .cli import main
2 +
3 +main()
added mld/cli.py +208 −0
@@ -0,0 +1,208 @@
1 +"""CLI `mld` — MacLustr Dispatch (tourne sur la passerelle M1M32)."""
2 +import argparse
3 +import json
4 +import sys
5 +from . import config, deploy, manifest, nodes, picker, registry
6 +
7 +
8 +def cmd_discover(a):
9 + nodes.discover()
10 +
11 +
12 +def cmd_scan(a):
13 + nodes.scan(aliases=a.nodes or None)
14 +
15 +
16 +def cmd_nodes(a):
17 + res = nodes.load_scan() or nodes.scan(verbose=False)
18 + nodes.print_scan(res)
19 +
20 +
21 +def cmd_apps(a):
22 + ms = manifest.load_all()
23 + r = registry.load()["apps"]
24 + print("%-18s %-8s %-6s %-30s %-8s %s" % ("app", "nœud", "port", "domaine", "épingle", "runtimes"))
25 + for name, m in ms.items():
26 + cur = r.get(name, {}).get("node") or "—"
27 + print("%-18s %-8s %-6s %-30s %-8s %s" % (name, cur, m.get("port") or "", m.get("domain") or "", m["placement"].get("pin") or "", ",".join(m["requires"]["runtimes"])))
28 +
29 +
30 +def cmd_show(a):
31 + print(json.dumps(manifest.load(a.app), indent=2, ensure_ascii=False))
32 +
33 +
34 +def cmd_pick(a):
35 + m = manifest.load(a.app) if a.app else {"app": "(ad hoc)", "requires": {"runtimes": a.runtime or ["pm2"], "ram_gb": a.ram or 1, "ports": []}, "ram_mb_observed": int((a.ram or 1) * 1024), "size_mb": 0, "placement": {"pin": None, "prefer": None, "avoid": [], "reason": ""}}
36 + scan = nodes.load_scan() if not a.rescan else nodes.scan(verbose=False)
37 + rows = picker.rank(m, scan, exclude=a.exclude or ())
38 + if a.json:
39 + print(json.dumps(rows, indent=2))
40 + return
41 + print("classement pour %s :" % m["app"])
42 + for r in rows:
43 + if r["ok"]:
44 + print(" %-8s score %.3f" % (r["node"], r["score"]))
45 + elif scan.get(r["node"], {}).get("online") and scan[r["node"]].get("role") == "worker":
46 + print(" %-8s%s" % (r["node"], r["why"]))
47 +
48 +
49 +def cmd_plan(a):
50 + ms = manifest.load_all()
51 + if a.apps:
52 + ms = {k: v for k, v in ms.items() if k in a.apps}
53 + scan = nodes.scan(verbose=False) if a.rescan else (nodes.load_scan() or nodes.scan(verbose=False))
54 + rows = picker.plan(ms, scan, exclude=a.exclude or ())
55 + if a.json:
56 + print(json.dumps(rows, indent=2, ensure_ascii=False))
57 + return
58 + print("%-19s %-8s %-8s %-6s %7s %s" % ("app", "actuel", "proposé", "score", "taille", "note"))
59 + counts = {}
60 + moving_mb = 0
61 + for r in rows:
62 + counts[r["node"]] = counts.get(r["node"], 0) + 1
63 + mv = r["node"] and r["current"] and r["node"] != r["current"]
64 + if mv:
65 + moving_mb += r["size_mb"]
66 + note = ("épinglé — " + r["reason"]) if r["pin"] else (("alt: " + ", ".join("%s %.2f" % x for x in r["alternatives"])) if r["alternatives"] else "")
67 + if not r["node"]:
68 + note = "AUCUN NŒUD : " + " ; ".join("%s (%s)" % b for b in r["blockers"])
69 + flag = "→" if mv else ("=" if r["node"] else "!")
70 + print("%-19s %-8s %-8s %-6s %6dM %s %s" % (r["app"], r["current"] or "—", r["node"] or "—", ("%.3f" % r["score"]) if r["score"] is not None else "", r["size_mb"], flag, note))
71 + print("\nrépartition : " + ", ".join("%s×%s" % (k, v) for k, v in sorted(counts.items(), key=lambda x: -x[1]) if k))
72 + print("données à déplacer : %.1f Go (→ = migration, = = redéploiement en place)" % (moving_mb / 1024.0))
73 +
74 +
75 +def cmd_fetch(a):
76 + deploy.fetch(a.app, a.from_node, final=a.final)
77 +
78 +
79 +def cmd_deploy(a):
80 + if a.node:
81 + node = a.node
82 + else:
83 + scan = nodes.load_scan() or nodes.scan(verbose=False)
84 + rows = picker.rank(manifest.load(a.app), scan)
85 + if not rows or not rows[0]["ok"]:
86 + raise SystemExit("aucun nœud éligible : " + "; ".join("%s: %s" % (r["node"], r["why"]) for r in rows[:6]))
87 + node = rows[0]["node"]
88 + print("nœud choisi : %s (score %.3f)" % (node, rows[0]["score"]))
89 + try:
90 + deploy.deploy(a.app, node, reinstall=a.reinstall, skip_sync=a.skip_sync)
91 + except deploy.DeployError as e:
92 + raise SystemExit("ÉCHEC : %s" % e)
93 +
94 +
95 +def cmd_move(a):
96 + to = a.to
97 + if not to:
98 + scan = nodes.load_scan() or nodes.scan(verbose=False)
99 + rows = picker.rank(manifest.load(a.app), scan, exclude=[registry.node_of(a.app)] if a.away else ())
100 + if not rows or not rows[0]["ok"]:
101 + raise SystemExit("aucun nœud éligible : " + "; ".join("%s: %s" % (r["node"], r["why"]) for r in rows[:6]))
102 + to = rows[0]["node"]
103 + print("nœud choisi : %s (score %.3f)" % (to, rows[0]["score"]))
104 + try:
105 + deploy.move(a.app, to, from_alias=a.from_node, keep_source=a.keep_source, reinstall=a.reinstall)
106 + except deploy.DeployError as e:
107 + raise SystemExit("ÉCHEC : %s" % e)
108 +
109 +
110 +def cmd_stop(a):
111 + deploy.stop(a.app, a.node or registry.node_of(a.app))
112 + registry.set_app(a.app, status="stopped")
113 +
114 +
115 +def cmd_start(a):
116 + node = a.node or registry.node_of(a.app)
117 + deploy.start(a.app, node)
118 + ok, det = deploy.health(a.app, node)
119 + registry.set_app(a.app, node=node, status="online" if ok else "unhealthy", health=det)
120 + print("santé :", det)
121 +
122 +
123 +def cmd_restart(a):
124 + cmd_stop(a)
125 + cmd_start(a)
126 +
127 +
128 +def cmd_retire(a):
129 + deploy.retire(a.app, a.node or registry.node_of(a.app), remove_dir=not a.keep_dir)
130 + if not a.keep_registry:
131 + registry.remove_app(a.app, note=a.note or "")
132 +
133 +
134 +def cmd_status(a):
135 + if a.live:
136 + for row in deploy.live_status(a.app):
137 + print("%-18s %-8s %-30s %-3s %s" % row)
138 + else:
139 + print("%-18s %-8s %-6s %-30s %-10s %s" % ("app", "nœud", "port", "domaine", "état", "déployée"))
140 + for row in deploy.status(a.app):
141 + print("%-18s %-8s %-6s %-30s %-10s %s" % tuple("" if x is None else x for x in row))
142 +
143 +
144 +def cmd_health(a):
145 + ok, det = deploy.health(a.app, a.node or registry.node_of(a.app))
146 + print("OK" if ok else "KO", det)
147 + sys.exit(0 if ok else 1)
148 +
149 +
150 +def cmd_registry(a):
151 + print(json.dumps(registry.load(), indent=2, ensure_ascii=False))
152 +
153 +
154 +def cmd_import(a):
155 + from . import importer
156 + for p in importer.import_node(a.node, apps_filter=a.apps or None):
157 + print("brouillon :", p)
158 +
159 +
160 +def cmd_prepare(a):
161 + from . import prepare
162 + prepare.prepare(a.node, runtimes=a.runtime or None, ka_helpers=a.ka_helpers)
163 +
164 +
165 +def cmd_bootstrap(a):
166 + from . import prepare
167 + prepare.bootstrap(a.node)
168 +
169 +
170 +def cmd_logs(a):
171 + node = a.node or registry.node_of(a.app)
172 + from . import ssh
173 + ip = nodes.ip_of(node)
174 + names = manifest.pm2_names(manifest.load(a.app))
175 + rc, out, err = ssh.run(ip, "pm2 logs %s --nostream --lines %d 2>/dev/null" % (" ".join(names), a.lines), timeout=60)
176 + print(out)
177 +
178 +
179 +def main(argv=None):
180 + p = argparse.ArgumentParser(prog="mld", description="MacLustr Dispatch — déploie les apps depuis la passerelle %s vers le nœud le plus efficient." % config.GATEWAY)
181 + sp = p.add_subparsers(dest="cmd", required=True)
182 + s = sp.add_parser("discover", help="balaye le LAN, met en cache les IP des nœuds"); s.set_defaults(f=cmd_discover)
183 + s = sp.add_parser("scan", help="sonde ressources + runtimes de chaque nœud"); s.add_argument("nodes", nargs="*"); s.set_defaults(f=cmd_scan)
184 + s = sp.add_parser("nodes", help="dernier scan"); s.set_defaults(f=cmd_nodes)
185 + s = sp.add_parser("apps", help="manifestes + emplacement courant"); s.set_defaults(f=cmd_apps)
186 + s = sp.add_parser("show", help="affiche un manifeste"); s.add_argument("app"); s.set_defaults(f=cmd_show)
187 + s = sp.add_parser("pick", help="classe les nœuds pour une app (ou un besoin ad hoc)"); s.add_argument("app", nargs="?"); s.add_argument("--ram", type=float); s.add_argument("--runtime", action="append"); s.add_argument("--exclude", action="append"); s.add_argument("--rescan", action="store_true"); s.add_argument("--json", action="store_true"); s.set_defaults(f=cmd_pick)
188 + s = sp.add_parser("plan", help="plan de placement pour toutes les apps (ou une liste)"); s.add_argument("apps", nargs="*"); s.add_argument("--exclude", action="append"); s.add_argument("--rescan", action="store_true"); s.add_argument("--json", action="store_true"); s.set_defaults(f=cmd_plan)
189 + s = sp.add_parser("fetch", help="copie l'app d'un nœud vers le staging"); s.add_argument("app"); s.add_argument("--from", dest="from_node", required=True); s.add_argument("--final", action="store_true"); s.set_defaults(f=cmd_fetch)
190 + s = sp.add_parser("deploy", help="staging -> nœud (choisi automatiquement sans --node)"); s.add_argument("app"); s.add_argument("--node"); s.add_argument("--reinstall", action="store_true"); s.add_argument("--skip-sync", action="store_true"); s.set_defaults(f=cmd_deploy)
191 + s = sp.add_parser("move", help="migre une app (fetch, stop, deploy, retire)"); s.add_argument("app"); s.add_argument("--to"); s.add_argument("--from", dest="from_node"); s.add_argument("--away", action="store_true", help="exclut le nœud actuel"); s.add_argument("--keep-source", action="store_true"); s.add_argument("--reinstall", action="store_true"); s.set_defaults(f=cmd_move)
192 + s = sp.add_parser("stop"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_stop)
193 + s = sp.add_parser("start"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_start)
194 + s = sp.add_parser("restart"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_restart)
195 + s = sp.add_parser("retire", help="arrête + efface la copie d'un nœud"); s.add_argument("app"); s.add_argument("--node"); s.add_argument("--keep-dir", action="store_true"); s.add_argument("--keep-registry", action="store_true"); s.add_argument("--note"); s.set_defaults(f=cmd_retire)
196 + s = sp.add_parser("status"); s.add_argument("app", nargs="?"); s.add_argument("--live", action="store_true"); s.set_defaults(f=cmd_status)
197 + s = sp.add_parser("health"); s.add_argument("app"); s.add_argument("--node"); s.set_defaults(f=cmd_health)
198 + s = sp.add_parser("logs"); s.add_argument("app"); s.add_argument("--node"); s.add_argument("--lines", type=int, default=40); s.set_defaults(f=cmd_logs)
199 + s = sp.add_parser("registry", help="registre JSON complet"); s.set_defaults(f=cmd_registry)
200 + s = sp.add_parser("import", help="brouillons de manifestes depuis les PM2 d'un nœud"); s.add_argument("node"); s.add_argument("apps", nargs="*"); s.set_defaults(f=cmd_import)
201 + s = sp.add_parser("prepare", help="installe runtimes/ngrok/pm2-startup sur un nœud"); s.add_argument("node"); s.add_argument("--runtime", action="append"); s.add_argument("--ka-helpers", action="store_true"); s.set_defaults(f=cmd_prepare)
202 + s = sp.add_parser("bootstrap", help="Xcode CLT + Homebrew sur un nœud vierge (long)"); s.add_argument("node"); s.set_defaults(f=cmd_bootstrap)
203 + a = p.parse_args(argv)
204 + a.f(a)
205 +
206 +
207 +if __name__ == "__main__":
208 + main()
added mld/config.py +76 −0
@@ -0,0 +1,76 @@
1 +"""maclustr-dispatch — configuration centrale.
2 +
3 +Tout tourne SUR la passerelle (M1M32). Python 3.9 (système macOS) : pas de syntaxe > 3.9.
4 +"""
5 +import os
6 +
7 +USER = "simon-pierreboucher"
8 +HOME = os.path.expanduser("~")
9 +GATEWAY = "M1M32"
10 +DOMAIN = "maclustr.io"
11 +LAN_PREFIX = "192.168.2."
12 +SSH_KEY = os.path.join(HOME, ".ssh", "id_ed25519") # clé « maclustr-m1m32 », autorisée sur tous les nœuds
13 +STATE = os.path.join(HOME, "dispatch") # état : apps/, stage/, registry.json, lan.json, logs/
14 +APPS_DIR = os.path.join(STATE, "apps")
15 +STAGE_DIR = os.path.join(STATE, "stage")
16 +LOG_DIR = os.path.join(STATE, "logs")
17 +REGISTRY = os.path.join(STATE, "registry.json")
18 +LAN_CACHE = os.path.join(STATE, "lan.json")
19 +SCAN_CACHE = os.path.join(STATE, "scan.json")
20 +NGROK_CFG = os.path.join(HOME, "Library", "Application Support", "ngrok", "ngrok.yml")
21 +RUN_DIR = "~/dispatch-run" # sur les nœuds : ecosystem PM2 générés
22 +SUDO_PW_FILE = os.path.join(STATE, ".sudo") # mot de passe sudo des nœuds (0600), pour `prepare`
23 +
24 +# Inventaire statique (alias -> cœurs, RAM Go, modèle). Les IP LAN sont découvertes en live.
25 +NODES = {
26 + "M3U96a": {"cores": 32, "ram_gb": 96, "model": "Mac Studio"},
27 + "M3U96b": {"cores": 32, "ram_gb": 96, "model": "Mac Studio"},
28 + "M2U64": {"cores": 24, "ram_gb": 64, "model": "Mac Studio"},
29 + "M4M64a": {"cores": 16, "ram_gb": 64, "model": "Mac Studio"},
30 + "M4M64b": {"cores": 16, "ram_gb": 64, "model": "Mac Studio"},
31 + "M4BP48": {"cores": 16, "ram_gb": 48, "model": "MacBook Pro"},
32 + "M4BP36": {"cores": 14, "ram_gb": 36, "model": "MacBook Pro"},
33 + "M4M36": {"cores": 14, "ram_gb": 36, "model": "Mac Studio"},
34 + "M2M32": {"cores": 12, "ram_gb": 32, "model": "Mac Studio"},
35 + "M2M32b": {"cores": 12, "ram_gb": 32, "model": "Mac Studio"},
36 + "M2M32c": {"cores": 12, "ram_gb": 32, "model": "Mac Studio"},
37 + "m4mc": {"cores": 12, "ram_gb": 24, "model": "Mac mini"},
38 + "M1M32": {"cores": 10, "ram_gb": 32, "model": "Mac Studio"},
39 + "m4ma": {"cores": 10, "ram_gb": 24, "model": "Mac mini"},
40 + "m4mb": {"cores": 10, "ram_gb": 16, "model": "Mac mini"},
41 + "m2m16": {"cores": 10, "ram_gb": 16, "model": "Mac mini"},
42 + "M3BA24": {"cores": 8, "ram_gb": 24, "model": "MacBook Air"},
43 + "M3BA16": {"cores": 8, "ram_gb": 16, "model": "MacBook Air"},
44 + "m2m8a": {"cores": 8, "ram_gb": 8, "model": "Mac mini"},
45 + "m2m8b": {"cores": 8, "ram_gb": 8, "model": "Mac mini"},
46 + "m4md": {"cores": 10, "ram_gb": 16, "model": "Mac mini"}, # ajouté 2026-09-04 (Mac16,10, M4, 100.78.227.79)
47 +}
48 +
49 +# Rôles : gateway = jamais d'app ; worker = éligible. Un nœud absent du scan est « offline ».
50 +ROLES = {alias: ("gateway" if alias == GATEWAY else "worker") for alias in NODES}
51 +
52 +# Runtimes que `prepare` sait installer via Homebrew, et comment les détecter sur un nœud.
53 +RUNTIME_CHECKS = {
54 + "node": "test -x /opt/homebrew/bin/node",
55 + "pnpm": "test -x /opt/homebrew/bin/pnpm",
56 + "pm2": "test -x /opt/homebrew/bin/pm2",
57 + "ngrok": "(test -x /opt/homebrew/bin/ngrok || test -x $HOME/bin/ngrok) && test -f \"$HOME/Library/Application Support/ngrok/ngrok.yml\"",
58 + "python@3.14": "test -x /opt/homebrew/opt/python@3.14/bin/python3.14",
59 + "python@3.13": "test -x /opt/homebrew/opt/python@3.13/bin/python3.13",
60 + "python@3.12": "test -x /opt/homebrew/opt/python@3.12/bin/python3.12",
61 + "uv": "test -x /opt/homebrew/bin/uv || test -x $HOME/.local/bin/uv",
62 + "uv-python@3.12": "ls -d $HOME/.local/share/uv/python/cpython-3.12* >/dev/null 2>&1",
63 + "docker": "test -x /opt/homebrew/bin/docker",
64 + "postgresql@17": "test -x /opt/homebrew/opt/postgresql@17/bin/postgres",
65 + "brew": "test -x /opt/homebrew/bin/brew",
66 + "git": "test -x /opt/homebrew/bin/git || test -x /usr/bin/git",
67 +}
68 +BREW_FORMULAE = {
69 + "node": "node", "pnpm": "pnpm", "pm2": "pm2", "python@3.14": "python@3.14", "python@3.13": "python@3.13",
70 + "python@3.12": "python@3.12", "uv": "uv", "docker": "docker", "postgresql@17": "postgresql@17", "git": "git",
71 +}
72 +BREW_CASKS = {"ngrok": "ngrok"}
73 +
74 +os.makedirs(APPS_DIR, exist_ok=True)
75 +os.makedirs(STAGE_DIR, exist_ok=True)
76 +os.makedirs(LOG_DIR, exist_ok=True)
added mld/deploy.py +334 −0
@@ -0,0 +1,334 @@
1 +"""Cycle de vie d'une app : fetch (nœud -> staging), deploy (staging -> nœud), stop, start, retire, move, health."""
2 +import json
3 +import os
4 +import shlex
5 +import time
6 +from . import config, manifest, nodes, registry, render, ssh
7 +
8 +SYNC_BASE_EXCLUDES = [".DS_Store", "*.log", "__pycache__/", ".pytest_cache/", ".mypy_cache/"]
9 +
10 +
11 +class DeployError(Exception):
12 + pass
13 +
14 +
15 +def _ctx(alias, ip):
16 + lan = nodes.load_lan()["ips"]
17 + return {"HOME": "/Users/%s" % config.USER, "NODE": alias, "LAN_IP": ip, "ips": lan, "USER": config.USER}
18 +
19 +
20 +def _abs(p):
21 + return p.replace("~", "/Users/%s" % config.USER, 1) if p.startswith("~") else p
22 +
23 +
24 +def _log(app, msg):
25 + line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
26 + print(line, flush=True)
27 + with open(os.path.join(config.LOG_DIR, app + ".log"), "a") as f:
28 + f.write(time.strftime("%Y-%m-%d ") + line + "\n")
29 +
30 +
31 +def _ip(alias):
32 + ip = nodes.ip_of(alias)
33 + if not ip:
34 + raise DeployError("nœud %s injoignable sur le LAN" % alias)
35 + return ip
36 +
37 +
38 +def _ensure_dir(ip, d):
39 + """Crée le répertoire cible ; sous /opt (racine root) passe par sudo -S avec le mot de passe de ~/dispatch/.sudo."""
40 + if ssh.ok(ip, "mkdir -p %s 2>/dev/null && test -w %s" % (shlex.quote(d), shlex.quote(d))):
41 + return
42 + pw = open(config.SUDO_PW_FILE).read().strip() if os.path.exists(config.SUDO_PW_FILE) else None
43 + if not pw:
44 + raise DeployError("impossible de créer %s (sudo requis, %s absent)" % (d, config.SUDO_PW_FILE))
45 + rc, out, err = ssh.run(ip, "echo %s | sudo -S -p '' mkdir -p %s && echo %s | sudo -S -p '' chown %s %s" % (shlex.quote(pw), shlex.quote(d), shlex.quote(pw), config.USER, shlex.quote(d)))
46 + if rc != 0:
47 + raise DeployError("création de %s impossible : %s" % (d, err.strip()[-200:]))
48 +
49 +
50 +def stage_dir(app):
51 + return os.path.join(config.STAGE_DIR, app)
52 +
53 +
54 +# ---------------------------------------------------------------- fetch : nœud -> staging
55 +def fetch(app, from_alias, final=False):
56 + m = manifest.load(app)
57 + ip = _ip(from_alias)
58 + _log(app, "fetch %s depuis %s (%s)%s" % (m["dir"], from_alias, ip, " [final]" if final else ""))
59 + excl = SYNC_BASE_EXCLUDES + list(m.get("sync_excludes") or [])
60 + rc, out, err = ssh.rsync_pull(ip, _abs(m["dir"]), os.path.join(stage_dir(app), "dir"), excludes=excl)
61 + if rc not in (0, 24):
62 + raise DeployError("rsync fetch a échoué (%s): %s" % (rc, err.strip()[-300:]))
63 + for extra in m.get("extra_paths") or []:
64 + dest = os.path.join(stage_dir(app), "extra", extra.strip("~/").replace("/", "__"))
65 + rc2, out2, err2 = ssh.run(ip, "test -e %s" % shlex.quote(_abs(extra)))
66 + if rc2 != 0:
67 + _log(app, " extra absent sur la source, ignoré : %s" % extra)
68 + continue
69 + isdir = ssh.ok(ip, "test -d %s" % shlex.quote(_abs(extra)))
70 + if isdir:
71 + rc2, out2, err2 = ssh.rsync_pull(ip, _abs(extra), dest, delete=True)
72 + else:
73 + os.makedirs(dest, exist_ok=True)
74 + import subprocess
75 + subprocess.run(["scp", "-p"] + ssh.SSH_OPTS + ["%s:%s" % (ssh.target(ip), _abs(extra)), os.path.join(dest, os.path.basename(extra))], capture_output=True)
76 + json.dump({"app": app, "from": from_alias, "ip": ip, "ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "final": final},
77 + open(os.path.join(stage_dir(app), "fetch.json"), "w"), indent=2)
78 + stat = [l for l in out.splitlines() if "Total transferred file size" in l or "Number of regular files transferred" in l or "Number of files transferred" in l or "Total bytes sent" in l]
79 + _log(app, " ok — " + " ; ".join(s.strip() for s in stat))
80 + return True
81 +
82 +
83 +# ---------------------------------------------------------------- stop / start / status sur un nœud
84 +def stop(app, alias, keep_dir=True):
85 + m = manifest.load(app)
86 + ip = _ip(alias)
87 + _log(app, "stop sur %s" % alias)
88 + names = manifest.pm2_names(m)
89 + script = ""
90 + if names:
91 + script += "for n in %s; do pm2 delete $n >/dev/null 2>&1 && echo \" pm2 delete $n\"; done; pm2 save --force >/dev/null 2>&1\n" % " ".join(names)
92 + for lbl in manifest.launchd_labels(m):
93 + script += "launchctl bootout gui/$(id -u)/%s >/dev/null 2>&1 && echo ' bootout %s'; rm -f ~/Library/LaunchAgents/%s.plist\n" % (lbl, lbl, lbl)
94 + if m.get("ngrok"):
95 + script += "pkill -f -- 'url=%s' 2>/dev/null; pkill -f -- 'url=https://%s' 2>/dev/null; pkill -f -- 'domain=%s' 2>/dev/null; true\n" % (m["ngrok"]["url"], m["ngrok"]["url"], m["ngrok"]["url"])
96 + for hook in (m.get("hooks") or {}).get("pre_stop") or []:
97 + script += hook + "\n"
98 + # tue ce qui écoute encore sur les ports de l'app (orphelins)
99 + for p in m["requires"].get("ports") or []:
100 + script += "sleep 1; for pid in $(lsof -nP -iTCP:%s -sTCP:LISTEN -t 2>/dev/null); do kill $pid 2>/dev/null && echo ' kill port %s pid '$pid; done\n" % (p, p)
101 + if m.get("ngrok"):
102 + script += "sleep 4\n" # laisse ngrok libérer le domaine réservé côté service
103 + rc, out, err = ssh.run(ip, script, timeout=180)
104 + for l in out.splitlines():
105 + _log(app, l)
106 + registry.log(app, "stopped", node=alias)
107 + return rc == 0
108 +
109 +
110 +def _install_hooks(m, ip, ctx):
111 + """Commandes post-sync : réécritures .env, hooks, inscription pousseur."""
112 + script = ""
113 + d = _abs(m["dir"])
114 + for fname, kv in (m.get("env_overrides") or {}).items():
115 + target = os.path.join(d, fname)
116 + kv = manifest.render(kv, ctx)
117 + py = "import re,sys,os\np=%r\nkv=%r\ns=open(p).read() if os.path.exists(p) else ''\nfor k,v in kv.items():\n line=k+'='+v\n if re.search(r'^'+re.escape(k)+r'=.*$', s, re.M): s=re.sub(r'^'+re.escape(k)+r'=.*$', line.replace('\\\\','\\\\\\\\'), s, flags=re.M)\n else: s=s.rstrip('\\n')+'\\n'+line+'\\n'\nopen(p,'w').write(s)\nprint(' .env réécrit: '+', '.join(kv))" % (target, kv)
118 + script += "python3 - <<'PYEOF'\n%s\nPYEOF\n" % py
119 + if m.get("ka_repo"):
120 + script += "grep -qx %s ~/.ka-pousseur-repos 2>/dev/null || echo %s >> ~/.ka-pousseur-repos; echo ' pousseur: %s inscrit'\n" % (shlex.quote(m["dir"]), shlex.quote(m["dir"]), m["dir"])
121 + for hook in (m.get("hooks") or {}).get("post_sync") or []:
122 + script += "cd %s && (%s)\n" % (shlex.quote(d), manifest.render(hook, ctx))
123 + return script
124 +
125 +
126 +def deploy(app, alias, reinstall=False, skip_sync=False):
127 + """staging -> nœud, puis démarrage (PM2 + launchd) et healthcheck."""
128 + m = manifest.load(app)
129 + ip = _ip(alias)
130 + ctx = _ctx(alias, ip)
131 + sd = os.path.join(stage_dir(app), "dir")
132 + if not skip_sync and not os.path.isdir(sd):
133 + raise DeployError("staging vide pour %s — faire `mld fetch %s --from <nœud>` ou `mld stage`" % (app, app))
134 + _log(app, "deploy → %s (%s)" % (alias, ip))
135 + # 1. runtimes
136 + scan = nodes.load_scan().get(alias) or nodes.probe(alias, ip)
137 + missing = [r for r in m["requires"]["runtimes"] if not scan.get("runtimes", {}).get(r)]
138 + if missing:
139 + raise DeployError("runtimes manquants sur %s : %s (→ `mld prepare %s`)" % (alias, ", ".join(missing), alias))
140 + # 2. sync
141 + d = _abs(m["dir"])
142 + if not skip_sync:
143 + _ensure_dir(ip, d)
144 + excl = SYNC_BASE_EXCLUDES + list(m.get("sync_excludes") or [])
145 + rc, out, err = ssh.rsync_push(sd, ip, d, excludes=excl, delete=True)
146 + if rc not in (0, 24):
147 + raise DeployError("rsync deploy a échoué (%s): %s" % (rc, err.strip()[-300:]))
148 + stat = [l.strip() for l in out.splitlines() if "Total transferred file size" in l or "Total bytes sent" in l]
149 + _log(app, " sync ok — %s" % (stat[0] if stat else ""))
150 + for extra in m.get("extra_paths") or []:
151 + src = os.path.join(stage_dir(app), "extra", extra.strip("~/").replace("/", "__"))
152 + if not os.path.exists(src):
153 + continue
154 + if os.path.isdir(src) and not os.path.exists(os.path.join(src, os.path.basename(extra))):
155 + ssh.rsync_push(src, ip, _abs(extra), delete=False)
156 + else:
157 + import subprocess
158 + ssh.run(ip, "mkdir -p %s" % shlex.quote(os.path.dirname(_abs(extra))))
159 + subprocess.run(["scp", "-p"] + ssh.SSH_OPTS + [os.path.join(src, os.path.basename(extra)), "%s:%s" % (ssh.target(ip), _abs(extra))], capture_output=True)
160 + _log(app, " extra: %s" % extra)
161 + # 3. install (optionnel) + hooks
162 + script = "cd %s || exit 3\n" % shlex.quote(d)
163 + if reinstall:
164 + script += "if [ -f pnpm-lock.yaml ]; then pnpm install --frozen-lockfile; elif [ -f package-lock.json ]; then npm ci; elif [ -f package.json ]; then npm install; fi\n"
165 + script += "if [ -f requirements.txt ] && [ -x .venv/bin/pip ]; then .venv/bin/pip install -q -r requirements.txt; fi\n"
166 + elif os.path.exists(os.path.join(sd, "package.json")) and os.path.isdir(os.path.join(sd, "node_modules")):
167 + script += "npm rebuild >/dev/null 2>&1 && echo ' npm rebuild ok' || echo ' npm rebuild: avertissement'\n"
168 + script += _install_hooks(m, ip, ctx)
169 + rc, out, err = ssh.run(ip, script, timeout=1800)
170 + for l in out.splitlines()[-15:]:
171 + _log(app, l)
172 + if rc != 0:
173 + raise DeployError("post-sync a échoué (%s): %s" % (rc, (err or out).strip()[-400:]))
174 + # 4. lancement
175 + start(app, alias, m=m, ctx=ctx, ip=ip)
176 + # 5. santé
177 + healthy, detail = health(app, alias, ip=ip, m=m)
178 + registry.set_app(app, node=alias, ip=ip, port=m.get("port"), domain=m.get("domain"), dir=m["dir"], label=m.get("label") or app,
179 + processes=manifest.pm2_names(m), launchd=manifest.launchd_labels(m), deployed=time.strftime("%Y-%m-%d %H:%M"),
180 + status="online" if healthy else "unhealthy", health=detail)
181 + registry.log(app, "deployed", node=alias, healthy=healthy)
182 + _log(app, " santé : %s" % detail)
183 + if not healthy:
184 + names = manifest.pm2_names(m)
185 + rc, out, _ = ssh.run(ip, "pm2 logs %s --nostream --lines 6 2>/dev/null | grep -v '^\\[TAILING' | tail -12" % " ".join(names), timeout=40)
186 + for l in out.splitlines():
187 + _log(app, " log: " + l[:160])
188 + raise DeployError("healthcheck KO sur %s : %s" % (alias, detail))
189 + return True
190 +
191 +
192 +def start(app, alias, m=None, ctx=None, ip=None):
193 + m = m or manifest.load(app)
194 + ip = ip or _ip(alias)
195 + ctx = ctx or _ctx(alias, ip)
196 + _log(app, "start sur %s" % alias)
197 + run_dir = _abs(config.RUN_DIR)
198 + eco = render.ecosystem(m, ctx)
199 + eco_path = "%s/%s.config.cjs" % (run_dir, app)
200 + ssh.write_remote_file(ip, eco_path, eco, mode="600")
201 + script = "mkdir -p %s\n" % run_dir
202 + names = manifest.pm2_names(m)
203 + if names:
204 + script += "for n in %s; do pm2 delete $n >/dev/null 2>&1; done\n" % " ".join(names)
205 + script += "pm2 start %s --update-env 2>&1 | grep -E 'error|Error|\\[PM2\\]' | head -5; pm2 save --force >/dev/null 2>&1; pm2 jlist | python3 -c \"import json,sys; L=json.load(sys.stdin); print(' pm2: '+', '.join(p['name']+'='+p['pm2_env']['status'] for p in L if p['name'] in %s))\" 2>/dev/null || pm2 ls | grep -E '%s'\n" % (shlex.quote(eco_path), json.dumps(names), "|".join(names))
206 + for item in m.get("launchd") or []:
207 + pl = render.plist(item, ctx)
208 + ppath = "%s/Library/LaunchAgents/%s.plist" % (ctx["HOME"], item["label"])
209 + ssh.write_remote_file(ip, ppath, pl, mode="644")
210 + script += "launchctl bootout gui/$(id -u)/%s >/dev/null 2>&1; launchctl bootstrap gui/$(id -u) %s && echo ' launchd: %s chargé' || echo ' launchd: %s ÉCHEC'\n" % (item["label"], shlex.quote(ppath), item["label"], item["label"])
211 + for hook in (m.get("hooks") or {}).get("post_start") or []:
212 + script += "cd %s && (%s)\n" % (shlex.quote(_abs(m["dir"])), manifest.render(hook, ctx))
213 + rc, out, err = ssh.run(ip, script, timeout=300)
214 + for l in out.splitlines():
215 + _log(app, l)
216 + if err.strip():
217 + _log(app, " stderr: " + err.strip()[-200:])
218 + return rc == 0
219 +
220 +
221 +def health(app, alias, ip=None, m=None, timeout=90):
222 + m = m or manifest.load(app)
223 + ip = ip or _ip(alias)
224 + if not m.get("port"):
225 + # pas de port HTTP : vérifie que les processus PM2 sont online
226 + names = manifest.pm2_names(m)
227 + if not names:
228 + return True, "aucun port/processus à vérifier"
229 + rc, out, _ = ssh.run(ip, "pm2 jlist | python3 -c \"import json,sys; L=json.load(sys.stdin); print(sum(1 for p in L if p['name'] in %s and p['pm2_env']['status']=='online'))\"" % json.dumps(names))
230 + n = int((out.strip() or "0").splitlines()[-1] or 0)
231 + return n == len(names), "%d/%d processus PM2 online" % (n, len(names))
232 + url = "http://127.0.0.1:%s%s" % (m["port"], m.get("health_path") or "/")
233 + deadline = time.time() + timeout
234 + code = "000"
235 + while time.time() < deadline:
236 + rc, out, _ = ssh.run(ip, "curl -s -o /dev/null -m 8 -w '%%{http_code}' %s" % shlex.quote(url), timeout=20)
237 + code = out.strip()[-3:] if out.strip() else "000"
238 + if code.isdigit() and code not in ("000",) and int(code) < 500:
239 + break
240 + time.sleep(4)
241 + local_ok = code.isdigit() and code != "000" and int(code) < 500
242 + pub = ""
243 + if m.get("domain") and local_ok and m.get("ngrok"):
244 + import subprocess
245 + for _ in range(20):
246 + p = subprocess.run(["curl", "-s", "-o", "/dev/null", "-m", "12", "-w", "%{http_code}", "https://%s%s" % (m["domain"], m.get("health_path") or "/")], capture_output=True, text=True)
247 + pub = p.stdout.strip()
248 + if pub.isdigit() and int(pub) < 500 and pub != "404":
249 + break
250 + time.sleep(5)
251 + detail = "local %s" % code + (" / public https://%s %s" % (m["domain"], pub) if pub else "")
252 + ok = local_ok and (not pub or (pub.isdigit() and int(pub) < 500 and pub != "404"))
253 + return ok, detail
254 +
255 +
256 +def retire(app, alias, remove_dir=True):
257 + """Arrête et efface la copie d'un nœud (après migration réussie)."""
258 + m = manifest.load(app)
259 + ip = _ip(alias)
260 + stop(app, alias)
261 + if remove_dir:
262 + d = _abs(m["dir"])
263 + rc, out, err = ssh.run(ip, "rm -rf %s && echo ' rm -rf %s'; rm -f %s/%s.config.cjs; grep -v -x %s ~/.ka-pousseur-repos > ~/.ka-pousseur-repos.tmp 2>/dev/null && mv ~/.ka-pousseur-repos.tmp ~/.ka-pousseur-repos; true" % (shlex.quote(d), d, _abs(config.RUN_DIR), app, shlex.quote(m["dir"])), timeout=600)
264 + for l in out.splitlines():
265 + _log(app, l)
266 + registry.log(app, "retired", node=alias)
267 + return True
268 +
269 +
270 +def move(app, to_alias, from_alias=None, keep_source=False, reinstall=False):
271 + """Migration complète : pré-sync à chaud, arrêt, sync final, déploiement, santé, retrait de la source (ou rollback)."""
272 + from_alias = from_alias or registry.node_of(app) or manifest.load(app).get("source_node")
273 + if not from_alias:
274 + raise DeployError("nœud source inconnu pour %s (préciser --from)" % app)
275 + if from_alias == to_alias:
276 + _log(app, "déjà sur %s → redéploiement en place" % to_alias)
277 + fetch(app, from_alias)
278 + stop(app, from_alias)
279 + fetch(app, from_alias, final=True)
280 + return deploy(app, to_alias, reinstall=reinstall)
281 + _log(app, "MOVE %s : %s%s" % (app, from_alias, to_alias))
282 + fetch(app, from_alias) # 1. copie à chaud (gros volume)
283 + stop(app, from_alias) # 2. arrêt source
284 + fetch(app, from_alias, final=True) # 3. delta final (cohérence des bases)
285 + try:
286 + deploy(app, to_alias, reinstall=reinstall) # 4. cible
287 + except DeployError as e:
288 + _log(app, "ÉCHEC sur %s : %s — rollback sur %s" % (to_alias, e, from_alias))
289 + try:
290 + stop(app, to_alias)
291 + except Exception:
292 + pass
293 + start(app, from_alias)
294 + ok, det = health(app, from_alias)
295 + registry.set_app(app, node=from_alias, status="online" if ok else "unhealthy", health=det)
296 + raise
297 + if not keep_source:
298 + retire(app, from_alias) # 5. nettoyage source
299 + registry.log(app, "moved", **{"from": from_alias, "to": to_alias})
300 + _log(app, "MOVE terminé : %s tourne sur %s" % (app, to_alias))
301 + return True
302 +
303 +
304 +def status(app=None):
305 + r = registry.load()
306 + apps = [app] if app else sorted(r["apps"])
307 + rows = []
308 + for a in apps:
309 + v = r["apps"].get(a)
310 + if not v:
311 + rows.append((a, "—", "", "", "non déployée", ""))
312 + continue
313 + rows.append((a, v.get("node"), v.get("port"), v.get("domain") or "", v.get("status"), v.get("deployed")))
314 + return rows
315 +
316 +
317 +def live_status(app=None, public=True):
318 + """Vérifie en live (PM2 + port + domaine) chaque app du registre."""
319 + r = registry.load()
320 + apps = [app] if app else sorted(r["apps"])
321 + out = []
322 + for a in apps:
323 + v = r["apps"].get(a)
324 + if not v:
325 + continue
326 + m = manifest.load(a)
327 + try:
328 + ip = _ip(v["node"])
329 + ok, det = health(a, v["node"], ip=ip, m=m, timeout=10) if public else (None, "")
330 + except DeployError as e:
331 + ok, det = False, str(e)
332 + registry.set_app(a, status="online" if ok else "unhealthy", health=det)
333 + out.append((a, v["node"], v.get("domain") or "", "OK" if ok else "KO", det))
334 + return out
added mld/importer.py +81 −0
@@ -0,0 +1,81 @@
1 +"""`mld import <nœud>` : brouillons de manifestes à partir des processus PM2 / launchd vivants d'un nœud.
2 +
3 +Regroupe par préfixe de nom (lou-ka-web, lou-ka-sync, lou-ka-ngrok → lou-ka). Les brouillons vont dans
4 +~/dispatch/apps/_drafts/<app>.json ; on les relit, on ajuste, puis on les déplace dans ~/dispatch/apps/.
5 +"""
6 +import json
7 +import os
8 +import re
9 +from . import config, manifest, nodes, ssh
10 +
11 +SUFFIXES = ("-web", "-sync", "-ngrok", "-api", "-tunnel", "-server", "-etl", "-scheduler", "-ingest", "-market",
12 + "-worker", "-proxy", "-crawler", "-enrichment", "-sitemaps", "-tun-search", "-tun-embed")
13 +ENV_DROP_PREFIX = ("PM2_", "npm_", "__", "TERM", "SSH_", "LC_", "HOMEBREW", "ITERM", "XPC_", "COLOR", "LS_", "SHELL", "PWD", "OLDPWD",
14 + "PATH", "HOME", "USER", "LOGNAME", "LANG", "TMPDIR", "SECURITYSESSIONID", "COMMAND_MODE", "unique_id", "FPATH",
15 + "CLAUDE", "OTEL", "DISPLAY", "MallocNanoZone", "ZSH", "EDITOR", "PAGER", "LESS", "NoDefaultCurrentDirectoryInExePath",
16 + "COREPACK", "AI_AGENT", "OSLogRateLimit", "GIT_", "ANTHROPIC_MODEL", "DISABLE_", "MANPATH", "INFOPATH", "STARSHIP",
17 + "TERMINFO", "WINDOWID", "NVM", "P9K", "TMUX", "SSH", "_")
18 +
19 +
20 +def app_of(name):
21 + for s in SUFFIXES:
22 + if name.endswith(s):
23 + return name[: -len(s)]
24 + return name
25 +
26 +
27 +def clean_env(env, name):
28 + out = {}
29 + for k, v in (env or {}).items():
30 + if k == name or k.startswith(ENV_DROP_PREFIX) or not isinstance(v, str):
31 + continue
32 + out[k] = v
33 + return out
34 +
35 +
36 +def import_node(alias, apps_filter=None):
37 + ip = nodes.ip_of(alias)
38 + rc, out, err = ssh.run(ip, "pm2 jlist 2>/dev/null; echo; echo __MEM__; pm2 jlist 2>/dev/null | python3 -c \"import json,sys; print(json.dumps({p['name']: p.get('monit',{}).get('memory',0) for p in json.load(sys.stdin)}))\"", timeout=60)
39 + raw, _, mem = out.partition("__MEM__")
40 + L = json.loads(raw.strip() or "[]")
41 + mems = json.loads(mem.strip() or "{}")
42 + groups = {}
43 + for p in L:
44 + e = p.get("pm2_env", {})
45 + app = app_of(p["name"])
46 + if apps_filter and app not in apps_filter:
47 + continue
48 + g = groups.setdefault(app, {"app": app, "processes": [], "ngrok": None, "cwd": None, "ram": 0})
49 + g["ram"] += int(mems.get(p["name"], 0)) // (1024 * 1024)
50 + args = e.get("args") or []
51 + script = e.get("pm_exec_path") or ""
52 + if "ngrok" in script or (args and "ngrok" in " ".join(map(str, args))):
53 + blob = " ".join([script] + [str(a) for a in args])
54 + mo = re.search(r"--(?:url|domain)=(?:https?://)?([^\s]+)", blob)
55 + port = re.search(r"\s(\d{2,5})(?:\s|$)", blob)
56 + g["ngrok"] = {"name": p["name"], "url": mo.group(1) if mo else None, "port": int(port.group(1)) if port else None}
57 + continue
58 + g["processes"].append({
59 + "name": p["name"], "manager": "pm2", "script": script, "args": args,
60 + "interpreter": e.get("exec_interpreter") if e.get("exec_interpreter") not in (None, "none", "N/A") else None,
61 + "cwd": e.get("pm_cwd"), "env": clean_env(e.get("env"), p["name"]), "cron_restart": e.get("cron_restart"),
62 + "autorestart": e.get("autorestart", True), "max_memory_restart": e.get("max_memory_restart"),
63 + })
64 + g["cwd"] = g["cwd"] or e.get("pm_cwd")
65 + drafts = os.path.join(config.APPS_DIR, "_drafts")
66 + os.makedirs(drafts, exist_ok=True)
67 + written = []
68 + for app, g in groups.items():
69 + cwd = g["cwd"] or ""
70 + m = json.loads(json.dumps(manifest.DEFAULTS))
71 + mo = re.match(r"(/Users/[^/]+/(?:apps/)?[^/]+|/opt/[^/]+)", cwd)
72 + d = mo.group(1) if mo else cwd
73 + m.update({"app": app, "label": app, "dir": d.replace("/Users/%s" % config.USER, "~"), "processes": g["processes"], "ngrok": g["ngrok"],
74 + "domain": (g["ngrok"] or {}).get("url"), "port": (g["ngrok"] or {}).get("port"), "ram_mb_observed": max(g["ram"], 128)})
75 + m["requires"]["ports"] = [m["port"]] if m["port"] else []
76 + m["placement"]["reason"] = "importé de %s" % alias
77 + manifest.detect_requirements(m)
78 + p = os.path.join(drafts, app + ".json")
79 + json.dump(m, open(p, "w"), indent=2, ensure_ascii=False)
80 + written.append(p)
81 + return written
added mld/manifest.py +112 −0
@@ -0,0 +1,112 @@
1 +"""Manifestes d'apps : ~/dispatch/apps/<app>.json (source de vérité de « comment » tourne une app).
2 +
3 +Schéma :
4 +{
5 + "app": "lou-ka", "label": "Lou-Ka", "domain": "www.lou-ka.com", "port": 8095, "health_path": "/",
6 + "dir": "~/apps/lou-ka", # répertoire de l'app (même chemin absolu sur tous les nœuds)
7 + "extra_paths": ["~/.ssh/trouveka_tunnel"], # autres fichiers/dossiers à copier avec l'app
8 + "sync_excludes": ["data/backups/"], # exclusions rsync (relatives à dir)
9 + "requires": {"runtimes": ["python@3.14", "pm2", "ngrok"], "ram_gb": 2, "ports": [8095]},
10 + "ram_mb_observed": 400, "size_mb": 4359,
11 + "placement": {"pin": null, "prefer": null, "avoid": [], "reason": ""},
12 + "processes": [{"name": "lou-ka-web", "manager": "pm2", "script": "...", "args": [...], "interpreter": null,
13 + "cwd": "...", "env": {...}, "cron_restart": null, "autorestart": true, "max_memory_restart": null}],
14 + "ngrok": {"name": "lou-ka-ngrok", "url": "www.lou-ka.com", "port": 8095} | null,
15 + "launchd": [{"label": "...", "program_arguments": [...], "working_directory": "...", "env": {...}, "keep_alive": true}],
16 + "env_overrides": {".env": {"DATABASE_URL": "postgresql://...@{{IP:M2M32}}:5432/x"}}, # réécritures de fichiers .env
17 + "hooks": {"post_sync": ["bash ..."], "post_start": []},
18 + "ka_repo": true # inscrire dir dans ~/.ka-pousseur-repos du nœud hôte
19 +}
20 +Gabarits acceptés dans env/args/env_overrides : {{HOME}}, {{NODE}}, {{LAN_IP}}, {{IP:<alias>}}.
21 +"""
22 +import glob
23 +import json
24 +import os
25 +import re
26 +from . import config
27 +
28 +DEFAULTS = {
29 + "label": None, "domain": None, "port": None, "health_path": "/", "extra_paths": [], "sync_excludes": [],
30 + "requires": {"runtimes": ["pm2"], "ram_gb": 1, "ports": []}, "ram_mb_observed": 512, "size_mb": 0,
31 + "placement": {"pin": None, "prefer": None, "avoid": [], "reason": ""}, "processes": [], "ngrok": None,
32 + "launchd": [], "env_overrides": {}, "hooks": {"post_sync": [], "post_start": []}, "ka_repo": False,
33 +}
34 +
35 +
36 +def path(app):
37 + return os.path.join(config.APPS_DIR, app + ".json")
38 +
39 +
40 +def load(app):
41 + p = path(app)
42 + if not os.path.exists(p):
43 + raise SystemExit("manifeste introuvable : %s" % p)
44 + m = json.load(open(p))
45 + for k, v in DEFAULTS.items():
46 + m.setdefault(k, json.loads(json.dumps(v)))
47 + m["requires"].setdefault("runtimes", ["pm2"])
48 + m["requires"].setdefault("ram_gb", 1)
49 + m["requires"].setdefault("ports", [m["port"]] if m.get("port") else [])
50 + return m
51 +
52 +
53 +def save(m):
54 + json.dump(m, open(path(m["app"]), "w"), indent=2, ensure_ascii=False)
55 + os.chmod(path(m["app"]), 0o600)
56 +
57 +
58 +def all_apps():
59 + return sorted(os.path.basename(p)[:-5] for p in glob.glob(os.path.join(config.APPS_DIR, "*.json")))
60 +
61 +
62 +def load_all():
63 + return {a: load(a) for a in all_apps()}
64 +
65 +
66 +def render(value, ctx):
67 + """Remplace {{HOME}}, {{NODE}}, {{LAN_IP}}, {{IP:alias}} dans une chaîne / liste / dict."""
68 + if isinstance(value, str):
69 + def sub(mo):
70 + key = mo.group(1)
71 + if key.startswith("IP:"):
72 + return ctx.get("ips", {}).get(key[3:], mo.group(0))
73 + return str(ctx.get(key, mo.group(0)))
74 + return re.sub(r"\{\{([A-Za-z0-9_:@.-]+)\}\}", sub, value)
75 + if isinstance(value, list):
76 + return [render(v, ctx) for v in value]
77 + if isinstance(value, dict):
78 + return {k: render(v, ctx) for k, v in value.items()}
79 + return value
80 +
81 +
82 +def pm2_names(m):
83 + names = [p["name"] for p in m["processes"] if p.get("manager", "pm2") == "pm2"]
84 + if m.get("ngrok"):
85 + names.append(m["ngrok"]["name"])
86 + return names
87 +
88 +
89 +def launchd_labels(m):
90 + return [l["label"] for l in m.get("launchd", [])]
91 +
92 +
93 +def detect_requirements(m):
94 + """Déduit les runtimes requis à partir des processus (complète requires.runtimes)."""
95 + req = set(m["requires"].get("runtimes") or [])
96 + req.add("pm2")
97 + if m.get("ngrok"):
98 + req.add("ngrok")
99 + blob = json.dumps(m["processes"]) + json.dumps(m.get("launchd", []))
100 + if "/opt/homebrew/bin/node" in blob or "next" in blob or "npm" in blob or ".mjs" in blob or ".js" in blob:
101 + req.add("node")
102 + if "pnpm" in blob:
103 + req.add("pnpm")
104 + vp = m.get("venv_python") or ""
105 + for ver in ("3.14", "3.13", "3.12"):
106 + if "python@%s" % ver in vp or "/%s/" % ver in vp or "python%s" % ver in vp:
107 + req.add("python@%s" % ver)
108 + if "uv/python/cpython-3.12" in vp:
109 + req.discard("python@3.12")
110 + req.add("uv-python@3.12")
111 + m["requires"]["runtimes"] = sorted(req)
112 + return m
added mld/nodes.py +155 −0
@@ -0,0 +1,155 @@
1 +"""Découverte LAN des nœuds + sonde de ressources/runtimes (scan)."""
2 +import json
3 +import os
4 +import time
5 +from concurrent.futures import ThreadPoolExecutor
6 +from . import config, ssh
7 +
8 +RUNTIME_PROBE = "\n".join('if %s; then echo "RT %s 1"; else echo "RT %s 0"; fi' % (cmd, name, name) for name, cmd in config.RUNTIME_CHECKS.items())
9 +
10 +
11 +def _probe_script():
12 + # sonde 100 % shell (les minis sans Xcode CLT n'ont pas de python3 fonctionnel) : lignes "KV clé valeur"
13 + sh = r'''
14 +export LC_ALL=C
15 +echo "KV host $(hostname -s)"
16 +echo "KV marker $(cat ~/.maclustr-node 2>/dev/null)"
17 +echo "KV ip $(ifconfig | awk '/inet 192\.168\.2\./{print $2; exit}')"
18 +echo "KV os $(sw_vers -productVersion)"
19 +echo "KV cores $(sysctl -n hw.ncpu)"
20 +echo "KV ram_gb $(( $(sysctl -n hw.memsize) / 1073741824 ))"
21 +echo "KV load1 $(sysctl -n vm.loadavg | tr -d '{}' | awk '{print $1}' | tr ',' '.')"
22 +echo "KV free_gb $(vm_stat | awk '/Pages (free|inactive|speculative|purgeable)/{gsub("\\.","",$NF); s+=$NF} END{printf "%.1f", s*16384/1073741824}')"
23 +echo "KV disk_free_gb $(df -g / | awk 'NR==2{print $4}')"
24 +echo "KV boot $(sysctl -n kern.boottime | sed -E 's/.*sec = ([0-9]+).*/\1/')"
25 +echo "KV ports $(lsof -nP -iTCP -sTCP:LISTEN 2>/dev/null | awk 'NR>1{n=split($9,a,":"); print a[n]}' | sort -un | tr '\n' ',')"
26 +echo "KV pm2 $(pm2 jlist 2>/dev/null | tr -d '\n' | grep -o '"name":"[^"]*"' | cut -d'"' -f4 | sort -u | tr '\n' ',')"
27 +'''
28 + return sh + "\n" + RUNTIME_PROBE + "\n"
29 +
30 +
31 +def load_lan():
32 + if os.path.exists(config.LAN_CACHE):
33 + return json.load(open(config.LAN_CACHE))
34 + return {"updated": None, "ips": {}}
35 +
36 +
37 +def save_lan(d):
38 + d["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
39 + json.dump(d, open(config.LAN_CACHE, "w"), indent=2)
40 +
41 +
42 +def _identify(ip):
43 + rc, out, _ = ssh.run(ip, "cat ~/.maclustr-node 2>/dev/null || hostname -s", timeout=12, path=False)
44 + if rc == 0 and out.strip():
45 + return ip, out.strip().splitlines()[0].strip()
46 + return ip, None
47 +
48 +
49 +def discover(rng=range(1, 255), verbose=True):
50 + """Balaye 192.168.2.0/24 en SSH (clé passerelle) et lit le marqueur ~/.maclustr-node."""
51 + ips = [config.LAN_PREFIX + str(i) for i in rng]
52 + found = {}
53 + with ThreadPoolExecutor(max_workers=48) as ex:
54 + for ip, alias in ex.map(_identify, ips):
55 + if alias:
56 + # normalise l'alias (le marqueur est la source de vérité, sinon hostname)
57 + for a in config.NODES:
58 + if a.lower() == alias.lower():
59 + alias = a
60 + break
61 + found[alias] = ip
62 + lan = load_lan()
63 + lan["ips"] = found
64 + save_lan(lan)
65 + if verbose:
66 + for a in sorted(found, key=lambda x: list(config.NODES).index(x) if x in config.NODES else 99):
67 + print(" %-8s %s" % (a, found[a]))
68 + missing = [a for a in config.NODES if a not in found]
69 + if missing:
70 + print(" hors LAN / éteints : %s" % ", ".join(missing))
71 + return found
72 +
73 +
74 +def ip_of(alias, rediscover=True):
75 + lan = load_lan()
76 + ip = lan["ips"].get(alias)
77 + if ip and ssh.ok(ip, "true", timeout=15):
78 + return ip
79 + if rediscover:
80 + found = discover(verbose=False)
81 + return found.get(alias)
82 + return ip
83 +
84 +
85 +def probe(alias, ip):
86 + rc, out, err = ssh.run(ip, _probe_script(), timeout=90)
87 + info = {"alias": alias, "ip": ip, "online": False}
88 + if "KV cores" not in out:
89 + info["error"] = (err or out).strip()[:200]
90 + return info
91 + rts = {}
92 + kv = {}
93 + for line in out.splitlines():
94 + if line.startswith("KV "):
95 + parts = line.split(" ", 2)
96 + kv[parts[1]] = parts[2].strip() if len(parts) > 2 else ""
97 + elif line.startswith("RT "):
98 + _, name, val = line.split(" ", 2)
99 + rts[name] = val.strip() == "1"
100 + def num(k, f=float, d=0):
101 + try:
102 + return f(kv.get(k, "") or d)
103 + except Exception:
104 + return d
105 + info.update({"host": kv.get("host", ""), "marker": kv.get("marker", ""), "os": kv.get("os", ""), "cores": num("cores", int),
106 + "ram_gb": num("ram_gb", int), "load1": num("load1"), "free_gb": num("free_gb"), "disk_free_gb": num("disk_free_gb", int)})
107 + if kv.get("ip"):
108 + info["ip"] = kv["ip"]
109 + try:
110 + info["uptime_h"] = round((time.time() - int(kv.get("boot") or 0)) / 3600, 1) if kv.get("boot") else None
111 + except Exception:
112 + info["uptime_h"] = None
113 + info["ports"] = sorted(set(int(p) for p in kv.get("ports", "").split(",") if p.isdigit()))
114 + info["pm2"] = [p for p in kv.get("pm2", "").split(",") if p]
115 + info["runtimes"] = rts
116 + info["online"] = info["cores"] > 0
117 + info.update({k: v for k, v in config.NODES.get(alias, {}).items() if k not in info or not info.get(k)})
118 + info["role"] = config.ROLES.get(alias, "worker")
119 + return info
120 +
121 +
122 +def scan(aliases=None, verbose=True):
123 + lan = load_lan()
124 + if not lan["ips"]:
125 + discover(verbose=False)
126 + lan = load_lan()
127 + targets = [(a, ip) for a, ip in lan["ips"].items() if not aliases or a in aliases]
128 + res = {}
129 + with ThreadPoolExecutor(max_workers=24) as ex:
130 + for info in ex.map(lambda t: probe(*t), targets):
131 + res[info["alias"]] = info
132 + for a in config.NODES:
133 + if a not in res:
134 + res[a] = {"alias": a, "online": False, "role": config.ROLES[a], "runtimes": {}, **config.NODES[a]}
135 + json.dump({"updated": time.strftime("%Y-%m-%dT%H:%M:%S"), "nodes": res}, open(config.SCAN_CACHE, "w"), indent=2)
136 + if verbose:
137 + print_scan(res)
138 + return res
139 +
140 +
141 +def load_scan():
142 + if os.path.exists(config.SCAN_CACHE):
143 + return json.load(open(config.SCAN_CACHE))["nodes"]
144 + return {}
145 +
146 +
147 +def print_scan(res):
148 + print("%-8s %-6s %-16s %5s %6s %6s %6s %7s %4s %s" % ("nœud", "rôle", "ip", "cœurs", "RAM", "libre", "load", "disque", "pm2", "runtimes"))
149 + for a in config.NODES:
150 + n = res.get(a, {})
151 + if not n.get("online"):
152 + print("%-8s %-6s %-16s — hors ligne" % (a, n.get("role", ""), n.get("ip", "")))
153 + continue
154 + rt = ",".join(k.replace("python@", "py").replace("uv-python@", "uvpy") for k, v in sorted(n["runtimes"].items()) if v)
155 + print("%-8s %-6s %-16s %5s %5sG %5sG %6.2f %6sG %4s %s" % (a, n["role"], n["ip"], n["cores"], n["ram_gb"], n["free_gb"], n["load1"], n["disk_free_gb"], len(n.get("pm2", [])), rt))
added mld/picker.py +96 −0
@@ -0,0 +1,96 @@
1 +"""Sélection du nœud le plus efficient pour une app (ou un lot d'apps : plan)."""
2 +from . import config, registry
3 +
4 +
5 +def eligible(node, m, projected, exclude=()):
6 + """Retourne (ok, raison)."""
7 + a = node["alias"]
8 + pin = m["placement"].get("pin")
9 + if node.get("role") != "worker" and not (pin == a):
10 + return False, "rôle %s" % node.get("role")
11 + if not node.get("online"):
12 + return False, "hors ligne"
13 + if a in exclude or a in (m["placement"].get("avoid") or []):
14 + return False, "exclu"
15 + if pin and pin != a:
16 + return False, "épinglé sur %s" % pin
17 + missing = [r for r in m["requires"]["runtimes"] if not node["runtimes"].get(r)]
18 + if missing:
19 + return False, "runtimes manquants : %s" % ",".join(missing)
20 + need = max(float(m["requires"].get("ram_gb") or 1), m.get("ram_mb_observed", 512) * 1.5 / 1024)
21 + free = node["free_gb"] - projected[a]["ram_gb"]
22 + if free - need < 2.0:
23 + return False, "RAM insuffisante (%.1f G libres, besoin %.1f G)" % (free, need)
24 + if node["disk_free_gb"] < (m.get("size_mb", 0) / 1024) * 1.5 + 20:
25 + return False, "disque insuffisant"
26 + busy = set(node.get("ports", [])) | projected[a]["ports"]
27 + if not _is_home(a, m):
28 + for p in m["requires"].get("ports") or []:
29 + if p in busy:
30 + return False, "port %s occupé" % p
31 + return True, ""
32 +
33 +
34 +def _is_home(alias, m):
35 + """Le nœud héberge déjà cette app (source d'import ou registre) : ses propres ports ne la bloquent pas."""
36 + return alias == m.get("source_node") or alias == registry.node_of(m["app"])
37 +
38 +
39 +STAY_BONUS = 0.05 # évite de déplacer des Go de données pour un gain de score marginal
40 +
41 +
42 +def score(node, m, projected):
43 + a = node["alias"]
44 + need = max(float(m["requires"].get("ram_gb") or 1), m.get("ram_mb_observed", 512) * 1.5 / 1024)
45 + free_after = node["free_gb"] - projected[a]["ram_gb"] - need
46 + ram_ratio = max(0.0, free_after / max(node["ram_gb"], 1))
47 + idle = max(0.0, 1.0 - (node["load1"] + 0.5 * projected[a]["apps"]) / max(node["cores"], 1))
48 + disk = min(1.0, node["disk_free_gb"] / 500.0)
49 + cores = min(1.0, node["cores"] / 32.0)
50 + hosted = len(registry.apps_on(a)) + projected[a]["apps"]
51 + s = 0.45 * ram_ratio + 0.25 * idle + 0.10 * disk + 0.15 * cores - 0.04 * hosted
52 + if m["placement"].get("prefer") == a:
53 + s += 0.15
54 + if _is_home(a, m):
55 + s += STAY_BONUS
56 + return round(s, 4)
57 +
58 +
59 +def rank(m, scan, projected=None, exclude=()):
60 + projected = projected or empty_projection(scan)
61 + rows = []
62 + for a, node in scan.items():
63 + ok, why = eligible(node, m, projected, exclude)
64 + rows.append({"node": a, "ok": ok, "why": why, "score": score(node, m, projected) if ok else -1})
65 + rows.sort(key=lambda r: (-r["ok"], -r["score"]))
66 + return rows
67 +
68 +
69 +def empty_projection(scan):
70 + return {a: {"ram_gb": 0.0, "apps": 0, "ports": set()} for a in scan}
71 +
72 +
73 +def plan(manifests, scan, exclude=()):
74 + """Placement glouton : apps les plus lourdes d'abord, projection des ressources consommées."""
75 + projected = empty_projection(scan)
76 + order = sorted(manifests.values(), key=lambda m: -(m.get("ram_mb_observed", 512) + m.get("size_mb", 0) / 20.0))
77 + out = []
78 + for m in order:
79 + rows = rank(m, scan, projected, exclude)
80 + best = rows[0] if rows and rows[0]["ok"] else None
81 + if best:
82 + a = best["node"]
83 + need = max(float(m["requires"].get("ram_gb") or 1), m.get("ram_mb_observed", 512) * 1.5 / 1024)
84 + projected[a]["ram_gb"] += need
85 + projected[a]["apps"] += 1
86 + projected[a]["ports"] |= set(m["requires"].get("ports") or [])
87 + pin = m["placement"].get("pin")
88 + blockers = []
89 + if not best:
90 + cand = [r for r in rows if not r["ok"] and (r["node"] == pin if pin else (scan[r["node"]].get("online") and scan[r["node"]].get("role") == "worker"))]
91 + cand.sort(key=lambda r: -(scan[r["node"]].get("ram_gb") or 0))
92 + blockers = [(r["node"], r["why"]) for r in cand][:8]
93 + out.append({"app": m["app"], "node": best["node"] if best else None, "score": best["score"] if best else None,
94 + "current": registry.node_of(m["app"]) or m.get("source_node"), "pin": pin, "reason": m["placement"].get("reason", ""),
95 + "size_mb": m.get("size_mb", 0), "alternatives": [(r["node"], r["score"]) for r in rows[1:4] if r["ok"]], "blockers": blockers})
96 + return out
added mld/prepare.py +129 −0
@@ -0,0 +1,129 @@
1 +"""`mld prepare <nœud>` : installe les runtimes manquants (Homebrew), copie la config ngrok, active pm2 au démarrage, pose le marqueur."""
2 +import os
3 +import shlex
4 +from . import config, nodes, ssh
5 +
6 +
7 +def _sudo_pw():
8 + if os.path.exists(config.SUDO_PW_FILE):
9 + return open(config.SUDO_PW_FILE).read().strip()
10 + return None
11 +
12 +
13 +def prepare(alias, runtimes=None, ka_helpers=False):
14 + ip = nodes.ip_of(alias)
15 + if not ip:
16 + raise SystemExit("nœud %s injoignable" % alias)
17 + info = nodes.probe(alias, ip)
18 + have = info.get("runtimes", {})
19 + want = runtimes or ["node", "pnpm", "pm2", "ngrok", "python@3.14", "python@3.13", "uv", "git"]
20 + print("prepare %s (%s) — présents : %s" % (alias, ip, ", ".join(k for k, v in have.items() if v)))
21 + if not have.get("brew"):
22 + print(" ✗ Homebrew absent : bootstrap requis (Xcode CLT). Lancer d'abord : mld bootstrap %s" % alias)
23 + return False
24 + missing = [r for r in want if not have.get(r)]
25 + script = ""
26 + for r in missing:
27 + if r in config.BREW_FORMULAE:
28 + script += "echo ' brew install %s'; brew install -q %s >/dev/null 2>&1 || echo ' ! échec %s'\n" % (r, config.BREW_FORMULAE[r], r)
29 + elif r in config.BREW_CASKS:
30 + script += "echo ' brew install --cask ngrok'; brew install -q --cask ngrok >/dev/null 2>&1 || brew install -q ngrok/ngrok/ngrok >/dev/null 2>&1 || echo ' ! échec ngrok'\n"
31 + # config ngrok (authtoken) depuis la passerelle
32 + if "ngrok" in missing or not have.get("ngrok"):
33 + if os.path.exists(config.NGROK_CFG):
34 + ssh.run(ip, "mkdir -p \"$HOME/Library/Application Support/ngrok\"")
35 + ssh.scp_to(ip, config.NGROK_CFG, "'/Users/%s/Library/Application Support/ngrok/ngrok.yml'" % config.USER)
36 + script += "echo ' ngrok.yml copié'\n"
37 + # marqueur
38 + script += "[ -f ~/.maclustr-node ] || echo %s > ~/.maclustr-node\n" % alias
39 + # pm2 startup (launchd) — nécessite sudo
40 + pw = _sudo_pw()
41 + script += "if ! ls ~/Library/LaunchAgents 2>/dev/null | grep -qi pm2; then cmd=$(pm2 startup launchd -u %s --hp /Users/%s 2>/dev/null | grep '^sudo' | head -1); if [ -n \"$cmd\" ]; then %s echo \" pm2 startup: $( ${cmd#sudo } >/dev/null 2>&1 && echo ok || echo 'à faire manuellement')\"; fi; fi\n" % (
42 + config.USER, config.USER, ("echo %s | sudo -S -v 2>/dev/null;" % shlex.quote(pw)) if pw else "")
43 + rc, out, err = ssh.run(ip, script, timeout=1800)
44 + if ka_helpers:
45 + out += ka_helpers_install(alias, ip)
46 + print(out)
47 + if err.strip():
48 + print(" stderr:", err.strip()[-300:])
49 + info = nodes.probe(alias, ip)
50 + still = [r for r in want if not info.get("runtimes", {}).get(r)]
51 + print(" résultat : %s" % ("tout est là" if not still else "manquent encore " + ", ".join(still)))
52 + return not still
53 +
54 +
55 +def bootstrap(alias):
56 + """Installe Xcode CLT + Homebrew sans GUI. Active un sudo NOPASSWD temporaire (retiré à la fin). Long (plusieurs Go)."""
57 + ip = nodes.ip_of(alias)
58 + pw = _sudo_pw()
59 + if not pw:
60 + raise SystemExit("mot de passe sudo requis dans %s" % config.SUDO_PW_FILE)
61 + script = r'''
62 +export LC_ALL=C
63 +echo %s | sudo -S -p '' sh -c 'echo "%s ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/mld-bootstrap && chmod 440 /etc/sudoers.d/mld-bootstrap'
64 +trap 'sudo rm -f /etc/sudoers.d/mld-bootstrap' EXIT
65 +if ! xcode-select -p >/dev/null 2>&1; then
66 + echo ' installation Command Line Tools...'
67 + touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
68 + PROD=$(softwareupdate -l 2>&1 | grep -oE 'Label: Command Line Tools for Xcode[^,]*' | sed 's/^Label: //' | sort -V | tail -1)
69 + echo " produit: ${PROD:-introuvable}"
70 + if [ -n "$PROD" ]; then sudo softwareupdate -i "$PROD" --verbose 2>&1 | tail -3; fi
71 + rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
72 + sudo xcode-select --switch /Library/Developer/CommandLineTools 2>/dev/null || true
73 +fi
74 +if [ ! -x /opt/homebrew/bin/brew ]; then
75 + echo ' installation Homebrew...'
76 + NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" 2>&1 | tail -4
77 + grep -q 'opt/homebrew/bin/brew shellenv' ~/.zprofile 2>/dev/null || echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile
78 +fi
79 +echo " CLT: $(xcode-select -p 2>/dev/null || echo absent) brew: $(/opt/homebrew/bin/brew --version 2>/dev/null | head -1 || echo absent)"
80 +''' % (shlex.quote(pw), config.USER)
81 + rc, out, err = ssh.run(ip, script, timeout=5400)
82 + print(out)
83 + if err.strip():
84 + print(" stderr:", err.strip()[-400:])
85 + return rc == 0
86 +
87 +
88 +def gateway_mdns():
89 + import subprocess
90 + name = subprocess.run(["scutil", "--get", "LocalHostName"], capture_output=True, text=True).stdout.strip()
91 + return (name + ".local") if name else nodes.load_lan()["ips"].get(config.GATEWAY, "")
92 +
93 +
94 +def ka_helpers_install(alias, ip):
95 + """Pousseur Ka (launchd zsh → git push vers spbgit) + alias SSH `gitsrv` → passerelle + clé autorisée sur la passerelle."""
96 + assets = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "assets")
97 + out = [" ka-helpers sur %s :" % alias]
98 + ssh.write_remote_file(ip, "/Users/%s/ka-pousseur.sh" % config.USER, open(os.path.join(assets, "ka-pousseur.sh")).read(), mode="755")
99 + ssh.write_remote_file(ip, "/Users/%s/Library/LaunchAgents/com.ka.pousseur.plist" % config.USER, open(os.path.join(assets, "com.ka.pousseur.plist")).read(), mode="644")
100 + mdns = gateway_mdns()
101 + block = "Host gitsrv\n HostName %s\n User %s\n IdentityFile ~/.ssh/ka_guardian_ed25519\n IdentitiesOnly yes\n ControlMaster no\n ControlPath none\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n LogLevel ERROR\n\n" % (mdns, config.USER)
102 + script = r"""
103 +mkdir -p ~/.ssh ~/ka-guardian-spool; touch ~/.ka-pousseur-repos; chmod 700 ~/.ssh
104 +[ -f ~/.ssh/ka_guardian_ed25519 ] || ssh-keygen -q -t ed25519 -N '' -C "ka-guardian@$(cat ~/.maclustr-node 2>/dev/null || hostname -s)" -f ~/.ssh/ka_guardian_ed25519
105 +# alias gitsrv en TÊTE du config (avant Host *) — remplace un bloc existant
106 +python3 - <<'PYEOF'
107 +import os,re
108 +p=os.path.expanduser('~/.ssh/config'); s=open(p).read() if os.path.exists(p) else ''
109 +s=re.sub(r'(?ms)^Host gitsrv\n(?:[ \t]+.*\n?)*\n?', '', s)
110 +open(p,'w').write(%r + s); os.chmod(p, 0o600); print(' gitsrv -> %s')
111 +PYEOF
112 +cat ~/.ssh/ka_guardian_ed25519.pub
113 +launchctl bootout gui/$(id -u)/com.ka.pousseur >/dev/null 2>&1; launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.ka.pousseur.plist && echo ' pousseur launchd chargé'
114 +""" % (block, mdns)
115 + rc, o, e = ssh.run(ip, script, timeout=120)
116 + pub = [l for l in o.splitlines() if l.startswith("ssh-ed25519")]
117 + out += [" " + l for l in o.splitlines() if not l.startswith("ssh-ed25519")]
118 + if pub:
119 + ak = os.path.expanduser("~/.ssh/authorized_keys")
120 + cur = open(ak).read() if os.path.exists(ak) else ""
121 + if pub[0].split()[1] not in cur:
122 + with open(ak, "a") as f:
123 + f.write(pub[0].strip() + "\n")
124 + out.append(" clé %s autorisée sur la passerelle" % (pub[0].split()[-1] if len(pub[0].split()) > 2 else alias))
125 + else:
126 + out.append(" clé déjà autorisée sur la passerelle")
127 + rc, o, e = ssh.run(ip, "ssh -o BatchMode=yes -o ConnectTimeout=6 gitsrv 'echo ok' 2>&1 | tail -1", timeout=30)
128 + out.append(" test ssh gitsrv : %s" % (o.strip() or e.strip()[-80:]))
129 + return "\n".join(out) + "\n"
added mld/registry.py +51 −0
@@ -0,0 +1,51 @@
1 +"""Registre des déploiements (source de vérité) : ~/dispatch/registry.json."""
2 +import json
3 +import os
4 +import time
5 +from . import config
6 +
7 +
8 +def load():
9 + if os.path.exists(config.REGISTRY):
10 + return json.load(open(config.REGISTRY))
11 + return {"updated": None, "gateway": config.GATEWAY, "apps": {}, "history": []}
12 +
13 +
14 +def save(r):
15 + r["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
16 + r["gateway"] = config.GATEWAY
17 + tmp = config.REGISTRY + ".tmp"
18 + json.dump(r, open(tmp, "w"), indent=2, ensure_ascii=False)
19 + os.replace(tmp, config.REGISTRY)
20 +
21 +
22 +def node_of(app):
23 + return load()["apps"].get(app, {}).get("node")
24 +
25 +
26 +def apps_on(alias):
27 + return [a for a, v in load()["apps"].items() if v.get("node") == alias]
28 +
29 +
30 +def set_app(app, **fields):
31 + r = load()
32 + cur = r["apps"].get(app, {})
33 + cur.update(fields)
34 + cur["updated"] = time.strftime("%Y-%m-%dT%H:%M:%S")
35 + r["apps"][app] = cur
36 + save(r)
37 +
38 +
39 +def remove_app(app, note=""):
40 + r = load()
41 + if app in r["apps"]:
42 + r["history"].append({"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "app": app, "event": "removed", "was": r["apps"][app], "note": note})
43 + del r["apps"][app]
44 + save(r)
45 +
46 +
47 +def log(app, event, **kw):
48 + r = load()
49 + r["history"].append(dict(ts=time.strftime("%Y-%m-%dT%H:%M:%S"), app=app, event=event, **kw))
50 + r["history"] = r["history"][-500:]
51 + save(r)
added mld/render.py +89 −0
@@ -0,0 +1,89 @@
1 +"""Génération des fichiers de lancement : ecosystem PM2 (.cjs) et plists launchd."""
2 +import json
3 +import plistlib
4 +import shlex
5 +
6 +
7 +def _mem(v):
8 + if not v:
9 + return None
10 + if isinstance(v, str):
11 + return v
12 + return "%dM" % (int(v) // (1024 * 1024))
13 +
14 +
15 +def _home(path, ctx):
16 + """PM2/launchd n'expandent pas « ~ » : chemin absolu obligatoire."""
17 + if isinstance(path, str) and path.startswith("~"):
18 + return ctx["HOME"] + path[1:]
19 + return path
20 +
21 +
22 +def ecosystem(m, ctx):
23 + """Retourne le contenu JS d'un ecosystem.config.cjs pour les processus PM2 + tunnel ngrok."""
24 + from .manifest import render
25 + apps = []
26 + for p in m["processes"]:
27 + if p.get("manager", "pm2") != "pm2":
28 + continue
29 + a = {
30 + "name": p["name"],
31 + "script": _home(render(p["script"], ctx), ctx),
32 + "cwd": _home(render(p.get("cwd") or m["dir"], ctx), ctx),
33 + "exec_mode": "fork",
34 + "instances": 1,
35 + "autorestart": p.get("autorestart", True),
36 + "merge_logs": True,
37 + "time": True,
38 + "env": render(p.get("env") or {}, ctx),
39 + }
40 + if p.get("args"):
41 + args = p["args"]
42 + a["args"] = " ".join(shlex.quote(str(x)) for x in render(args, ctx)) if isinstance(args, list) else render(args, ctx)
43 + interp = p.get("interpreter")
44 + if interp and interp not in ("none", "N/A"):
45 + a["interpreter"] = render(interp, ctx)
46 + elif interp in ("none",) or (p["script"].endswith((".sh", ".mjs", ".js", ".py", ".cjs")) is False):
47 + a["interpreter"] = "none"
48 + if p.get("cron_restart"):
49 + a["cron_restart"] = p["cron_restart"]
50 + if p.get("max_memory_restart"):
51 + a["max_memory_restart"] = _mem(p["max_memory_restart"])
52 + if p.get("restart_delay"):
53 + a["restart_delay"] = p["restart_delay"]
54 + if p.get("node_args"):
55 + a["node_args"] = p["node_args"]
56 + apps.append(a)
57 + if m.get("ngrok"):
58 + n = m["ngrok"]
59 + args = ["http", "--url=%s" % n["url"], str(n["port"]), "--log=stdout"] + list(n.get("extra_args") or [])
60 + apps.append({
61 + "name": n["name"], "script": n.get("binary") or "/opt/homebrew/bin/ngrok", "args": " ".join(args),
62 + "interpreter": "none", "cwd": _home(render(m["dir"], ctx), ctx), "exec_mode": "fork", "autorestart": True,
63 + "restart_delay": 5000, "merge_logs": True, "time": True,
64 + })
65 + head = "// généré par maclustr-dispatch — ne pas éditer à la main (app: %s, nœud: %s)\n" % (m["app"], ctx.get("NODE"))
66 + return head + "module.exports = { apps: %s };\n" % json.dumps(apps, indent=2, ensure_ascii=False)
67 +
68 +
69 +def plist(item, ctx):
70 + from .manifest import render
71 + d = {
72 + "Label": item["label"],
73 + "ProgramArguments": render(item["program_arguments"], ctx),
74 + "RunAtLoad": item.get("run_at_load", True),
75 + "KeepAlive": item.get("keep_alive", True),
76 + "StandardOutPath": render(item.get("stdout") or "{{HOME}}/Library/Logs/%s.log" % item["label"], ctx),
77 + "StandardErrorPath": render(item.get("stderr") or "{{HOME}}/Library/Logs/%s.err.log" % item["label"], ctx),
78 + }
79 + if item.get("working_directory"):
80 + d["WorkingDirectory"] = _home(render(item["working_directory"], ctx), ctx)
81 + env = dict(item.get("env") or {})
82 + env.setdefault("HOME", ctx["HOME"])
83 + env.setdefault("PATH", "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin")
84 + d["EnvironmentVariables"] = render(env, ctx)
85 + if item.get("start_interval"):
86 + d["StartInterval"] = int(item["start_interval"])
87 + if item.get("throttle_interval"):
88 + d["ThrottleInterval"] = int(item["throttle_interval"])
89 + return plistlib.dumps(d).decode()
added mld/ssh.py +75 −0
@@ -0,0 +1,75 @@
1 +"""SSH / rsync vers les nœuds via le LAN 192.168.2.x (le SSH inter-nœuds Tailscale est bloqué par ACL)."""
2 +import os
3 +import shlex
4 +import subprocess
5 +from . import config
6 +
7 +SSH_OPTS = [
8 + "-o", "BatchMode=yes", "-o", "ConnectTimeout=8", "-o", "StrictHostKeyChecking=no",
9 + "-o", "UserKnownHostsFile=/dev/null", "-o", "LogLevel=ERROR", "-o", "ServerAliveInterval=15",
10 + "-i", config.SSH_KEY,
11 +]
12 +# PATH complet sur le nœud (les shells non interactifs n'ont pas Homebrew)
13 +REMOTE_PATH = "export PATH=/opt/homebrew/bin:/opt/homebrew/sbin:$HOME/bin:$HOME/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin; "
14 +
15 +
16 +def target(ip):
17 + return "%s@%s" % (config.USER, ip)
18 +
19 +
20 +def run(ip, cmd, timeout=120, stdin=None, path=True):
21 + """Exécute `cmd` (bash) sur le nœud. Retourne (rc, stdout, stderr)."""
22 + full = (REMOTE_PATH if path else "") + cmd
23 + try:
24 + p = subprocess.run(["ssh"] + SSH_OPTS + [target(ip), "bash -s"], input=full if stdin is None else stdin,
25 + capture_output=True, text=True, timeout=timeout)
26 + return p.returncode, p.stdout, p.stderr
27 + except subprocess.TimeoutExpired:
28 + return 124, "", "timeout after %ss" % timeout
29 +
30 +
31 +def run_script(ip, script, timeout=300):
32 + """Envoie un script complet sur stdin de `bash -s`."""
33 + return run(ip, "", timeout=timeout, stdin=REMOTE_PATH + "\n" + script)
34 +
35 +
36 +def ok(ip, cmd, timeout=60):
37 + return run(ip, cmd, timeout)[0] == 0
38 +
39 +
40 +def rsync_push(local_dir, ip, remote_dir, excludes=(), delete=True, timeout=7200, extra=()):
41 + """Passerelle -> nœud. remote_dir peut contenir ~ (résolu côté nœud)."""
42 + args = ["rsync", "-a", "--stats", "-e", "ssh " + " ".join(shlex.quote(o) for o in SSH_OPTS)]
43 + if delete:
44 + args.append("--delete")
45 + for e in excludes:
46 + args += ["--exclude", e]
47 + args += list(extra)
48 + args += ["--rsync-path", "mkdir -p %s && rsync" % shlex.quote(remote_dir), local_dir.rstrip("/") + "/", "%s:%s/" % (target(ip), remote_dir)]
49 + p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
50 + return p.returncode, p.stdout, p.stderr
51 +
52 +
53 +def rsync_pull(ip, remote_dir, local_dir, excludes=(), delete=True, timeout=7200):
54 + """Nœud -> passerelle (staging)."""
55 + os.makedirs(local_dir, exist_ok=True)
56 + args = ["rsync", "-a", "--stats", "-e", "ssh " + " ".join(shlex.quote(o) for o in SSH_OPTS)]
57 + if delete:
58 + args.append("--delete")
59 + for e in excludes:
60 + args += ["--exclude", e]
61 + args += ["%s:%s/" % (target(ip), remote_dir), local_dir.rstrip("/") + "/"]
62 + p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
63 + return p.returncode, p.stdout, p.stderr
64 +
65 +
66 +def scp_to(ip, local_file, remote_path):
67 + p = subprocess.run(["scp"] + SSH_OPTS + [local_file, "%s:%s" % (target(ip), remote_path)], capture_output=True, text=True)
68 + return p.returncode == 0
69 +
70 +
71 +def write_remote_file(ip, remote_path, content, mode="644"):
72 + """Écrit un fichier sur le nœud via heredoc (contenu texte)."""
73 + q = shlex.quote(remote_path)
74 + script = "mkdir -p $(dirname %s) && cat > %s <<'__MLD_EOF__'\n%s\n__MLD_EOF__\nchmod %s %s" % (q, q, content, mode, q)
75 + return run(ip, script, timeout=60)
76