Repository Public
1# Architecture23```4 Internet (TLS, Let's Encrypt)5 │6 Caddy on BHS64 (MacLustr Tunnel gateway)7 │ WireGuard8 ┌─────────────────▼──────────────────────────────┐9 │ node M2M32c (macOS, PM2, placed by mld) │10 │ │11 visitor ─────▶ │ clip-web Next.js 16 (App Router) │12 admin ─────▶ │ public cinema /, /feed, /film, /series … │13 │ studio /studio/* (session + RBAC) │14 │ API /api/* (public + studio) │15 │ media /media/* (range, signed) │16 │ │ enqueue ▲ read │17 │ ▼ │ │18 │ Redis 8 (BullMQ) PostgreSQL 17 (db clip) │19 │ │ ▲ │20 │ ▼ │ │21 │ clip-worker Node (tsx) │22 │ pipeline stages, scheduler, GC, backups │23 │ FFmpeg / ffprobe │24 │ │ │25 │ ▼ │26 │ asset store ~/data/clip-spboucher-ai │27 │ content-addressed (SHA-256), outside the repo│28 └────────┬───────────────────────────────────────┘29 │ HTTPS (server side only)30 ▼31 xAI API: grok-4.3 · Grok Imagine image/video · TTS32```3334## Processes3536| Process | Role |37|---|---|38| `clip-web` | Next.js server: public pages (ISR-free, server-rendered from PostgreSQL), studio, API, media delivery with HTTP range |39| `clip-worker` | BullMQ workers (`pipeline` queue, concurrency from the budget policy), the programming scheduler, garbage collection, nightly backup |4041They share one codebase (`src/`). Only server code imports `src/lib/env.ts`; the client bundle never sees a secret42(`scripts/secrets-scan.ts` greps the built client chunks in CI).4344## Pipeline4546```47Idea ─▶ Pitches ─▶ [Research] ─▶ Script ─▶ Shot plan + bibles ─▶ Keyframes ─▶ Voice ─▶ Video ─▶ Assembly ─▶ QA ─▶ Approval ─▶ Publication48 Creative Research Screen- Storyboard / Visual Visual Audio Motion Editor Critic policy Publisher49 Director Editor writer Director Director Director Director50```5152Stage order differs from the brief in one place: **voice is synthesised before video**. Narration length is the53only reliable clock, so each shot's video duration is derived from its measured narration (cheap call first,54expensive call sized correctly, no re-buying a clip because it was a second short).5556Each stage is a pure function `(project state) → (rows + assets)`, run by one BullMQ job:5758- **Typed contracts** — `src/lib/contracts.ts` (Zod): `CreativeBrief`, `Pitch`, `Script`, `ShotPlan`, `Bible`,59 `QualityReport`, `FilmManifest`. LLM output is parsed against the schema; a parse failure is a stage failure.60- **Idempotency** — every paid call goes through `runProviderJob()` (`src/pipeline/provider-job.ts`) with an61 idempotency key = SHA-256 of (kind, model, canonical input). The `provider_jobs` row is unique on that key:62 a retry or a double click finds the finished row and reuses its asset; a crashed video job is resumed from63 its stored `request_id` rather than resubmitted.64- **Budget gate** — before any paid call: kill switch, per-shot / per-film / daily / monthly ceilings, and the65 estimate from `src/providers/pricing.ts`. A refusal raises `BudgetError` and parks the project in `blocked`.66- **Bounded retries** — HTTP layer: exponential backoff + jitter, 3 attempts, circuit breaker per endpoint67 (`src/providers/http.ts`). Shot regeneration: capped by `max_regenerations_per_shot`, then human approval.68- **Provenance** — each `provider_jobs` row stores provider, model, settings, prompt, input hash, timestamps,69 cost and output asset. The published `FilmManifest` is a public projection without prompts.7071Autonomy modes (per project): `manual` (each stage is started by an operator), `assisted` (free and cheap stages72run on their own, the pipeline pauses before paid video generation), `autonomous` (runs to QA inside the budget).73Publication always requires a human approval unless `auto_publish` is explicitly enabled in the policy, and never74happens on a failed QA without a recorded override.7576## Data model7778`db/migrations/001_init.sql`. Groups:7980- Identity: `users`, `sessions`, `audit_log`, `rate_limits`81- Editorial: `series`, `collections`, `collection_films`, `tags`, `film_tags`82- Production (private): `projects` (brief), `concept_candidates`, `script_versions`, `bibles`, `storyboards`,83 `shots`, `prompt_versions`, `provider_jobs` (= generation attempts), `assets`, `audio_tracks`, `subtitle_tracks`,84 `film_versions` (manifest), `quality_reviews`, `approvals`85- Public: `films` (metadata only), `publications`, `reactions`, `analytics_events`, `reports`86- Control: `budget_policy`, `cost_entries`, `schedules`, `settings`8788Public pages query `films` + delivery assets only; prompts live in `shots` / `prompt_versions` / `provider_jobs`89and are never selected by a public query (`src/lib/films.ts` is the single public read path).9091## Storage9293`AssetStorage` (`src/lib/storage.ts`) with one driver today: a content-addressed filesystem store94(`ASSET_ROOT/ab/cd/<sha256>.<ext>`). Published delivery assets are served at `/media/<sha>.<ext>` with range95requests and immutable caching; everything else needs a studio session or an HMAC-signed, expiring URL96(`getSignedUrl`). An S3 driver is the documented migration path (same interface) — it is **not** implemented,97because the cluster has no object store and the brief asks to confirm real infrastructure first.9899## Search and recommendations100101No xAI embedding model exists (live check), so CLIP ships an explainable lexical embedding: hashed TF-IDF over102title, synopsis, thesis, tags, series and style (512 dimensions, cosine). Search = PostgreSQL full-text rank103+ trigram similarity + cosine, and every result reports which signal matched. Related films = cosine + shared104tags/series, with the reason shown ("same series", "shared tags: …"). No engagement signal feeds the ranking.105106## Migration path to distributed workers107108Workers are stateless apart from the asset store. To scale out: run `clip-worker` on more nodes against the same109Redis/PostgreSQL and move `AssetStorage` to S3-compatible storage (MinIO on the LAN). `docker-compose.yml`110describes that topology for a non-macOS host.111