Repository Public
1# PROGRESS23Running log of the CLIP build. Times are America/Toronto, 2026-11-10.45## Status by milestone67| Milestone | State | Evidence |8|---|---|---|9| M0 Discovery | done | `docs/provider-capabilities.md` (live API + official docs), `docs/architecture.md`, `docs/cost-model.md`, `docs/security.md`, `docs/implementation-plan.md`, `docs/deployment.md` |10| M1 Foundation | done | `db/migrations/001_init.sql`, auth (password + TOTP, sessions, RBAC, audit), storage, queue, design tokens |11| M2 Public cinema | done | home, feed, film, series, explore, search, about, report, saved; EN + FR |12| M3 First real generation | done | image → image-to-video → stored asset → playable draft, ledger from provider ticks (see "Real runs") |13| M4 Intelligent studio | done | Concept Lab, script + research, storyboard, prompt workbench, voice, captions, FFmpeg assembly, QA, approval |14| M5 Autonomous programming | done (scheduler shipped **disabled**) | series templates, scheduler with daily quota, budgets, duplicate prevention, publication policy |15| M6 Hardening | done, with the gaps listed below | CI gate, e2e against production, health, backups + restore drill, runbook |1617## Decisions18191. **REST adapters instead of `@ai-sdk/xai`.** The AI SDK hides the video `request_id` while a job is pending and20 does not surface `usage.cost_in_usd_ticks`. CLIP needs the first to resume a paid job after a crash instead of21 buying it twice, and the second as billing truth. Providers stay behind interfaces (`src/providers/types.ts`).222. **Voice before video.** Narration length is the only reliable clock; each clip is generated for the duration23 its line needs. The brief's order (video, then voice) would mean re-buying clips that turn out too short.243. **Narration over cinematic shots; no lip-sync.** The 1.5 models can lip-sync, but exact captions and reliable25 timing matter more here. Word-timed captions come from the TTS character timestamps.264. **No music.** Each clip's own generated ambience is the sound bed, ducked under the voice. Nothing licensed,27 nothing third-party.285. **Filesystem asset store** (content-addressed) behind the `AssetStorage` interface. The cluster has no object29 store; S3 is the documented migration path, not implemented.306. **Lexical, explainable search and recommendations.** xAI lists no embedding model for this key, so CLIP uses a31 hashed TF-IDF embedding plus PostgreSQL full-text and trigram; each result states why it matched.327. **`grok-4.7` as the language model** after a measured comparison with `grok-4.3` (provider doc, "Model choice").338. **Informational films are fact-checked line by line**, not only on the claims the script declares.349. **PM2 on a cluster node, deployed with `mld`**, because that is the real infrastructure (macOS nodes, Caddy on35 the tunnel gateway). `docker-compose.yml` is provided for portability and is not exercised.3610. **TypeScript 6**, because typescript-eslint does not support TypeScript 7 yet.3738## Commands used3940```bash41scripts/ci.sh # typecheck, lint, 28 unit tests, build, secrets scan42PW_BASE_URL=https://clip.spboucher.ai pnpm test:e2e # public + API + studio access control + performance budget43mld stage ~/Desktop/Cluster/projets/clip-spboucher-ai clip-spboucher-ai && mld deploy clip-spboucher-ai --node M2M32c44ssh M2M32c 'cd ~/apps/clip-spboucher-ai && scripts/backup.sh && scripts/restore.sh ~/backups/clip-spboucher-ai'45```4647## Known gaps (not implemented, stated plainly)4849- **Passkeys (WebAuthn)**: not implemented. Studio auth is password + TOTP; TOTP is enforced once the owner enrols.50- **Studio in French**: the studio UI is English only. The public site is fully bilingual.51- **C2PA signing**: not implemented (no signing certificate). Provenance is a public JSON block plus MP4 container metadata.52- **HLS adaptive streaming**: not implemented. Delivery is progressive MP4 with range requests, plus a 480p low-data rendition.53- **1:1 and 16:9 renditions**: not produced. Films are composed for 9:16; cropping them would break the framing.54- **Burned-in subtitles**: not produced (the node's FFmpeg has no libass). Captions are sidecar WebVTT rendered by the player.55- **OpenTelemetry and external error tracking**: not implemented. Observability is JSON logs (PM2), `/api/health`, the studio dashboard and the cluster's own `mld heal`.56- **Hosted CI**: the git forge (spbgit) has no runner. The CI gate is `scripts/ci.sh`, installable as a pre-push hook.57- **Docker Compose**: written, not run.58- **S3 storage driver**: interface only.59- **Reference-to-video / video editing / image editing**: adapters exist per the documented API, but the pipeline does not use them and they were **not** exercised against the live API.60- **Accessibility**: built to WCAG 2.2 AA patterns (labels, focus, contrast, reduced motion, captions); no formal audit was run.6162## Real runs (provider calls actually made on 2026-11-10)6364| # | Where | What happened | Result |65|---|---|---|---|66| 1 | laptop, smoke test | one image, one image-to-video (lite, 4 s), one French TTS with timestamps, one structured LLM call, one web-search call | all succeeded; response shapes and costs recorded in `docs/provider-capabilities.md` |67| 2 | laptop, CLI | *The Keeper's Debt*, 5 shots, draft model | full pipeline on the first run, QA 7.9 passed, $1.20; used as the local test film |68| 3 | laptop, studio + worker | French informational project in **assisted** mode | ran concept → script → research → storyboard → keyframes → voice through the queue and **paused before video**, as designed |69| 4 | production, studio, autonomous | *What the Distance Keeps* (One-Minute Ideas) | research found an inaccuracy → script rewritten and re-checked; QA **failed** on a blocking artifact in shot 4 → prompt revised in the workbench, **only shot 4 regenerated** → QA 8.4 passed → approved → **published** |70| 5 | production, studio, autonomous | *Weight on a Seed* (Objects with Stories, French) | QA 7.4, four blocking issues (the images do not show what the narration says) → **held, not published** |71| 6 | production, studio, autonomous | *Noon Under the Tide* (Impossible Worlds) | one clip failed with a provider `internal_error` (not billed); stage re-run from the API, the three finished clips were reused; QA 8.1 passed → approved → **published** |7273| 7 | production, studio, autonomous | *The Outward Bearing / Le cap du large* (Micro Cinema, French) | QA **failed** (two near-black shots, caught by both the blank-frame check and the critic) → three keyframes revised and regenerated in one pass → QA 7.9, one blocking issue left in shot 4's motion → the motion prompt was fixed, but the regeneration was **refused by the per-shot ceiling** ($2.06 spent + $0.99 > $2.50) → **held, not published** |7475Production spend for these runs: **$20.87** (video $18.83, LLM $0.93, images $0.84, web search $0.26, speech $0.02),7684 ledger entries, all from provider-reported usage except speech (computed). Local development and smoke tests77added about $2.40 on the same key. Two films are published; two are held by the quality gate.7879Things these runs proved, beyond the happy path:8081- **The budget gate refuses real spend**: a regeneration was blocked by the per-shot ceiling and the project was parked with the reason shown in the studio.8283- **Publishing before approval is refused** (`409 This film version has not been approved`).84- **A repeated request does not start a second job** (`duplicate: true`, same job id).85- **A worker restart in the middle of a stage recovers**: the stage was picked up again by the queue and the86 interrupted call was retried once; nothing was paid twice.87- **Backup and restore**: `scripts/backup.sh` (dump + assets + copy to a second node) and `scripts/restore.sh`88 (scratch database, row counts, 0 missing assets) both ran on the production node.8990Problems found by running for real, and fixed:9192| Problem | Fix |93|---|---|94| `grok-4.3` wrote a weak, off-topic informational script and under-declared its claims | default LLM → `grok-4.7`; informational films are checked line by line; stricter Screenwriter and Research Editor prompts |95| `grok-4.7` structured calls take 2–3 minutes; the 240 s HTTP timeout fired | LLM timeout 600 s, one retry |96| The search-grounded answer arrives as the last of several messages | parse the last `message`, tolerate text around the JSON |97| A restart left an in-flight LLM call "submitted" for 15 minutes | worker releases its own orphaned calls at start-up (`releaseOrphans`) |98| A transient provider error on one shot ended the stage instead of retrying | the shot pool now rethrows the original (retryable) error |99| A fiction script asked for images generative video cannot hold (precise multi-object states) | Screenwriter rule: every line must be showable by one still with simple motion |100| Dashboard and project list relied on the layout for authorization | every studio page now calls `requireRole()` itself |101| Unknown pages returned 200 because of a streaming `loading.tsx` | removed; real 404 status |102103## Live vs blocked104105**Live at https://clip.spboucher.ai** (TLS by Let's Encrypt through the tunnel gateway):106107- public cinema in English and French with real published films, vertical feed, film pages, series, explore, search, report form, sitemap, OpenAPI document;108- private studio with password + optional TOTP, roles, audit log, Concept Lab, storyboard, prompt workbench, timeline, QA report, approval, release, programme, costs, kill switch;109- worker with the full pipeline, real xAI image / video / speech / LLM / web-search calls, cost ledger from provider ticks, budgets, idempotency, recovery;110- health endpoint, PM2 supervision, the cluster's `mld heal`, nightly backup at 04:20 with an off-node copy.111112**Deliberately off until the owner turns them on** (Settings): the programming scheduler and automatic113publication. Default ceilings: $2.50 per shot, $8 per film, $25 per day, $150 per month.114115**Waiting on the owner:**1161171. Sign in at `/studio` with the password in `~/Desktop/Cluster/secrets/clip-spboucher-ai/owner-password.txt`, choose a new one (forced at first sign-in), and turn on two-factor authentication in **Account**.1182. Decide what to do with the two films held by QA:119 - *The Outward Bearing* (French) is one clip away: the corrected motion prompt for shot 4 is saved. Raise "Per shot" to about $3.50 in Settings, then press **Run** on Video in the project; assembly and QA follow on their own.120 - *Weight on a Seed* (French) needs shots 2–5 redone or its lines rewritten (about $4 more, above the $8 film ceiling), or archive it.1213. If CAD figures are wanted, enter a USD→CAD rate in Settings (none is fetched or assumed).1224. Enable the scheduler and add programmes only once the first films look right.123124**Blocked: nothing.** Everything not delivered is listed under "Known gaps" above and is a scope choice, not a missing credential.125126## Second pass (2026-11-10, evening): better films, new interface, Claude Code director127128The owner judged the first films poor and the interface dated. What changed:129130**Film quality (pipeline v2).** The xAI image guide and the official storyboard-to-film example were read and applied.131132| Before | Now |133|---|---|134| every keyframe generated independently at 1k, low quality | shot 1 generated at 2k / medium; shots 2+ made by **editing shot 1** (`/v1/images/edits`), which keeps the character identical |135| image prompts led with mood and lists of things to avoid, often dark | subject first, then details, art direction, light, colour, composition; bright, readable exposure |136| 720p video | **native 1080p** (`grok-imagine-video-1.5`), measured at $0.2517/s (6 s = $1.51) |137| 24 fps clips converted to 30 fps (judder) | 24 fps master |138| objects and landscapes, slow static shots | a protagonist who wants something, visible action in every shot, a payoff |139140Films made with it: *The Painted Wall* (QA 8.3, published), *The One Stool* (QA 9.0, published), *Perle, gardienne de la141lumière* (directed by Claude Code, French, QA 8.7, published, hero), *Rust on the Bloom* (French, QA 7.7, **held** for the142owner: two lines do not match their pictures). The two first-generation films were withdrawn (restorable in Studio → Films).143A 5-shot film now costs about $9.50–13.144145**Ceilings raised at the owner's request**: $6 per shot, $25 per film, $100 per day, $500 per month (audit log entry).146147**Interface.** Light theme by default with a dark toggle (saved per browser), Bricolage Grotesque headings, larger touch148targets, app-style bottom tab bar on phones, immersive feed, studio restyled with the same tokens.149150**Claude Code director** (Studio → Director, or `tsx scripts/director-start.ts --idea "…"`). One idea in; the worker starts151Claude Code headless (`claude -p`, stream-json) in an empty directory with three tools only: `./clip` (studio commands),152Read (to look at the frames it made) and Write. Claude writes the plan, generates and inspects keyframes, runs **cheap153previews** (draft model, 480p, a few cents) before paying for 1080p finals, re-renders what is wrong, cuts the film and154runs QA. Every paid call goes through the same budget gate and ledger, plus a per-run budget; Claude's own cost is added155to the film's ledger. It cannot publish. First run: 43 turns, $0.76 of Claude, $12.81 of media, QA 8.7.156157Found and fixed during this pass:158159| Problem | Fix |160|---|---|161| bursts of `fetch failed` when three films rendered at once (the node has no IPv6 route; Node's dual-stack racing timed out) | IPv4 first and a 3 s family attempt timeout (`scripts/_env.ts`) |162| a network blip while polling lost clips that were already paid for | polling tolerates outages; a failed job keeps its provider request id when the failure was on our side |163| Claude Code tried shell loops and pipes, which the permission rules refuse | playbook: one `./clip` command per call |164165Still true: publication by the operator CLI names a studio user as approver; the three films above were approved by the166operator (Claude) after viewing each cut, because the owner asked for a better programme.167