# Electron Rendering Integration — Plan 2 (Review Findings + Remaining Work) Date: 2026-06-05 This document supersedes `electron-rendering-integration-plan.md` for ongoing work. It is self-contained: you do not need the original plan to continue from here. It records what is already implemented, what was found during review, and the concrete remaining TODOs in priority order. ## Goal Render MP4 inside the Electron desktop app without the standalone HTTP render server, while keeping that standalone server working for the web/self-hosted build. Desktop uses Electron's own Chromium (a hidden render window) for frame capture and a bundled `ffmpeg` for H.264 encoding. ## Architecture (as built) - **Vue renderer** owns project state and the export UI (`ExportDialog.vue`). For MP4 it calls `renderMp4OnDesktop()` when `window.ruwioDesktop.renderMp4` exists, otherwise falls back to the HTTP server path (`downloadFromServer`). - **Preload** (`electron/preload.cts`) exposes a narrow API on `window.ruwioDesktop`: `renderMp4`, `onRenderProgress`, `cancelRender`. - **Main process** (`electron/main.cts`) validates every IPC payload, shows a native save dialog, owns job IDs + `AbortController`s, and forwards progress events. IPC channels: `ruwio:render-mp4`, `ruwio:render-cancel`, progress on `ruwio:render-progress`. - **Render worker** (`electron/renderWorker.cts`) creates a hidden `BrowserWindow`, loads the packaged render page, seeks each frame, captures it, and pipes PNG bytes into `ffmpeg` stdin. Mirrors the server frame loop. - **Shared validation** (`shared/renderValidation.cjs`) is the single source of render limits, `playbackRange`, and `validateRenderRequest`, used by both `server/server.js` and `electron/main.cts`. - **ffmpeg resolution** (`electron/ffmpegPath.cts`): `FFMPEG_PATH` env → `process.resourcesPath/bin//` (packaged) → `ffmpeg` on PATH. - **Staging script** (`scripts/stage-ffmpeg.cjs`) copies an ffmpeg binary into `bin//` for packaging. Wired into `desktop:build`/`desktop:pack` via `stage:ffmpeg -- --optional`. - **Packaging** (`package.json` `build`): `extraResources` ships `server/render.html` and `bin/`; `files` ships `shared/renderValidation.cjs`. ## Key files | Concern | File | | --- | --- | | Renderer export routing | `src/components/ExportDialog.vue` (`exportMp4Server`) | | Renderer desktop client | `src/lib/desktopRenderExport.ts` | | Desktop API types | `src/types/desktop.d.ts` | | Preload bridge | `electron/preload.cts` | | Main IPC + save dialog | `electron/main.cts` (`handleRenderMp4`, `handleRenderCancel`) | | Frame capture + ffmpeg | `electron/renderWorker.cts` | | ffmpeg path resolver | `electron/ffmpegPath.cts` | | Shared validation | `shared/renderValidation.cjs` | | Render page (capture surface) | `server/render.html` (`window.__animatrInit`, `window.__animatrSeek`) | | Standalone server | `server/server.js` | | ffmpeg staging | `scripts/stage-ffmpeg.cjs` | | Desktop render verification | `scripts/verify-*.cjs` + runner `scripts/verify-desktop-render.cjs` (`npm run test:render-desktop`) | ## Current verified state Last verified: 2026-06-06 (Linux, DISPLAY available, system ffmpeg/ffprobe 7.1.4). - `tsc -p tsconfig.electron.json` — passes. - `npm run build` (vue-tsc + vite) — passes. - `npm run test:render-validation` — 7/7 pass. - `npm run test:render-server` — pass. - `npm run test:render-server:e2e` — pass (server still produces a playable H.264 MP4; confirms the shared `render.html` change below is server-safe). - `npm run test:render-desktop` — **all 6 desktop checks pass.** New aggregated Electron harness suite (`scripts/verify-*.cjs`, runner `scripts/verify-desktop-render.cjs`) that drives the real worker headlessly: - capture: hidden-window `capturePage` yields advancing, non-blank frames. - export: the real `renderMp4Job` produces a 25-frame H.264 MP4 with motion. - cancel: mid-export abort rejects cleanly, no partial file. - timeout: the inactivity watchdog aborts a wedged render (~2s). - assets: local `data:` assets load; remote `http:` is blocked, no hang. - background: transparent regions flatten to white (not black). - `npm run desktop:pack` — succeeds. Packaged resources verified: `resources/server/render.html`, executable `resources/bin/linux/ffmpeg` (runs `-version`), and `app.asar` contains `dist-electron/{main,renderWorker, ffmpegPath}.cjs` + `shared/renderValidation.cjs`. Findings #1–#6 are all resolved (see status tags below). Not yet done: - A manual GUI desktop export (dev or packaged) through the save dialog. The worker logic itself is proven end-to-end by `test:render-desktop`; only the click-through has not been recorded. - **`bin/linux/ffmpeg` is the dynamically-linked Debian system binary (~358 KB), not a redistributable static build.** Fine for local dev; NOT release-safe. - Windows/macOS ffmpeg binaries and licensing notes (Phase C #10–11). - Platform installer builds (AppImage / NSIS / DMG) and their manual checks. ## Review findings (ordered by severity) ### 1. CRITICAL — `capturePage()` on a non-shown, non-offscreen window — RESOLVED (does not reproduce) Verified 2026-06-06 via `scripts/verify-capture.cjs`: a hidden, non-offscreen window's `capturePage()` returns real, non-blank frames whose contents advance frame-to-frame (linearly-stepping position hash over a moving rect). No offscreen mode was needed. The remaining staleness risk was finding #2. Original analysis below. `renderWorker.cts` creates the render window with `show: false` but **not** `webPreferences.offscreen: true`, then calls `webContents.capturePage()` once per frame. Hidden, non-offscreen windows are not composited by Chromium, so `capturePage()` can return blank or stale frames. This is the single biggest risk and cannot be confirmed by reading code — it needs a real export of a moving scene, frame-stepped, to confirm frames actually advance. Fix direction if it reproduces: - Switch the render window to `webPreferences.offscreen: true` and drive capture off the `paint` event (`win.webContents.on('paint', (e, dirty, image) => ...)` / `beginFrameSubscription`), **or** - Show the window positioned off-screen (e.g. far negative coordinates) so the compositor paints it, then `capturePage`. Offscreen mode also naturally solves finding #2. ### 2. HIGH — No wait for a compositor paint between seek and capture — RESOLVED (was real; fixed) Confirmed real and reproducible: without a settle, `capturePage` intermittently returned the *previous* frame (stale/off-by-one) in ~1/3 of runs. Fixed in `server/render.html`: `window.__animatrSeek` now returns a Promise that resolves only after a double `requestAnimationFrame` (with a 1s anti-hang fallback), so both the desktop worker (`await executeJavaScript`) and the server (Puppeteer `await evaluate`) wait for a paint before capturing. 6/6 PASS after the fix; `test:render-server:e2e` confirms the server path is unaffected. Original analysis below. `__animatrSeek` runs a synchronous 2D-canvas `drawFrame`, but the worker calls `capturePage` immediately after `executeJavaScript(...seek)` with no wait for a paint. This risks an off-by-one frame or tearing. The original plan explicitly required "wait for the render function to settle." Fix direction: await a double `requestAnimationFrame` inside the page (or, in offscreen mode, await the next `paint` event) before capturing. ### 3. MEDIUM — No render timeout in the desktop worker — RESOLVED Added an inactivity watchdog in `renderWorker.cts`: a combined `AbortController` (`jobAbort`) trips on user cancel or after `RENDER_INACTIVITY_TIMEOUT_MS` (default 2 min, override via `RUWIO_RENDER_TIMEOUT_MS`) with no progress. The timer is re-armed on every `onProgress`, and each per-frame await is wrapped in `raceAbort(...)` so a wedged `executeJavaScript`/`capturePage` is interrupted, ffmpeg is killed, and the partial file removed. Verified by `scripts/verify-timeout.cjs` (aborts ~2s with message "render timed out"). Original analysis below. `server/server.js` auto-aborts a stuck job via `JOB_TIMEOUT_MS`. The desktop worker has no watchdog, so a hung `executeJavaScript`/`capturePage` wedges the export with only manual cancel as escape. Fix direction: add a timeout that aborts the job's `AbortController` after N seconds of no progress (reset the timer on each `onProgress`). ### 4. MEDIUM — Transparent window flattens to black under `yuv420p` — RESOLVED Decision: match the server (flatten onto white). The render window is now `transparent: false` / `backgroundColor: '#ffffff'`, so transparent regions (e.g. `backgroundVideo` scenes, where `render.html` skips the bg fill) capture as white instead of black. Verified by `scripts/verify-bg.cjs` (corner pixel `rgb(255,255,255)`, zero near-black pixels). Original analysis below. The render window is `transparent: true`. Opaque scenes are fine because `render.html` fills `canvas.background` over the full frame, but any genuinely transparent region flattens onto **black** (vs the server's Puppeteer path, which flattens onto white via `omitBackground:false`). Decide this consciously: either make the render window background opaque, or document MP4 transparency behavior as black-filled. ### 5. LOW — Outbound network not blocked in the render window — RESOLVED The render window now uses an isolated session partition (`ruwio-render`) with a `session.webRequest.onBeforeRequest` filter on `['*://*/*']` that cancels all http/https requests; local schemes (`file:`/`data:`/`blob:`) are unmatched and load normally. Verified by `scripts/verify-assets.cjs`: a local `data:` image draws (1600 px), a remote `http:` asset is blocked (0 px) and does not hang. Original analysis below. The render window denies `window.open` and `will-navigate`, but does not block network requests. A project asset with an `http(s)` `src` will be fetched by the hidden window. The original plan listed "block external network requests unless explicitly allowed." Low risk for a local app; close it for defense-in-depth via `session.webRequest.onBeforeRequest` or a CSP on the render page. ### 6. LOW — Desktop jobs are not serialized in main — RESOLVED `main.cts` now guards with a module-level `renderActive` flag set at the top of `handleRenderMp4` (before the save dialog) and reset in a `finally`; a second concurrent IPC invoke throws "A render is already in progress". Mirrors the server's `MAX_ACTIVE_JOBS=1`. Original analysis below. `handleRenderMp4` will run concurrent jobs; only the `ExportDialog` `if (progress.value) return` guard prevents it from the UI, and IPC is directly invokable. The server enforces `MAX_ACTIVE_JOBS`. Add a one-line guard in main (reject or queue when a job is already active). ## What is already correct (do not re-do) - Partial output file is removed on failure/cancel; ffmpeg is drained and the window closed in `finally`. - Cancel-before-jobId race handled in `desktopRenderExport.ts` via `pendingAbort`, including the window where main is blocked on the save dialog. - ffmpeg ENOENT surfaces the friendly `missingFfmpegMessage()` through both the `spawn`-error and `exit`-`!started` branches. - Packaging wiring is consistent: render page + `bin/` in `extraResources`, `shared/renderValidation.cjs` in `files`, resolvers read `process.resourcesPath`. - Validation parity: both server and main call the same `validateRenderRequest`; desktop uses frozen `DEFAULT_LIMITS`, server overlays env-configurable limits. - Filename sanitization (`sanitizeMp4Name`) strips path separators before the save dialog. ## Remaining TODOs ### Phase A — Prove frame capture works — DONE Findings #1 and #2 resolved (see status tags above). The acceptance criteria (moving scene → correct MP4; clean cancel; no partial file) are covered by `npm run test:render-desktop` (capture/export/cancel checks). ### Phase B — Robustness — DONE Findings #3, #4, #5, #6 all implemented and verified (status tags above). ### Phase C — Packaging + cross-platform (partly done) 8. `npm run desktop:pack` + packaged-resource verification — DONE for Linux (see "Current verified state"). 9. **TODO (manual GUI):** launch the unpacked app and export MP4 with no standalone server running. Click-through not yet recorded. 10. **TODO:** stage reviewed *redistributable static* ffmpeg binaries for Linux (`bin/linux/ffmpeg`, replace the current dynamic system copy), Windows (`bin/win32/ffmpeg.exe`), and macOS (`bin/darwin/ffmpeg`). 11. **TODO:** record ffmpeg release notes per bundled binary: source URL, upstream version, platform/arch, static vs dynamic, codecs (esp. `libx264`), GPL/LGPL obligations, attribution/license files to ship. ### Phase D — Verification matrix Automated checks — DONE and recorded under "Current verified state" (`build`, `desktop:pack`, `test:render-validation`, `test:render-server`, `test:render-server:e2e`, `test:render-desktop`). **TODO (manual GUI):** - desktop dev export of a small scene through the save dialog - desktop packaged export of a small scene - scenes with: image assets; text/non-default fonts; non-default fps; hidden layers skipped - server-mode MP4 export via the browser fallback - Platform builds: Linux AppImage, Windows NSIS, macOS DMG ## Manual check commands (reference) Server fallback smoke (needs Chromium, ffmpeg, ffprobe): ```bash npm run test:render-server npm run test:render-server:e2e ``` Browser fallback against the standalone server: ```bash # terminal 1 cd server HOST=127.0.0.1 PORT=6000 RENDER_TOKEN="choose-a-long-random-token" npm start # terminal 2 VITE_RENDER_SERVER=http://localhost:6000 VITE_RENDER_TOKEN="choose-a-long-random-token" npm run dev ``` Stage a platform ffmpeg for packaging: ```bash npm run stage:ffmpeg -- --source /path/to/ffmpeg ``` ## Risks carried forward - **Frame capture on hidden windows** (finding #1) is the gating risk for the whole desktop path. Resolve Phase A before any packaging work. - **Render drift** between `server/render.html` and `src/lib/render.ts`: keep validation/parity in mind; consider generating the render page from shared code later to avoid divergence. - **ffmpeg packaging varies by platform**: solve with explicit binary resolution (done) plus an unpacked-app smoke check before installer work.