RUWIO/dev_docs/electron-rendering-integration-plan_2.md

# Electron Rendering Integration — Plan 2 (Review Findings + Remaining Work)

Date: 2026-06-05

This document supersedes `electron-rendering-integration-plan.md` for ongoing
work. It is self-contained: you do not need the original plan to continue from
here. It records what is already implemented, what was found during review, and
the concrete remaining TODOs in priority order.

## Goal

Render MP4 inside the Electron desktop app without the standalone HTTP render
server, while keeping that standalone server working for the web/self-hosted
build. Desktop uses Electron's own Chromium (a hidden render window) for frame
capture and a bundled `ffmpeg` for H.264 encoding.

## Architecture (as built)

- **Vue renderer** owns project state and the export UI (`ExportDialog.vue`).
  For MP4 it calls `renderMp4OnDesktop()` when `window.ruwioDesktop.renderMp4`
  exists, otherwise falls back to the HTTP server path (`downloadFromServer`).
- **Preload** (`electron/preload.cts`) exposes a narrow API on
  `window.ruwioDesktop`: `renderMp4`, `onRenderProgress`, `cancelRender`.
- **Main process** (`electron/main.cts`) validates every IPC payload, shows a
  native save dialog, owns job IDs + `AbortController`s, and forwards progress
  events. IPC channels: `ruwio:render-mp4`, `ruwio:render-cancel`, progress on
  `ruwio:render-progress`.
- **Render worker** (`electron/renderWorker.cts`) creates a hidden
  `BrowserWindow`, loads the packaged render page, seeks each frame, captures
  it, and pipes PNG bytes into `ffmpeg` stdin. Mirrors the server frame loop.
- **Shared validation** (`shared/renderValidation.cjs`) is the single source of
  render limits, `playbackRange`, and `validateRenderRequest`, used by both
  `server/server.js` and `electron/main.cts`.
- **ffmpeg resolution** (`electron/ffmpegPath.cts`): `FFMPEG_PATH` env →
  `process.resourcesPath/bin/<platform>/<ffmpeg>` (packaged) → `ffmpeg` on PATH.
- **Staging script** (`scripts/stage-ffmpeg.cjs`) copies an ffmpeg binary into
  `bin/<platform>/` for packaging. Wired into `desktop:build`/`desktop:pack`
  via `stage:ffmpeg -- --optional`.
- **Packaging** (`package.json` `build`): `extraResources` ships
  `server/render.html` and `bin/`; `files` ships `shared/renderValidation.cjs`.

## Key files

| Concern | File |
| --- | --- |
| Renderer export routing | `src/components/ExportDialog.vue` (`exportMp4Server`) |
| Renderer desktop client | `src/lib/desktopRenderExport.ts` |
| Desktop API types | `src/types/desktop.d.ts` |
| Preload bridge | `electron/preload.cts` |
| Main IPC + save dialog | `electron/main.cts` (`handleRenderMp4`, `handleRenderCancel`) |
| Frame capture + ffmpeg | `electron/renderWorker.cts` |
| ffmpeg path resolver | `electron/ffmpegPath.cts` |
| Shared validation | `shared/renderValidation.cjs` |
| Render page (capture surface) | `server/render.html` (`window.__animatrInit`, `window.__animatrSeek`) |
| Standalone server | `server/server.js` |
| ffmpeg staging | `scripts/stage-ffmpeg.cjs` |
| Desktop render verification | `scripts/verify-*.cjs` + runner `scripts/verify-desktop-render.cjs` (`npm run test:render-desktop`) |

## Current verified state

Last verified: 2026-06-06 (Linux, DISPLAY available, system ffmpeg/ffprobe 7.1.4).

- `tsc -p tsconfig.electron.json` — passes.
- `npm run build` (vue-tsc + vite) — passes.
- `npm run test:render-validation` — 7/7 pass.
- `npm run test:render-server` — pass.
- `npm run test:render-server:e2e` — pass (server still produces a playable
  H.264 MP4; confirms the shared `render.html` change below is server-safe).
- `npm run test:render-desktop` — **all 6 desktop checks pass.** New aggregated
  Electron harness suite (`scripts/verify-*.cjs`, runner
  `scripts/verify-desktop-render.cjs`) that drives the real worker headlessly:
  - capture: hidden-window `capturePage` yields advancing, non-blank frames.
  - export: the real `renderMp4Job` produces a 25-frame H.264 MP4 with motion.
  - cancel: mid-export abort rejects cleanly, no partial file.
  - timeout: the inactivity watchdog aborts a wedged render (~2s).
  - assets: local `data:` assets load; remote `http:` is blocked, no hang.
  - background: transparent regions flatten to white (not black).
- `npm run desktop:pack` — succeeds. Packaged resources verified:
  `resources/server/render.html`, executable `resources/bin/linux/ffmpeg`
  (runs `-version`), and `app.asar` contains `dist-electron/{main,renderWorker,
  ffmpegPath}.cjs` + `shared/renderValidation.cjs`.

Findings #1–#6 are all resolved (see status tags below).

Not yet done:

- A manual GUI desktop export (dev or packaged) through the save dialog. The
  worker logic itself is proven end-to-end by `test:render-desktop`; only the
  click-through has not been recorded.
- **`bin/linux/ffmpeg` is the dynamically-linked Debian system binary (~358 KB),
  not a redistributable static build.** Fine for local dev; NOT release-safe.
- Windows/macOS ffmpeg binaries and licensing notes (Phase C #10–11).
- Platform installer builds (AppImage / NSIS / DMG) and their manual checks.

## Review findings (ordered by severity)

### 1. CRITICAL — `capturePage()` on a non-shown, non-offscreen window — RESOLVED (does not reproduce)

Verified 2026-06-06 via `scripts/verify-capture.cjs`: a hidden, non-offscreen
window's `capturePage()` returns real, non-blank frames whose contents advance
frame-to-frame (linearly-stepping position hash over a moving rect). No
offscreen mode was needed. The remaining staleness risk was finding #2.

Original analysis below.



`renderWorker.cts` creates the render window with `show: false` but **not**
`webPreferences.offscreen: true`, then calls `webContents.capturePage()` once
per frame. Hidden, non-offscreen windows are not composited by Chromium, so
`capturePage()` can return blank or stale frames. This is the single biggest
risk and cannot be confirmed by reading code — it needs a real export of a
moving scene, frame-stepped, to confirm frames actually advance.

Fix direction if it reproduces:

- Switch the render window to `webPreferences.offscreen: true` and drive capture
  off the `paint` event (`win.webContents.on('paint', (e, dirty, image) => ...)`
  / `beginFrameSubscription`), **or**
- Show the window positioned off-screen (e.g. far negative coordinates) so the
  compositor paints it, then `capturePage`.

Offscreen mode also naturally solves finding #2.

### 2. HIGH — No wait for a compositor paint between seek and capture — RESOLVED (was real; fixed)

Confirmed real and reproducible: without a settle, `capturePage` intermittently
returned the *previous* frame (stale/off-by-one) in ~1/3 of runs. Fixed in
`server/render.html`: `window.__animatrSeek` now returns a Promise that resolves
only after a double `requestAnimationFrame` (with a 1s anti-hang fallback), so
both the desktop worker (`await executeJavaScript`) and the server (Puppeteer
`await evaluate`) wait for a paint before capturing. 6/6 PASS after the fix;
`test:render-server:e2e` confirms the server path is unaffected.

Original analysis below.



`__animatrSeek` runs a synchronous 2D-canvas `drawFrame`, but the worker calls
`capturePage` immediately after `executeJavaScript(...seek)` with no wait for a
paint. This risks an off-by-one frame or tearing. The original plan explicitly
required "wait for the render function to settle."

Fix direction: await a double `requestAnimationFrame` inside the page (or, in
offscreen mode, await the next `paint` event) before capturing.

### 3. MEDIUM — No render timeout in the desktop worker — RESOLVED

Added an inactivity watchdog in `renderWorker.cts`: a combined `AbortController`
(`jobAbort`) trips on user cancel or after `RENDER_INACTIVITY_TIMEOUT_MS`
(default 2 min, override via `RUWIO_RENDER_TIMEOUT_MS`) with no progress. The
timer is re-armed on every `onProgress`, and each per-frame await is wrapped in
`raceAbort(...)` so a wedged `executeJavaScript`/`capturePage` is interrupted,
ffmpeg is killed, and the partial file removed. Verified by
`scripts/verify-timeout.cjs` (aborts ~2s with message "render timed out").

Original analysis below.



`server/server.js` auto-aborts a stuck job via `JOB_TIMEOUT_MS`. The desktop
worker has no watchdog, so a hung `executeJavaScript`/`capturePage` wedges the
export with only manual cancel as escape.

Fix direction: add a timeout that aborts the job's `AbortController` after N
seconds of no progress (reset the timer on each `onProgress`).

### 4. MEDIUM — Transparent window flattens to black under `yuv420p` — RESOLVED

Decision: match the server (flatten onto white). The render window is now
`transparent: false` / `backgroundColor: '#ffffff'`, so transparent regions
(e.g. `backgroundVideo` scenes, where `render.html` skips the bg fill) capture
as white instead of black. Verified by `scripts/verify-bg.cjs` (corner pixel
`rgb(255,255,255)`, zero near-black pixels).

Original analysis below.



The render window is `transparent: true`. Opaque scenes are fine because
`render.html` fills `canvas.background` over the full frame, but any genuinely
transparent region flattens onto **black** (vs the server's Puppeteer path,
which flattens onto white via `omitBackground:false`). Decide this consciously:
either make the render window background opaque, or document MP4 transparency
behavior as black-filled.

### 5. LOW — Outbound network not blocked in the render window — RESOLVED

The render window now uses an isolated session partition (`ruwio-render`) with a
`session.webRequest.onBeforeRequest` filter on `['*://*/*']` that cancels all
http/https requests; local schemes (`file:`/`data:`/`blob:`) are unmatched and
load normally. Verified by `scripts/verify-assets.cjs`: a local `data:` image
draws (1600 px), a remote `http:` asset is blocked (0 px) and does not hang.

Original analysis below.



The render window denies `window.open` and `will-navigate`, but does not block
network requests. A project asset with an `http(s)` `src` will be fetched by the
hidden window. The original plan listed "block external network requests unless
explicitly allowed." Low risk for a local app; close it for defense-in-depth via
`session.webRequest.onBeforeRequest` or a CSP on the render page.

### 6. LOW — Desktop jobs are not serialized in main — RESOLVED

`main.cts` now guards with a module-level `renderActive` flag set at the top of
`handleRenderMp4` (before the save dialog) and reset in a `finally`; a second
concurrent IPC invoke throws "A render is already in progress". Mirrors the
server's `MAX_ACTIVE_JOBS=1`.

Original analysis below.



`handleRenderMp4` will run concurrent jobs; only the `ExportDialog`
`if (progress.value) return` guard prevents it from the UI, and IPC is directly
invokable. The server enforces `MAX_ACTIVE_JOBS`. Add a one-line guard in main
(reject or queue when a job is already active).

## What is already correct (do not re-do)

- Partial output file is removed on failure/cancel; ffmpeg is drained and the
  window closed in `finally`.
- Cancel-before-jobId race handled in `desktopRenderExport.ts` via
  `pendingAbort`, including the window where main is blocked on the save dialog.
- ffmpeg ENOENT surfaces the friendly `missingFfmpegMessage()` through both the
  `spawn`-error and `exit`-`!started` branches.
- Packaging wiring is consistent: render page + `bin/` in `extraResources`,
  `shared/renderValidation.cjs` in `files`, resolvers read `process.resourcesPath`.
- Validation parity: both server and main call the same `validateRenderRequest`;
  desktop uses frozen `DEFAULT_LIMITS`, server overlays env-configurable limits.
- Filename sanitization (`sanitizeMp4Name`) strips path separators before the
  save dialog.

## Remaining TODOs

### Phase A — Prove frame capture works — DONE

Findings #1 and #2 resolved (see status tags above). The acceptance criteria
(moving scene → correct MP4; clean cancel; no partial file) are covered by
`npm run test:render-desktop` (capture/export/cancel checks).

### Phase B — Robustness — DONE

Findings #3, #4, #5, #6 all implemented and verified (status tags above).

### Phase C — Packaging + cross-platform (partly done)

8. `npm run desktop:pack` + packaged-resource verification — DONE for Linux
   (see "Current verified state").
9. **TODO (manual GUI):** launch the unpacked app and export MP4 with no
   standalone server running. Click-through not yet recorded.
10. **TODO:** stage reviewed *redistributable static* ffmpeg binaries for Linux
    (`bin/linux/ffmpeg`, replace the current dynamic system copy), Windows
    (`bin/win32/ffmpeg.exe`), and macOS (`bin/darwin/ffmpeg`).
11. **TODO:** record ffmpeg release notes per bundled binary: source URL,
    upstream version, platform/arch, static vs dynamic, codecs (esp.
    `libx264`), GPL/LGPL obligations, attribution/license files to ship.

### Phase D — Verification matrix

Automated checks — DONE and recorded under "Current verified state"
(`build`, `desktop:pack`, `test:render-validation`, `test:render-server`,
`test:render-server:e2e`, `test:render-desktop`).

**TODO (manual GUI):**
- desktop dev export of a small scene through the save dialog
- desktop packaged export of a small scene
- scenes with: image assets; text/non-default fonts; non-default fps;
  hidden layers skipped
- server-mode MP4 export via the browser fallback
- Platform builds: Linux AppImage, Windows NSIS, macOS DMG

## Manual check commands (reference)

Server fallback smoke (needs Chromium, ffmpeg, ffprobe):

```bash
npm run test:render-server
npm run test:render-server:e2e
```

Browser fallback against the standalone server:

```bash
# terminal 1
cd server
HOST=127.0.0.1 PORT=6000 RENDER_TOKEN="choose-a-long-random-token" npm start
# terminal 2
VITE_RENDER_SERVER=http://localhost:6000 VITE_RENDER_TOKEN="choose-a-long-random-token" npm run dev
```

Stage a platform ffmpeg for packaging:

```bash
npm run stage:ffmpeg -- --source /path/to/ffmpeg
```

## Risks carried forward

- **Frame capture on hidden windows** (finding #1) is the gating risk for the
  whole desktop path. Resolve Phase A before any packaging work.
- **Render drift** between `server/render.html` and `src/lib/render.ts`: keep
  validation/parity in mind; consider generating the render page from shared
  code later to avoid divergence.
- **ffmpeg packaging varies by platform**: solve with explicit binary
  resolution (done) plus an unpacked-app smoke check before installer work.