RUWIO/dev_docs/electron-rendering-integration-plan_2.md

Electron Rendering Integration — Plan 2 (Review Findings + Remaining Work)

Date: 2026-06-05

This document supersedes electron-rendering-integration-plan.md for ongoing work. It is self-contained: you do not need the original plan to continue from here. It records what is already implemented, what was found during review, and the concrete remaining TODOs in priority order.

Goal

Render MP4 inside the Electron desktop app without the standalone HTTP render server, while keeping that standalone server working for the web/self-hosted build. Desktop uses Electron's own Chromium (a hidden render window) for frame capture and a bundled ffmpeg for H.264 encoding.

Architecture (as built)

• Vue renderer owns project state and the export UI (ExportDialog.vue). For MP4 it calls renderMp4OnDesktop() when window.ruwioDesktop.renderMp4 exists, otherwise falls back to the HTTP server path (downloadFromServer).
• Preload (electron/preload.cts) exposes a narrow API on window.ruwioDesktop: renderMp4, onRenderProgress, cancelRender.
• Main process (electron/main.cts) validates every IPC payload, shows a native save dialog, owns job IDs + AbortControllers, and forwards progress events. IPC channels: ruwio:render-mp4, ruwio:render-cancel, progress on ruwio:render-progress.
• Render worker (electron/renderWorker.cts) creates a hidden BrowserWindow, loads the packaged render page, seeks each frame, captures it, and pipes PNG bytes into ffmpeg stdin. Mirrors the server frame loop.
• Shared validation (shared/renderValidation.cjs) is the single source of render limits, playbackRange, and validateRenderRequest, used by both server/server.js and electron/main.cts.
• ffmpeg resolution (electron/ffmpegPath.cts): FFMPEG_PATH env → process.resourcesPath/bin/<platform>/<ffmpeg> (packaged) → ffmpeg on PATH.
• Staging script (scripts/stage-ffmpeg.cjs) copies an ffmpeg binary into bin/<platform>/ for packaging. Wired into desktop:build/desktop:pack via stage:ffmpeg -- --optional.
• Packaging (package.json build): extraResources ships server/render.html and bin/; files ships shared/renderValidation.cjs.

Key files

Concern	File
Renderer export routing	src/components/ExportDialog.vue (exportMp4Server)
Renderer desktop client	src/lib/desktopRenderExport.ts
Desktop API types	src/types/desktop.d.ts
Preload bridge	electron/preload.cts
Main IPC + save dialog	electron/main.cts (handleRenderMp4, handleRenderCancel)
Frame capture + ffmpeg	electron/renderWorker.cts
ffmpeg path resolver	electron/ffmpegPath.cts
Shared validation	shared/renderValidation.cjs
Render page (capture surface)	server/render.html (window.__animatrInit, window.__animatrSeek)
Standalone server	server/server.js
ffmpeg staging	scripts/stage-ffmpeg.cjs
Desktop render verification	scripts/verify-*.cjs + runner scripts/verify-desktop-render.cjs (npm run test:render-desktop)

Current verified state

Last verified: 2026-06-06 (Linux, DISPLAY available, system ffmpeg/ffprobe 7.1.4).

• tsc -p tsconfig.electron.json — passes.
• npm run build (vue-tsc + vite) — passes.
• npm run test:render-validation — 7/7 pass.
• npm run test:render-server — pass.
• npm run test:render-server:e2e — pass (server still produces a playable H.264 MP4; confirms the shared render.html change below is server-safe).
• npm run test:render-desktop — all 6 desktop checks pass. New aggregated Electron harness suite (scripts/verify-*.cjs, runner scripts/verify-desktop-render.cjs) that drives the real worker headlessly:
  • capture: hidden-window capturePage yields advancing, non-blank frames.
  • export: the real renderMp4Job produces a 25-frame H.264 MP4 with motion.
  • cancel: mid-export abort rejects cleanly, no partial file.
  • timeout: the inactivity watchdog aborts a wedged render (~2s).
  • assets: local data: assets load; remote http: is blocked, no hang.
  • background: transparent regions flatten to white (not black).
• npm run desktop:pack — succeeds. Packaged resources verified: resources/server/render.html, executable resources/bin/linux/ffmpeg (runs -version), and app.asar contains dist-electron/{main,renderWorker, ffmpegPath}.cjs + shared/renderValidation.cjs.

Findings #1–#6 are all resolved (see status tags below).

Not yet done:

• A manual GUI desktop export (dev or packaged) through the save dialog. The worker logic itself is proven end-to-end by test:render-desktop; only the click-through has not been recorded.
• bin/linux/ffmpeg is the dynamically-linked Debian system binary (~358 KB), not a redistributable static build. Fine for local dev; NOT release-safe.
• Windows/macOS ffmpeg binaries and licensing notes (Phase C #10–11).
• Platform installer builds (AppImage / NSIS / DMG) and their manual checks.

Review findings (ordered by severity)

1. CRITICAL — capturePage() on a non-shown, non-offscreen window — RESOLVED (does not reproduce)

Verified 2026-06-06 via scripts/verify-capture.cjs: a hidden, non-offscreen window's capturePage() returns real, non-blank frames whose contents advance frame-to-frame (linearly-stepping position hash over a moving rect). No offscreen mode was needed. The remaining staleness risk was finding #2.

Original analysis below.

renderWorker.cts creates the render window with show: false but not webPreferences.offscreen: true, then calls webContents.capturePage() once per frame. Hidden, non-offscreen windows are not composited by Chromium, so capturePage() can return blank or stale frames. This is the single biggest risk and cannot be confirmed by reading code — it needs a real export of a moving scene, frame-stepped, to confirm frames actually advance.

Fix direction if it reproduces:

• Switch the render window to webPreferences.offscreen: true and drive capture off the paint event (win.webContents.on('paint', (e, dirty, image) => ...) / beginFrameSubscription), or
• Show the window positioned off-screen (e.g. far negative coordinates) so the compositor paints it, then capturePage.

Offscreen mode also naturally solves finding #2.

2. HIGH — No wait for a compositor paint between seek and capture — RESOLVED (was real; fixed)

Confirmed real and reproducible: without a settle, capturePage intermittently returned the previous frame (stale/off-by-one) in ~1/3 of runs. Fixed in server/render.html: window.__animatrSeek now returns a Promise that resolves only after a double requestAnimationFrame (with a 1s anti-hang fallback), so both the desktop worker (await executeJavaScript) and the server (Puppeteer await evaluate) wait for a paint before capturing. 6/6 PASS after the fix; test:render-server:e2e confirms the server path is unaffected.

Original analysis below.

__animatrSeek runs a synchronous 2D-canvas drawFrame, but the worker calls capturePage immediately after executeJavaScript(...seek) with no wait for a paint. This risks an off-by-one frame or tearing. The original plan explicitly required "wait for the render function to settle."

Fix direction: await a double requestAnimationFrame inside the page (or, in offscreen mode, await the next paint event) before capturing.

3. MEDIUM — No render timeout in the desktop worker — RESOLVED

Added an inactivity watchdog in renderWorker.cts: a combined AbortController (jobAbort) trips on user cancel or after RENDER_INACTIVITY_TIMEOUT_MS (default 2 min, override via RUWIO_RENDER_TIMEOUT_MS) with no progress. The timer is re-armed on every onProgress, and each per-frame await is wrapped in raceAbort(...) so a wedged executeJavaScript/capturePage is interrupted, ffmpeg is killed, and the partial file removed. Verified by scripts/verify-timeout.cjs (aborts ~2s with message "render timed out").

Original analysis below.

server/server.js auto-aborts a stuck job via JOB_TIMEOUT_MS. The desktop worker has no watchdog, so a hung executeJavaScript/capturePage wedges the export with only manual cancel as escape.

Fix direction: add a timeout that aborts the job's AbortController after N seconds of no progress (reset the timer on each onProgress).

4. MEDIUM — Transparent window flattens to black under yuv420p — RESOLVED

Decision: match the server (flatten onto white). The render window is now transparent: false / backgroundColor: '#ffffff', so transparent regions (e.g. backgroundVideo scenes, where render.html skips the bg fill) capture as white instead of black. Verified by scripts/verify-bg.cjs (corner pixel rgb(255,255,255), zero near-black pixels).

Original analysis below.

The render window is transparent: true. Opaque scenes are fine because render.html fills canvas.background over the full frame, but any genuinely transparent region flattens onto black (vs the server's Puppeteer path, which flattens onto white via omitBackground:false). Decide this consciously: either make the render window background opaque, or document MP4 transparency behavior as black-filled.

5. LOW — Outbound network not blocked in the render window — RESOLVED

The render window now uses an isolated session partition (ruwio-render) with a session.webRequest.onBeforeRequest filter on ['*://*/*'] that cancels all http/https requests; local schemes (file:/data:/blob:) are unmatched and load normally. Verified by scripts/verify-assets.cjs: a local data: image draws (1600 px), a remote http: asset is blocked (0 px) and does not hang.

Original analysis below.

The render window denies window.open and will-navigate, but does not block network requests. A project asset with an http(s) src will be fetched by the hidden window. The original plan listed "block external network requests unless explicitly allowed." Low risk for a local app; close it for defense-in-depth via session.webRequest.onBeforeRequest or a CSP on the render page.

6. LOW — Desktop jobs are not serialized in main — RESOLVED

main.cts now guards with a module-level renderActive flag set at the top of handleRenderMp4 (before the save dialog) and reset in a finally; a second concurrent IPC invoke throws "A render is already in progress". Mirrors the server's MAX_ACTIVE_JOBS=1.

Original analysis below.

handleRenderMp4 will run concurrent jobs; only the ExportDialog if (progress.value) return guard prevents it from the UI, and IPC is directly invokable. The server enforces MAX_ACTIVE_JOBS. Add a one-line guard in main (reject or queue when a job is already active).

What is already correct (do not re-do)

• Partial output file is removed on failure/cancel; ffmpeg is drained and the window closed in finally.
• Cancel-before-jobId race handled in desktopRenderExport.ts via pendingAbort, including the window where main is blocked on the save dialog.
• ffmpeg ENOENT surfaces the friendly missingFfmpegMessage() through both the spawn-error and exit-!started branches.
• Packaging wiring is consistent: render page + bin/ in extraResources, shared/renderValidation.cjs in files, resolvers read process.resourcesPath.
• Validation parity: both server and main call the same validateRenderRequest; desktop uses frozen DEFAULT_LIMITS, server overlays env-configurable limits.
• Filename sanitization (sanitizeMp4Name) strips path separators before the save dialog.

Remaining TODOs

Phase A — Prove frame capture works — DONE

Findings #1 and #2 resolved (see status tags above). The acceptance criteria (moving scene → correct MP4; clean cancel; no partial file) are covered by npm run test:render-desktop (capture/export/cancel checks).

Phase B — Robustness — DONE

Findings #3, #4, #5, #6 all implemented and verified (status tags above).

Phase C — Packaging + cross-platform (partly done)

8. npm run desktop:pack + packaged-resource verification — DONE for Linux (see "Current verified state").
9. TODO (manual GUI): launch the unpacked app and export MP4 with no standalone server running. Click-through not yet recorded.
10. TODO: stage reviewed redistributable static ffmpeg binaries for Linux (bin/linux/ffmpeg, replace the current dynamic system copy), Windows (bin/win32/ffmpeg.exe), and macOS (bin/darwin/ffmpeg).
11. TODO: record ffmpeg release notes per bundled binary: source URL, upstream version, platform/arch, static vs dynamic, codecs (esp. libx264), GPL/LGPL obligations, attribution/license files to ship.

Phase D — Verification matrix

Automated checks — DONE and recorded under "Current verified state" (build, desktop:pack, test:render-validation, test:render-server, test:render-server:e2e, test:render-desktop).

TODO (manual GUI):

• desktop dev export of a small scene through the save dialog
• desktop packaged export of a small scene
• scenes with: image assets; text/non-default fonts; non-default fps; hidden layers skipped
• server-mode MP4 export via the browser fallback
• Platform builds: Linux AppImage, Windows NSIS, macOS DMG

Manual check commands (reference)

Server fallback smoke (needs Chromium, ffmpeg, ffprobe):

npm run test:render-server
npm run test:render-server:e2e

Browser fallback against the standalone server:

# terminal 1
cd server
HOST=127.0.0.1 PORT=6000 RENDER_TOKEN="choose-a-long-random-token" npm start
# terminal 2
VITE_RENDER_SERVER=http://localhost:6000 VITE_RENDER_TOKEN="choose-a-long-random-token" npm run dev

Stage a platform ffmpeg for packaging:

npm run stage:ffmpeg -- --source /path/to/ffmpeg

Risks carried forward

• Frame capture on hidden windows (finding #1) is the gating risk for the whole desktop path. Resolve Phase A before any packaging work.
• Render drift between server/render.html and src/lib/render.ts: keep validation/parity in mind; consider generating the render page from shared code later to avoid divergence.
• ffmpeg packaging varies by platform: solve with explicit binary resolution (done) plus an unpacked-app smoke check before installer work.