RUWIO/dev_docs/electron-rendering-integration-plan.md

# Electron Rendering Integration Plan

Date: 2026-06-05

## Verdict

Yes, the MP4 rendering part can be included in the Electron app.

The current render server is already made from desktop-compatible pieces:

- Node orchestration in `server/server.js`
- a browser render surface in `server/render.html`
- frame capture through Chromium
- MP4 encoding through `ffmpeg`
- progress, cancellation, temp files, validation, and cleanup around the job

The main caveat is that the server cannot be bundled as-is and expected to work in a packaged desktop app. It currently assumes:

- `puppeteer-core` is installed in the server package
- Chromium exists at `PUPPETEER_EXECUTABLE_PATH` or `/usr/bin/chromium`
- `ffmpeg` exists on `PATH`
- the Vue app talks to `http://localhost:6000` with `VITE_RENDER_SERVER`
- Electron Builder packages only `dist/`, `dist-electron/`, and root `package.json`

So the feature is feasible, but the right implementation is to move the render job into the Electron main process behind IPC, not to expose an HTTP render server from the desktop app.

## Recommended Architecture

Use Electron as the local render host:

- Vue renderer keeps owning project state and export UI.
- Preload exposes a narrow desktop render API on `window.ruwioDesktop`.
- Electron main validates render requests, starts/cancels jobs, captures frames, runs `ffmpeg`, and returns a saved MP4 path or file bytes.
- The standalone `server/` package remains available for web or self-hosted server deployments.

Preferred render surface:

- Use a hidden/offscreen Electron `BrowserWindow` to load a packaged render page.
- Call `webContents.executeJavaScript` or `webContents.send` to initialize the scene and seek each frame.
- Use `webContents.capturePage()` for PNG frame capture.
- Pipe PNG frames into `ffmpeg` the same way `server/server.js` does today.

This avoids bundling a second Chromium through Puppeteer. Electron already ships Chromium, so the desktop build only needs to solve `ffmpeg` packaging.

Fallback/spike option:

- Start the existing Express render server as a child process bound to `127.0.0.1` with a random token.
- Point the renderer at that local URL.
- This is useful for a quick feasibility spike, but it keeps the localhost server, token plumbing, Puppeteer, and external Chromium dependency in the desktop app. It should not be the final architecture.

## Dependency Strategy

### Chromium

Do not add a separate Chromium binary for the first native implementation. Use Electron's bundled Chromium through a hidden render window.

Only keep Puppeteer for the standalone server unless a spike proves Electron capture cannot match render quality or performance.

### ffmpeg

`ffmpeg` is the real packaged-app dependency.

Implementation options:

1. Bundle per-platform `ffmpeg` binaries as `extraResources`.
2. Use a maintained npm package that supplies platform-specific binaries.
3. Support a user-configured/system `ffmpeg` path as a fallback.

Recommended first production path:

- Add an internal resolver such as `electron/ffmpegPath.cts`.
- In development, prefer `process.env.FFMPEG_PATH`, then `ffmpeg` on `PATH`.
- In packaged builds, prefer `process.resourcesPath/bin/<platform>/ffmpeg`.
- Show a friendly export error if no usable binary is found.

## Implementation Plan

## Phase 1: Extract Shared Render Validation

Goal: keep server and desktop limits identical.

Tasks:

1. Move request validation and playback range helpers out of `server/server.js` into a shared module.
2. Keep the module Node-compatible and browser-safe where practical.
3. Export:
   - `validateRenderRequest(scene, options)`
   - `playbackRange(scene)`
   - render limits and allowed ffmpeg presets
4. Update `server/server.js` to use the shared validator.
5. Add focused tests for invalid dimensions, fps, duration, frame count, layers, keyframes, `crf`, and preset.

Acceptance:

- Existing server behavior is unchanged.
- Desktop rendering can reuse the same validation before launching any render surface or child process.

## Phase 2: Add Desktop Render IPC Contract

Goal: let the Vue app request MP4 rendering without knowing whether the backend is HTTP or Electron.

Tasks:

1. Extend `src/types/desktop.d.ts` with a render API:

```ts
renderMp4(options: {
  scene: Scene
  fps?: number
  crf?: number
  preset?: string
  suggestedName?: string
}): Promise<{ path: string } | null>
onRenderProgress(callback: (event: {
  jobId: string
  progress: number
  done: boolean
  error?: string
}) => void): () => void
cancelRender(jobId: string): Promise<void>
```

2. Add preload wrappers that call `ipcRenderer.invoke` and subscribe to progress events.
3. Add main-process handlers:
   - `ruwio:render-mp4`
   - `ruwio:render-cancel`
4. Validate every IPC payload in main before starting a job.
5. Keep job IDs and abort controllers in main, not in the Vue renderer.

Acceptance:

- A renderer call can start a fake/stub render and receive progress.
- Cancel reaches the main process.
- Invalid payloads fail before any child process or hidden window is created.

## Phase 3: Build The Electron Render Worker

Goal: replace Puppeteer with Electron's own Chromium for desktop exports.

Tasks:

1. Create an Electron-only render worker module, for example `electron/renderWorker.cts`.
2. Package a render page:
   - first pass: reuse `server/render.html` as an extra resource
   - later pass: consider generating a render page from the shared browser renderer to reduce drift
3. Create a hidden `BrowserWindow` per job or a serialized reusable worker window:
   - `show: false`
   - `webPreferences.sandbox: true`
   - no Node integration
   - restricted navigation
4. Load the render page from a local file URL.
5. Initialize the scene.
6. For each frame:
   - seek to the target time
   - wait for the render function to settle
   - capture the page or canvas bounds
   - write PNG bytes to `ffmpeg.stdin`
   - emit progress over IPC
7. On cancel or timeout:
   - abort the frame loop
   - kill `ffmpeg`
   - close the worker window
   - remove temp files

Acceptance:

- A tiny scene renders to MP4 in `npm run desktop:dev`.
- Progress updates reach `ExportDialog`.
- Cancel stops capture and `ffmpeg`.
- Temp directories are removed on success, failure, and cancel.

## Phase 4: Package ffmpeg

Goal: make MP4 export work in packaged desktop builds.

Tasks:

1. Choose the binary source:
   - committed/release-managed binaries under a `vendor/` or `bin/` folder, or
   - npm package binaries copied into Electron Builder resources
2. Add Electron Builder `extraResources` entries for:
   - Linux ffmpeg
   - Windows ffmpeg.exe
   - macOS ffmpeg
   - render worker HTML/assets
3. Add `ffmpeg` path resolution for dev and packaged builds.
4. Verify executable permissions on macOS/Linux packaged output.
5. Document licensing obligations for the selected `ffmpeg` build.

Acceptance:

- `npm run desktop:pack` includes the render page and ffmpeg.
- The unpacked app exports MP4 without a separately installed render server.
- Missing ffmpeg produces a clear user-facing error.

## Phase 5: Switch ExportDialog To Desktop Rendering

Goal: use native rendering in Electron and keep the current server path for the web app.

Tasks:

1. Add `src/lib/desktopRenderExport.ts`.
2. In `ExportDialog.vue`, route MP4 export as:
   - if `window.ruwioDesktop?.renderMp4` exists: use desktop IPC render
   - else: use current `downloadFromServer`
3. Reuse current progress UI and cancellation behavior.
4. Save the finished MP4 through a native save dialog from main.
5. Keep `VITE_RENDER_SERVER` support for browser builds and remote/self-hosted workflows.

Acceptance:

- Desktop users do not need `cd server && npm run start`.
- Browser users can still use the external render server.
- The MP4 button reports a helpful error when no server is configured in browser mode.

## Phase 6: Keep Standalone Server Working

Goal: avoid breaking existing server deployments.

Tasks:

1. Keep `server/server.js` as an Express wrapper around shared validation/render helpers where useful.
2. Keep `server/render.html` or replace it only after desktop and server render outputs are compared.
3. Preserve `/health`, `/render`, `/events`, `/file`, and `DELETE` behavior.
4. Add a smoke test or manual check for server MP4 export after shared refactors.

Acceptance:

- `cd server && npm run start` still starts the local render server.
- The Vue app can still render through `VITE_RENDER_SERVER` and `VITE_RENDER_TOKEN`.

## Phase 7: Verification Matrix

Run these before calling the integration complete:

- `npm run build`
- `npm run desktop:pack`
- `npm run test:render-server`
- `npm run test:render-server:e2e`
- desktop dev MP4 export of a small scene
- desktop packaged MP4 export of a small scene
- cancel an MP4 export after progress starts
- export a scene with image assets
- export a scene with text/fonts
- export a scene at non-default fps
- export a scene with hidden layers skipped
- server-mode MP4 export still works

Manual platform checks:

- Linux AppImage
- Windows NSIS build
- macOS DMG build

## Risks And Mitigations

- **ffmpeg packaging varies by platform**: solve with explicit binary resolution and an unpacked-app smoke check before installer work.
- **render drift between `server/render.html` and `src/lib/render.ts`**: keep parity tests, then consider generating the desktop/server render page from shared code.
- **large scenes can freeze the desktop app**: keep render work in main-controlled worker windows, serialize jobs initially, enforce the same render limits as the server.
- **IPC can move large payloads**: scenes are JSON-sized today, but avoid sending frame buffers over IPC; frame capture and `ffmpeg` piping should stay in main.
- **untrusted project assets reach Chromium**: keep the hidden render window sandboxed, restrict navigation, block external network requests unless explicitly allowed, and keep existing project validation work moving.

## Suggested First Pull Request

Keep the first PR small and proof-oriented:

1. Extract shared render validation from `server/server.js`.
2. Add a stubbed desktop render IPC contract.
3. Add `desktopRenderExport.ts` and route the MP4 button to desktop IPC when available.
4. Return a fake progress/done result in development or behind a temporary flag.

That proves the app integration and cancellation shape before dealing with frame capture and ffmpeg packaging.

## Suggested Second Pull Request

Implement the real Electron render worker:

1. Hidden render window.
2. Frame capture loop.
3. `ffmpeg` stdin piping.
4. Temp directory cleanup.
5. Dev-only `ffmpeg` path resolution.

## Suggested Third Pull Request

Make it shippable:

1. Package ffmpeg and render worker assets.
2. Verify `desktop:pack`.
3. Add packaged-app manual test notes.
4. Document ffmpeg licensing and update README desktop export notes.