rename docs
This commit is contained in:
th committed
1 parent
518d43855b
commit
196aa2e8ba
8 files changed
No files matched your search
@@ -0,0 +1,308 @@
|
||||
# Electron Rendering Integration — Plan 2 (Review Findings + Remaining Work)
|
||||
|
||||
Date: 2026-06-05
|
||||
|
||||
This document supersedes `electron-rendering-integration-plan.md` for ongoing
|
||||
work. It is self-contained: you do not need the original plan to continue from
|
||||
here. It records what is already implemented, what was found during review, and
|
||||
the concrete remaining TODOs in priority order.
|
||||
|
||||
## Goal
|
||||
|
||||
Render MP4 inside the Electron desktop app without the standalone HTTP render
|
||||
server, while keeping that standalone server working for the web/self-hosted
|
||||
build. Desktop uses Electron's own Chromium (a hidden render window) for frame
|
||||
capture and a bundled `ffmpeg` for H.264 encoding.
|
||||
|
||||
## Architecture (as built)
|
||||
|
||||
- **Vue renderer** owns project state and the export UI (`ExportDialog.vue`).
|
||||
For MP4 it calls `renderMp4OnDesktop()` when `window.ruwioDesktop.renderMp4`
|
||||
exists, otherwise falls back to the HTTP server path (`downloadFromServer`).
|
||||
- **Preload** (`electron/preload.cts`) exposes a narrow API on
|
||||
`window.ruwioDesktop`: `renderMp4`, `onRenderProgress`, `cancelRender`.
|
||||
- **Main process** (`electron/main.cts`) validates every IPC payload, shows a
|
||||
native save dialog, owns job IDs + `AbortController`s, and forwards progress
|
||||
events. IPC channels: `ruwio:render-mp4`, `ruwio:render-cancel`, progress on
|
||||
`ruwio:render-progress`.
|
||||
- **Render worker** (`electron/renderWorker.cts`) creates a hidden
|
||||
`BrowserWindow`, loads the packaged render page, seeks each frame, captures
|
||||
it, and pipes PNG bytes into `ffmpeg` stdin. Mirrors the server frame loop.
|
||||
- **Shared validation** (`shared/renderValidation.cjs`) is the single source of
|
||||
render limits, `playbackRange`, and `validateRenderRequest`, used by both
|
||||
`server/server.js` and `electron/main.cts`.
|
||||
- **ffmpeg resolution** (`electron/ffmpegPath.cts`): `FFMPEG_PATH` env →
|
||||
`process.resourcesPath/bin/<platform>/<ffmpeg>` (packaged) → `ffmpeg` on PATH.
|
||||
- **Staging script** (`scripts/stage-ffmpeg.cjs`) copies an ffmpeg binary into
|
||||
`bin/<platform>/` for packaging. Wired into `desktop:build`/`desktop:pack`
|
||||
via `stage:ffmpeg -- --optional`.
|
||||
- **Packaging** (`package.json` `build`): `extraResources` ships
|
||||
`server/render.html` and `bin/`; `files` ships `shared/renderValidation.cjs`.
|
||||
|
||||
## Key files
|
||||
|
||||
| Concern | File |
|
||||
| --- | --- |
|
||||
| Renderer export routing | `src/components/ExportDialog.vue` (`exportMp4Server`) |
|
||||
| Renderer desktop client | `src/lib/desktopRenderExport.ts` |
|
||||
| Desktop API types | `src/types/desktop.d.ts` |
|
||||
| Preload bridge | `electron/preload.cts` |
|
||||
| Main IPC + save dialog | `electron/main.cts` (`handleRenderMp4`, `handleRenderCancel`) |
|
||||
| Frame capture + ffmpeg | `electron/renderWorker.cts` |
|
||||
| ffmpeg path resolver | `electron/ffmpegPath.cts` |
|
||||
| Shared validation | `shared/renderValidation.cjs` |
|
||||
| Render page (capture surface) | `server/render.html` (`window.__animatrInit`, `window.__animatrSeek`) |
|
||||
| Standalone server | `server/server.js` |
|
||||
| ffmpeg staging | `scripts/stage-ffmpeg.cjs` |
|
||||
| Desktop render verification | `scripts/verify-*.cjs` + runner `scripts/verify-desktop-render.cjs` (`npm run test:render-desktop`) |
|
||||
|
||||
## Current verified state
|
||||
|
||||
Last verified: 2026-06-06 (Linux, DISPLAY available, system ffmpeg/ffprobe 7.1.4).
|
||||
|
||||
- `tsc -p tsconfig.electron.json` — passes.
|
||||
- `npm run build` (vue-tsc + vite) — passes.
|
||||
- `npm run test:render-validation` — 7/7 pass.
|
||||
- `npm run test:render-server` — pass.
|
||||
- `npm run test:render-server:e2e` — pass (server still produces a playable
|
||||
H.264 MP4; confirms the shared `render.html` change below is server-safe).
|
||||
- `npm run test:render-desktop` — **all 6 desktop checks pass.** New aggregated
|
||||
Electron harness suite (`scripts/verify-*.cjs`, runner
|
||||
`scripts/verify-desktop-render.cjs`) that drives the real worker headlessly:
|
||||
- capture: hidden-window `capturePage` yields advancing, non-blank frames.
|
||||
- export: the real `renderMp4Job` produces a 25-frame H.264 MP4 with motion.
|
||||
- cancel: mid-export abort rejects cleanly, no partial file.
|
||||
- timeout: the inactivity watchdog aborts a wedged render (~2s).
|
||||
- assets: local `data:` assets load; remote `http:` is blocked, no hang.
|
||||
- background: transparent regions flatten to white (not black).
|
||||
- `npm run desktop:pack` — succeeds. Packaged resources verified:
|
||||
`resources/server/render.html`, executable `resources/bin/linux/ffmpeg`
|
||||
(runs `-version`), and `app.asar` contains `dist-electron/{main,renderWorker,
|
||||
ffmpegPath}.cjs` + `shared/renderValidation.cjs`.
|
||||
|
||||
Findings #1–#6 are all resolved (see status tags below).
|
||||
|
||||
Not yet done:
|
||||
|
||||
- A manual GUI desktop export (dev or packaged) through the save dialog. The
|
||||
worker logic itself is proven end-to-end by `test:render-desktop`; only the
|
||||
click-through has not been recorded.
|
||||
- **`bin/linux/ffmpeg` is the dynamically-linked Debian system binary (~358 KB),
|
||||
not a redistributable static build.** Fine for local dev; NOT release-safe.
|
||||
- Windows/macOS ffmpeg binaries and licensing notes (Phase C #10–11).
|
||||
- Platform installer builds (AppImage / NSIS / DMG) and their manual checks.
|
||||
|
||||
## Review findings (ordered by severity)
|
||||
|
||||
### 1. CRITICAL — `capturePage()` on a non-shown, non-offscreen window — RESOLVED (does not reproduce)
|
||||
|
||||
Verified 2026-06-06 via `scripts/verify-capture.cjs`: a hidden, non-offscreen
|
||||
window's `capturePage()` returns real, non-blank frames whose contents advance
|
||||
frame-to-frame (linearly-stepping position hash over a moving rect). No
|
||||
offscreen mode was needed. The remaining staleness risk was finding #2.
|
||||
|
||||
Original analysis below.
|
||||
|
||||
|
||||
|
||||
`renderWorker.cts` creates the render window with `show: false` but **not**
|
||||
`webPreferences.offscreen: true`, then calls `webContents.capturePage()` once
|
||||
per frame. Hidden, non-offscreen windows are not composited by Chromium, so
|
||||
`capturePage()` can return blank or stale frames. This is the single biggest
|
||||
risk and cannot be confirmed by reading code — it needs a real export of a
|
||||
moving scene, frame-stepped, to confirm frames actually advance.
|
||||
|
||||
Fix direction if it reproduces:
|
||||
|
||||
- Switch the render window to `webPreferences.offscreen: true` and drive capture
|
||||
off the `paint` event (`win.webContents.on('paint', (e, dirty, image) => ...)`
|
||||
/ `beginFrameSubscription`), **or**
|
||||
- Show the window positioned off-screen (e.g. far negative coordinates) so the
|
||||
compositor paints it, then `capturePage`.
|
||||
|
||||
Offscreen mode also naturally solves finding #2.
|
||||
|
||||
### 2. HIGH — No wait for a compositor paint between seek and capture — RESOLVED (was real; fixed)
|
||||
|
||||
Confirmed real and reproducible: without a settle, `capturePage` intermittently
|
||||
returned the *previous* frame (stale/off-by-one) in ~1/3 of runs. Fixed in
|
||||
`server/render.html`: `window.__animatrSeek` now returns a Promise that resolves
|
||||
only after a double `requestAnimationFrame` (with a 1s anti-hang fallback), so
|
||||
both the desktop worker (`await executeJavaScript`) and the server (Puppeteer
|
||||
`await evaluate`) wait for a paint before capturing. 6/6 PASS after the fix;
|
||||
`test:render-server:e2e` confirms the server path is unaffected.
|
||||
|
||||
Original analysis below.
|
||||
|
||||
|
||||
|
||||
`__animatrSeek` runs a synchronous 2D-canvas `drawFrame`, but the worker calls
|
||||
`capturePage` immediately after `executeJavaScript(...seek)` with no wait for a
|
||||
paint. This risks an off-by-one frame or tearing. The original plan explicitly
|
||||
required "wait for the render function to settle."
|
||||
|
||||
Fix direction: await a double `requestAnimationFrame` inside the page (or, in
|
||||
offscreen mode, await the next `paint` event) before capturing.
|
||||
|
||||
### 3. MEDIUM — No render timeout in the desktop worker — RESOLVED
|
||||
|
||||
Added an inactivity watchdog in `renderWorker.cts`: a combined `AbortController`
|
||||
(`jobAbort`) trips on user cancel or after `RENDER_INACTIVITY_TIMEOUT_MS`
|
||||
(default 2 min, override via `RUWIO_RENDER_TIMEOUT_MS`) with no progress. The
|
||||
timer is re-armed on every `onProgress`, and each per-frame await is wrapped in
|
||||
`raceAbort(...)` so a wedged `executeJavaScript`/`capturePage` is interrupted,
|
||||
ffmpeg is killed, and the partial file removed. Verified by
|
||||
`scripts/verify-timeout.cjs` (aborts ~2s with message "render timed out").
|
||||
|
||||
Original analysis below.
|
||||
|
||||
|
||||
|
||||
`server/server.js` auto-aborts a stuck job via `JOB_TIMEOUT_MS`. The desktop
|
||||
worker has no watchdog, so a hung `executeJavaScript`/`capturePage` wedges the
|
||||
export with only manual cancel as escape.
|
||||
|
||||
Fix direction: add a timeout that aborts the job's `AbortController` after N
|
||||
seconds of no progress (reset the timer on each `onProgress`).
|
||||
|
||||
### 4. MEDIUM — Transparent window flattens to black under `yuv420p` — RESOLVED
|
||||
|
||||
Decision: match the server (flatten onto white). The render window is now
|
||||
`transparent: false` / `backgroundColor: '#ffffff'`, so transparent regions
|
||||
(e.g. `backgroundVideo` scenes, where `render.html` skips the bg fill) capture
|
||||
as white instead of black. Verified by `scripts/verify-bg.cjs` (corner pixel
|
||||
`rgb(255,255,255)`, zero near-black pixels).
|
||||
|
||||
Original analysis below.
|
||||
|
||||
|
||||
|
||||
The render window is `transparent: true`. Opaque scenes are fine because
|
||||
`render.html` fills `canvas.background` over the full frame, but any genuinely
|
||||
transparent region flattens onto **black** (vs the server's Puppeteer path,
|
||||
which flattens onto white via `omitBackground:false`). Decide this consciously:
|
||||
either make the render window background opaque, or document MP4 transparency
|
||||
behavior as black-filled.
|
||||
|
||||
### 5. LOW — Outbound network not blocked in the render window — RESOLVED
|
||||
|
||||
The render window now uses an isolated session partition (`ruwio-render`) with a
|
||||
`session.webRequest.onBeforeRequest` filter on `['*://*/*']` that cancels all
|
||||
http/https requests; local schemes (`file:`/`data:`/`blob:`) are unmatched and
|
||||
load normally. Verified by `scripts/verify-assets.cjs`: a local `data:` image
|
||||
draws (1600 px), a remote `http:` asset is blocked (0 px) and does not hang.
|
||||
|
||||
Original analysis below.
|
||||
|
||||
|
||||
|
||||
The render window denies `window.open` and `will-navigate`, but does not block
|
||||
network requests. A project asset with an `http(s)` `src` will be fetched by the
|
||||
hidden window. The original plan listed "block external network requests unless
|
||||
explicitly allowed." Low risk for a local app; close it for defense-in-depth via
|
||||
`session.webRequest.onBeforeRequest` or a CSP on the render page.
|
||||
|
||||
### 6. LOW — Desktop jobs are not serialized in main — RESOLVED
|
||||
|
||||
`main.cts` now guards with a module-level `renderActive` flag set at the top of
|
||||
`handleRenderMp4` (before the save dialog) and reset in a `finally`; a second
|
||||
concurrent IPC invoke throws "A render is already in progress". Mirrors the
|
||||
server's `MAX_ACTIVE_JOBS=1`.
|
||||
|
||||
Original analysis below.
|
||||
|
||||
|
||||
|
||||
`handleRenderMp4` will run concurrent jobs; only the `ExportDialog`
|
||||
`if (progress.value) return` guard prevents it from the UI, and IPC is directly
|
||||
invokable. The server enforces `MAX_ACTIVE_JOBS`. Add a one-line guard in main
|
||||
(reject or queue when a job is already active).
|
||||
|
||||
## What is already correct (do not re-do)
|
||||
|
||||
- Partial output file is removed on failure/cancel; ffmpeg is drained and the
|
||||
window closed in `finally`.
|
||||
- Cancel-before-jobId race handled in `desktopRenderExport.ts` via
|
||||
`pendingAbort`, including the window where main is blocked on the save dialog.
|
||||
- ffmpeg ENOENT surfaces the friendly `missingFfmpegMessage()` through both the
|
||||
`spawn`-error and `exit`-`!started` branches.
|
||||
- Packaging wiring is consistent: render page + `bin/` in `extraResources`,
|
||||
`shared/renderValidation.cjs` in `files`, resolvers read `process.resourcesPath`.
|
||||
- Validation parity: both server and main call the same `validateRenderRequest`;
|
||||
desktop uses frozen `DEFAULT_LIMITS`, server overlays env-configurable limits.
|
||||
- Filename sanitization (`sanitizeMp4Name`) strips path separators before the
|
||||
save dialog.
|
||||
|
||||
## Remaining TODOs
|
||||
|
||||
### Phase A — Prove frame capture works — DONE
|
||||
|
||||
Findings #1 and #2 resolved (see status tags above). The acceptance criteria
|
||||
(moving scene → correct MP4; clean cancel; no partial file) are covered by
|
||||
`npm run test:render-desktop` (capture/export/cancel checks).
|
||||
|
||||
### Phase B — Robustness — DONE
|
||||
|
||||
Findings #3, #4, #5, #6 all implemented and verified (status tags above).
|
||||
|
||||
### Phase C — Packaging + cross-platform (partly done)
|
||||
|
||||
8. `npm run desktop:pack` + packaged-resource verification — DONE for Linux
|
||||
(see "Current verified state").
|
||||
9. **TODO (manual GUI):** launch the unpacked app and export MP4 with no
|
||||
standalone server running. Click-through not yet recorded.
|
||||
10. **TODO:** stage reviewed *redistributable static* ffmpeg binaries for Linux
|
||||
(`bin/linux/ffmpeg`, replace the current dynamic system copy), Windows
|
||||
(`bin/win32/ffmpeg.exe`), and macOS (`bin/darwin/ffmpeg`).
|
||||
11. **TODO:** record ffmpeg release notes per bundled binary: source URL,
|
||||
upstream version, platform/arch, static vs dynamic, codecs (esp.
|
||||
`libx264`), GPL/LGPL obligations, attribution/license files to ship.
|
||||
|
||||
### Phase D — Verification matrix
|
||||
|
||||
Automated checks — DONE and recorded under "Current verified state"
|
||||
(`build`, `desktop:pack`, `test:render-validation`, `test:render-server`,
|
||||
`test:render-server:e2e`, `test:render-desktop`).
|
||||
|
||||
**TODO (manual GUI):**
|
||||
- desktop dev export of a small scene through the save dialog
|
||||
- desktop packaged export of a small scene
|
||||
- scenes with: image assets; text/non-default fonts; non-default fps;
|
||||
hidden layers skipped
|
||||
- server-mode MP4 export via the browser fallback
|
||||
- Platform builds: Linux AppImage, Windows NSIS, macOS DMG
|
||||
|
||||
## Manual check commands (reference)
|
||||
|
||||
Server fallback smoke (needs Chromium, ffmpeg, ffprobe):
|
||||
|
||||
```bash
|
||||
npm run test:render-server
|
||||
npm run test:render-server:e2e
|
||||
```
|
||||
|
||||
Browser fallback against the standalone server:
|
||||
|
||||
```bash
|
||||
# terminal 1
|
||||
cd server
|
||||
HOST=127.0.0.1 PORT=6000 RENDER_TOKEN="choose-a-long-random-token" npm start
|
||||
# terminal 2
|
||||
VITE_RENDER_SERVER=http://localhost:6000 VITE_RENDER_TOKEN="choose-a-long-random-token" npm run dev
|
||||
```
|
||||
|
||||
Stage a platform ffmpeg for packaging:
|
||||
|
||||
```bash
|
||||
npm run stage:ffmpeg -- --source /path/to/ffmpeg
|
||||
```
|
||||
|
||||
## Risks carried forward
|
||||
|
||||
- **Frame capture on hidden windows** (finding #1) is the gating risk for the
|
||||
whole desktop path. Resolve Phase A before any packaging work.
|
||||
- **Render drift** between `server/render.html` and `src/lib/render.ts`: keep
|
||||
validation/parity in mind; consider generating the render page from shared
|
||||
code later to avoid divergence.
|
||||
- **ffmpeg packaging varies by platform**: solve with explicit binary
|
||||
resolution (done) plus an unpacked-app smoke check before installer work.
|
||||
Reference in new issue
Block a user