# Electron E2E & Capture Verification — Test Plan Date: 2026-06-06 This plan defines how we verify (a) the **capture/render fidelity** of the desktop and server export paths and (b) **all Electron-specific behaviour**, and answers the open question of **whether the browser build and the Electron build need separate test coverage**. It builds on what already exists — do not re-invent these: - the in-app **Test Runner** (`src/devtools/{testRunner.ts,tests.ts,TestOverlay.vue}`), - the Node/Electron **render harnesses** (`scripts/verify-*.cjs`), - the **server** tests (`server/{server.smoke,render.e2e}.test.cjs`), - the **shared validation** unit tests (`shared/renderValidation.test.cjs`), - the manual checklist (`06_electron-rendering-manual-checks.md`). > **TL;DR** > 1. **Don't fork the suite per build.** The frontend is ~99% shared and gated > only on `window.ruwioDesktop`. Test the shared core **once**; cover the small, > enumerable set of divergences with **seam-targeted** tests. > 2. **Capture fidelity** is verified with one **golden-scene matrix** run through > all three renderers (editor `drawSceneFrame`, desktop `capturePage`, server > `screenshot`). The same matrix doubles as the **render.ts ⇄ render.html > parity** check — the highest-value new test, because those two renderers are > hand-duplicated and silently drift. > 3. **Unlock the ~130 existing in-app suites for CI** by adding one headless > auto-run hook, then host it under both a browser runner and an Electron > runner. This is the cheapest large coverage win. > 4. **Add the missing layer:** an Electron **app-level e2e** that launches the > real app and drives menu commands / the export→save-dialog→file flow. --- ## 1. The system under test (what actually diverges) ### 1.1 Two renderer implementations (critical) | Renderer | Code | Used by | | --- | --- | --- | | **#1 TS `drawSceneFrame`** | `src/lib/render.ts` | Live editor preview (`CanvasStage.vue` `drawPreview`), **and every client export**: GIF (`gifExport.ts`), APNG (`apngExport.ts`), WebM (`videoExport.ts`), still PNG/JPG (`stillImageExport.ts`), clipboard frame — all via `src/lib/frameExport.ts` `renderFrames()`. | | **#2 vanilla-JS reimpl** | inlined in `server/render.html` (`drawFrame`/`sample`/`drawLayerNode`/`drawMaskShape`), exposed as `window.__animatrInit` / `window.__animatrSeek` | **Both MP4 backends**: the Puppeteer **server** (`server/server.js`) *and* the Electron **desktop worker** (`electron/renderWorker.cts`). | These two are **functionally the same renderer maintained twice**. Renderer #1 is heavily covered by in-app pixel tests; Renderer #2 has only a non-blank/motion fingerprint today. A feature added to `render.ts` (new asset kind, easing, shadow/blur tweak, mask mode) that is not mirrored into `render.html` produces **correct previews/GIFs but wrong MP4s** with no failing test. **Parity between #1 and #2 is the single biggest coverage gap.** ### 1.2 Browser vs Electron — the full divergence set Detection is **only** `window.ruwioDesktop` presence (set by `electron/preload.cts`). No user-agent sniffing anywhere. The divergences are: | # | Divergence | Where | Build | Test owner (proposed) | | --- | --- | --- | --- | --- | | D1 | **MP4 backend switch** `hasDesktopRender()` → desktop IPC vs server HTTP | `src/lib/desktopRenderExport.ts:12`, `ExportDialog.vue` (`exportMp4Server`) | both | unit (inject/remove bridge) + each backend's e2e | | D2 | **Native save dialog** (`dialog.showSaveDialog`); `null` return on cancel | `electron/main.cts` `runRenderMp4` | electron | Electron app e2e (stubbed dialog) | | D3 | **18 IPC menu commands** via `onMenuCommand` → `runDesktopMenuCommand` | `electron/main.cts` `createMenu`, `src/App.vue`, `src/types/desktop.d.ts` | electron | Electron app e2e | | D4 | **Service worker** registered only in browser | `src/main.ts` (`!window.ruwioDesktop`) | browser | jsdom/browser unit | | D5 | **Bundled ffmpeg + capturePage** pipeline | `electron/renderWorker.cts`, `electron/ffmpegPath.cts` | electron | capture/render e2e | | D6 | **`renderActive` single-job guard** (mirrors server `MAX_ACTIVE_JOBS=1`) | `electron/main.cts` `handleRenderMp4` | electron | Electron app e2e / worker e2e | | D7 | **Canvas-larger-than-display** hard error (#11) | `electron/main.cts` `runRenderMp4` | electron | Electron e2e / main unit | | D8 | **Hardened render window** — `show:false`, white bg (#4), `backgroundThrottling:false` (#12), isolated partition, `http(s)` request block (#5) | `electron/renderWorker.cts` `createRenderWindow` | electron | capture/render e2e (extend `verify-assets`/`verify-bg`) | | D9 | **Navigation/external-link hardening** (`setWindowOpenHandler`, `will-navigate`, `shell.openExternal` https-only) | `electron/main.cts` | electron | Electron app e2e (or main unit) | | D10 | **`darwin` menu-role branches** | `electron/main.cts` `createMenu` | electron (mac) | smoke / manual | | D11 | **HiDPI frame normalization** (#10), **odd-dim pad** (#13) | `electron/renderWorker.cts`, `server/server.js` | electron + server | capture/render e2e | **Conclusion on the user's question:** *Yes, differentiate — but minimally.* The shared frontend (stores, `drawSceneFrame`, UI components, all non-MP4 exports) is build-agnostic and is tested **once**. Only D1–D11 need build-specific tests, and each is a small, named seam. We do **not** maintain two parallel frontend suites. --- ## 2. Current coverage inventory & gaps ### Layer A — in-app Test Runner (`src/devtools/`) — renderer, **manual-only** - Custom `suite`/`test`/`expect` DSL (`testRunner.ts`), ~130 suites in `tests.ts`, results UI in `TestOverlay.vue`. Triggered by the dev menu `test-runner` command (`App.vue`, gated on `import.meta.env.DEV`). - Runs **in the live renderer** against real Pinia store, Konva stage, and DOM. - Covers: scene store, interpolation/`sampleLayer`, **pixel-level `drawSceneFrame`** (asset kinds, masks, groups, clip instances, shadow, blur, crop), client exports (GIF/APNG magic bytes + `renderFrames` counts/scaling/pixels), serialization round-trips, and a wide set of UI components + a live-DOM smoke suite. - **Gap:** no headless invocation, **no exit code**, results only in UI/clipboard → invisible to CI. It is the largest existing asset and is currently un-automatable. ### Layer B — Node/Electron harnesses — **CI-shaped (exit codes)** - `scripts/verify-*.cjs` via `npm run test:render-desktop` (spawns the real `electron` binary): `verify-capture` (hidden window → `render.html` → `capturePage` → BGRA fingerprint: non-blank, right size, motion advances), `verify-export` (real worker → ffmpeg → ffprobe), `verify-cancel`, `verify-timeout` (watchdog), `verify-assets` (D8 network block), `verify-bg` (D8 white-flatten). - `server/render.e2e.test.cjs` (full server lifecycle → SSE → download → ffprobe), `server/server.smoke.test.cjs` (routes/auth/validation), `shared/renderValidation.test.cjs`. - **Gaps:** capture is verified for *liveness/motion*, **not fidelity** (no golden comparison, no breadth across asset kinds for the MP4 path); no render.ts⇄render.html parity; no throughput (#12) assertion; #10/#11/#13 fixes have no regression test; no app-level e2e (menus, dialog, `renderActive`); the in-app suite isn't reachable. ### Cross-cutting gaps - **No CI** (`.github/workflows` absent) — every layer is run by hand. - **No frontend unit framework** (no vitest/jsdom/test-utils) — only `node:test` (`.cjs`) and the in-app runner. --- ## 3. Target architecture — the test pyramid ``` ┌─────────────────────────────────────────────────────────┐ L3 app │ Electron app e2e (Playwright _electron): launch real app,│ NEW │ menu commands D3, export→save-dialog→file D2, renderActive│ │ D6, oversize D7, nav-hardening D9, preload bridge shape │ ├─────────────────────────────────────────────────────────┤ L2 e2e │ Capture & render e2e (extend scripts/verify-*): │ EXTEND render │ • golden-scene matrix across 3 renderers (fidelity) │ │ • render.ts ⇄ render.html PARITY (§4) │ │ • #10 resolution / #11 oversize / #12 throughput / #13 │ │ • server render.e2e breadth parity │ ├─────────────────────────────────────────────────────────┤ L1 comp │ In-app Test Runner, run HEADLESS under two hosts: │ UNLOCK /unit │ • browser host (no ruwioDesktop) │ │ • electron host (ruwioDesktop present) → covers D1/D4 │ ├─────────────────────────────────────────────────────────┤ L0 unit │ node:test: shared/renderValidation (have), pure helpers, │ KEEP/ADD │ hasDesktopRender() seam, sanitizeMp4Name, ffmpegPath │ └─────────────────────────────────────────────────────────┘ ``` The shared frontend is exercised once at **L1**; build divergences live at **L0 seam tests** and **L2/L3**. --- ## 4. Capture verification — the core design **Goal:** prove that, for a known scene at a known time, every capture backend produces the **correct** pixels — not merely a non-blank, advancing frame. ### 4.1 Golden-scene matrix A small set of deterministic scene JSONs (`test/fixtures/scenes/*.json`) covering the rendering feature matrix: - asset kinds: `rect`, `circle`, `text`, `polygon` (open/closed, bezier), `image` (`data:` PNG) - modifiers: shadow (opacity/blur/distance/angle), blur, stroke, opacity, rotation/scale - compositing: top-level mask (in/out), grouped mask, multi-layer z-order via `trackOrder` - clip instances (trim/startTime), background solid vs `backgroundVideo`/transparent - a few sampled times per scene (e.g. t = 0, mid, end) Each scene is **renderer-agnostic** (the same object the editor serializes via `activeSceneClone()` and ships to both backends). ### 4.2 Three capture backends, one fixture set Run every (scene, time) through: 1. **Editor renderer #1** — `drawSceneFrame(ctx, scene, t, images)` into an offscreen canvas → `getImageData`. (Runs in the L1 host; no Electron.) 2. **Desktop capture** — hidden `BrowserWindow` → `render.html` → `__animatrInit`/`__animatrSeek(t)` → `capturePage(rect)` → bitmap. (Extend `scripts/verify-capture.cjs`.) 3. **Server capture** — Puppeteer page → `render.html` → `__animatrSeek(t)` → `page.screenshot(clip)`. (Extend `server/render.e2e.test.cjs` or a sibling.) ### 4.3 What we assert - **Parity (#1 ⇄ #2):** backend-2/3 output must match backend-1 (`drawSceneFrame`) within a per-pixel tolerance. **This is the render.ts ⇄ render.html drift guard.** Treat `drawSceneFrame` (the unit-tested TS renderer) as the spec. - **Self-fidelity of `capturePage`:** compare `capturePage` output to render.html's own `canvas.toDataURL()` for the same frame → proves the *capture step* is faithful (no scale-factor inflation #10, no stale/off-by-one frame, no size clipping #11). - **Golden regression:** optionally commit reference PNGs; diff with tolerance to catch unintended renderer changes over time. ### 4.4 Determinism hazards (must be controlled or tests flake) | Hazard | Mitigation | | --- | --- | | **Fonts** — `render.html` & `render.ts` use `sans-serif`; glyphs differ per OS | bundle a single test webfont and force it in fixtures; **or** exclude `text` from pixel-exact parity (assert bounding-box/coverage instead); pin CI OS for text goldens | | **Anti-aliasing / GPU vs software** | run capture with `--disable-gpu`; compare with tolerance (e.g. ≤2/255 per channel, ≤0.1% pixels over threshold) | | **Device scale factor** | pin to 1 in tests; #10 normalization makes desktop match; server already pins `deviceScaleFactor:1` | | **Image decode / paint timing** | always `__animatrSeek` (double-rAF settle) before capture; `__animatrInit` resolves after first decode | | **#10/#11 only manifest off scale-1 / large displays** | mark those assertions; run once on a HiDPI box / large virtual display (note in manual-checks) | ### 4.5 Fold the recent fixes in as regressions - **#10 resolution:** `verify-resolution.cjs` — capture a scene, assert `frame.getSize()`/ffprobe dims **exactly equal** the canvas (no-op on scale-1 CI; real value on HiDPI — note it). - **#11 oversize:** assert `main.runRenderMp4` rejects a canvas > display with the clear error (drive via L3 app e2e, or extract the guard to a testable pure fn). - **#12 throughput:** extend `verify-export.cjs` to time N frames and **fail if per-frame wall-clock approaches the 1 s `__animatrSeek` fallback** (guards the `backgroundThrottling` regression). - **#13 odd dims:** render a 101×101 scene, ffprobe → expect a clean 102×102 MP4 (already proven by a throwaway harness; promote it to `verify-odd-dims.cjs` and add the server equivalent). Mirror in `shared/renderValidation.test.cjs` if we ever switch to reject-mode. --- ## 5. Unlocking the in-app Test Runner for CI (cheapest big win) The ~130 suites already exercise the shared frontend. Make them headless: 1. **Auto-run hook.** On boot, if `location.search` has `?__runtests=1` (or `import.meta.env.MODE === 'test'`), after mount call `runAll(registerAllTests)` and publish results: ```ts // src/devtools/headless.ts (new) const results = await runAll(registerAllTests, { delayMs: 0 }) ;(window as any).__ruwioTestResults = results document.body.setAttribute('data-tests-done', results.some(r => !r.ok) ? 'fail' : 'pass') ``` Keep it dev/test-only (tree-shaken from prod as the overlay already is). 2. **Two hosts, one suite:** - **Browser host:** serve the built/dev app under Playwright (Chromium), open `?__runtests=1`, wait for `data-tests-done`, read `__ruwioTestResults`, set exit code. `window.ruwioDesktop` absent → exercises the **browser** branch (D1 server path selection, D4 SW gating present at module load). - **Electron host:** Playwright `_electron` launches the dev app, same hook, `page.evaluate(() => window.__ruwioTestResults)`. `window.ruwioDesktop` present → exercises the **electron** branch. 3. **Tagging divergent suites.** A handful of suites assert build-specific behaviour; add a convention (e.g. suite name prefix `desktop:` / `browser:`) and let the headless hook skip the inapplicable set per host. Most suites are build-agnostic and run under both. This turns the existing investment into the L1 layer for **both** builds with a single hook — and is itself the cleanest demonstration of "differentiate by host, not by duplicating tests." > Caveat: the runner mutates the live app and pins locale to `en` (it snapshots and > restores store state). That is fine for a dedicated test page/run; do not point it > at a user session. --- ## 6. The missing layer — Electron app-level e2e (L3) Nothing today launches the **real assembled app** and drives the main↔renderer seam (menus, native dialog, IPC round-trips, window hardening). Proposed: **Tooling: Playwright `_electron`** (`electron.launch({ args: ['.'] })`), because it (a) launches the packaged/dev main, (b) hands back the renderer as a `Page` for UI assertions, (c) can `evaluate` in both main and renderer, and (d) the same Playwright install serves the L1 browser host. (Alternative: `wdio-electron-service` — stronger native-API mocking, heavier. Decision in §9.) **Test-mode hook in `electron/main.cts`** (guarded by `RUWIO_E2E=1`), so native, human-only surfaces become deterministic: - **Stub the save dialog:** when `RUWIO_E2E`, `runRenderMp4` uses a fixed temp path (or honours `RUWIO_E2E_SAVE_PATH`) instead of `dialog.showSaveDialog`. Lets the full export→encode→file flow run unattended (covers D2, D5). - **Fire menu commands programmatically:** expose `ipcMain.handle('ruwio:test-menu', cmd => sendAppCommand(cmd))` (or call the existing `sendAppCommand`) so the e2e can assert each of the 18 commands routes correctly in `App.vue` (covers D3). - Keep the hook tiny and entirely behind the env flag. **L3 scenarios:** - Each menu command D3 produces its renderer effect (export dialog opens, undo/redo, play/pause, panel toggle, etc.). - Export MP4 end-to-end via the stubbed dialog → file exists, plays (ffprobe) → D2/D5. - Cancel mid-export → no partial at the final path (we already fixed #7; assert it at app level too). - Second concurrent export rejected by `renderActive` → D6. - Oversize canvas → clear error surfaced to the renderer → D7/#11. - Preload bridge shape: `window.ruwioDesktop` exposes exactly the documented surface (`platform`, `versions`, `onMenuCommand`, `renderMp4`, `onRenderProgress`, `cancelRender`) and nothing more → contract test for `preload.cts`/`desktop.d.ts`. - Navigation hardening D9: `window.open`/in-page nav to a non-allowlisted URL is denied (can assert via main-process evaluate of the handlers). --- ## 7. Browser-only & seam unit tests (L0) - **`hasDesktopRender()` switch (D1):** unit test that injects a fake `window.ruwioDesktop.renderMp4` and asserts MP4 export calls the desktop path; removes it and asserts the server path. Pure, fast, no Electron. - **Service-worker gating (D4):** import `src/main.ts` logic under jsdom with/without `window.ruwioDesktop` and assert `navigator.serviceWorker.register` is called only in the browser case. - **`sanitizeMp4Name`, `resolveFfmpegPath`, `playbackRange`, validation limits:** plain `node:test` (validation already covered; add the others). These need a tiny **jsdom + node:test** (or vitest) setup — the first frontend unit harness in the repo. Keep it minimal; the bulk of component coverage stays in L1. --- ## 8. Deliverables & sequencing Phased so each phase is independently valuable and CI-wireable. **Phase 1 — Regression-proof the fixes we just shipped (small, high value)** - `scripts/verify-odd-dims.cjs` (#13) + server odd-dims case; add to `verify-desktop-render.cjs` aggregator. - `scripts/verify-resolution.cjs` (#10, no-op on scale-1 — documented). - throughput assertion in `verify-export.cjs` (#12). - `npm run test:render-desktop` picks these up automatically. **Phase 2 — Capture fidelity & renderer parity (the core ask)** - `test/fixtures/scenes/*.json` golden matrix (§4.1). - `scripts/verify-capture-fidelity.cjs` — desktop `capturePage` vs `drawSceneFrame` vs render.html `toDataURL`, with the §4.4 tolerance/determinism controls. - server-side sibling reusing the same fixtures (extend `render.e2e.test.cjs`). - a shared pixel-diff util (`test/lib/pixelDiff.cjs`). - **Outcome:** render.ts ⇄ render.html drift is now caught. **Phase 3 — Unlock the in-app suite headlessly** - `src/devtools/headless.ts` auto-run hook + boot wiring (§5). - `test/e2e/inapp.browser.spec.ts` (Playwright Chromium) and `test/e2e/inapp.electron.spec.ts` (Playwright `_electron`). - suite tagging convention for build-specific suites. **Phase 4 — Electron app e2e (L3)** - `RUWIO_E2E` test-mode hook in `electron/main.cts` (dialog stub + menu-fire IPC). - `test/e2e/app.electron.spec.ts` covering D2/D3/D5/D6/D7/D9 + preload contract. **Phase 5 — L0 seam units + CI** - jsdom/node:test for D1/D4 + pure helpers. - A root `test` script aggregating the CI-safe layers, and a first `.github/workflows/test.yml` (matrix: lint/build, node `test:*`, `test:render-*` with ffmpeg/Chromium, Playwright browser + electron jobs). - Update `06_electron-rendering-manual-checks.md` to point at the new automated checks and list what still needs a HiDPI/macOS box (#10, D10). ### Proposed `package.json` scripts (final shape) ```jsonc "test": "npm run test:unit && npm run test:render-validation && npm run test:render-server && npm run test:capture-fidelity", "test:unit": "node --test test/unit/*.test.cjs", "test:capture-fidelity": "tsc -p tsconfig.electron.json && node scripts/verify-capture-fidelity.cjs", "test:e2e:inapp": "playwright test test/e2e/inapp.*.spec.ts", "test:e2e:app": "playwright test test/e2e/app.electron.spec.ts" // existing: test:render-validation, test:render-server[:e2e], test:render-desktop ``` --- ## 9. Open decisions (need a call before building Phases 3–4) 1. **App-e2e tool:** Playwright `_electron` (recommended — one tool for browser + electron, renderer Page access) vs `wdio-electron-service` (better native-API mocking) vs a lighter spawn-electron-and-assert-IPC harness (no UI driving). 2. **Golden images:** commit reference PNGs (catches *any* renderer change, but needs a blessing workflow + pins CI OS for fonts) **or** parity-only (compare the three live renderers to each other, no committed baselines — less brittle, but can't catch a change that drifts all three together). Recommendation: **parity-first**, add goldens later for a curated non-text subset. 3. **Text fidelity:** bundle a test font for deterministic glyphs, or exempt `text` from pixel-exact parity (structural assertion only). Affects fixture design. 4. **In-app runner in CI:** accept that it mutates a live app on a dedicated test page (recommended) — or carve a subset that is side-effect-free for faster CI. 5. **New frontend unit harness:** introduce **vitest + jsdom** (richer, watch mode, Vue ecosystem) vs stay on **node:test + jsdom** (zero new deps). Recommendation: vitest, since L0/L1 will grow. --- ## 10. Out of scope / explicitly not doing - Re-testing the shared, build-agnostic frontend twice. It is covered once at L1 and run under both hosts. - Driving real native OS menus / native save dialogs in automation (brittle); we use the IPC menu-fire hook and the dialog stub instead, and keep one **manual** pass in `06_electron-rendering-manual-checks.md` for true native behaviour and macOS roles (D10). - The offscreen-rendering rewrite (deferred per the fix-plan); if it lands later, the §4 capture matrix is exactly the harness that re-validates it.