RUWIO/dev_docs/electron-e2e-and-capture-test-plan.md

Electron E2E & Capture Verification — Test Plan

Date: 2026-06-06

This plan defines how we verify (a) the capture/render fidelity of the desktop and server export paths and (b) all Electron-specific behaviour, and answers the open question of whether the browser build and the Electron build need separate test coverage.

It builds on what already exists — do not re-invent these:

• the in-app Test Runner (src/devtools/{testRunner.ts,tests.ts,TestOverlay.vue}),
• the Node/Electron render harnesses (scripts/verify-*.cjs),
• the server tests (server/{server.smoke,render.e2e}.test.cjs),
• the shared validation unit tests (shared/renderValidation.test.cjs),
• the manual checklist (electron-rendering-manual-checks.md).

TL;DR

1. Don't fork the suite per build. The frontend is ~99% shared and gated only on window.ruwioDesktop. Test the shared core once; cover the small, enumerable set of divergences with seam-targeted tests.
2. Capture fidelity is verified with one golden-scene matrix run through all three renderers (editor drawSceneFrame, desktop capturePage, server screenshot). The same matrix doubles as the render.ts ⇄ render.html parity check — the highest-value new test, because those two renderers are hand-duplicated and silently drift.
3. Unlock the ~130 existing in-app suites for CI by adding one headless auto-run hook, then host it under both a browser runner and an Electron runner. This is the cheapest large coverage win.
4. Add the missing layer: an Electron app-level e2e that launches the real app and drives menu commands / the export→save-dialog→file flow.

---

1. The system under test (what actually diverges)

1.1 Two renderer implementations (critical)

Renderer	Code	Used by
#1 TS drawSceneFrame	src/lib/render.ts	Live editor preview (CanvasStage.vue drawPreview), and every client export: GIF (gifExport.ts), APNG (apngExport.ts), WebM (videoExport.ts), still PNG/JPG (stillImageExport.ts), clipboard frame — all via src/lib/frameExport.ts renderFrames().
#2 vanilla-JS reimpl	inlined in server/render.html (drawFrame/sample/drawLayerNode/drawMaskShape), exposed as window.__animatrInit / window.__animatrSeek	Both MP4 backends: the Puppeteer server (server/server.js) and the Electron desktop worker (electron/renderWorker.cts).

These two are functionally the same renderer maintained twice. Renderer #1 is heavily covered by in-app pixel tests; Renderer #2 has only a non-blank/motion fingerprint today. A feature added to render.ts (new asset kind, easing, shadow/blur tweak, mask mode) that is not mirrored into render.html produces correct previews/GIFs but wrong MP4s with no failing test. Parity between #1 and #2 is the single biggest coverage gap.

1.2 Browser vs Electron — the full divergence set

Detection is only window.ruwioDesktop presence (set by electron/preload.cts). No user-agent sniffing anywhere. The divergences are:

#	Divergence	Where	Build	Test owner (proposed)
D1	MP4 backend switch hasDesktopRender() → desktop IPC vs server HTTP	src/lib/desktopRenderExport.ts:12, ExportDialog.vue (exportMp4Server)	both	unit (inject/remove bridge) + each backend's e2e
D2	Native save dialog (dialog.showSaveDialog); null return on cancel	electron/main.cts runRenderMp4	electron	Electron app e2e (stubbed dialog)
D3	18 IPC menu commands via onMenuCommand → runDesktopMenuCommand	electron/main.cts createMenu, src/App.vue, src/types/desktop.d.ts	electron	Electron app e2e
D4	Service worker registered only in browser	src/main.ts (!window.ruwioDesktop)	browser	jsdom/browser unit
D5	Bundled ffmpeg + capturePage pipeline	electron/renderWorker.cts, electron/ffmpegPath.cts	electron	capture/render e2e
D6	renderActive single-job guard (mirrors server MAX_ACTIVE_JOBS=1)	electron/main.cts handleRenderMp4	electron	Electron app e2e / worker e2e
D7	Canvas-larger-than-display hard error (#11)	electron/main.cts runRenderMp4	electron	Electron e2e / main unit
D8	Hardened render window — show:false, white bg (#4), backgroundThrottling:false (#12), isolated partition, http(s) request block (#5)	electron/renderWorker.cts createRenderWindow	electron	capture/render e2e (extend verify-assets/verify-bg)
D9	Navigation/external-link hardening (setWindowOpenHandler, will-navigate, shell.openExternal https-only)	electron/main.cts	electron	Electron app e2e (or main unit)
D10	darwin menu-role branches	electron/main.cts createMenu	electron (mac)	smoke / manual
D11	HiDPI frame normalization (#10), odd-dim pad (#13)	electron/renderWorker.cts, server/server.js	electron + server	capture/render e2e

Conclusion on the user's question: Yes, differentiate — but minimally. The shared frontend (stores, drawSceneFrame, UI components, all non-MP4 exports) is build-agnostic and is tested once. Only D1–D11 need build-specific tests, and each is a small, named seam. We do not maintain two parallel frontend suites.

---

2. Current coverage inventory & gaps

Layer A — in-app Test Runner (src/devtools/) — renderer, manual-only

• Custom suite/test/expect DSL (testRunner.ts), ~130 suites in tests.ts, results UI in TestOverlay.vue. Triggered by the dev menu test-runner command (App.vue, gated on import.meta.env.DEV).
• Runs in the live renderer against real Pinia store, Konva stage, and DOM.
• Covers: scene store, interpolation/sampleLayer, pixel-level drawSceneFrame (asset kinds, masks, groups, clip instances, shadow, blur, crop), client exports (GIF/APNG magic bytes + renderFrames counts/scaling/pixels), serialization round-trips, and a wide set of UI components + a live-DOM smoke suite.
• Gap: no headless invocation, no exit code, results only in UI/clipboard → invisible to CI. It is the largest existing asset and is currently un-automatable.

Layer B — Node/Electron harnesses — CI-shaped (exit codes)

• scripts/verify-*.cjs via npm run test:render-desktop (spawns the real electron binary): verify-capture (hidden window → render.html → capturePage → BGRA fingerprint: non-blank, right size, motion advances), verify-export (real worker → ffmpeg → ffprobe), verify-cancel, verify-timeout (watchdog), verify-assets (D8 network block), verify-bg (D8 white-flatten).
• server/render.e2e.test.cjs (full server lifecycle → SSE → download → ffprobe), server/server.smoke.test.cjs (routes/auth/validation), shared/renderValidation.test.cjs.
• Gaps: capture is verified for liveness/motion, not fidelity (no golden comparison, no breadth across asset kinds for the MP4 path); no render.ts⇄render.html parity; no throughput (#12) assertion; #10/#11/#13 fixes have no regression test; no app-level e2e (menus, dialog, renderActive); the in-app suite isn't reachable.

Cross-cutting gaps

• No CI (.github/workflows absent) — every layer is run by hand.
• No frontend unit framework (no vitest/jsdom/test-utils) — only node:test (.cjs) and the in-app runner.

---

3. Target architecture — the test pyramid

            ┌─────────────────────────────────────────────────────────┐
   L3  app  │ Electron app e2e (Playwright _electron): launch real app,│  NEW
            │ menu commands D3, export→save-dialog→file D2, renderActive│
            │ D6, oversize D7, nav-hardening D9, preload bridge shape   │
            ├─────────────────────────────────────────────────────────┤
   L2  e2e  │ Capture & render e2e (extend scripts/verify-*):          │  EXTEND
   render   │ • golden-scene matrix across 3 renderers (fidelity)      │
            │ • render.ts ⇄ render.html PARITY (§4)                    │
            │ • #10 resolution / #11 oversize / #12 throughput / #13   │
            │ • server render.e2e breadth parity                       │
            ├─────────────────────────────────────────────────────────┤
   L1  comp │ In-app Test Runner, run HEADLESS under two hosts:        │  UNLOCK
   /unit    │ • browser host (no ruwioDesktop)                         │
            │ • electron host (ruwioDesktop present)  → covers D1/D4   │
            ├─────────────────────────────────────────────────────────┤
   L0  unit │ node:test: shared/renderValidation (have), pure helpers, │  KEEP/ADD
            │ hasDesktopRender() seam, sanitizeMp4Name, ffmpegPath     │
            └─────────────────────────────────────────────────────────┘

The shared frontend is exercised once at L1; build divergences live at L0 seam tests and L2/L3.

---

4. Capture verification — the core design

Goal: prove that, for a known scene at a known time, every capture backend produces the correct pixels — not merely a non-blank, advancing frame.

4.1 Golden-scene matrix

A small set of deterministic scene JSONs (test/fixtures/scenes/*.json) covering the rendering feature matrix:

• asset kinds: rect, circle, text, polygon (open/closed, bezier), image (data: PNG)
• modifiers: shadow (opacity/blur/distance/angle), blur, stroke, opacity, rotation/scale
• compositing: top-level mask (in/out), grouped mask, multi-layer z-order via trackOrder
• clip instances (trim/startTime), background solid vs backgroundVideo/transparent
• a few sampled times per scene (e.g. t = 0, mid, end)

Each scene is renderer-agnostic (the same object the editor serializes via activeSceneClone() and ships to both backends).

4.2 Three capture backends, one fixture set

Run every (scene, time) through:

1. Editor renderer #1 — drawSceneFrame(ctx, scene, t, images) into an offscreen canvas → getImageData. (Runs in the L1 host; no Electron.)
2. Desktop capture — hidden BrowserWindow → render.html → __animatrInit/__animatrSeek(t) → capturePage(rect) → bitmap. (Extend scripts/verify-capture.cjs.)
3. Server capture — Puppeteer page → render.html → __animatrSeek(t) → page.screenshot(clip). (Extend server/render.e2e.test.cjs or a sibling.)

4.3 What we assert

• Parity (#1 ⇄ #2): backend-2/3 output must match backend-1 (drawSceneFrame) within a per-pixel tolerance. This is the render.ts ⇄ render.html drift guard. Treat drawSceneFrame (the unit-tested TS renderer) as the spec.
• Self-fidelity of capturePage: compare capturePage output to render.html's own canvas.toDataURL() for the same frame → proves the capture step is faithful (no scale-factor inflation #10, no stale/off-by-one frame, no size clipping #11).
• Golden regression: optionally commit reference PNGs; diff with tolerance to catch unintended renderer changes over time.

4.4 Determinism hazards (must be controlled or tests flake)

Hazard	Mitigation
Fonts — render.html & render.ts use sans-serif; glyphs differ per OS	bundle a single test webfont and force it in fixtures; or exclude text from pixel-exact parity (assert bounding-box/coverage instead); pin CI OS for text goldens
Anti-aliasing / GPU vs software	run capture with --disable-gpu; compare with tolerance (e.g. ≤2/255 per channel, ≤0.1% pixels over threshold)
Device scale factor	pin to 1 in tests; #10 normalization makes desktop match; server already pins deviceScaleFactor:1
Image decode / paint timing	always __animatrSeek (double-rAF settle) before capture; __animatrInit resolves after first decode
#10/#11 only manifest off scale-1 / large displays	mark those assertions; run once on a HiDPI box / large virtual display (note in manual-checks)

4.5 Fold the recent fixes in as regressions

• #10 resolution: verify-resolution.cjs — capture a scene, assert frame.getSize()/ffprobe dims exactly equal the canvas (no-op on scale-1 CI; real value on HiDPI — note it).
• #11 oversize: assert main.runRenderMp4 rejects a canvas > display with the clear error (drive via L3 app e2e, or extract the guard to a testable pure fn).
• #12 throughput: extend verify-export.cjs to time N frames and fail if per-frame wall-clock approaches the 1 s __animatrSeek fallback (guards the backgroundThrottling regression).
• #13 odd dims: render a 101×101 scene, ffprobe → expect a clean 102×102 MP4 (already proven by a throwaway harness; promote it to verify-odd-dims.cjs and add the server equivalent). Mirror in shared/renderValidation.test.cjs if we ever switch to reject-mode.

---

5. Unlocking the in-app Test Runner for CI (cheapest big win)

The ~130 suites already exercise the shared frontend. Make them headless:

1. Auto-run hook. On boot, if location.search has ?__runtests=1 (or import.meta.env.MODE === 'test'), after mount call runAll(registerAllTests) and publish results:

   // src/devtools/headless.ts (new)
   const results = await runAll(registerAllTests, { delayMs: 0 })
   ;(window as any).__ruwioTestResults = results
   document.body.setAttribute('data-tests-done',
     results.some(r => !r.ok) ? 'fail' : 'pass')
   

   Keep it dev/test-only (tree-shaken from prod as the overlay already is).
2. Two hosts, one suite:
   • Browser host: serve the built/dev app under Playwright (Chromium), open ?__runtests=1, wait for data-tests-done, read __ruwioTestResults, set exit code. window.ruwioDesktop absent → exercises the browser branch (D1 server path selection, D4 SW gating present at module load).
   • Electron host: Playwright _electron launches the dev app, same hook, page.evaluate(() => window.__ruwioTestResults). window.ruwioDesktop present → exercises the electron branch.
3. Tagging divergent suites. A handful of suites assert build-specific behaviour; add a convention (e.g. suite name prefix desktop: / browser:) and let the headless hook skip the inapplicable set per host. Most suites are build-agnostic and run under both.

This turns the existing investment into the L1 layer for both builds with a single hook — and is itself the cleanest demonstration of "differentiate by host, not by duplicating tests."

Caveat: the runner mutates the live app and pins locale to en (it snapshots and restores store state). That is fine for a dedicated test page/run; do not point it at a user session.

---

6. The missing layer — Electron app-level e2e (L3)

Nothing today launches the real assembled app and drives the main↔renderer seam (menus, native dialog, IPC round-trips, window hardening). Proposed:

Tooling: Playwright _electron (electron.launch({ args: ['.'] })), because it (a) launches the packaged/dev main, (b) hands back the renderer as a Page for UI assertions, (c) can evaluate in both main and renderer, and (d) the same Playwright install serves the L1 browser host. (Alternative: wdio-electron-service — stronger native-API mocking, heavier. Decision in §9.)

Test-mode hook in electron/main.cts (guarded by RUWIO_E2E=1), so native, human-only surfaces become deterministic:

• Stub the save dialog: when RUWIO_E2E, runRenderMp4 uses a fixed temp path (or honours RUWIO_E2E_SAVE_PATH) instead of dialog.showSaveDialog. Lets the full export→encode→file flow run unattended (covers D2, D5).
• Fire menu commands programmatically: expose ipcMain.handle('ruwio:test-menu', cmd => sendAppCommand(cmd)) (or call the existing sendAppCommand) so the e2e can assert each of the 18 commands routes correctly in App.vue (covers D3).
• Keep the hook tiny and entirely behind the env flag.

L3 scenarios:

• Each menu command D3 produces its renderer effect (export dialog opens, undo/redo, play/pause, panel toggle, etc.).
• Export MP4 end-to-end via the stubbed dialog → file exists, plays (ffprobe) → D2/D5.
• Cancel mid-export → no partial at the final path (we already fixed #7; assert it at app level too).
• Second concurrent export rejected by renderActive → D6.
• Oversize canvas → clear error surfaced to the renderer → D7/#11.
• Preload bridge shape: window.ruwioDesktop exposes exactly the documented surface (platform, versions, onMenuCommand, renderMp4, onRenderProgress, cancelRender) and nothing more → contract test for preload.cts/desktop.d.ts.
• Navigation hardening D9: window.open/in-page nav to a non-allowlisted URL is denied (can assert via main-process evaluate of the handlers).

---

7. Browser-only & seam unit tests (L0)

• hasDesktopRender() switch (D1): unit test that injects a fake window.ruwioDesktop.renderMp4 and asserts MP4 export calls the desktop path; removes it and asserts the server path. Pure, fast, no Electron.
• Service-worker gating (D4): import src/main.ts logic under jsdom with/without window.ruwioDesktop and assert navigator.serviceWorker.register is called only in the browser case.
• sanitizeMp4Name, resolveFfmpegPath, playbackRange, validation limits: plain node:test (validation already covered; add the others).

These need a tiny jsdom + node:test (or vitest) setup — the first frontend unit harness in the repo. Keep it minimal; the bulk of component coverage stays in L1.

---

8. Deliverables & sequencing

Phased so each phase is independently valuable and CI-wireable.

Phase 1 — Regression-proof the fixes we just shipped (small, high value)

• scripts/verify-odd-dims.cjs (#13) + server odd-dims case; add to verify-desktop-render.cjs aggregator.
• scripts/verify-resolution.cjs (#10, no-op on scale-1 — documented).
• throughput assertion in verify-export.cjs (#12).
• npm run test:render-desktop picks these up automatically.

Phase 2 — Capture fidelity & renderer parity (the core ask)

• test/fixtures/scenes/*.json golden matrix (§4.1).
• scripts/verify-capture-fidelity.cjs — desktop capturePage vs drawSceneFrame vs render.html toDataURL, with the §4.4 tolerance/determinism controls.
• server-side sibling reusing the same fixtures (extend render.e2e.test.cjs).
• a shared pixel-diff util (test/lib/pixelDiff.cjs).
• Outcome: render.ts ⇄ render.html drift is now caught.

Phase 3 — Unlock the in-app suite headlessly

• src/devtools/headless.ts auto-run hook + boot wiring (§5).
• test/e2e/inapp.browser.spec.ts (Playwright Chromium) and test/e2e/inapp.electron.spec.ts (Playwright _electron).
• suite tagging convention for build-specific suites.

Phase 4 — Electron app e2e (L3)

• RUWIO_E2E test-mode hook in electron/main.cts (dialog stub + menu-fire IPC).
• test/e2e/app.electron.spec.ts covering D2/D3/D5/D6/D7/D9 + preload contract.

Phase 5 — L0 seam units + CI

• jsdom/node:test for D1/D4 + pure helpers.
• A root test script aggregating the CI-safe layers, and a first .github/workflows/test.yml (matrix: lint/build, node test:*, test:render-* with ffmpeg/Chromium, Playwright browser + electron jobs).
• Update electron-rendering-manual-checks.md to point at the new automated checks and list what still needs a HiDPI/macOS box (#10, D10).

Proposed package.json scripts (final shape)

"test": "npm run test:unit && npm run test:render-validation && npm run test:render-server && npm run test:capture-fidelity",
"test:unit": "node --test test/unit/*.test.cjs",
"test:capture-fidelity": "tsc -p tsconfig.electron.json && node scripts/verify-capture-fidelity.cjs",
"test:e2e:inapp": "playwright test test/e2e/inapp.*.spec.ts",
"test:e2e:app":   "playwright test test/e2e/app.electron.spec.ts"
// existing: test:render-validation, test:render-server[:e2e], test:render-desktop

---

9. Open decisions (need a call before building Phases 3–4)

1. App-e2e tool: Playwright _electron (recommended — one tool for browser + electron, renderer Page access) vs wdio-electron-service (better native-API mocking) vs a lighter spawn-electron-and-assert-IPC harness (no UI driving).
2. Golden images: commit reference PNGs (catches any renderer change, but needs a blessing workflow + pins CI OS for fonts) or parity-only (compare the three live renderers to each other, no committed baselines — less brittle, but can't catch a change that drifts all three together). Recommendation: parity-first, add goldens later for a curated non-text subset.
3. Text fidelity: bundle a test font for deterministic glyphs, or exempt text from pixel-exact parity (structural assertion only). Affects fixture design.
4. In-app runner in CI: accept that it mutates a live app on a dedicated test page (recommended) — or carve a subset that is side-effect-free for faster CI.
5. New frontend unit harness: introduce vitest + jsdom (richer, watch mode, Vue ecosystem) vs stay on node:test + jsdom (zero new deps). Recommendation: vitest, since L0/L1 will grow.

---

10. Out of scope / explicitly not doing

• Re-testing the shared, build-agnostic frontend twice. It is covered once at L1 and run under both hosts.
• Driving real native OS menus / native save dialogs in automation (brittle); we use the IPC menu-fire hook and the dialog stub instead, and keep one manual pass in electron-rendering-manual-checks.md for true native behaviour and macOS roles (D10).
• The offscreen-rendering rewrite (deferred per the fix-plan); if it lands later, the §4 capture matrix is exactly the harness that re-validates it.