Files
RUWIO/dev_docs/08_electron-e2e-and-capture-test-plan.md
T
2026-06-07 11:10:05 +02:00

367 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Electron E2E & Capture Verification — Test Plan
Date: 2026-06-06
This plan defines how we verify (a) the **capture/render fidelity** of the
desktop and server export paths and (b) **all Electron-specific behaviour**, and
answers the open question of **whether the browser build and the Electron build
need separate test coverage**.
It builds on what already exists — do not re-invent these:
- the in-app **Test Runner** (`src/devtools/{testRunner.ts,tests.ts,TestOverlay.vue}`),
- the Node/Electron **render harnesses** (`scripts/verify-*.cjs`),
- the **server** tests (`server/{server.smoke,render.e2e}.test.cjs`),
- the **shared validation** unit tests (`shared/renderValidation.test.cjs`),
- the manual checklist (`06_electron-rendering-manual-checks.md`).
> **TL;DR**
> 1. **Don't fork the suite per build.** The frontend is ~99% shared and gated
> only on `window.ruwioDesktop`. Test the shared core **once**; cover the small,
> enumerable set of divergences with **seam-targeted** tests.
> 2. **Capture fidelity** is verified with one **golden-scene matrix** run through
> all three renderers (editor `drawSceneFrame`, desktop `capturePage`, server
> `screenshot`). The same matrix doubles as the **render.ts ⇄ render.html
> parity** check — the highest-value new test, because those two renderers are
> hand-duplicated and silently drift.
> 3. **Unlock the ~130 existing in-app suites for CI** by adding one headless
> auto-run hook, then host it under both a browser runner and an Electron
> runner. This is the cheapest large coverage win.
> 4. **Add the missing layer:** an Electron **app-level e2e** that launches the
> real app and drives menu commands / the export→save-dialog→file flow.
---
## 1. The system under test (what actually diverges)
### 1.1 Two renderer implementations (critical)
| Renderer | Code | Used by |
| --- | --- | --- |
| **#1 TS `drawSceneFrame`** | `src/lib/render.ts` | Live editor preview (`CanvasStage.vue` `drawPreview`), **and every client export**: GIF (`gifExport.ts`), APNG (`apngExport.ts`), WebM (`videoExport.ts`), still PNG/JPG (`stillImageExport.ts`), clipboard frame — all via `src/lib/frameExport.ts` `renderFrames()`. |
| **#2 vanilla-JS reimpl** | inlined in `server/render.html` (`drawFrame`/`sample`/`drawLayerNode`/`drawMaskShape`), exposed as `window.__animatrInit` / `window.__animatrSeek` | **Both MP4 backends**: the Puppeteer **server** (`server/server.js`) *and* the Electron **desktop worker** (`electron/renderWorker.cts`). |
These two are **functionally the same renderer maintained twice**. Renderer #1
is heavily covered by in-app pixel tests; Renderer #2 has only a non-blank/motion
fingerprint today. A feature added to `render.ts` (new asset kind, easing,
shadow/blur tweak, mask mode) that is not mirrored into `render.html` produces
**correct previews/GIFs but wrong MP4s** with no failing test. **Parity between
#1 and #2 is the single biggest coverage gap.**
### 1.2 Browser vs Electron — the full divergence set
Detection is **only** `window.ruwioDesktop` presence (set by `electron/preload.cts`).
No user-agent sniffing anywhere. The divergences are:
| # | Divergence | Where | Build | Test owner (proposed) |
| --- | --- | --- | --- | --- |
| D1 | **MP4 backend switch** `hasDesktopRender()` → desktop IPC vs server HTTP | `src/lib/desktopRenderExport.ts:12`, `ExportDialog.vue` (`exportMp4Server`) | both | unit (inject/remove bridge) + each backend's e2e |
| D2 | **Native save dialog** (`dialog.showSaveDialog`); `null` return on cancel | `electron/main.cts` `runRenderMp4` | electron | Electron app e2e (stubbed dialog) |
| D3 | **18 IPC menu commands** via `onMenuCommand` → `runDesktopMenuCommand` | `electron/main.cts` `createMenu`, `src/App.vue`, `src/types/desktop.d.ts` | electron | Electron app e2e |
| D4 | **Service worker** registered only in browser | `src/main.ts` (`!window.ruwioDesktop`) | browser | jsdom/browser unit |
| D5 | **Bundled ffmpeg + capturePage** pipeline | `electron/renderWorker.cts`, `electron/ffmpegPath.cts` | electron | capture/render e2e |
| D6 | **`renderActive` single-job guard** (mirrors server `MAX_ACTIVE_JOBS=1`) | `electron/main.cts` `handleRenderMp4` | electron | Electron app e2e / worker e2e |
| D7 | **Canvas-larger-than-display** hard error (#11) | `electron/main.cts` `runRenderMp4` | electron | Electron e2e / main unit |
| D8 | **Hardened render window** — `show:false`, white bg (#4), `backgroundThrottling:false` (#12), isolated partition, `http(s)` request block (#5) | `electron/renderWorker.cts` `createRenderWindow` | electron | capture/render e2e (extend `verify-assets`/`verify-bg`) |
| D9 | **Navigation/external-link hardening** (`setWindowOpenHandler`, `will-navigate`, `shell.openExternal` https-only) | `electron/main.cts` | electron | Electron app e2e (or main unit) |
| D10 | **`darwin` menu-role branches** | `electron/main.cts` `createMenu` | electron (mac) | smoke / manual |
| D11 | **HiDPI frame normalization** (#10), **odd-dim pad** (#13) | `electron/renderWorker.cts`, `server/server.js` | electron + server | capture/render e2e |
**Conclusion on the user's question:** *Yes, differentiate — but minimally.* The
shared frontend (stores, `drawSceneFrame`, UI components, all non-MP4 exports) is
build-agnostic and is tested **once**. Only D1–D11 need build-specific tests, and
each is a small, named seam. We do **not** maintain two parallel frontend suites.
---
## 2. Current coverage inventory & gaps
### Layer A — in-app Test Runner (`src/devtools/`) — renderer, **manual-only**
- Custom `suite`/`test`/`expect` DSL (`testRunner.ts`), ~130 suites in `tests.ts`,
results UI in `TestOverlay.vue`. Triggered by the dev menu `test-runner` command
(`App.vue`, gated on `import.meta.env.DEV`).
- Runs **in the live renderer** against real Pinia store, Konva stage, and DOM.
- Covers: scene store, interpolation/`sampleLayer`, **pixel-level `drawSceneFrame`**
(asset kinds, masks, groups, clip instances, shadow, blur, crop), client exports
(GIF/APNG magic bytes + `renderFrames` counts/scaling/pixels), serialization
round-trips, and a wide set of UI components + a live-DOM smoke suite.
- **Gap:** no headless invocation, **no exit code**, results only in UI/clipboard
→ invisible to CI. It is the largest existing asset and is currently un-automatable.
### Layer B — Node/Electron harnesses — **CI-shaped (exit codes)**
- `scripts/verify-*.cjs` via `npm run test:render-desktop` (spawns the real
`electron` binary): `verify-capture` (hidden window → `render.html` →
`capturePage` → BGRA fingerprint: non-blank, right size, motion advances),
`verify-export` (real worker → ffmpeg → ffprobe), `verify-cancel`,
`verify-timeout` (watchdog), `verify-assets` (D8 network block), `verify-bg`
(D8 white-flatten).
- `server/render.e2e.test.cjs` (full server lifecycle → SSE → download → ffprobe),
`server/server.smoke.test.cjs` (routes/auth/validation), `shared/renderValidation.test.cjs`.
- **Gaps:** capture is verified for *liveness/motion*, **not fidelity** (no golden
comparison, no breadth across asset kinds for the MP4 path); no render.ts⇄render.html
parity; no throughput (#12) assertion; #10/#11/#13 fixes have no regression test;
no app-level e2e (menus, dialog, `renderActive`); the in-app suite isn't reachable.
### Cross-cutting gaps
- **No CI** (`.github/workflows` absent) — every layer is run by hand.
- **No frontend unit framework** (no vitest/jsdom/test-utils) — only `node:test`
(`.cjs`) and the in-app runner.
---
## 3. Target architecture — the test pyramid
```
┌─────────────────────────────────────────────────────────┐
L3 app │ Electron app e2e (Playwright _electron): launch real app,│ NEW
│ menu commands D3, export→save-dialog→file D2, renderActive│
│ D6, oversize D7, nav-hardening D9, preload bridge shape │
├─────────────────────────────────────────────────────────┤
L2 e2e │ Capture & render e2e (extend scripts/verify-*): │ EXTEND
render │ • golden-scene matrix across 3 renderers (fidelity) │
│ • render.ts ⇄ render.html PARITY (§4) │
│ • #10 resolution / #11 oversize / #12 throughput / #13 │
│ • server render.e2e breadth parity │
├─────────────────────────────────────────────────────────┤
L1 comp │ In-app Test Runner, run HEADLESS under two hosts: │ UNLOCK
/unit │ • browser host (no ruwioDesktop) │
│ • electron host (ruwioDesktop present) → covers D1/D4 │
├─────────────────────────────────────────────────────────┤
L0 unit │ node:test: shared/renderValidation (have), pure helpers, │ KEEP/ADD
│ hasDesktopRender() seam, sanitizeMp4Name, ffmpegPath │
└─────────────────────────────────────────────────────────┘
```
The shared frontend is exercised once at **L1**; build divergences live at **L0
seam tests** and **L2/L3**.
---
## 4. Capture verification — the core design
**Goal:** prove that, for a known scene at a known time, every capture backend
produces the **correct** pixels — not merely a non-blank, advancing frame.
### 4.1 Golden-scene matrix
A small set of deterministic scene JSONs (`test/fixtures/scenes/*.json`) covering
the rendering feature matrix:
- asset kinds: `rect`, `circle`, `text`, `polygon` (open/closed, bezier), `image` (`data:` PNG)
- modifiers: shadow (opacity/blur/distance/angle), blur, stroke, opacity, rotation/scale
- compositing: top-level mask (in/out), grouped mask, multi-layer z-order via `trackOrder`
- clip instances (trim/startTime), background solid vs `backgroundVideo`/transparent
- a few sampled times per scene (e.g. t = 0, mid, end)
Each scene is **renderer-agnostic** (the same object the editor serializes via
`activeSceneClone()` and ships to both backends).
### 4.2 Three capture backends, one fixture set
Run every (scene, time) through:
1. **Editor renderer #1** — `drawSceneFrame(ctx, scene, t, images)` into an
offscreen canvas → `getImageData`. (Runs in the L1 host; no Electron.)
2. **Desktop capture** — hidden `BrowserWindow` → `render.html` →
`__animatrInit`/`__animatrSeek(t)` → `capturePage(rect)` → bitmap.
(Extend `scripts/verify-capture.cjs`.)
3. **Server capture** — Puppeteer page → `render.html` → `__animatrSeek(t)` →
`page.screenshot(clip)`. (Extend `server/render.e2e.test.cjs` or a sibling.)
### 4.3 What we assert
- **Parity (#1 ⇄ #2):** backend-2/3 output must match backend-1 (`drawSceneFrame`)
within a per-pixel tolerance. **This is the render.ts ⇄ render.html drift guard.**
Treat `drawSceneFrame` (the unit-tested TS renderer) as the spec.
- **Self-fidelity of `capturePage`:** compare `capturePage` output to render.html's
own `canvas.toDataURL()` for the same frame → proves the *capture step* is faithful
(no scale-factor inflation #10, no stale/off-by-one frame, no size clipping #11).
- **Golden regression:** optionally commit reference PNGs; diff with tolerance to
catch unintended renderer changes over time.
### 4.4 Determinism hazards (must be controlled or tests flake)
| Hazard | Mitigation |
| --- | --- |
| **Fonts** — `render.html` & `render.ts` use `sans-serif`; glyphs differ per OS | bundle a single test webfont and force it in fixtures; **or** exclude `text` from pixel-exact parity (assert bounding-box/coverage instead); pin CI OS for text goldens |
| **Anti-aliasing / GPU vs software** | run capture with `--disable-gpu`; compare with tolerance (e.g. ≤2/255 per channel, ≤0.1% pixels over threshold) |
| **Device scale factor** | pin to 1 in tests; #10 normalization makes desktop match; server already pins `deviceScaleFactor:1` |
| **Image decode / paint timing** | always `__animatrSeek` (double-rAF settle) before capture; `__animatrInit` resolves after first decode |
| **#10/#11 only manifest off scale-1 / large displays** | mark those assertions; run once on a HiDPI box / large virtual display (note in manual-checks) |
### 4.5 Fold the recent fixes in as regressions
- **#10 resolution:** `verify-resolution.cjs` — capture a scene, assert
`frame.getSize()`/ffprobe dims **exactly equal** the canvas (no-op on scale-1 CI;
real value on HiDPI — note it).
- **#11 oversize:** assert `main.runRenderMp4` rejects a canvas > display with the
clear error (drive via L3 app e2e, or extract the guard to a testable pure fn).
- **#12 throughput:** extend `verify-export.cjs` to time N frames and **fail if
per-frame wall-clock approaches the 1 s `__animatrSeek` fallback** (guards the
`backgroundThrottling` regression).
- **#13 odd dims:** render a 101×101 scene, ffprobe → expect a clean 102×102 MP4
(already proven by a throwaway harness; promote it to `verify-odd-dims.cjs` and add
the server equivalent). Mirror in `shared/renderValidation.test.cjs` if we ever
switch to reject-mode.
---
## 5. Unlocking the in-app Test Runner for CI (cheapest big win)
The ~130 suites already exercise the shared frontend. Make them headless:
1. **Auto-run hook.** On boot, if `location.search` has `?__runtests=1` (or
`import.meta.env.MODE === 'test'`), after mount call
`runAll(registerAllTests)` and publish results:
```ts
// src/devtools/headless.ts (new)
const results = await runAll(registerAllTests, { delayMs: 0 })
;(window as any).__ruwioTestResults = results
document.body.setAttribute('data-tests-done',
results.some(r => !r.ok) ? 'fail' : 'pass')
```
Keep it dev/test-only (tree-shaken from prod as the overlay already is).
2. **Two hosts, one suite:**
- **Browser host:** serve the built/dev app under Playwright (Chromium), open
`?__runtests=1`, wait for `data-tests-done`, read `__ruwioTestResults`, set
exit code. `window.ruwioDesktop` absent → exercises the **browser** branch (D1
server path selection, D4 SW gating present at module load).
- **Electron host:** Playwright `_electron` launches the dev app, same hook,
`page.evaluate(() => window.__ruwioTestResults)`. `window.ruwioDesktop`
present → exercises the **electron** branch.
3. **Tagging divergent suites.** A handful of suites assert build-specific behaviour;
add a convention (e.g. suite name prefix `desktop:` / `browser:`) and let the
headless hook skip the inapplicable set per host. Most suites are build-agnostic
and run under both.
This turns the existing investment into the L1 layer for **both** builds with a
single hook — and is itself the cleanest demonstration of "differentiate by host,
not by duplicating tests."
> Caveat: the runner mutates the live app and pins locale to `en` (it snapshots and
> restores store state). That is fine for a dedicated test page/run; do not point it
> at a user session.
---
## 6. The missing layer — Electron app-level e2e (L3)
Nothing today launches the **real assembled app** and drives the main↔renderer
seam (menus, native dialog, IPC round-trips, window hardening). Proposed:
**Tooling: Playwright `_electron`** (`electron.launch({ args: ['.'] })`), because
it (a) launches the packaged/dev main, (b) hands back the renderer as a `Page` for
UI assertions, (c) can `evaluate` in both main and renderer, and (d) the same
Playwright install serves the L1 browser host. (Alternative: `wdio-electron-service`
— stronger native-API mocking, heavier. Decision in §9.)
**Test-mode hook in `electron/main.cts`** (guarded by `RUWIO_E2E=1`), so native,
human-only surfaces become deterministic:
- **Stub the save dialog:** when `RUWIO_E2E`, `runRenderMp4` uses a fixed temp path
(or honours `RUWIO_E2E_SAVE_PATH`) instead of `dialog.showSaveDialog`. Lets the
full export→encode→file flow run unattended (covers D2, D5).
- **Fire menu commands programmatically:** expose `ipcMain.handle('ruwio:test-menu',
cmd => sendAppCommand(cmd))` (or call the existing `sendAppCommand`) so the e2e can
assert each of the 18 commands routes correctly in `App.vue` (covers D3).
- Keep the hook tiny and entirely behind the env flag.
**L3 scenarios:**
- Each menu command D3 produces its renderer effect (export dialog opens, undo/redo,
play/pause, panel toggle, etc.).
- Export MP4 end-to-end via the stubbed dialog → file exists, plays (ffprobe) → D2/D5.
- Cancel mid-export → no partial at the final path (we already fixed #7; assert it at
app level too).
- Second concurrent export rejected by `renderActive` → D6.
- Oversize canvas → clear error surfaced to the renderer → D7/#11.
- Preload bridge shape: `window.ruwioDesktop` exposes exactly the documented surface
(`platform`, `versions`, `onMenuCommand`, `renderMp4`, `onRenderProgress`,
`cancelRender`) and nothing more → contract test for `preload.cts`/`desktop.d.ts`.
- Navigation hardening D9: `window.open`/in-page nav to a non-allowlisted URL is denied
(can assert via main-process evaluate of the handlers).
---
## 7. Browser-only & seam unit tests (L0)
- **`hasDesktopRender()` switch (D1):** unit test that injects a fake
`window.ruwioDesktop.renderMp4` and asserts MP4 export calls the desktop path;
removes it and asserts the server path. Pure, fast, no Electron.
- **Service-worker gating (D4):** import `src/main.ts` logic under jsdom with/without
`window.ruwioDesktop` and assert `navigator.serviceWorker.register` is called only
in the browser case.
- **`sanitizeMp4Name`, `resolveFfmpegPath`, `playbackRange`, validation limits:** plain
`node:test` (validation already covered; add the others).
These need a tiny **jsdom + node:test** (or vitest) setup — the first frontend unit
harness in the repo. Keep it minimal; the bulk of component coverage stays in L1.
---
## 8. Deliverables & sequencing
Phased so each phase is independently valuable and CI-wireable.
**Phase 1 — Regression-proof the fixes we just shipped (small, high value)**
- `scripts/verify-odd-dims.cjs` (#13) + server odd-dims case; add to `verify-desktop-render.cjs` aggregator.
- `scripts/verify-resolution.cjs` (#10, no-op on scale-1 — documented).
- throughput assertion in `verify-export.cjs` (#12).
- `npm run test:render-desktop` picks these up automatically.
**Phase 2 — Capture fidelity & renderer parity (the core ask)**
- `test/fixtures/scenes/*.json` golden matrix (§4.1).
- `scripts/verify-capture-fidelity.cjs` — desktop `capturePage` vs `drawSceneFrame`
vs render.html `toDataURL`, with the §4.4 tolerance/determinism controls.
- server-side sibling reusing the same fixtures (extend `render.e2e.test.cjs`).
- a shared pixel-diff util (`test/lib/pixelDiff.cjs`).
- **Outcome:** render.ts ⇄ render.html drift is now caught.
**Phase 3 — Unlock the in-app suite headlessly**
- `src/devtools/headless.ts` auto-run hook + boot wiring (§5).
- `test/e2e/inapp.browser.spec.ts` (Playwright Chromium) and
`test/e2e/inapp.electron.spec.ts` (Playwright `_electron`).
- suite tagging convention for build-specific suites.
**Phase 4 — Electron app e2e (L3)**
- `RUWIO_E2E` test-mode hook in `electron/main.cts` (dialog stub + menu-fire IPC).
- `test/e2e/app.electron.spec.ts` covering D2/D3/D5/D6/D7/D9 + preload contract.
**Phase 5 — L0 seam units + CI**
- jsdom/node:test for D1/D4 + pure helpers.
- A root `test` script aggregating the CI-safe layers, and a first
`.github/workflows/test.yml` (matrix: lint/build, node `test:*`,
`test:render-*` with ffmpeg/Chromium, Playwright browser + electron jobs).
- Update `06_electron-rendering-manual-checks.md` to point at the new automated checks
and list what still needs a HiDPI/macOS box (#10, D10).
### Proposed `package.json` scripts (final shape)
```jsonc
"test": "npm run test:unit && npm run test:render-validation && npm run test:render-server && npm run test:capture-fidelity",
"test:unit": "node --test test/unit/*.test.cjs",
"test:capture-fidelity": "tsc -p tsconfig.electron.json && node scripts/verify-capture-fidelity.cjs",
"test:e2e:inapp": "playwright test test/e2e/inapp.*.spec.ts",
"test:e2e:app": "playwright test test/e2e/app.electron.spec.ts"
// existing: test:render-validation, test:render-server[:e2e], test:render-desktop
```
---
## 9. Open decisions (need a call before building Phases 3–4)
1. **App-e2e tool:** Playwright `_electron` (recommended — one tool for browser +
electron, renderer Page access) vs `wdio-electron-service` (better native-API
mocking) vs a lighter spawn-electron-and-assert-IPC harness (no UI driving).
2. **Golden images:** commit reference PNGs (catches *any* renderer change, but
needs a blessing workflow + pins CI OS for fonts) **or** parity-only (compare the
three live renderers to each other, no committed baselines — less brittle, but
can't catch a change that drifts all three together). Recommendation: **parity-first**,
add goldens later for a curated non-text subset.
3. **Text fidelity:** bundle a test font for deterministic glyphs, or exempt `text`
from pixel-exact parity (structural assertion only). Affects fixture design.
4. **In-app runner in CI:** accept that it mutates a live app on a dedicated test page
(recommended) — or carve a subset that is side-effect-free for faster CI.
5. **New frontend unit harness:** introduce **vitest + jsdom** (richer, watch mode,
Vue ecosystem) vs stay on **node:test + jsdom** (zero new deps). Recommendation:
vitest, since L0/L1 will grow.
---
## 10. Out of scope / explicitly not doing
- Re-testing the shared, build-agnostic frontend twice. It is covered once at L1 and
run under both hosts.
- Driving real native OS menus / native save dialogs in automation (brittle); we use
the IPC menu-fire hook and the dialog stub instead, and keep one **manual** pass in
`06_electron-rendering-manual-checks.md` for true native behaviour and macOS roles (D10).
- The offscreen-rendering rewrite (deferred per the fix-plan); if it lands later,
the §4 capture matrix is exactly the harness that re-validates it.