Play videos on sprites, converted on the server with clips (#402)

A video becomes an asset like a sound: uploaded once, converted to one
format by a worker of its own, cut into clips in its tab, and worn by a
sprite through its Texture row. Only Chrome is a target, without alpha.

- Server: POST ?r=videos queues the upload under the hash of its bytes,
  so the MP4's name is known at once. The worker (VideoWorker over
  VideoFormat) writes H.264/AAC, yuv420p, constant frame rate, a
  keyframe every 15 frames, at most 1920x1080, with a poster and the
  details beside it. A failed job says why and is tried again on the
  next upload.
- An upload may be up to 2 GB. It is streamed from the request to disk
  and hashed there, so php-fpm holds none of it in memory. The other
  routes keep their 256 MB.
- The worker runs in the container game-maker-video on stein2, a
  compose project of its own (tools/video/), and in CI beside the PHP
  server. The CI image has ffmpeg for it.
- Model: asset type video with size, length, frame rate and sound, and
  clips like a sound's. The clip helpers and commands serve both.
- A sprite's Texture row offers videos and clips after the pictures.
  resolvePicture names a playing video by a key, and the texture library
  hands it to a VideoLibrary with one player per key.
- The editor shows a placed video as a quiet preview. Play starts it
  over at its clip's start and lets it be heard through the master bus,
  a pause holds it, Esc makes it quiet again.
- Video tab: the picture, a ruler, the clip strip and a track on the
  Sound tab's time axis. A drag spans a clip, a handle moves an edge,
  Play plays the picked clip.
- The picker shows a video on its way with its progress and adopts an
  upload from it once it is ready. Dropping a video file does the same.

Decided on my own:

- Two sprites wearing the same clip share one player and show the same
  picture. Control per instance belongs to the follow-up with nodes.
- A video plays in real time, the game's time scale does not slow it.
- The container may use 3 of the 4 cores of stein2.

Findings:

- The first real upload (279 MB) got 413: the API's own ceiling of
  256 MB for every request, not nginx or the Brandmauer.
- Chrome has no picture of a video that never played, and a Pixi
  VideoSource drawn before the first frame stays empty for good. The
  texture is made in the first frame callback (openVideo).
- A player nobody asks for in 2 s is let go. The inspector's PREVIEW
  asked only on an edit, so with the sprite not on screen its texture
  was destroyed under it and the batcher threw on every frame. The
  preview now redraws every frame while it shows a video, like a Time
  shader. A let-go texture keeps an empty source instead of none.
- VideoSource.load() reloads the element and cuts off an earlier
  play(). Playing starts after loadeddata.

Tests:

- PHP videos.test.php: 68 checks, with the real ffmpeg
- 300 MB upload through Brandmauer and nginx: 201 in 21 s, php-fpm at
  13.7 MB RSS at most
- Unit videos.test.ts: 20 tests
- e2e 66 (stein2, atx was off): ALL PASSED, 18 checks; e2e 04 with the
  new asset count green
- Gate (atx): 294 files, 4520 tests. serialize.test.ts used video as
  its example of an unknown type and now uses a made-up one.
- Counter-checks: a keyframe gap of 30 fails the PHP check, a key that
  keeps the video's end fails 2 unit tests, a clip without its end
  fails "plays only its stretch" in e2e 66

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/viewport/VideoLibrary.ts

@@ -0,0 +1,259 @@
+/**
+ * The players behind every video a sprite wears (#402): one `<video>` per key (see `videoKey`), its
+ * picture a Pixi texture that Chrome refreshes with every decoded frame. Two sprites wearing the
+ * same clip share a player and show the same picture.
+ *
+ * In the editor a video plays without sound, as a preview. A run starts every player over at its
+ * clip's start and lets it be heard through the master bus; a pause holds them, and the end of the
+ * run makes them quiet previews again.
+ */
+import { Assets, Texture, VideoSource } from 'pixi.js';
+import { parseVideoKey, type VideoPlay } from '../core/model/video';
+import { FileLibrary } from './FileLibrary';
+import { soundLibrary } from './SoundLibrary';
+
+const VIDEO_BASE = '/game-maker/data/videos';
+/** How close to its end a clip counts as there: half a frame at 30 fps. */
+const END_SLACK = 1 / 60;
+/** A player nothing has asked for in this long is let go: no sprite wears it any more. */
+const IDLE_MS = 2000;
+
+interface Player {
+  play: VideoPlay;
+  element: HTMLVideoElement;
+  /** Made once the video has shown its first frame, see `openVideo`. */
+  texture: Texture | null;
+  /** The sound on its way into the master, made the first time a run lets it be heard. */
+  gain: GainNode | null;
+  gone: boolean;
+  /** When a sprite last asked for its picture, by `performance.now()`. */
+  asked: number;
+}
+
+export class VideoLibrary {
+  private readonly players = new Map<string, Player>();
+  private readonly listeners = new Set<() => void>();
+  private heard = false;
+  private paused = false;
+  private sweeper: ReturnType<typeof setInterval> | null = null;
+
+  /** Runs `fn` whenever a player has its first picture; returns the unsubscribe. */
+  onChange(fn: () => void): () => void {
+    this.listeners.add(fn);
+    return () => this.listeners.delete(fn);
+  }
+
+  /** The picture of a playing video, made on the first ask; null until it has one. */
+  get(key: string): Texture | null {
+    const player = this.players.get(key) ?? this.start(key);
+    if (!player) return null;
+    player.asked = performance.now();
+    return player.texture;
+  }
+
+  /** A run begins: every player from its start, heard. */
+  startRun(): void {
+    this.heard = true;
+    this.paused = false;
+    for (const player of this.players.values()) {
+      this.hear(player);
+      this.rewind(player);
+    }
+  }
+
+  /** The run is over: back to quiet previews, from their start. */
+  endRun(): void {
+    this.heard = false;
+    this.paused = false;
+    for (const player of this.players.values()) {
+      player.element.muted = true;
+      this.rewind(player);
+    }
+  }
+
+  setPaused(paused: boolean): void {
+    this.paused = paused;
+    for (const player of this.players.values())
+      if (paused) player.element.pause();
+      else void player.element.play().catch(() => {});
+  }
+
+  /** Lets every player go: a session that ends takes its videos along. */
+  clear(): void {
+    for (const player of this.players.values()) this.drop(player);
+    this.players.clear();
+    this.stopSweeping();
+  }
+
+  /** Drops the players nobody asked for lately; a paused run asks for nothing and keeps them. */
+  private sweep(): void {
+    if (this.paused) return;
+    const now = performance.now();
+    for (const [key, player] of this.players)
+      if (now - player.asked > IDLE_MS) {
+        this.drop(player);
+        this.players.delete(key);
+      }
+    if (this.players.size === 0) this.stopSweeping();
+  }
+
+  private stopSweeping(): void {
+    if (this.sweeper !== null) clearInterval(this.sweeper);
+    this.sweeper = null;
+  }
+
+  /** What each player is doing (test hook). */
+  debugInfo(): { key: string; time: number; paused: boolean; muted: boolean; ready: boolean }[] {
+    return [...this.players].map(([key, { element, texture }]) => ({
+      key,
+      time: element.currentTime,
+      paused: element.paused,
+      muted: element.muted,
+      ready: texture !== null,
+    }));
+  }
+
+  private start(key: string): Player | null {
+    const play = parseVideoKey(key);
+    if (!play) return null;
+    const player: Player = {
+      play,
+      element: openVideo(`${VIDEO_BASE}/${play.file}`, play.start, (texture) => {
+        if (player.gone) return texture.destroy(true);
+        player.texture = texture;
+        // Started even in a paused run, and held once there is a picture.
+        if (this.paused) player.element.pause();
+        this.notify();
+      }),
+      texture: null,
+      gain: null,
+      gone: false,
+      asked: performance.now(),
+    };
+    this.players.set(key, player);
+    this.sweeper ??= setInterval(() => this.sweep(), IDLE_MS);
+    this.watch(player);
+    if (this.heard) this.hear(player);
+    return player;
+  }
+
+  /**
+   * Sends a clip back to its start at its end, or holds it there when it does not repeat. A clip
+   * that runs to the end of the video is sent back when the video ends; the whole video loops by
+   * itself.
+   */
+  private watch(player: Player): void {
+    const { element, play } = player;
+    element.loop = play.loop && play.start === 0 && !Number.isFinite(play.end);
+    element.addEventListener('ended', () => {
+      if (player.gone || !play.loop) return;
+      element.currentTime = play.start;
+      void element.play().catch(() => {});
+    });
+    if (!Number.isFinite(play.end)) return;
+    const check = (_now: number, frame: VideoFrameCallbackMetadata) => {
+      if (player.gone) return;
+      if (frame.mediaTime >= play.end - END_SLACK || element.currentTime >= play.end) {
+        if (play.loop) element.currentTime = play.start;
+        else element.pause();
+      }
+      element.requestVideoFrameCallback(check);
+    };
+    element.requestVideoFrameCallback(check);
+  }
+
+  private hear(player: Player): void {
+    player.gain ??= soundLibrary.mediaInput(player.element);
+    if (player.gain) player.gain.gain.value = player.play.volume;
+    player.element.muted = false;
+  }
+
+  private rewind(player: Player): void {
+    player.element.currentTime = player.play.start;
+    if (!this.paused) void player.element.play().catch(() => {});
+  }
+
+  private drop(player: Player): void {
+    player.gone = true;
+    player.element.pause();
+    player.gain?.disconnect();
+    player.element.removeAttribute('src');
+    player.element.load();
+    // The texture itself stays whole: a wearer that has not synced since still draws it, and a
+    // destroyed one has no source and threw in the batcher. It shows nothing until the wearer asks
+    // again; only the video behind it is let go.
+    const texture = player.texture;
+    if (!texture) return;
+    const video = texture.source;
+    texture.source = Texture.EMPTY.source;
+    video.destroy();
+  }
+
+  private notify(): void {
+    for (const listener of this.listeners)
+      try {
+        listener();
+      } catch (error) {
+        console.error('[videos] a listener failed', error);
+      }
+  }
+}
+
+export const videoLibrary = new VideoLibrary();
+
+/**
+ * A `<video>` of `url`, started without sound at `at` seconds, and its picture as a texture once it
+ * has shown its first frame. Chrome has no picture of a video that never played, and a Pixi source
+ * drawn before it has one stays empty for good (measured with Pixi 8.20: playing and `update()`
+ * afterwards changed nothing, a source made after the first frame showed every frame). The caller
+ * holds the element where it wants it once `onPicture` has run.
+ */
+export function openVideo(url: string, at: number, onPicture: (texture: Texture) => void): HTMLVideoElement {
+  const element = document.createElement('video');
+  element.muted = true;
+  element.playsInline = true;
+  element.preload = 'auto';
+  element.src = url;
+  if (at > 0) element.currentTime = at;
+  element.addEventListener(
+    'loadeddata',
+    () => {
+      element.requestVideoFrameCallback(() =>
+        onPicture(new Texture({ source: new VideoSource({ resource: element, autoPlay: false }) })),
+      );
+      void element.play().catch(() => {});
+    },
+    { once: true },
+  );
+  return element;
+}
+
+/** The stills the worker cut from each video, for the picker (see `posterFileOf`). */
+class PosterLibrary extends FileLibrary<Texture> {
+  private readonly loaded = new Map<string, Texture>();
+
+  constructor() {
+    super(VIDEO_BASE, 'posters');
+  }
+
+  /** The still if it is in memory, otherwise null while it loads. */
+  load(file: string): Texture | null {
+    const texture = this.loaded.get(file);
+    if (!texture) this.startLoading(file);
+    return texture ?? null;
+  }
+
+  protected fetchFile(url: string): Promise<Texture> {
+    return Assets.load<Texture>(url);
+  }
+
+  protected keep(file: string, texture: Texture): void {
+    this.loaded.set(file, texture);
+  }
+
+  protected holds(file: string): boolean {
+    return this.loaded.has(file);
+  }
+}
+
+export const posterLibrary = new PosterLibrary();