Record the game view from a behavior into a video asset (#405)
game-maker / gate (push) Successful in 4m14s
game-maker / suite (push) Successful in 17m7s

A game can now film itself. A behavior starts and stops a take, and
the editor turns it into a video asset with the run's picture and the
sound the player hears. The exported game hands the file over as a
download, since there is no project to keep it in.

- New action nodes `Start recording` (Name, Resolution, FPS) and
  `Stop recording`. An empty name becomes `Recording <timestamp>`.
- Resolution is `Game view`, `720p` or `1080p`. For the fixed sizes
  the height is set and the width follows the camera.
- The picture is the camera only, without hints, overlays or the
  letterbox. It is drawn into a render texture of the target size,
  so a smaller window still gives full 1080p.
- `PictureReader` reads the texture back through pixel buffers with
  a fence. The editor never waits on the GPU, and a frame with all
  three buffers busy is dropped and counted.
- Nothing of the take reaches the visible canvas. A first version
  drew tiles into it and copied them out, and with the low-latency
  canvas that midway state flickered on screen.
- The sound is tapped behind the master limiter in the FX worklet
  and encoded as Opus.
- H.264 is chosen when the browser has it, VP9 otherwise. A small
  own Matroska writer packs both tracks, no new dependency. It names
  the frame rate and BT.709 so ffmpeg keeps both.
- A second `Start recording` is ignored with a warning in the log.
  A take goes on through `Load scene` and a restart, pauses with the
  editor and is saved when the run ends. It stops at 2 GB.
- The editor uploads the take on the existing video route and makes
  the asset once the run is over. The video tab gets a Download
  button for the converted MP4.
- `downloadBlob` in `src/ui` replaces the song editor's own copy.

Findings:
- `On start` runs before the game screen knows the world. A take
  therefore fixes its size on the first frame that has a camera.
- Without a frame rate in the file, ffmpeg turned 30 fps into 59.94.
- The worker's poster JPEG is off in colour for every upload (red
  255 comes out as 229, BT.601). Not touched here.

Tests:
- Unit recording + matroska: 21 green, plus the node catalog cases.
  The Matroska test reads the file back with its own EBML reader.
- e2e 67 (new) on melektaus-atx: ALL PASSED. 1080p30 from a
  1280x800 window gives a 1920x1080 MP4 with sound and no dropped
  frame. Every 4th pixel is checked, a green quarter catches an
  upside-down picture. A take ended by Esc is saved with the
  timestamp name.
- e2e 60j: the exported game downloads an `.mkv` with EBML magic.
- The flicker is gone on the user's machine without `?vsync`.
- Gate on melektaus-atx: CHECK EXIT 0, 299 files, 4586 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-11 21:11:05 +02:00
1 parent 36ffd9a61a
commit e191dc4b2e
35 files changed
+2211 -26

No files matched your search

+3
View File
@@ -6,6 +6,7 @@ import type { SaveArea } from '../core/graph/savedValues';
import type { SceneAsset } from '../core/model/Asset';
import { findScene, type Project } from '../core/model/Project';
import { mixOf } from '../core/music/mixer';
import type { RecordingSink } from '../core/recording/recording';
import type { AudioSink } from './systems/Audio';
import { addGameSystems, type BehaviorSystem, type TimeScaled } from './systems/BehaviorSystem';
import type { Input } from './systems/Input';
@@ -18,6 +19,8 @@ export interface RunServices {
physics: Rapier;
clock: TimeScaled;
saves: SaveArea;
/** Outlives a `Load scene` and a restart, so a recording goes on through both (#405). */
recorder?: RecordingSink;
}
/** What a scene holds of objects, which is all a run writes into it. */
+9 -1
View File
@@ -55,6 +55,7 @@ import { CollisionSystem } from './CollisionSystem';
import type { CollisionSource } from './solids';
import { AnimationSystem } from './AnimationSystem';
import type { Input } from './Input';
import type { RecordingSink } from '../../core/recording/recording';
import type { System } from './System';
export type { TimeScaled } from './behaviorContext';
@@ -115,6 +116,8 @@ export interface GameSystemOptions {
clock?: TimeScaled | null;
/** Where persistent Variables assets are read from at the start and written to as they change (#166). */
saves?: SaveArea | null;
/** Where `Start recording` records to (#405): only a played game has one. */
recorder?: RecordingSink | null;
}
/** Collisions and animations are ticked first, so behaviors see this frame's events. */
@@ -130,11 +133,12 @@ export function addGameSystems(
physics = null,
clock = null,
saves = null,
recorder = null,
}: GameSystemOptions = {},
): BehaviorSystem {
const collisions = physics ? new PhysicsSystem(physics) : new CollisionSystem();
const animations = new AnimationSystem();
const sources = { collisions, animations, audio, clock, saves };
const sources = { collisions, animations, audio, clock, saves, recorder };
const behaviors = new BehaviorSystem(project, input, sources, shouldRun);
if (carried) behaviors.carryOver(carried);
if (kept) {
@@ -155,6 +159,7 @@ export interface EventSources {
audio?: AudioSink | null;
clock?: TimeScaled | null;
saves?: SaveArea | null;
recorder?: RecordingSink | null;
}
/** A behavior's `runWhilePaused` switch: whether it is ticked while Pause game holds the world. */
@@ -198,6 +203,7 @@ export class BehaviorSystem implements System {
private readonly audio: AudioSink;
private readonly clock: TimeScaled | null;
private readonly saves: SaveArea | null;
private readonly recorder: RecordingSink | null;
/** What was last written to `saves`, so a frame that changed nothing writes nothing. */
private lastSaved = '';
@@ -216,6 +222,7 @@ export class BehaviorSystem implements System {
this.audio = sources.audio ?? SILENT_AUDIO;
this.clock = sources.clock ?? null;
this.saves = sources.saves ?? null;
this.recorder = sources.recorder ?? null;
}
/** The values `global` declarations share this run – read-only to callers, rebuilt by `start`. */
@@ -750,6 +757,7 @@ export class BehaviorSystem implements System {
collisions: this.collisions,
audio: this.audio,
clock: this.clock,
recorder: this.recorder,
shared: this.sharedState,
stateChanged: (machineId, from, to) => this.pending.push({ machineId, from, to, source: instance }),
});
+14
View File
@@ -42,6 +42,7 @@ import type { World } from '../World';
import type { AudioSink, NoteTiming } from './Audio';
import type { Instance } from './behaviorPulses';
import type { Input } from './Input';
import type { RecordingSink } from '../../core/recording/recording';
import { bodyOf, type CollisionSource, layerHolderOf, NOT_BLOCKED } from './solids';
export interface TimeScaled {
@@ -55,6 +56,8 @@ export interface ContextSources {
collisions: CollisionSource | null;
audio: AudioSink;
clock: TimeScaled | null;
/** Where a recording of the game view goes (#405); only a played game has one. */
recorder: RecordingSink | null;
shared: SharedState;
/** A graph moved one of this instance's machines; its events follow between exec flows. */
stateChanged(machineId: string, from: string, to: string): void;
@@ -265,9 +268,20 @@ export function behaviorContext(
setPaused: (paused) => {
world.paused = paused;
},
startRecording: (settings) => tell(world, run.recorder?.start(settings) ?? NO_RECORDER),
stopRecording: () => tell(world, run.recorder?.stop() ?? NO_RECORDER),
};
}
const NO_RECORDER = 'Recording works only while the game is played, not in the editor run';
/** A line of the host for the log, as the `Log` node writes one; nothing for an empty one. */
function tell(world: World, line: string): void {
if (!line) return;
world.log.push(line);
console.warn('[recording]', line);
}
/**
* What the graph asked for, as the sink understands it: the asset's own volume scaled by the
* node's, and its loop flag unless the node overrules it either way. The asset is what a sound