meleki/feuerwerk
1
0 Fork 0
Code Issues 33 Actions Projects 1 Wiki Activity

Record the game view from a behavior into a video asset (#405)
game-maker / gate (push) Successful in 4m14s
Details
game-maker / suite (push) Successful in 17m7s
Details

Browse Source 
A game can now film itself. A behavior starts and stops a take, and
the editor turns it into a video asset with the run's picture and the
sound the player hears. The exported game hands the file over as a
download, since there is no project to keep it in.

- New action nodes `Start recording` (Name, Resolution, FPS) and
  `Stop recording`. An empty name becomes `Recording <timestamp>`.
- Resolution is `Game view`, `720p` or `1080p`. For the fixed sizes
  the height is set and the width follows the camera.
- The picture is the camera only, without hints, overlays or the
  letterbox. It is drawn into a render texture of the target size,
  so a smaller window still gives full 1080p.
- `PictureReader` reads the texture back through pixel buffers with
  a fence. The editor never waits on the GPU, and a frame with all
  three buffers busy is dropped and counted.
- Nothing of the take reaches the visible canvas. A first version
  drew tiles into it and copied them out, and with the low-latency
  canvas that midway state flickered on screen.
- The sound is tapped behind the master limiter in the FX worklet
  and encoded as Opus.
- H.264 is chosen when the browser has it, VP9 otherwise. A small
  own Matroska writer packs both tracks, no new dependency. It names
  the frame rate and BT.709 so ffmpeg keeps both.
- A second `Start recording` is ignored with a warning in the log.
  A take goes on through `Load scene` and a restart, pauses with the
  editor and is saved when the run ends. It stops at 2 GB.
- The editor uploads the take on the existing video route and makes
  the asset once the run is over. The video tab gets a Download
  button for the converted MP4.
- `downloadBlob` in `src/ui` replaces the song editor's own copy.

Findings:
- `On start` runs before the game screen knows the world. A take
  therefore fixes its size on the first frame that has a camera.
- Without a frame rate in the file, ffmpeg turned 30 fps into 59.94.
- The worker's poster JPEG is off in colour for every upload (red
  255 comes out as 229, BT.601). Not touched here.

Tests:
- Unit recording + matroska: 21 green, plus the node catalog cases.
  The Matroska test reads the file back with its own EBML reader.
- e2e 67 (new) on melektaus-atx: ALL PASSED. 1080p30 from a
  1280x800 window gives a 1920x1080 MP4 with sound and no dropped
  frame. Every 4th pixel is checked, a green quarter catches an
  upside-down picture. A take ended by Esc is saved with the
  timestamp name.
- e2e 60j: the exported game downloads an `.mkv` with EBML magic.
- The flicker is gone on the user's machine without `?vsync`.
- Gate on melektaus-atx: CHECK EXIT 0, 299 files, 4586 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-11 21:11:05 +02:00
1 parent 36ffd9a61a
commit e191dc4b2e
35 files changed
+2211 -26
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/viewport/GameRecorder.ts
+532
View File
@@ -0,0 +1,532 @@
import { Matrix, type RenderTexture } from 'pixi.js';
import { preSkipOf } from '../core/music/ogg';
import { AVC_CODEC, colourOf, MatroskaWriter, VP9_CODEC } from '../core/recording/matroska';
import {
avcCodec,
FrameGate,
isKeyFrame,
pictureTransform,
RECORDING_MOST_BYTES,
RecordingClock,
recordingName,
type RecordingSettings,
type RecordingSink,
recordingSize,
videoBitrate,
vp9Codec,
} from '../core/recording/recording';
import { ui } from '../ui';
import { PictureReader } from './PictureReader';
import { soundLibrary } from './SoundLibrary';
import type { PictureArea } from './WorldViewport';
const OPUS_BITRATE = 160_000;
/** Frames the video encoder may have waiting before the next is dropped instead of piling up. */
const MOST_QUEUED = 6;
/** Encoded bytes held as loose pieces before they are folded into a Blob, out of the page's heap. */
const FOLD_BYTES = 16 * 1024 * 1024;
/** How long a stop waits for the last sound the worklet holds; a suspended context sends none. */
const LAST_SOUND_MS = 300;
/** How long a stop waits for the frames still on their way back from the GPU. */
const LAST_FRAMES_MS = 500;
/** The view a recording takes its frames from (`WorldViewport`). */
export interface RecordingView {
picture(): { shown: PictureArea; whole: PictureArea } | null;
drawPicture(transform: Matrix, target: RenderTexture): void;
}
/** A recording once it is made. */
export interface MadeRecording {
name: string;
blob: Blob;
durationMs: number;
width: number;
height: number;
}
/** Where the recorder looks, and what it tells. */
export interface RecordingHost {
view(): RecordingView | null;
/** A line for the log of the run that plays. */
log(line: string): void;
saved(recording: MadeRecording): void;
}
/**
* `waiting` for a picture to take its size from: `On start` runs before the view is handed the
* world, and a game may switch its camera on later.
*/
type TakeState = 'waiting' | 'starting' | 'recording' | 'finishing' | 'done' | 'failed';
/** What a check reads of a recording (test hook). */
export interface TakeInfo {
name: string;
state: TakeState;
width: number;
height: number;
fps: number;
codec: string;
sound: boolean;
frames: number;
dropped: number;
soundFrames: number;
/** From the first frame to the end of the last, in ms. */
durationMs: number;
/** What the video encoder said its colour is, null while it said nothing. */
colour: VideoColorSpaceInit | null;
bytes: number;
failure: string | null;
}
/**
* Records the game view (#405): `Start recording` and `Stop recording` reach it through the run, the
* host hands it the view and draws a frame for it each tick (`capture`). One recording at a time; it
* outlives the world, so a `Load scene` or a restart goes on in the same file.
*/
export class GameRecorder implements RecordingSink {
private take: Take | null = null;
private paused = false;
/** Every recording of this page, newest last (test hook). */
private readonly takes: Take[] = [];
constructor(private readonly host: RecordingHost) {}
get isRecording(): boolean {
return this.take !== null;
}
start(settings: RecordingSettings): string {
if (this.take)
return `Start recording is ignored: "${this.take.name}" is recording already – Stop recording ends it`;
const take = new Take(recordingName(settings.name, new Date()), settings, this.host, () => {
if (this.take === take) this.take = null;
});
this.take = take;
this.takes.push(take);
if (this.paused) take.setPaused(true);
return '';
}
stop(): string {
if (!this.take) return 'Stop recording: nothing is recording';
this.take.finish('');
this.take = null;
return '';
}
/** The editor holds the run: the recording holds with it, and its time leaves the pause out. */
setPaused(paused: boolean): void {
this.paused = paused;
this.take?.setPaused(paused);
}
/** The view has drawn this frame: the recording takes it, if it is due. */
capture(): void {
const view = this.take && this.host.view();
if (view) this.take!.capture(view);
}
/** The run is over: a recording still going is saved, not lost. */
end(): void {
this.stop();
}
info(): { recording: boolean; takes: TakeInfo[] } {
return { recording: this.isRecording, takes: this.takes.map((take) => take.info()) };
}
}
/** One recording, from its encoders to its file. */
class Take {
private state: TakeState = 'waiting';
private size = { w: 0, h: 0 };
private readonly clock = new RecordingClock();
private readonly gate: FrameGate;
private video: VideoEncoder | null = null;
private audio: AudioEncoder | null = null;
private tap: { node: AudioWorkletNode; rate: number; release(): void } | null = null;
private writer: MatroskaWriter | null = null;
private codec = '';
private videoHead: Uint8Array | null = null;
private colour: VideoColorSpaceInit | null = null;
private audioHead: Uint8Array | null = null;
private reader: PictureReader | null = null;
private toldNoCamera = false;
private folded: Blob[] = [];
private loose: Uint8Array[] = [];
private looseBytes = 0;
private frames = 0;
private dropped = 0;
private lastMs = 0;
private soundFrames = 0;
/** The recording's time of the first sample heard, in ms; null before it. */
private soundStart: number | null = null;
private soundEnded: (() => void) | null = null;
private stopAsked = false;
private paused = false;
private failure: string | null = null;
constructor(
readonly name: string,
private readonly settings: RecordingSettings,
private readonly host: RecordingHost,
private readonly gone: () => void,
) {
this.gate = new FrameGate(settings.fps);
}
/**
* Takes the size from the first picture there is and sets up the encoders; until they are ready
* frames are not taken, and a stop asked meanwhile ends it after.
*/
private open(picture: { shown: PictureArea; whole: PictureArea }): void {
const ratio = ui().app.renderer.resolution;
const device = (area: PictureArea) => ({ w: area.w * ratio, h: area.h * ratio });
this.size = recordingSize(this.settings.resolution, device(picture.shown), device(picture.whole));
this.state = 'starting';
try {
this.reader = new PictureReader(this.size.w, this.size.h);
} catch (error) {
this.fail(error instanceof Error ? error.message : String(error));
return;
}
void this.openEncoders().then(
() => {
if (this.state !== 'starting') return;
this.state = 'recording';
if (this.stopAsked) this.finish('');
},
(error: unknown) => this.fail(error instanceof Error ? error.message : String(error)),
);
}
private async openEncoders(): Promise<void> {
if (typeof VideoEncoder === 'undefined')
throw new Error('this browser cannot encode video – recording needs Chrome');
const { w, h } = this.size;
const fps = this.settings.fps;
const base = { width: w, height: h, framerate: fps, bitrate: videoBitrate(w, h, fps) };
const configs: VideoEncoderConfig[] = [
{ ...base, codec: avcCodec(w, h, fps), avc: { format: 'avc' }, latencyMode: 'realtime' },
{ ...base, codec: vp9Codec(w, h), latencyMode: 'realtime' },
];
let config: VideoEncoderConfig | null = null;
for (const candidate of configs)
if (
await VideoEncoder.isConfigSupported(candidate).then(
(s) => s.supported === true,
() => false,
)
) {
config = candidate;
break;
}
if (!config) throw new Error(`this browser cannot encode ${w}x${h} at ${fps} fps`);
this.codec = config.codec;
await this.openSound();
const writer = new MatroskaWriter(this.audio ? 2 : 1);
this.writer = writer;
const video = new VideoEncoder({
output: (chunk, meta) => {
const head = meta?.decoderConfig?.description;
if (head && !this.videoHead) this.videoHead = copyOf(head);
this.colour ??= meta?.decoderConfig?.colorSpace ?? null;
writer.add(1, chunk.timestamp, chunk.type === 'key', chunkBytes(chunk));
this.collect();
},
error: (error) => this.fail(error.message),
});
video.configure(config);
this.video = video;
}
/** The master behind the limiter, as Opus; without it the recording goes on silent and says so. */
private async openSound(): Promise<void> {
const tap = await soundLibrary.outputTap().catch(() => null);
const config: AudioEncoderConfig = {
codec: 'opus',
sampleRate: tap?.rate ?? 48000,
numberOfChannels: 2,
bitrate: OPUS_BITRATE,
};
const supported =
tap !== null &&
typeof AudioEncoder !== 'undefined' &&
(await AudioEncoder.isConfigSupported(config).then(
(s) => s.supported === true,
() => false,
));
if (!tap || !supported) {
tap?.release();
this.host.log(`Recording "${this.name}" has no sound: this browser cannot record it`);
return;
}
const audio = new AudioEncoder({
output: (chunk, meta) => {
const head = meta?.decoderConfig?.description;
if (head && !this.audioHead) this.audioHead = copyOf(head);
this.writer?.add(2, chunk.timestamp, true, chunkBytes(chunk));
this.collect();
},
error: (error) => this.fail(error.message),
});
audio.configure(config);
this.audio = audio;
this.tap = tap;
tap.node.port.onmessage = (event: MessageEvent<Float32Array[]>) => this.heard(event.data);
}
setPaused(paused: boolean): void {
if (paused === this.paused) return;
this.paused = paused;
const now = performance.now();
if (paused) this.clock.pause(now);
else this.clock.resume(now);
}
capture(view: RecordingView): void {
if (this.state === 'waiting') {
const picture = view.picture();
if (picture) this.open(picture);
else if (!this.toldNoCamera) {
this.toldNoCamera = true;
this.host.log(
`Recording "${this.name}" waits: no camera is switched on, so there is nothing to record`,
);
}
return;
}
const video = this.video;
const reader = this.reader;
if (this.state !== 'recording' || !video || !reader) return;
reader.collect((bytes, ms) => this.encode(bytes, ms));
// A frame taken may have reached the most a file holds and ended the recording.
if (!this.taking || this.paused) return;
const now = performance.now();
if (!this.clock.started) this.clock.start(now);
const ms = this.clock.at(now);
if (!this.gate.take(ms)) return;
const picture = view.picture();
if (!picture || reader.busy || video.encodeQueueSize >= MOST_QUEUED) {
this.dropped++;
return;
}
const m = pictureTransform(
this.settings.resolution === 'view' ? picture.shown : picture.whole,
this.size,
);
view.drawPicture(new Matrix(m.a, 0, 0, m.d, m.tx, m.ty), reader.texture);
reader.read(ms);
}
private get taking(): boolean {
return this.state === 'recording';
}
/** A frame back from the GPU, into the video encoder. */
private encode(bytes: Uint8Array, ms: number): void {
const video = this.video;
if (!video || video.state !== 'configured') return;
const frame = new VideoFrame(bytes, {
format: 'RGBX',
codedWidth: this.size.w,
codedHeight: this.size.h,
timestamp: Math.round(ms * 1000),
duration: Math.round(1e6 / this.settings.fps),
});
video.encode(frame, { keyFrame: isKeyFrame(this.frames, this.settings.fps) });
frame.close();
this.frames++;
this.lastMs = ms;
if (this.taking && (this.writer?.bytes ?? 0) > RECORDING_MOST_BYTES)
this.finish(`Recording "${this.name}" reached 2 GB, the most the server takes, and was stopped`);
}
/** A piece of the master from the worklet; an empty one is the last. */
private heard(sides: Float32Array[]): void {
const audio = this.audio;
const frames = sides[0]?.length ?? 0;
if (frames === 0) {
this.soundEnded?.();
return;
}
// Before the first frame there is no time to put it at. Pieces keep coming during a pause only
// as long as they were on their way, so they belong before it and are kept.
if (!audio || !this.tap || this.state !== 'recording' || !this.clock.started) return;
const rate = this.tap.rate;
this.soundStart ??= Math.max(0, this.clock.at(performance.now()) - (frames / rate) * 1000);
const planar = new Float32Array(frames * 2);
planar.set(sides[0]!, 0);
planar.set(sides[1] ?? sides[0]!, frames);
const data = new AudioData({
format: 'f32-planar',
sampleRate: rate,
numberOfFrames: frames,
numberOfChannels: 2,
timestamp: Math.round(this.soundStart * 1000 + (this.soundFrames / rate) * 1e6),
data: planar,
});
audio.encode(data);
data.close();
this.soundFrames += frames;
}
/** Ends the recording and saves what it has; `why` goes to the log when it is not the node's stop. */
finish(why: string): void {
if (why) this.host.log(why);
if (this.state === 'waiting') {
this.state = 'done';
this.gone();
this.host.log(`Recording "${this.name}" ended before its first frame, so nothing was saved`);
return;
}
if (this.state === 'starting') {
this.stopAsked = true;
return;
}
if (this.state !== 'recording') return;
this.state = 'finishing';
this.gone();
void this.save().catch((error: unknown) =>
this.fail(error instanceof Error ? error.message : String(error)),
);
}
private async save(): Promise<void> {
await Promise.all([this.lastFrames(), this.lastSound()]);
if (this.video?.state === 'configured') await this.video.flush();
if (this.audio?.state === 'configured') await this.audio.flush();
this.close();
const writer = this.writer;
if (!writer || this.frames === 0) {
this.state = 'done';
this.host.log(`Recording "${this.name}" ended before its first frame, so nothing was saved`);
return;
}
const frame = 1000 / this.settings.fps;
const { head, rest } = writer.finish({
video: {
codec: this.codec.startsWith('avc1') ? AVC_CODEC : VP9_CODEC,
width: this.size.w,
height: this.size.h,
private: this.videoHead,
fps: this.settings.fps,
colour: colourOf(this.colour),
},
audio: this.audio
? {
channels: 2,
private: this.audioHead,
preSkip: preSkipOf(this.audioHead ?? new Uint8Array()) ?? 0,
}
: null,
durationMs: this.lastMs + frame,
app: 'game-maker',
});
this.collect();
const blob = new Blob([head, ...this.folded, ...this.loose, ...rest] as BlobPart[], {
type: 'video/x-matroska',
});
this.folded = [];
this.loose = [];
this.state = 'done';
this.host.saved({
name: this.name,
blob,
durationMs: this.lastMs + frame,
width: this.size.w,
height: this.size.h,
});
}
/** Takes the frames still on their way back from the GPU, a moment at most. */
private async lastFrames(): Promise<void> {
const until = performance.now() + LAST_FRAMES_MS;
for (;;) {
this.reader?.collect((bytes, ms) => this.encode(bytes, ms));
if (!this.reader?.pending || performance.now() > until) return;
await new Promise((resolve) => setTimeout(resolve, 4));
}
}
/** Asks the worklet for what it still holds and waits for it, a moment at most. */
private lastSound(): Promise<void> {
const tap = this.tap;
if (!tap) return Promise.resolve();
return new Promise<void>((resolve) => {
const timer = setTimeout(done, LAST_SOUND_MS);
function done(): void {
clearTimeout(timer);
resolve();
}
this.soundEnded = done;
tap.node.port.postMessage('stop');
}).finally(() => {
tap.release();
this.tap = null;
});
}
/** Moves the clusters the writer finished into the file, folding them into Blobs as they pile up. */
private collect(): void {
for (const part of this.writer?.take() ?? []) {
this.loose.push(part);
this.looseBytes += part.length;
}
if (this.looseBytes < FOLD_BYTES) return;
this.folded.push(new Blob(this.loose as BlobPart[]));
this.loose = [];
this.looseBytes = 0;
}
private close(): void {
for (const encoder of [this.video, this.audio])
if (encoder && encoder.state !== 'closed') encoder.close();
this.reader?.destroy();
this.reader = null;
}
private fail(reason: string): void {
if (this.state === 'done' || this.state === 'failed') return;
this.failure = reason;
this.state = 'failed';
this.tap?.release();
this.tap = null;
this.close();
this.gone();
this.host.log(`Recording "${this.name}" failed: ${reason}`);
}
info(): TakeInfo {
return {
name: this.name,
state: this.state,
width: this.size.w,
height: this.size.h,
fps: this.settings.fps,
codec: this.codec,
sound: this.audio !== null,
frames: this.frames,
dropped: this.dropped,
soundFrames: this.soundFrames,
durationMs: this.frames > 0 ? Math.round(this.lastMs + 1000 / this.settings.fps) : 0,
colour: this.colour,
bytes: this.writer?.bytes ?? 0,
failure: this.failure,
};
}
}
function chunkBytes(chunk: EncodedVideoChunk | EncodedAudioChunk): Uint8Array {
const data = new Uint8Array(chunk.byteLength);
chunk.copyTo(data);
return data;
}
function copyOf(source: AllowSharedBufferSource): Uint8Array {
const view = ArrayBuffer.isView(source)
? new Uint8Array(source.buffer, source.byteOffset, source.byteLength)
: new Uint8Array(source);
return view.slice();
}