meleki/feuerwerk
1
0 Fork 0
Code Issues 33 Actions Projects 1 Wiki Activity

Record the game view from a behavior into a video asset (#405)
game-maker / gate (push) Successful in 4m14s
Details
game-maker / suite (push) Successful in 17m7s
Details

Browse Source 
A game can now film itself. A behavior starts and stops a take, and
the editor turns it into a video asset with the run's picture and the
sound the player hears. The exported game hands the file over as a
download, since there is no project to keep it in.

- New action nodes `Start recording` (Name, Resolution, FPS) and
  `Stop recording`. An empty name becomes `Recording <timestamp>`.
- Resolution is `Game view`, `720p` or `1080p`. For the fixed sizes
  the height is set and the width follows the camera.
- The picture is the camera only, without hints, overlays or the
  letterbox. It is drawn into a render texture of the target size,
  so a smaller window still gives full 1080p.
- `PictureReader` reads the texture back through pixel buffers with
  a fence. The editor never waits on the GPU, and a frame with all
  three buffers busy is dropped and counted.
- Nothing of the take reaches the visible canvas. A first version
  drew tiles into it and copied them out, and with the low-latency
  canvas that midway state flickered on screen.
- The sound is tapped behind the master limiter in the FX worklet
  and encoded as Opus.
- H.264 is chosen when the browser has it, VP9 otherwise. A small
  own Matroska writer packs both tracks, no new dependency. It names
  the frame rate and BT.709 so ffmpeg keeps both.
- A second `Start recording` is ignored with a warning in the log.
  A take goes on through `Load scene` and a restart, pauses with the
  editor and is saved when the run ends. It stops at 2 GB.
- The editor uploads the take on the existing video route and makes
  the asset once the run is over. The video tab gets a Download
  button for the converted MP4.
- `downloadBlob` in `src/ui` replaces the song editor's own copy.

Findings:
- `On start` runs before the game screen knows the world. A take
  therefore fixes its size on the first frame that has a camera.
- Without a frame rate in the file, ffmpeg turned 30 fps into 59.94.
- The worker's poster JPEG is off in colour for every upload (red
  255 comes out as 229, BT.601). Not touched here.

Tests:
- Unit recording + matroska: 21 green, plus the node catalog cases.
  The Matroska test reads the file back with its own EBML reader.
- e2e 67 (new) on melektaus-atx: ALL PASSED. 1080p30 from a
  1280x800 window gives a 1920x1080 MP4 with sound and no dropped
  frame. Every 4th pixel is checked, a green quarter catches an
  upside-down picture. A take ended by Esc is saved with the
  timestamp name.
- e2e 60j: the exported game downloads an `.mkv` with EBML magic.
- The flicker is gone on the user's machine without `?vsync`.
- Gate on melektaus-atx: CHECK EXIT 0, 299 files, 4586 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-11 21:11:05 +02:00
1 parent 36ffd9a61a
commit e191dc4b2e
35 files changed
+2211 -26
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/viewport/GameRecorder.ts
+532
View File
@@ -0,0 +1,532 @@
import { Matrix, type RenderTexture } from 'pixi.js';
import { preSkipOf } from '../core/music/ogg';
import { AVC_CODEC, colourOf, MatroskaWriter, VP9_CODEC } from '../core/recording/matroska';
import {
avcCodec,
FrameGate,
isKeyFrame,
pictureTransform,
RECORDING_MOST_BYTES,
RecordingClock,
recordingName,
type RecordingSettings,
type RecordingSink,
recordingSize,
videoBitrate,
vp9Codec,
} from '../core/recording/recording';
import { ui } from '../ui';
import { PictureReader } from './PictureReader';
import { soundLibrary } from './SoundLibrary';
import type { PictureArea } from './WorldViewport';
const OPUS_BITRATE = 160_000;
/** Frames the video encoder may have waiting before the next is dropped instead of piling up. */
const MOST_QUEUED = 6;
/** Encoded bytes held as loose pieces before they are folded into a Blob, out of the page's heap. */
const FOLD_BYTES = 16 * 1024 * 1024;
/** How long a stop waits for the last sound the worklet holds; a suspended context sends none. */
const LAST_SOUND_MS = 300;
/** How long a stop waits for the frames still on their way back from the GPU. */
const LAST_FRAMES_MS = 500;
/** The view a recording takes its frames from (`WorldViewport`). */
export interface RecordingView {
picture(): { shown: PictureArea; whole: PictureArea } | null;
drawPicture(transform: Matrix, target: RenderTexture): void;
}
/** A recording once it is made. */
export interface MadeRecording {
name: string;
blob: Blob;
durationMs: number;
width: number;
height: number;
}
/** Where the recorder looks, and what it tells. */
export interface RecordingHost {
view(): RecordingView | null;
/** A line for the log of the run that plays. */
log(line: string): void;
saved(recording: MadeRecording): void;
}
/**
* `waiting` for a picture to take its size from: `On start` runs before the view is handed the
* world, and a game may switch its camera on later.
*/
type TakeState = 'waiting' | 'starting' | 'recording' | 'finishing' | 'done' | 'failed';
/** What a check reads of a recording (test hook). */
export interface TakeInfo {
name: string;
state: TakeState;
width: number;
height: number;
fps: number;
codec: string;
sound: boolean;
frames: number;
dropped: number;
soundFrames: number;
/** From the first frame to the end of the last, in ms. */
durationMs: number;
/** What the video encoder said its colour is, null while it said nothing. */
colour: VideoColorSpaceInit | null;
bytes: number;
failure: string | null;
}
/**
* Records the game view (#405): `Start recording` and `Stop recording` reach it through the run, the
* host hands it the view and draws a frame for it each tick (`capture`). One recording at a time; it
* outlives the world, so a `Load scene` or a restart goes on in the same file.
*/
export class GameRecorder implements RecordingSink {
private take: Take | null = null;
private paused = false;
/** Every recording of this page, newest last (test hook). */
private readonly takes: Take[] = [];
constructor(private readonly host: RecordingHost) {}
get isRecording(): boolean {
return this.take !== null;
}
start(settings: RecordingSettings): string {
if (this.take)
return `Start recording is ignored: "${this.take.name}" is recording already – Stop recording ends it`;
const take = new Take(recordingName(settings.name, new Date()), settings, this.host, () => {
if (this.take === take) this.take = null;
});
this.take = take;
this.takes.push(take);
if (this.paused) take.setPaused(true);
return '';
}
stop(): string {
if (!this.take) return 'Stop recording: nothing is recording';
this.take.finish('');
this.take = null;
return '';
}
/** The editor holds the run: the recording holds with it, and its time leaves the pause out. */
setPaused(paused: boolean): void {
this.paused = paused;
this.take?.setPaused(paused);
}
/** The view has drawn this frame: the recording takes it, if it is due. */
capture(): void {
const view = this.take && this.host.view();
if (view) this.take!.capture(view);
}
/** The run is over: a recording still going is saved, not lost. */
end(): void {
this.stop();
}
info(): { recording: boolean; takes: TakeInfo[] } {
return { recording: this.isRecording, takes: this.takes.map((take) => take.info()) };
}
}
/** One recording, from its encoders to its file. */
class Take {
private state: TakeState = 'waiting';
private size = { w: 0, h: 0 };
private readonly clock = new RecordingClock();
private readonly gate: FrameGate;
private video: VideoEncoder | null = null;
private audio: AudioEncoder | null = null;
private tap: { node: AudioWorkletNode; rate: number; release(): void } | null = null;
private writer: MatroskaWriter | null = null;
private codec = '';
private videoHead: Uint8Array | null = null;
private colour: VideoColorSpaceInit | null = null;
private audioHead: Uint8Array | null = null;
private reader: PictureReader | null = null;
private toldNoCamera = false;
private folded: Blob[] = [];
private loose: Uint8Array[] = [];
private looseBytes = 0;
private frames = 0;
private dropped = 0;
private lastMs = 0;
private soundFrames = 0;
/** The recording's time of the first sample heard, in ms; null before it. */
private soundStart: number | null = null;
private soundEnded: (() => void) | null = null;
private stopAsked = false;
private paused = false;
private failure: string | null = null;
constructor(
readonly name: string,
private readonly settings: RecordingSettings,
private readonly host: RecordingHost,
private readonly gone: () => void,
) {
this.gate = new FrameGate(settings.fps);
}
/**
* Takes the size from the first picture there is and sets up the encoders; until they are ready
* frames are not taken, and a stop asked meanwhile ends it after.
*/
private open(picture: { shown: PictureArea; whole: PictureArea }): void {
const ratio = ui().app.renderer.resolution;
const device = (area: PictureArea) => ({ w: area.w * ratio, h: area.h * ratio });
this.size = recordingSize(this.settings.resolution, device(picture.shown), device(picture.whole));
this.state = 'starting';
try {
this.reader = new PictureReader(this.size.w, this.size.h);
} catch (error) {
this.fail(error instanceof Error ? error.message : String(error));
return;
}
void this.openEncoders().then(
() => {
if (this.state !== 'starting') return;
this.state = 'recording';
if (this.stopAsked) this.finish('');
},
(error: unknown) => this.fail(error instanceof Error ? error.message : String(error)),
);
}
private async openEncoders(): Promise<void> {
if (typeof VideoEncoder === 'undefined')
throw new Error('this browser cannot encode video – recording needs Chrome');
const { w, h } = this.size;
const fps = this.settings.fps;
const base = { width: w, height: h, framerate: fps, bitrate: videoBitrate(w, h, fps) };
const configs: VideoEncoderConfig[] = [
{ ...base, codec: avcCodec(w, h, fps), avc: { format: 'avc' }, latencyMode: 'realtime' },
{ ...base, codec: vp9Codec(w, h), latencyMode: 'realtime' },
];
let config: VideoEncoderConfig | null = null;
for (const candidate of configs)
if (
await VideoEncoder.isConfigSupported(candidate).then(
(s) => s.supported === true,
() => false,
)
) {
config = candidate;
break;
}
if (!config) throw new Error(`this browser cannot encode ${w}x${h} at ${fps} fps`);
this.codec = config.codec;
await this.openSound();
const writer = new MatroskaWriter(this.audio ? 2 : 1);
this.writer = writer;
const video = new VideoEncoder({
output: (chunk, meta) => {
const head = meta?.decoderConfig?.description;
if (head && !this.videoHead) this.videoHead = copyOf(head);
this.colour ??= meta?.decoderConfig?.colorSpace ?? null;
writer.add(1, chunk.timestamp, chunk.type === 'key', chunkBytes(chunk));
this.collect();
},
error: (error) => this.fail(error.message),
});
video.configure(config);
this.video = video;
}
/** The master behind the limiter, as Opus; without it the recording goes on silent and says so. */
private async openSound(): Promise<void> {
const tap = await soundLibrary.outputTap().catch(() => null);
const config: AudioEncoderConfig = {
codec: 'opus',
sampleRate: tap?.rate ?? 48000,
numberOfChannels: 2,
bitrate: OPUS_BITRATE,
};
const supported =
tap !== null &&
typeof AudioEncoder !== 'undefined' &&
(await AudioEncoder.isConfigSupported(config).then(
(s) => s.supported === true,
() => false,
));
if (!tap || !supported) {
tap?.release();
this.host.log(`Recording "${this.name}" has no sound: this browser cannot record it`);
return;
}
const audio = new AudioEncoder({
output: (chunk, meta) => {
const head = meta?.decoderConfig?.description;
if (head && !this.audioHead) this.audioHead = copyOf(head);
this.writer?.add(2, chunk.timestamp, true, chunkBytes(chunk));
this.collect();
},
error: (error) => this.fail(error.message),
});
audio.configure(config);
this.audio = audio;
this.tap = tap;
tap.node.port.onmessage = (event: MessageEvent<Float32Array[]>) => this.heard(event.data);
}
setPaused(paused: boolean): void {
if (paused === this.paused) return;
this.paused = paused;
const now = performance.now();
if (paused) this.clock.pause(now);
else this.clock.resume(now);
}
capture(view: RecordingView): void {
if (this.state === 'waiting') {
const picture = view.picture();
if (picture) this.open(picture);
else if (!this.toldNoCamera) {
this.toldNoCamera = true;
this.host.log(
`Recording "${this.name}" waits: no camera is switched on, so there is nothing to record`,
);
}
return;
}
const video = this.video;
const reader = this.reader;
if (this.state !== 'recording' || !video || !reader) return;
reader.collect((bytes, ms) => this.encode(bytes, ms));
// A frame taken may have reached the most a file holds and ended the recording.
if (!this.taking || this.paused) return;
const now = performance.now();
if (!this.clock.started) this.clock.start(now);
const ms = this.clock.at(now);
if (!this.gate.take(ms)) return;
const picture = view.picture();
if (!picture || reader.busy || video.encodeQueueSize >= MOST_QUEUED) {
this.dropped++;
return;
}
const m = pictureTransform(
this.settings.resolution === 'view' ? picture.shown : picture.whole,
this.size,
);
view.drawPicture(new Matrix(m.a, 0, 0, m.d, m.tx, m.ty), reader.texture);
reader.read(ms);
}
private get taking(): boolean {
return this.state === 'recording';
}
/** A frame back from the GPU, into the video encoder. */
private encode(bytes: Uint8Array, ms: number): void {
const video = this.video;
if (!video || video.state !== 'configured') return;
const frame = new VideoFrame(bytes, {
format: 'RGBX',
codedWidth: this.size.w,
codedHeight: this.size.h,
timestamp: Math.round(ms * 1000),
duration: Math.round(1e6 / this.settings.fps),
});
video.encode(frame, { keyFrame: isKeyFrame(this.frames, this.settings.fps) });
frame.close();
this.frames++;
this.lastMs = ms;
if (this.taking && (this.writer?.bytes ?? 0) > RECORDING_MOST_BYTES)
this.finish(`Recording "${this.name}" reached 2 GB, the most the server takes, and was stopped`);
}
/** A piece of the master from the worklet; an empty one is the last. */
private heard(sides: Float32Array[]): void {
const audio = this.audio;
const frames = sides[0]?.length ?? 0;
if (frames === 0) {
this.soundEnded?.();
return;
}
// Before the first frame there is no time to put it at. Pieces keep coming during a pause only
// as long as they were on their way, so they belong before it and are kept.
if (!audio || !this.tap || this.state !== 'recording' || !this.clock.started) return;
const rate = this.tap.rate;
this.soundStart ??= Math.max(0, this.clock.at(performance.now()) - (frames / rate) * 1000);
const planar = new Float32Array(frames * 2);
planar.set(sides[0]!, 0);
planar.set(sides[1] ?? sides[0]!, frames);
const data = new AudioData({
format: 'f32-planar',
sampleRate: rate,
numberOfFrames: frames,
numberOfChannels: 2,
timestamp: Math.round(this.soundStart * 1000 + (this.soundFrames / rate) * 1e6),
data: planar,
});
audio.encode(data);
data.close();
this.soundFrames += frames;
}
/** Ends the recording and saves what it has; `why` goes to the log when it is not the node's stop. */
finish(why: string): void {
if (why) this.host.log(why);
if (this.state === 'waiting') {
this.state = 'done';
this.gone();
this.host.log(`Recording "${this.name}" ended before its first frame, so nothing was saved`);
return;
}
if (this.state === 'starting') {
this.stopAsked = true;
return;
}
if (this.state !== 'recording') return;
this.state = 'finishing';
this.gone();
void this.save().catch((error: unknown) =>
this.fail(error instanceof Error ? error.message : String(error)),
);
}
private async save(): Promise<void> {
await Promise.all([this.lastFrames(), this.lastSound()]);
if (this.video?.state === 'configured') await this.video.flush();
if (this.audio?.state === 'configured') await this.audio.flush();
this.close();
const writer = this.writer;
if (!writer || this.frames === 0) {
this.state = 'done';
this.host.log(`Recording "${this.name}" ended before its first frame, so nothing was saved`);
return;
}
const frame = 1000 / this.settings.fps;
const { head, rest } = writer.finish({
video: {
codec: this.codec.startsWith('avc1') ? AVC_CODEC : VP9_CODEC,
width: this.size.w,
height: this.size.h,
private: this.videoHead,
fps: this.settings.fps,
colour: colourOf(this.colour),
},
audio: this.audio
? {
channels: 2,
private: this.audioHead,
preSkip: preSkipOf(this.audioHead ?? new Uint8Array()) ?? 0,
}
: null,
durationMs: this.lastMs + frame,
app: 'game-maker',
});
this.collect();
const blob = new Blob([head, ...this.folded, ...this.loose, ...rest] as BlobPart[], {
type: 'video/x-matroska',
});
this.folded = [];
this.loose = [];
this.state = 'done';
this.host.saved({
name: this.name,
blob,
durationMs: this.lastMs + frame,
width: this.size.w,
height: this.size.h,
});
}
/** Takes the frames still on their way back from the GPU, a moment at most. */
private async lastFrames(): Promise<void> {
const until = performance.now() + LAST_FRAMES_MS;
for (;;) {
this.reader?.collect((bytes, ms) => this.encode(bytes, ms));
if (!this.reader?.pending || performance.now() > until) return;
await new Promise((resolve) => setTimeout(resolve, 4));
}
}
/** Asks the worklet for what it still holds and waits for it, a moment at most. */
private lastSound(): Promise<void> {
const tap = this.tap;
if (!tap) return Promise.resolve();
return new Promise<void>((resolve) => {
const timer = setTimeout(done, LAST_SOUND_MS);
function done(): void {
clearTimeout(timer);
resolve();
}
this.soundEnded = done;
tap.node.port.postMessage('stop');
}).finally(() => {
tap.release();
this.tap = null;
});
}
/** Moves the clusters the writer finished into the file, folding them into Blobs as they pile up. */
private collect(): void {
for (const part of this.writer?.take() ?? []) {
this.loose.push(part);
this.looseBytes += part.length;
}
if (this.looseBytes < FOLD_BYTES) return;
this.folded.push(new Blob(this.loose as BlobPart[]));
this.loose = [];
this.looseBytes = 0;
}
private close(): void {
for (const encoder of [this.video, this.audio])
if (encoder && encoder.state !== 'closed') encoder.close();
this.reader?.destroy();
this.reader = null;
}
private fail(reason: string): void {
if (this.state === 'done' || this.state === 'failed') return;
this.failure = reason;
this.state = 'failed';
this.tap?.release();
this.tap = null;
this.close();
this.gone();
this.host.log(`Recording "${this.name}" failed: ${reason}`);
}
info(): TakeInfo {
return {
name: this.name,
state: this.state,
width: this.size.w,
height: this.size.h,
fps: this.settings.fps,
codec: this.codec,
sound: this.audio !== null,
frames: this.frames,
dropped: this.dropped,
soundFrames: this.soundFrames,
durationMs: this.frames > 0 ? Math.round(this.lastMs + 1000 / this.settings.fps) : 0,
colour: this.colour,
bytes: this.writer?.bytes ?? 0,
failure: this.failure,
};
}
}
function chunkBytes(chunk: EncodedVideoChunk | EncodedAudioChunk): Uint8Array {
const data = new Uint8Array(chunk.byteLength);
chunk.copyTo(data);
return data;
}
function copyOf(source: AllowSharedBufferSource): Uint8Array {
const view = ArrayBuffer.isView(source)
? new Uint8Array(source.buffer, source.byteOffset, source.byteLength)
: new Uint8Array(source);
return view.slice();
}
src/viewport/PictureReader.ts
+102
View File
@@ -0,0 +1,102 @@
import { RenderTexture, type WebGLRenderer } from 'pixi.js';
import { ui } from '../ui';
/** Frames on their way back from the GPU at once; while all are, a frame due is dropped. */
const MOST_IN_FLIGHT = 3;
interface InFlight {
buffer: WebGLBuffer;
fence: WebGLSync;
ms: number;
}
/**
* Hands a recording its frames without drawing into the canvas on screen (#405): the low latency
* canvas shows what is drawn into it at once, so a frame drawn there flickered in its corner. The
* view is drawn into `texture` instead and read back through pixel buffers, so the main thread does
* not wait for the GPU: a frame comes out of `collect` a tick or two after it was drawn.
*/
export class PictureReader {
readonly texture: RenderTexture;
private readonly gl: WebGL2RenderingContext;
private readonly free: WebGLBuffer[] = [];
private readonly inFlight: InFlight[] = [];
private readonly bytes: Uint8Array;
constructor(
readonly w: number,
readonly h: number,
) {
const renderer = ui().app.renderer as WebGLRenderer;
const gl = renderer.gl as WebGL2RenderingContext | undefined;
if (!gl || typeof gl.fenceSync !== 'function') throw new Error('recording needs WebGL 2');
this.gl = gl;
this.texture = RenderTexture.create({ width: w, height: h, resolution: 1, antialias: true });
for (let i = 0; i < MOST_IN_FLIGHT; i++) {
const buffer = gl.createBuffer();
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, buffer);
gl.bufferData(gl.PIXEL_PACK_BUFFER, w * h * 4, gl.STREAM_READ);
this.free.push(buffer);
}
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, null);
this.bytes = new Uint8Array(w * h * 4);
}
/** Whether a frame drawn now would have no buffer to go back in. */
get busy(): boolean {
return this.free.length === 0;
}
get pending(): number {
return this.inFlight.length;
}
/** Starts reading back what `texture` holds now, the frame of recording time `ms`. */
read(ms: number): void {
const buffer = this.free.pop();
if (!buffer) return;
const gl = this.gl;
const renderer = ui().app.renderer as WebGLRenderer;
const target = renderer.renderTarget.getRenderTarget(this.texture);
// Resolves the multisampled picture into the texture; a second time is only a copy.
renderer.renderTarget.adaptor.finishRenderPass(target);
const gpu = renderer.renderTarget.getGpuRenderTarget(target);
// Through the adaptor, so Pixi knows which framebuffer is bound.
renderer.renderTarget.adaptor.bindFramebuffer(gpu.resolveTargetFramebuffer);
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, buffer);
gl.readPixels(0, 0, this.w, this.h, gl.RGBA, gl.UNSIGNED_BYTE, 0);
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, null);
const fence = gl.fenceSync(gl.SYNC_GPU_COMMANDS_COMPLETE, 0)!;
gl.flush();
this.inFlight.push({ buffer, fence, ms });
}
/**
* Hands over each frame the GPU has finished, oldest first, its rows top first as Pixi draws a
* texture. The bytes are reused for the next frame, so `take` copies what it keeps.
*/
collect(take: (bytes: Uint8Array, ms: number) => void): void {
const gl = this.gl;
while (this.inFlight.length > 0) {
const next = this.inFlight[0]!;
if (gl.clientWaitSync(next.fence, 0, 0) === gl.TIMEOUT_EXPIRED) return;
gl.deleteSync(next.fence);
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, next.buffer);
gl.getBufferSubData(gl.PIXEL_PACK_BUFFER, 0, this.bytes);
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, null);
this.inFlight.shift();
this.free.push(next.buffer);
take(this.bytes, next.ms);
}
}
destroy(): void {
const gl = this.gl;
for (const { buffer, fence } of this.inFlight.splice(0)) {
gl.deleteSync(fence);
gl.deleteBuffer(buffer);
}
for (const buffer of this.free.splice(0)) gl.deleteBuffer(buffer);
this.texture.destroy(true);
}
}
src/viewport/SoundLibrary.ts
+19
View File
@@ -614,6 +614,25 @@ export class SoundLibrary implements AudioSink {
return readMeter((this.guardMeter ??= this.tapMeter(guard.limiter)));
}
/**
* A node that hears what leaves the output, limiter and all (#405): a recording of the game view
* takes its sound there, in pieces of both sides (`record-tap`). Null without a live output.
*/
async outputTap(): Promise<{ node: AudioWorkletNode; rate: number; release(): void } | null> {
const context = this.audio();
if (!(context instanceof AudioContext) || !(await this.fxWorklet())) return null;
const guard = this.guard;
if (!guard) return null;
const node = new AudioWorkletNode(context, 'record-tap', {
numberOfOutputs: 0,
channelCount: 2,
channelCountMode: 'explicit',
});
const source = guard.limiter ?? guard.input;
source.connect(node);
return { node, rate: context.sampleRate, release: () => source.disconnect(node) };
}
/** Lets every level meter go. */
unmeter(): void {
for (const bus of this.buses.values()) this.dropMeter(bus);
src/viewport/VideoLibrary.ts
+6 -1
View File
@@ -27,6 +27,11 @@ import { soundLibrary } from './SoundLibrary';
import { followVideo, readSpeed, type SpeedReading } from './videoFollow';
const VIDEO_BASE = '/game-maker/data/videos';
/** Where the MP4 of a video asset's `file` is served. */
export function videoFileUrl(file: string): string {
return `${VIDEO_BASE}/${file}`;
}
/** How close to its end a clip counts as there: half a frame at 30 fps. */
const END_SLACK = 1 / 60;
/** A player nothing has asked for in this long is let go: no sprite wears it any more. */
@@ -281,7 +286,7 @@ export class VideoLibrary {
const at = drive ? drivenTimeOf(play, time, NaN) : play.start;
const player: Player = {
play,
element: openVideo(`${VIDEO_BASE}/${play.file}`, at, (texture) => {
element: openVideo(videoFileUrl(play.file), at, (texture) => {
if (player.gone) return texture.destroy(true);
player.texture = texture;
if (drive) this.follow(player, drive);
src/viewport/WorldViewport.ts
+60 -1
View File
@@ -1,4 +1,12 @@
import { Container, type FederatedPointerEvent, Graphics, type Mesh, Texture } from 'pixi.js';
import {
Container,
type FederatedPointerEvent,
Graphics,
type Matrix,
type Mesh,
type RenderTexture,
Texture,
} from 'pixi.js';
import { isSymbolType } from '../core/model/AssetTypes';
import { type Extent, scaleOf } from '../core/model/matrix';
import type { Vec2 } from '../core/model/Properties';
@@ -51,6 +59,14 @@ export interface Frame {
background: number;
}
/** A part of the view by its centre and size, in local CSS pixels. */
export interface PictureArea {
x: number;
y: number;
w: number;
h: number;
}
export abstract class WorldViewport<S extends ViewSource = ViewSource> extends Widget {
protected readonly bg = new Graphics();
protected readonly content = new Container();
@@ -512,6 +528,49 @@ export abstract class WorldViewport<S extends ViewSource = ViewSource> extends W
return (shading && shadedPicture(shading, shaders, ppu, true)?.grid) || view.grid;
}
/**
* The camera's picture as the view lays it out (#405), in local CSS pixels by its centre: the part
* on screen, and all of it, which is larger only while a fixed scale cuts it to the view. Null
* without a camera.
*/
picture(): { shown: PictureArea; whole: PictureArea } | null {
const view = this.currentView();
const camera = view && this.mainCamera();
if (!view || !camera) return null;
const { rect } = view;
const frame = cameraFrame(camera.props);
const x = rect.x + rect.w / 2;
const y = rect.y + rect.h / 2;
return {
shown: { x, y, w: rect.w, h: rect.h },
whole: { x, y, w: frame.w * view.scale, h: frame.h * view.scale },
};
}
/**
* Draws the view into `target` mapped by `transform`, without what lies over the picture
* (`chrome`): a recording reads it from there (#405). Never into the canvas on screen, which shows
* at once what is drawn into it while its context is low latency. Drawn even while its tab is
* hidden, so a recording goes on behind another tab.
*/
drawPicture(transform: Matrix, target: RenderTexture): void {
const shown = this.chrome().filter((part) => part.visible);
for (const part of shown) part.visible = false;
const visible = this.visible;
this.visible = true;
try {
ui().app.renderer.render({ container: this, target, transform, clear: true, clearColor: [0, 0, 0, 1] });
} finally {
this.visible = visible;
for (const part of shown) part.visible = true;
}
}
/** What the view lays over the game's picture and a recording leaves out. */
protected chrome(): Container[] {
return [];
}
/** Whether the game view is at a fixed scale rather than fit: the only one overlays snap at. */
protected fixedScale(): boolean {
return false;
src/viewport/fxWorklet.ts
+49
View File
@@ -391,3 +391,52 @@ class ChannelProbe extends AudioWorkletProcessor {
}
registerProcessor('channel-probe', ChannelProbe);
/** Frames a recording's sound is handed over in: about a tenth of a second. */
const RECORD_BLOCK = 4096;
/**
* Hands what reaches it to the page, both sides, `RECORD_BLOCK` frames at a time (#405): the master
* as it leaves the limiter, for a recording of the game view. Told to stop, it sends what it holds
* and an empty piece after it, the sign that nothing more comes.
*/
class RecordTap extends Stoppable {
private readonly sides = [new Float32Array(RECORD_BLOCK), new Float32Array(RECORD_BLOCK)];
private filled = 0;
constructor() {
super();
this.port.onmessage = () => {
this.send();
this.port.postMessage([new Float32Array(0), new Float32Array(0)]);
this.stop();
};
}
process(inputs: Float32Array[][]): boolean {
if (this.stopped) return false;
const input = inputs[0] ?? [];
const left = input[0];
const frames = left?.length ?? 128;
this.sides.forEach((side, c) => {
const data = input[c] ?? left;
if (data) side.set(data, this.filled);
else side.fill(0, this.filled, this.filled + frames);
});
this.filled += frames;
if (this.filled + frames > RECORD_BLOCK) this.send();
return true;
}
private send(): void {
if (this.filled === 0) return;
const out = this.sides.map((side) => side.slice(0, this.filled));
this.filled = 0;
this.port.postMessage(
out,
out.map((side) => side.buffer),
);
}
}
registerProcessor('record-tap', RecordTap);