A video becomes an asset like a sound: uploaded once, converted to one format by a worker of its own, cut into clips in its tab, and worn by a sprite through its Texture row. Only Chrome is a target, without alpha. - Server: POST ?r=videos queues the upload under the hash of its bytes, so the MP4's name is known at once. The worker (VideoWorker over VideoFormat) writes H.264/AAC, yuv420p, constant frame rate, a keyframe every 15 frames, at most 1920x1080, with a poster and the details beside it. A failed job says why and is tried again on the next upload. - An upload may be up to 2 GB. It is streamed from the request to disk and hashed there, so php-fpm holds none of it in memory. The other routes keep their 256 MB. - The worker runs in the container game-maker-video on stein2, a compose project of its own (tools/video/), and in CI beside the PHP server. The CI image has ffmpeg for it. - Model: asset type video with size, length, frame rate and sound, and clips like a sound's. The clip helpers and commands serve both. - A sprite's Texture row offers videos and clips after the pictures. resolvePicture names a playing video by a key, and the texture library hands it to a VideoLibrary with one player per key. - The editor shows a placed video as a quiet preview. Play starts it over at its clip's start and lets it be heard through the master bus, a pause holds it, Esc makes it quiet again. - Video tab: the picture, a ruler, the clip strip and a track on the Sound tab's time axis. A drag spans a clip, a handle moves an edge, Play plays the picked clip. - The picker shows a video on its way with its progress and adopts an upload from it once it is ready. Dropping a video file does the same. Decided on my own: - Two sprites wearing the same clip share one player and show the same picture. Control per instance belongs to the follow-up with nodes. - A video plays in real time, the game's time scale does not slow it. - The container may use 3 of the 4 cores of stein2. Findings: - The first real upload (279 MB) got 413: the API's own ceiling of 256 MB for every request, not nginx or the Brandmauer. - Chrome has no picture of a video that never played, and a Pixi VideoSource drawn before the first frame stays empty for good. The texture is made in the first frame callback (openVideo). - A player nobody asks for in 2 s is let go. The inspector's PREVIEW asked only on an edit, so with the sprite not on screen its texture was destroyed under it and the batcher threw on every frame. The preview now redraws every frame while it shows a video, like a Time shader. A let-go texture keeps an empty source instead of none. - VideoSource.load() reloads the element and cuts off an earlier play(). Playing starts after loadeddata. Tests: - PHP videos.test.php: 68 checks, with the real ffmpeg - 300 MB upload through Brandmauer and nginx: 201 in 21 s, php-fpm at 13.7 MB RSS at most - Unit videos.test.ts: 20 tests - e2e 66 (stein2, atx was off): ALL PASSED, 18 checks; e2e 04 with the new asset count green - Gate (atx): 294 files, 4520 tests. serialize.test.ts used video as its example of an unknown type and now uses a made-up one. - Counter-checks: a keyframe gap of 30 fails the PHP check, a key that keeps the video's end fails 2 unit tests, a clip without its end fails "plays only its stretch" in e2e 66 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
169 lines
8.8 KiB
TypeScript
169 lines
8.8 KiB
TypeScript
import type { Application } from 'pixi.js';
|
||
import { physicsStatus } from '../runtime/physics/engine';
|
||
import type { SchedulerOptions } from '../core/music/Scheduler';
|
||
import type { Sequence } from '../core/music/sequence';
|
||
import type { FxEnvelope, FxStage } from '../core/soundfx/nodes';
|
||
import { Music } from '../viewport/music';
|
||
import { bothAtCenter, envelopeDriver, followCurve, releaseEnvelope, stageNodes } from '../viewport/fxNodes';
|
||
import { soundLibrary } from '../viewport/SoundLibrary';
|
||
import { saturationAutoGain } from '../core/soundfx/saturation';
|
||
import { compressorMakeup } from '../core/soundfx/dynamics';
|
||
import { bakeHooks } from './debug/bakeHooks';
|
||
import { curveHooks } from './debug/curveHooks';
|
||
import { soundHooks } from './debug/soundHooks';
|
||
import { videoHooks } from './debug/videoHooks';
|
||
import { patternHooks } from './debug/patternHooks';
|
||
import { routingHooks } from './debug/routingHooks';
|
||
import { songHooks } from './debug/songHooks';
|
||
import { documentHooks } from './debug/documentHooks';
|
||
import { graphHooks } from './debug/graphHooks';
|
||
import { centreOf, type HookContext } from './debug/hookHelpers';
|
||
import { pixelArtHooks } from './debug/pixelArtHooks';
|
||
import { renderHooks } from './debug/renderHooks';
|
||
import { sceneHooks } from './debug/sceneHooks';
|
||
import { uiHooks } from './debug/uiHooks';
|
||
import { vectorArtHooks } from './debug/vectorArtHooks';
|
||
import type { DocumentId, Editor, TabId } from './Editor';
|
||
import { NodeBoard } from './plugins/NodeBoard';
|
||
import type { EditorState } from './state';
|
||
import { rectOf } from './textProbe';
|
||
|
||
/**
|
||
* Installs `window.__gm`, which the check scripts in tools/ use to find widgets by their
|
||
* visible text and to read editor state. The hooks of each area live in `debug/`.
|
||
*/
|
||
export function installDebugHook(app: Application, state: EditorState, editor: Editor): void {
|
||
const context: HookContext = { app, state, editor };
|
||
const music = new Music(() => state.project);
|
||
(window as unknown as { __gm: unknown }).__gm = {
|
||
state,
|
||
/** The Pixi application, for probes that need the renderer or the stage. */
|
||
app,
|
||
...uiHooks(context),
|
||
...sceneHooks(context),
|
||
...graphHooks(context),
|
||
...documentHooks(context),
|
||
...pixelArtHooks(context),
|
||
...vectorArtHooks(context),
|
||
...curveHooks(context),
|
||
...patternHooks(context),
|
||
...songHooks(context),
|
||
...routingHooks(context),
|
||
...soundHooks(context),
|
||
...videoHooks(context),
|
||
...bakeHooks(context),
|
||
...renderHooks(context),
|
||
/**
|
||
* Voices started, stopped and in flight. CI runs the browser with `--mute-audio`, so nothing
|
||
* is ever heard there: these counters are the only way a check can see that a graph played
|
||
* anything at all, the way `renderStats` is for a repaint that did not happen.
|
||
*/
|
||
sound: {
|
||
stats: () => soundLibrary.stats(),
|
||
duration: (file: string) => soundLibrary.duration(file),
|
||
/** Where the knobs of each voice of that file are now, oldest first, a pitch as its speed (#315). */
|
||
levels: (file: string) => soundLibrary.levels(file),
|
||
fxWorkletUrl: () => soundLibrary.fxWorkletUrl(),
|
||
/** Seconds on the audio clock: the time a voice really runs on. */
|
||
clock: () => soundLibrary.clock(),
|
||
/** Decodes ahead of a check, so a play can be asserted without racing the network. */
|
||
load: (file: string) => soundLibrary.load(file).then((buffer) => buffer?.duration ?? null),
|
||
stopAll: () => soundLibrary.stopAll(),
|
||
/** What Stop sound asks for: every voice of that file, its echoes cut or left to ring (#324). */
|
||
stop: (file: string, letRing: boolean) => soundLibrary.stop(file, letRing),
|
||
/** The patterns and songs a game plays, with the notes and clips last planned of each. */
|
||
playbacks: () => soundLibrary.playbacks(),
|
||
/** Every bus there is, the master and those of the project among them (#325), and how loud one is. */
|
||
buses: () => soundLibrary.busLevels(),
|
||
mixVolume: (name: string) => soundLibrary.mixVolume(name),
|
||
/** Whether the audio clock is held by the pause of the toolbar or runs. */
|
||
clockState: () => soundLibrary.clockState(),
|
||
/** The second of the audio clock a voice started at, and the rate that clock runs at (#346). */
|
||
startedAt: (voice: number) => soundLibrary.startedAt(voice),
|
||
playing: (voice: number) => soundLibrary.playing(voice),
|
||
sampleRate: () => soundLibrary.liveContext().sampleRate,
|
||
/**
|
||
* The nodes a stage of a voice becomes, in any context – an offline one renders what it does.
|
||
* `at` is when its voice starts, which a Flanger's sweep counts from (#341), `impulse` the
|
||
* sound a Reverb that takes one has (#328).
|
||
*/
|
||
stageNodes: (context: BaseAudioContext, stage: FxStage, at?: number, impulse?: AudioBuffer) =>
|
||
stageNodes(context, stage, undefined, at, impulse),
|
||
/** The gain in dB a Saturation's auto-gain gives at a drive (#323). */
|
||
saturationAutoGain,
|
||
/** The gain after a Compressor as a factor, from its makeup and auto-gain (#326, #378). */
|
||
compressorMakeup,
|
||
/** What drives the two shares of a Mix "Both at center", in any context (unstarted). */
|
||
bothAtCenter: (context: BaseAudioContext, mix: number, a: GainNode, b: GainNode) =>
|
||
bothAtCenter(context, mix, a, b),
|
||
/** The curve of an Envelope from `at` (unstarted), its release, and how it moves a number (#335). */
|
||
envelopeDriver: (context: BaseAudioContext, envelope: FxEnvelope, at: number) =>
|
||
envelopeDriver(context, envelope, at),
|
||
releaseEnvelope: (level: AudioParam, envelope: FxEnvelope, startedAt: number, at: number) =>
|
||
releaseEnvelope(level, envelope, startedAt, at),
|
||
stageWith: (context: BaseAudioContext, stage: FxStage, driver: ConstantSourceNode) =>
|
||
stageNodes(context, stage, (mod, param, ratio, map) =>
|
||
followCurve(context, driver, mod, param, ratio, map),
|
||
),
|
||
},
|
||
/**
|
||
* The scheduler (#332), which nothing in the editor plays yet: a check hands it a sequence of
|
||
* notes and reads back what was planned for which second of the audio clock.
|
||
*/
|
||
music: {
|
||
play: (sequence: Sequence, options?: SchedulerOptions & { from?: number }) =>
|
||
music.play(sequence, options),
|
||
edit: (sequence: Sequence) => music.edit(sequence),
|
||
stop: () => music.stop(),
|
||
position: () => music.current?.position() ?? null,
|
||
playing: () => music.current?.playing ?? false,
|
||
planned: () => [...music.planned],
|
||
latency: () => soundLibrary.latency(),
|
||
},
|
||
world: () => editor.worldSnapshot(),
|
||
playInfo: () => editor.playInfo(),
|
||
/** Which scene Play would start, and under which of the two rules (#183: set in the settings). */
|
||
playFrom: () => editor.playFromInfo(),
|
||
/** The two run buttons of the top bar: the picture each wears, and whether it takes a click. */
|
||
runButtons: () => editor.runButtons(),
|
||
/** Where to click one of the icon buttons: menu, undo, redo, play, pause. */
|
||
toolbarAt: (name: keyof Editor['toolbarControls']) => centreOf(editor.toolbarControls[name]),
|
||
/** Behaviors running without Play (`Run in editor`), and how long they have been at it. */
|
||
editorRun: () => editor.editorRunInfo(),
|
||
physics: () => physicsStatus(),
|
||
tab: () => editor.tab,
|
||
showTab: (tab: TabId) => editor.showTab(tab),
|
||
/** The open documents in tab order and the one showing (#171); a document is `asset:<id>` or `list`. */
|
||
documents: () => editor.openDocuments(),
|
||
openDocument: (doc: DocumentId) => editor.openDocument(doc),
|
||
closeDocument: (doc: DocumentId) => editor.closeDocument(doc),
|
||
/** The tabs in windows of their own (#371): kind, document, box, and where to press. */
|
||
floatWindows: () => editor.floatWindows(),
|
||
floatDocument: (doc: DocumentId) => editor.floatDocument(doc),
|
||
/** The kind Delete, Copy and Paste go to: the window pressed last, else the tab showing. */
|
||
keyTarget: () => editor.keyTarget(),
|
||
plugins: {
|
||
/** Every file in `data/plugins/` and what became of it. */
|
||
records: () => [...(editor.plugins?.records ?? [])],
|
||
/** The plugins in the user management tab (#179): one row per file, with its switch. */
|
||
pane: () => ({ ...editor.pluginsPane.debugInfo(), visible: editor.tab === 'management' }),
|
||
/** Where to click the switch of one plugin; null when it has no row. */
|
||
switchAt: (file: string) => {
|
||
const box = editor.pluginsPane.switchOf(file);
|
||
return box ? centreOf(box) : null;
|
||
},
|
||
/** A plugin tab's node board as drawn, with its canvas and Fit button; null without one. */
|
||
board: (tabId: string) => {
|
||
const pane = editor.pluginPane(tabId);
|
||
if (!(pane instanceof NodeBoard)) return null;
|
||
return {
|
||
...pane.debugInfo(),
|
||
visible: pane.visible,
|
||
rect: rectOf(pane),
|
||
fit: centreOf(pane.fitButton),
|
||
};
|
||
},
|
||
},
|
||
};
|
||
}
|