Play videos on sprites, converted on the server with clips (#402)
A video becomes an asset like a sound: uploaded once, converted to one format by a worker of its own, cut into clips in its tab, and worn by a sprite through its Texture row. Only Chrome is a target, without alpha. - Server: POST ?r=videos queues the upload under the hash of its bytes, so the MP4's name is known at once. The worker (VideoWorker over VideoFormat) writes H.264/AAC, yuv420p, constant frame rate, a keyframe every 15 frames, at most 1920x1080, with a poster and the details beside it. A failed job says why and is tried again on the next upload. - An upload may be up to 2 GB. It is streamed from the request to disk and hashed there, so php-fpm holds none of it in memory. The other routes keep their 256 MB. - The worker runs in the container game-maker-video on stein2, a compose project of its own (tools/video/), and in CI beside the PHP server. The CI image has ffmpeg for it. - Model: asset type video with size, length, frame rate and sound, and clips like a sound's. The clip helpers and commands serve both. - A sprite's Texture row offers videos and clips after the pictures. resolvePicture names a playing video by a key, and the texture library hands it to a VideoLibrary with one player per key. - The editor shows a placed video as a quiet preview. Play starts it over at its clip's start and lets it be heard through the master bus, a pause holds it, Esc makes it quiet again. - Video tab: the picture, a ruler, the clip strip and a track on the Sound tab's time axis. A drag spans a clip, a handle moves an edge, Play plays the picked clip. - The picker shows a video on its way with its progress and adopts an upload from it once it is ready. Dropping a video file does the same. Decided on my own: - Two sprites wearing the same clip share one player and show the same picture. Control per instance belongs to the follow-up with nodes. - A video plays in real time, the game's time scale does not slow it. - The container may use 3 of the 4 cores of stein2. Findings: - The first real upload (279 MB) got 413: the API's own ceiling of 256 MB for every request, not nginx or the Brandmauer. - Chrome has no picture of a video that never played, and a Pixi VideoSource drawn before the first frame stays empty for good. The texture is made in the first frame callback (openVideo). - A player nobody asks for in 2 s is let go. The inspector's PREVIEW asked only on an edit, so with the sprite not on screen its texture was destroyed under it and the batcher threw on every frame. The preview now redraws every frame while it shows a video, like a Time shader. A let-go texture keeps an empty source instead of none. - VideoSource.load() reloads the element and cuts off an earlier play(). Playing starts after loadeddata. Tests: - PHP videos.test.php: 68 checks, with the real ffmpeg - 300 MB upload through Brandmauer and nginx: 201 in 21 s, php-fpm at 13.7 MB RSS at most - Unit videos.test.ts: 20 tests - e2e 66 (stein2, atx was off): ALL PASSED, 18 checks; e2e 04 with the new asset count green - Gate (atx): 294 files, 4520 tests. serialize.test.ts used video as its example of an unknown type and now uses a made-up one. - Counter-checks: a keyframe gap of 30 fails the PHP check, a key that keeps the video's end fails 2 unit tests, a clip without its end fails "plays only its stretch" in e2e 66 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed
1 parent
2f376262d3
commit
d48f174caf
63 files changed
+3500
-103
No files matched your search
@@ -0,0 +1,233 @@
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { uploadVideo } from '../lib/projects.mjs';
|
||||
import { MAKE_OBJECT, pickAssetType } from './lib/scaffold.mjs';
|
||||
|
||||
/**
|
||||
* Videos (#402). A WebM goes up and the worker turns it into an MP4 (in CI beside the PHP server,
|
||||
* on stein2 in the container game-maker-video). "+ → Video" offers it with what it is; Use makes
|
||||
* the asset and opens its tab, which shows the picture. A drag in the clip strip spans a clip, one
|
||||
* undo step. A sprite wearing the clip plays only that stretch, over and over, without sound in the
|
||||
* editor; Play starts it over from the clip's start and lets it be heard, Esc makes it quiet again.
|
||||
*
|
||||
* The test video is 2 s at 30 fps, a quarter each red, green, blue and white, with a tone. Colours
|
||||
* are read from the video the tab holds and from the screen.
|
||||
*
|
||||
* Everything it adds carries a `vd66` id or is the asset it made, and goes again at the end.
|
||||
*/
|
||||
const SPRITE = 'vd66-sprite';
|
||||
const OBJECT = 'vd66-object';
|
||||
const COLORS = { red: '#fe0000', green: '#00ff01', blue: '#0000ff', white: '#ffffff' };
|
||||
|
||||
export async function video(t) {
|
||||
const { check, sleep, json, mutate, click, clickText, drag, shot, showTab, play, stop } = t;
|
||||
const waitFor = async (read, done, tries = 40) => {
|
||||
let value = await read();
|
||||
for (let i = 0; i < tries && !done(value); i++) {
|
||||
await sleep(150);
|
||||
value = await read();
|
||||
}
|
||||
return value;
|
||||
};
|
||||
|
||||
// 66a. Converted on the server: the asked size, length and frame rate, as an MP4 with a poster.
|
||||
const uploaded = await uploadVideo(
|
||||
t.baseUrl,
|
||||
readFileSync(new URL('./fixtures/colors.webm', import.meta.url)),
|
||||
'colors.webm',
|
||||
);
|
||||
check(
|
||||
'an uploaded WebM is converted to an MP4 with its size, length, frame rate and sound',
|
||||
uploaded.status === 'ready' &&
|
||||
/^[a-f0-9]{12}\/[a-f0-9]{64}\.mp4$/.test(uploaded.file) &&
|
||||
uploaded.width === 320 &&
|
||||
uploaded.height === 180 &&
|
||||
Math.abs(uploaded.duration - 2) < 0.05 &&
|
||||
uploaded.fps === 30 &&
|
||||
uploaded.audio === true &&
|
||||
uploaded.poster === uploaded.file.replace(/\.mp4$/, '.jpg'),
|
||||
JSON.stringify(uploaded),
|
||||
);
|
||||
|
||||
// 66b. "+ → Video" offers it with what it is, Use makes the asset and opens the tab.
|
||||
await showTab('scene');
|
||||
await clickText('+');
|
||||
await pickAssetType(t, 'Video');
|
||||
const picker = await waitFor(
|
||||
() => json('__gm.videoPicker()'),
|
||||
(p) => p?.files.includes(uploaded.file),
|
||||
);
|
||||
check(
|
||||
'the + menu opens the video picker on what the server holds',
|
||||
Boolean(picker?.files.includes(uploaded.file)),
|
||||
JSON.stringify(picker),
|
||||
);
|
||||
const row = await json(`__gm.videoPickerRow(${JSON.stringify(uploaded.file)})`);
|
||||
await click(row.x, row.y);
|
||||
const picked = await json('__gm.videoPicker()');
|
||||
check(
|
||||
'a ready video says its size, length, frame rate and sound',
|
||||
picked.detail.includes('320 × 180 · 2 s · 30 fps · with sound'),
|
||||
picked.detail,
|
||||
);
|
||||
await shot('66-video-picker');
|
||||
await clickText('Use video');
|
||||
const assetId = await waitFor(
|
||||
() => json('__gm.videoTab.current()'),
|
||||
(id) => Boolean(id),
|
||||
);
|
||||
const asset = await json(`__gm.state.project.assets.find((a) => a.id === ${JSON.stringify(assetId)})`);
|
||||
check(
|
||||
'Use makes a video asset of it and opens its tab',
|
||||
asset?.type === 'video' &&
|
||||
asset.file === uploaded.file &&
|
||||
asset.width === 320 &&
|
||||
asset.fps === 30 &&
|
||||
asset.name === 'colors',
|
||||
JSON.stringify(asset),
|
||||
);
|
||||
|
||||
// 66c. The tab shows the picture: red at the start, read from the video and from the screen.
|
||||
const preview = await waitFor(
|
||||
() => json('__gm.videoTab.preview()'),
|
||||
(p) => p?.shown && p.pixel?.[0] > 200,
|
||||
);
|
||||
const shownColor = (p) => json(`__gm.pixelAt(${Math.round(p.x + p.w / 2)}, ${Math.round(p.y + p.h / 2)})`);
|
||||
check(
|
||||
'the tab shows the video, its first frame red',
|
||||
preview?.shown &&
|
||||
preview.pixel[0] > 200 &&
|
||||
preview.pixel[1] < 50 &&
|
||||
(await shownColor(preview)) === COLORS.red,
|
||||
JSON.stringify({ preview, screen: await shownColor(preview) }),
|
||||
);
|
||||
|
||||
// 66d. A drag in the strip spans a clip from 0.6 s to 1.4 s, which is then picked; one undo step.
|
||||
const from = await json('__gm.videoTab.stripAt(0.6)');
|
||||
const to = await json('__gm.videoTab.stripAt(1.4)');
|
||||
await drag(from.x, from.y, to.x, to.y);
|
||||
await sleep(300);
|
||||
const clips = await json('__gm.videoTab.clips()');
|
||||
const clip = clips[0];
|
||||
check(
|
||||
'a drag in the strip spans a clip, which is picked',
|
||||
clips.length === 1 &&
|
||||
Math.abs(clip.props.start - 0.6) < 0.02 &&
|
||||
Math.abs(clip.props.end - 1.4) < 0.02 &&
|
||||
(await json('__gm.videoTab.pickedClip()')) === clip.id,
|
||||
JSON.stringify({ clips, picked: await json('__gm.videoTab.pickedClip()') }),
|
||||
);
|
||||
const atClip = await waitFor(
|
||||
() => json('__gm.videoTab.preview()'),
|
||||
(p) => p?.pixel?.[1] > 200 && p.pixel[0] < 50,
|
||||
);
|
||||
check(
|
||||
'the tab stands at the picked clip’s start, green',
|
||||
atClip.pixel[1] > 200 && atClip.pixel[0] < 50,
|
||||
JSON.stringify(atClip),
|
||||
);
|
||||
await t.undo();
|
||||
const undone = await json('__gm.videoTab.clips()');
|
||||
await t.redo();
|
||||
check(
|
||||
'the clip is one undo step',
|
||||
undone.length === 0 && (await json('__gm.videoTab.clips()')).length === 1,
|
||||
JSON.stringify(undone),
|
||||
);
|
||||
|
||||
// 66e. The undo left the video picked; a click on the band picks the clip again. Play in the tab
|
||||
// runs the clip and moves the playhead; Stop holds it.
|
||||
const band = await json('__gm.videoTab.stripAt(1)');
|
||||
await click(band.x, band.y);
|
||||
check('a click on its band picks the clip', (await json('__gm.videoTab.pickedClip()')) === clip.id);
|
||||
const playButton = await json('__gm.videoTab.playButton()');
|
||||
await click(playButton.x, playButton.y);
|
||||
await sleep(500);
|
||||
const running = await json(
|
||||
'({ playing: __gm.videoTab.playing(), head: __gm.videoTab.playhead(), button: __gm.videoTab.playButton().text })',
|
||||
);
|
||||
check(
|
||||
'Play in the tab plays the picked clip from its start',
|
||||
running.playing && running.head > 0.6 && running.head < 1.4 && running.button === 'Stop',
|
||||
JSON.stringify(running),
|
||||
);
|
||||
await click(playButton.x, playButton.y);
|
||||
check('Stop holds it', (await json('__gm.videoTab.playing()')) === false);
|
||||
await shot('66-video-tab');
|
||||
|
||||
// 66f. A sprite wearing the clip shows only its stretch, over and over: green and blue, never red
|
||||
// or white, without sound in the editor.
|
||||
await showTab('scene');
|
||||
await mutate(`
|
||||
const p = __gm.state.project;
|
||||
const scene = __gm.state.activeScene;
|
||||
${MAKE_OBJECT}
|
||||
p.assets.unshift({ id: '${SPRITE}', type: 'sprite', name: '${SPRITE}', props: { width: 320, height: 180, color: 0xffffff, texture: ${JSON.stringify(clip.id)}, shader: '' } });
|
||||
scene.objects.push(make('${OBJECT}', '${SPRITE}', '', 0, 'video'));
|
||||
__gm.state.select(null);
|
||||
__gm.lookAt(0, 0);`);
|
||||
await waitFor(
|
||||
() => json('__gm.videos()'),
|
||||
(players) => players.some((p) => p.ready),
|
||||
);
|
||||
const seen = new Set();
|
||||
for (let i = 0; i < 14; i++) {
|
||||
const r = await json(`__gm.objectRect('${OBJECT}')`);
|
||||
seen.add(await json(`__gm.pixelAt(${Math.round(r.x)}, ${Math.round(r.y)})`));
|
||||
await sleep(120);
|
||||
}
|
||||
const players = await json('__gm.videos()');
|
||||
check(
|
||||
'a sprite wearing the clip plays only its stretch, round and round',
|
||||
seen.has(COLORS.green) && seen.has(COLORS.blue) && !seen.has(COLORS.red) && !seen.has(COLORS.white),
|
||||
JSON.stringify([...seen]),
|
||||
);
|
||||
check(
|
||||
'the editor plays it without sound',
|
||||
players.length === 1 && players[0].muted && !players[0].paused && players[0].key.includes('#0.6~1.4~1~1'),
|
||||
JSON.stringify(players),
|
||||
);
|
||||
await shot('66-video-sprite');
|
||||
|
||||
// 66g. Play starts it over from the clip's start and lets it be heard; Esc makes it quiet again.
|
||||
await play();
|
||||
await sleep(200);
|
||||
const heard = (await json('__gm.videos()'))[0];
|
||||
check(
|
||||
'Play starts the clip over and lets it be heard',
|
||||
heard && !heard.muted && !heard.paused && heard.time >= 0.6 && heard.time < 1.1,
|
||||
JSON.stringify(heard),
|
||||
);
|
||||
await stop();
|
||||
await sleep(200);
|
||||
const quiet = (await json('__gm.videos()'))[0];
|
||||
check('Esc makes it a quiet preview again', quiet && quiet.muted, JSON.stringify(quiet));
|
||||
|
||||
// 66h. The sprite picked but not in the scene: only the inspector's preview wears the video. It
|
||||
// asks for it every frame, so the player is not let go after 2 s; a let-go texture used to
|
||||
// leave the preview drawing a destroyed one, and the batcher threw on every frame.
|
||||
await mutate(`
|
||||
const p = __gm.state.project;
|
||||
for (const holder of p.assets) if (holder.objects) holder.objects = holder.objects.filter((o) => o.id !== '${OBJECT}');
|
||||
__gm.state.select({ kind: 'asset', assetId: '${SPRITE}' });`);
|
||||
const logFrom = t.consoleLog.length;
|
||||
await sleep(4500);
|
||||
const inspected = await json('__gm.inspectorSpritePreview()');
|
||||
const thrown = t.consoleLog.slice(logFrom).filter((line) => /EXCEPTION|alphaMode/.test(line));
|
||||
const kept = await json('__gm.videos()');
|
||||
const inPreview = inspected && (await shownColor(inspected));
|
||||
check(
|
||||
'a sprite shown only in the inspector keeps its video playing there',
|
||||
thrown.length === 0 &&
|
||||
inspected?.animates === true &&
|
||||
kept.length === 1 &&
|
||||
kept[0].ready &&
|
||||
(inPreview === COLORS.green || inPreview === COLORS.blue),
|
||||
JSON.stringify({ thrown, inspected, kept, inPreview }),
|
||||
);
|
||||
|
||||
await mutate(`
|
||||
const p = __gm.state.project;
|
||||
for (const holder of p.assets) if (holder.objects) holder.objects = holder.objects.filter((o) => o.id !== '${OBJECT}');
|
||||
p.assets = p.assets.filter((a) => a.id !== '${SPRITE}' && a.id !== ${JSON.stringify(assetId)});
|
||||
__gm.state.select(null);`);
|
||||
}
|
||||
Reference in new issue
Block a user