Play videos on sprites, converted on the server with clips (#402)
A video becomes an asset like a sound: uploaded once, converted to one format by a worker of its own, cut into clips in its tab, and worn by a sprite through its Texture row. Only Chrome is a target, without alpha. - Server: POST ?r=videos queues the upload under the hash of its bytes, so the MP4's name is known at once. The worker (VideoWorker over VideoFormat) writes H.264/AAC, yuv420p, constant frame rate, a keyframe every 15 frames, at most 1920x1080, with a poster and the details beside it. A failed job says why and is tried again on the next upload. - An upload may be up to 2 GB. It is streamed from the request to disk and hashed there, so php-fpm holds none of it in memory. The other routes keep their 256 MB. - The worker runs in the container game-maker-video on stein2, a compose project of its own (tools/video/), and in CI beside the PHP server. The CI image has ffmpeg for it. - Model: asset type video with size, length, frame rate and sound, and clips like a sound's. The clip helpers and commands serve both. - A sprite's Texture row offers videos and clips after the pictures. resolvePicture names a playing video by a key, and the texture library hands it to a VideoLibrary with one player per key. - The editor shows a placed video as a quiet preview. Play starts it over at its clip's start and lets it be heard through the master bus, a pause holds it, Esc makes it quiet again. - Video tab: the picture, a ruler, the clip strip and a track on the Sound tab's time axis. A drag spans a clip, a handle moves an edge, Play plays the picked clip. - The picker shows a video on its way with its progress and adopts an upload from it once it is ready. Dropping a video file does the same. Decided on my own: - Two sprites wearing the same clip share one player and show the same picture. Control per instance belongs to the follow-up with nodes. - A video plays in real time, the game's time scale does not slow it. - The container may use 3 of the 4 cores of stein2. Findings: - The first real upload (279 MB) got 413: the API's own ceiling of 256 MB for every request, not nginx or the Brandmauer. - Chrome has no picture of a video that never played, and a Pixi VideoSource drawn before the first frame stays empty for good. The texture is made in the first frame callback (openVideo). - A player nobody asks for in 2 s is let go. The inspector's PREVIEW asked only on an edit, so with the sprite not on screen its texture was destroyed under it and the batcher threw on every frame. The preview now redraws every frame while it shows a video, like a Time shader. A let-go texture keeps an empty source instead of none. - VideoSource.load() reloads the element and cuts off an earlier play(). Playing starts after loadeddata. Tests: - PHP videos.test.php: 68 checks, with the real ffmpeg - 300 MB upload through Brandmauer and nginx: 201 in 21 s, php-fpm at 13.7 MB RSS at most - Unit videos.test.ts: 20 tests - e2e 66 (stein2, atx was off): ALL PASSED, 18 checks; e2e 04 with the new asset count green - Gate (atx): 294 files, 4520 tests. serialize.test.ts used video as its example of an unknown type and now uses a made-up one. - Counter-checks: a keyframe gap of 30 fails the PHP check, a key that keeps the video's end fails 2 unit tests, a clip without its end fails "plays only its stretch" in e2e 66 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed
1 parent
2f376262d3
commit
d48f174caf
63 files changed
+3500
-103
No files matched your search
@@ -0,0 +1,219 @@
|
||||
<?php
|
||||
declare(strict_types=1);
|
||||
|
||||
/**
|
||||
* Videos (#402): the conversion rules, the store and its queue, and the worker with the real
|
||||
* ffmpeg. Plain PHP like files.test.php; CI runs it with `php tests/php/videos.test.php`. It needs
|
||||
* ffmpeg and ffprobe on the PATH, which the CI image and the container `game-maker-video` have.
|
||||
*/
|
||||
|
||||
require __DIR__ . '/../../api/lib/AtomicFile.php';
|
||||
require __DIR__ . '/../../api/lib/OriginalName.php';
|
||||
require __DIR__ . '/../../api/lib/UserStore.php';
|
||||
require __DIR__ . '/../../api/lib/HashedFileStore.php';
|
||||
require __DIR__ . '/../../api/lib/VideoFormat.php';
|
||||
require __DIR__ . '/../../api/lib/VideoQueue.php';
|
||||
require __DIR__ . '/../../api/lib/VideoStore.php';
|
||||
require __DIR__ . '/../../api/lib/VideoWorker.php';
|
||||
|
||||
$failed = 0;
|
||||
$passed = 0;
|
||||
|
||||
function check(string $name, bool $ok, mixed $got = null): void
|
||||
{
|
||||
global $failed, $passed;
|
||||
if ($ok) {
|
||||
$passed++;
|
||||
echo "PASS {$name}\n";
|
||||
return;
|
||||
}
|
||||
$failed++;
|
||||
echo "FAIL {$name}", $got === null ? '' : ' — got ' . json_encode($got, JSON_UNESCAPED_UNICODE), "\n";
|
||||
}
|
||||
|
||||
/** @param array<string,mixed> $extra */
|
||||
function stream(string $type, array $extra = []): array
|
||||
{
|
||||
return ['codec_type' => $type, ...$extra];
|
||||
}
|
||||
|
||||
// The rules, without ffmpeg.
|
||||
check('4K fits into 1080p', VideoFormat::fit(3840, 2160) === [1920, 1080], VideoFormat::fit(3840, 2160));
|
||||
check('a standing 4K fits standing up', VideoFormat::fit(2160, 3840) === [1080, 1920], VideoFormat::fit(2160, 3840));
|
||||
check('4:3 is bound by its height', VideoFormat::fit(4000, 3000) === [1440, 1080], VideoFormat::fit(4000, 3000));
|
||||
check('1080p stays', VideoFormat::fit(1920, 1080) === [1920, 1080]);
|
||||
check('a small one is not made larger', VideoFormat::fit(640, 360) === [640, 360]);
|
||||
check('odd sides become even, never larger', VideoFormat::fit(641, 361) === [640, 360], VideoFormat::fit(641, 361));
|
||||
check('an ultra-wide one keeps its shape', VideoFormat::fit(5120, 1440) === [1920, 540], VideoFormat::fit(5120, 1440));
|
||||
|
||||
$phone = [
|
||||
'streams' => [
|
||||
stream('video', ['width' => 1920, 'height' => 1080, 'avg_frame_rate' => '30000/1001', 'side_data_list' => [['rotation' => -90]]]),
|
||||
stream('audio'),
|
||||
],
|
||||
'format' => ['duration' => '12.5'],
|
||||
];
|
||||
$plan = VideoFormat::plan($phone);
|
||||
check('a phone video turned on its side is planned standing', [$plan['width'], $plan['height']] === [1080, 1920], $plan);
|
||||
check('its frame rate stays a fraction', $plan['fps'] === '30000/1001', $plan['fps']);
|
||||
check('its sound is kept', $plan['audio'] === true);
|
||||
check('its length comes from the container', $plan['duration'] === 12.5, $plan['duration']);
|
||||
check('an old rotate tag counts too', VideoFormat::plan(['streams' => [stream('video', ['width' => 640, 'height' => 480, 'tags' => ['rotate' => '90']])]])['width'] === 480);
|
||||
check('120 fps is written at 60', VideoFormat::fps(['avg_frame_rate' => '120/1']) === '60');
|
||||
check('an unknown average falls back to the stream rate', VideoFormat::fps(['avg_frame_rate' => '0/0', 'r_frame_rate' => '25/1']) === '25');
|
||||
check('and with none at all to 30', VideoFormat::fps([]) === '30');
|
||||
check('30000/1001 is 29.97 frames a second', abs(VideoFormat::fpsValue('30000/1001') - 29.97) < 0.001);
|
||||
$silent = VideoFormat::plan(['streams' => [stream('video', ['width' => 320, 'height' => 240, 'avg_frame_rate' => '24/1'])]]);
|
||||
check('a file without sound has none', $silent['audio'] === false);
|
||||
check('and is converted without', in_array('-an', VideoFormat::convertArgs('in', 'out', $silent), true) && !in_array('0:a:0', VideoFormat::convertArgs('in', 'out', $silent), true));
|
||||
$noPicture = static function (array $probe): bool {
|
||||
try {
|
||||
VideoFormat::plan($probe);
|
||||
return false;
|
||||
} catch (InvalidArgumentException) {
|
||||
return true;
|
||||
}
|
||||
};
|
||||
check('a sound alone is no video', $noPicture(['streams' => [stream('audio')]]));
|
||||
check('nor is a song with a cover picture', $noPicture(['streams' => [stream('audio'), stream('video', ['width' => 600, 'height' => 600, 'disposition' => ['attached_pic' => 1]])]]));
|
||||
check('a progress line says how far', VideoFormat::progress("out_time_us=2500000\n", 10.0) === 0.25);
|
||||
check('other lines say nothing', VideoFormat::progress('frame=12', 10.0) === null && VideoFormat::progress('out_time_us=1', 0.0) === null);
|
||||
check('progress never passes the end', VideoFormat::progress('out_time_us=99000000', 10.0) === 1.0);
|
||||
|
||||
// The store and its queue.
|
||||
$data = sys_get_temp_dir() . '/game-maker-php-videos-' . bin2hex(random_bytes(4));
|
||||
$store = new VideoStore("{$data}/videos");
|
||||
$folder = 'a1b2c3d4e5f6';
|
||||
$rejected = static function (string $bytes) use ($store, $folder): bool {
|
||||
try {
|
||||
$store->store($bytes, 'x', $folder);
|
||||
return false;
|
||||
} catch (InvalidVideoException) {
|
||||
return true;
|
||||
}
|
||||
};
|
||||
check('a sound is turned away', $rejected('RIFF' . pack('V', 36) . 'WAVE' . str_repeat("\0", 32)));
|
||||
check('so is text', $rejected(str_repeat('hello ', 10)));
|
||||
check('so is something too short to be anything', $rejected("\0\0\0\x18ftyp"));
|
||||
|
||||
$fake = "\0\0\0\x18ftypisom" . str_repeat("\0", 64) . 'not really a video';
|
||||
$hash = hash('sha256', $fake);
|
||||
$record = $store->store($fake, 'holiday.mov', $folder);
|
||||
check('an upload is named after its bytes, before it is converted', $record['file'] === "{$folder}/{$hash}.mp4", $record);
|
||||
check('and waits', $record['status'] === 'queued' && $record['name'] === 'holiday.mov', $record);
|
||||
check('the upload is hidden from the web server', (fileperms($store->queue()->source($folder, $hash)) & 0777) === 0640);
|
||||
check('it is there for the picker', array_column($store->list($folder, false, [$folder => 'ann']), 'status') === ['queued']);
|
||||
check('but not for another account', $store->list('ffffffffffff', false, []) === []);
|
||||
check('an administrator sees it with its owner', $store->list('', true, [$folder => 'ann'])[0]['owner'] === 'ann');
|
||||
check('it counts as there for deleting', $store->exists("{$folder}/{$hash}.mp4"));
|
||||
check('deleting it drops the job and the upload', $store->delete("{$folder}/{$hash}.mp4") && $store->queue()->jobs() === [] && !is_file($store->queue()->source($folder, $hash)));
|
||||
check('and its name', !is_file("{$data}/videos/{$folder}/{$hash}.mp4.json"));
|
||||
|
||||
$store->store($fake, 'holiday.mov', $folder);
|
||||
$store->queue()->update($folder, $hash, ['state' => VideoQueue::CONVERTING, 'progress' => 0.4]);
|
||||
check('a job cut off by a stopped worker is queued again', $store->queue()->requeueInterrupted() === 1 && $store->queue()->job($folder, $hash)['state'] === 'queued');
|
||||
|
||||
// An upload straight from the request, as the API takes it: to disk, never more than allowed.
|
||||
$streamOf = static function (string $bytes) {
|
||||
$in = fopen('php://memory', 'r+b');
|
||||
fwrite($in, $bytes);
|
||||
rewind($in);
|
||||
return $in;
|
||||
};
|
||||
$tooLarge = static function (string $bytes, int $max) use ($store, $folder, $streamOf): bool {
|
||||
try {
|
||||
$store->storeStream($streamOf($bytes), $max, 'x', $folder);
|
||||
return false;
|
||||
} catch (VideoTooLargeException) {
|
||||
return true;
|
||||
}
|
||||
};
|
||||
$streamed = $fake . 'streamed';
|
||||
check('one byte over the ceiling is turned away', $tooLarge($streamed, strlen($streamed) - 1));
|
||||
$record = $store->storeStream($streamOf($streamed), strlen($streamed), 'streamed.mp4', $folder);
|
||||
check('right at the ceiling it is taken, named after its bytes', $record['file'] === "{$folder}/" . hash('sha256', $streamed) . '.mp4' && $record['status'] === 'queued', $record);
|
||||
check('the streamed upload is hidden as well', (fileperms($store->queue()->source($folder, hash('sha256', $streamed))) & 0777) === 0640);
|
||||
check('text from a stream is turned away too', $rejected(str_repeat('hello ', 10)));
|
||||
check('no received upload is left lying around', glob("{$data}/videos/queue/upload-*") === []);
|
||||
$store->delete($record['file']);
|
||||
|
||||
exec('command -v ffmpeg && command -v ffprobe', $found, $code);
|
||||
check('ffmpeg and ffprobe are on the PATH', $code === 0);
|
||||
if ($code !== 0) {
|
||||
exec('rm -rf ' . escapeshellarg($data));
|
||||
echo "{$failed} FAILED, {$passed} passed\n";
|
||||
exit(1);
|
||||
}
|
||||
|
||||
$worker = new VideoWorker($store, "{$data}/videos");
|
||||
check('the worker takes the oldest job', $worker->runOnce());
|
||||
$broken = $store->record("{$folder}/{$hash}.mp4");
|
||||
check('bytes ffmpeg cannot read fail the job, with its words', $broken['status'] === 'failed' && str_starts_with($broken['error'], 'ffmpeg cannot read this file'), $broken);
|
||||
check('the upload goes, the job stays to say why', !is_file($store->queue()->source($folder, $hash)) && $store->queue()->job($folder, $hash) !== null);
|
||||
check('nothing is left half written', glob("{$data}/videos/{$folder}/*.tmp.*") === []);
|
||||
check('nothing waits any more', $worker->runOnce() === false);
|
||||
|
||||
/** A test picture with a tone, written by ffmpeg itself. */
|
||||
function makeVideo(string $path, string $size, int $rate, float $seconds, bool $sound): void
|
||||
{
|
||||
$args = ['ffmpeg', '-v', 'error', '-y', '-f', 'lavfi', '-i', "testsrc2=size={$size}:rate={$rate}:duration={$seconds}"];
|
||||
if ($sound) {
|
||||
$args = [...$args, '-f', 'lavfi', '-i', "sine=frequency=440:duration={$seconds}", '-c:a', 'pcm_s16le'];
|
||||
}
|
||||
exec(implode(' ', array_map('escapeshellarg', [...$args, '-c:v', 'mjpeg', '-q:v', '5', '-shortest', $path])), $out, $code);
|
||||
if ($code !== 0) {
|
||||
throw new RuntimeException("cannot make {$path}");
|
||||
}
|
||||
}
|
||||
|
||||
$tmp = "{$data}/in";
|
||||
mkdir($tmp, 0775, true);
|
||||
makeVideo("{$tmp}/big.mov", '3840x2160', 25, 1.2, true);
|
||||
makeVideo("{$tmp}/small.mov", '320x240', 50, 1.0, false);
|
||||
$big = $store->store((string) file_get_contents("{$tmp}/big.mov"), 'big.mov', $folder);
|
||||
$small = $store->store((string) file_get_contents("{$tmp}/small.mov"), 'small.mov', $folder);
|
||||
$started = microtime(true);
|
||||
check('the worker converts both, one after the other', $worker->runOnce() && $worker->runOnce() && !$worker->runOnce());
|
||||
echo sprintf(" (converted in %.1f s)\n", microtime(true) - $started);
|
||||
|
||||
$done = $store->record($big['file']);
|
||||
check('a converted video is ready', $done['status'] === 'ready', $done);
|
||||
check('4K came out as 1080p', [$done['width'], $done['height']] === [1920, 1080], $done);
|
||||
check('it says its length and frame rate', abs($done['duration'] - 1.2) < 0.05 && $done['fps'] === 25.0, $done);
|
||||
check('and that it has sound', $done['audio'] === true);
|
||||
check('its poster lies beside it, named like the file', $done['poster'] === substr($big['file'], 0, -4) . '.jpg' && is_file("{$data}/videos/{$done['poster']}"), $done);
|
||||
check('both are served as they are', (fileperms("{$data}/videos/{$big['file']}") & 0777) === 0664 && (fileperms("{$data}/videos/{$done['poster']}") & 0777) === 0664);
|
||||
check('the job and the upload are gone', $store->queue()->job($folder, basename($big['file'], '.mp4')) === null && !is_file($store->queue()->source($folder, basename($big['file'], '.mp4'))));
|
||||
|
||||
$probe = json_decode((string) shell_exec('ffprobe -v error -print_format json -show_streams ' . escapeshellarg("{$data}/videos/{$big['file']}")), true);
|
||||
$codecs = array_column($probe['streams'], 'codec_name');
|
||||
check('it is H.264 with AAC', $codecs === ['h264', 'aac'], $codecs);
|
||||
check('in yuv420p', $probe['streams'][0]['pix_fmt'] === 'yuv420p', $probe['streams'][0]['pix_fmt']);
|
||||
$packets = trim((string) shell_exec('ffprobe -v error -select_streams v -show_entries packet=flags -of csv=p=0 ' . escapeshellarg("{$data}/videos/{$big['file']}")));
|
||||
$keyframes = array_keys(array_filter(explode("\n", $packets), static fn(string $flags): bool => str_contains($flags, 'K')));
|
||||
check('with a keyframe every 15 frames', $keyframes === [0, 15], $keyframes);
|
||||
$bytes = (string) file_get_contents("{$data}/videos/{$big['file']}");
|
||||
check('and its index up front, so it starts before it is loaded', strpos($bytes, 'moov') < strpos($bytes, 'mdat'));
|
||||
|
||||
$quiet = $store->record($small['file']);
|
||||
check('a small one stays small', [$quiet['width'], $quiet['height']] === [320, 240], $quiet);
|
||||
check('a silent one says so', $quiet['audio'] === false && $quiet['fps'] === 50.0, $quiet);
|
||||
|
||||
$again = $store->store((string) file_get_contents("{$tmp}/big.mov"), 'big again.mov', $folder);
|
||||
check('the same upload again is the converted file, without a job', $again['status'] === 'ready' && $store->queue()->job($folder, basename($big['file'], '.mp4')) === null, $again);
|
||||
check('under its new name', $again['name'] === 'big again.mov');
|
||||
$listed = array_column($store->list($folder, false, [$folder => 'ann']), null, 'file');
|
||||
check('the picker lists ready and failed ones', $listed[$big['file']]['status'] === 'ready' && $listed["{$folder}/{$hash}.mp4"]['status'] === 'failed', array_column($listed, 'status', 'file'));
|
||||
|
||||
$fakeAgain = $store->store($fake, 'holiday.mov', $folder);
|
||||
check('a failed upload sent again is tried again', $fakeAgain['status'] === 'queued', $fakeAgain);
|
||||
|
||||
check('deleting a converted video takes poster and details with it', $store->delete($big['file'])
|
||||
&& glob("{$data}/videos/{$folder}/" . basename($big['file'], '.mp4') . '*') === []);
|
||||
check('deleting a file that is not there says so', $store->delete($big['file']) === false);
|
||||
check('a path out of the store is no file to delete', $store->delete('../queue/x.mp4') === false);
|
||||
|
||||
exec('rm -rf ' . escapeshellarg($data));
|
||||
|
||||
echo $failed === 0 ? "ALL PASSED – {$passed} checks\n" : "{$failed} FAILED, {$passed} passed\n";
|
||||
exit($failed === 0 ? 0 : 1);
|
||||
Reference in new issue
Block a user