Stream long sounds from an Opus copy instead of decoding them (#349)
A sound was decoded whole as soon as the project opened, in the editor and in a released game: five minutes of stereo held some 115 MB for as long as the page was open. A sound with Stream on now plays from a copy as Opus in Ogg, fetched a piece at a time and decoded a few seconds ahead by WebCodecs, so a game holds a few megabytes for it. - The Stream switch is a row of the sound, off by default. The Sound tab suggests it from a minute on and says how far the copy has got. - StreamKeeper makes the copies in the browser, a moment after the last change, one at a time: the sound's own and one per clip of its cut, fades and Reverse baked in. A copy that is no longer wanted is taken off and deleted once the project is saved without it. - streamTable says which played names stream from which copy. A clip without a fitting copy, a Sample and a Reverb impulse stay decoded. A sound with Stream on is never decoded ahead. - The page fetches by Range and takes the whole file from a server that ignores it, as the CI's PHP server does. It keeps the bytes and the first two seconds decoded, the rest goes four seconds ahead. - A jump decodes 32 packets before the place first. Measured in Chrome, what follows is then the same to the bit as decoding from the start. With 4 packets it was 0.06 off. - The stream-player on the FX worklet reads the samples at the rate of the context along a Catmull-Rom curve. What comes late is silent and skipped once it is there, so a clip keeps its beat. - The opening stretch goes with the player as it is made. Sent by message it came a few milliseconds late, and 557 samples of the attack were lost. - Pitch on a streamed sound is a shifter after the player, as on a bus: the length stays and it is heard 42.7 ms late. - Bakes of patterns and songs (#340) always stream now. - The song timeline draws a streamed sound from its copy decoded at 8 kHz, keeping only the peaks. Tests: - Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3, stream-queue 6, streams 11, streaming 6, sound-library +5). - e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono, 4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by 5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and a reversed clip are the copy decoded whole to the sample (0). - Counter-checks red as they should be: no pre-roll, no catch-up of late samples at the end, no opening stretch, a streamed sound decoded ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed
1 parent
84cd36f902
commit
2764919bb8
30 files changed
+2443
-146
No files matched your search
@@ -1,4 +1,7 @@
|
||||
import { execSync } from 'node:child_process';
|
||||
import { sineWav } from '../lib/audio.mjs';
|
||||
import { edge, node, port, variable, written } from '../lib/builders.mjs';
|
||||
import { uploadSound } from '../lib/projects.mjs';
|
||||
import { LANES, MAKE_OBJECT, pickAssetType } from './lib/scaffold.mjs';
|
||||
|
||||
/**
|
||||
@@ -2570,6 +2573,201 @@ export async function sound(t) {
|
||||
p.assets = p.assets.filter((a) => a.type !== 'bus');
|
||||
p.routing = { nodes: [], edges: [] };`);
|
||||
|
||||
// 34h-4. A long sound streamed (#349): five minutes of mono, decoded whole some 55 MB. With Stream
|
||||
// on, the editor makes a copy of it as Opus and a game plays from that, through its bus and
|
||||
// the master; the renderer grows by a few megabytes while it plays, not by the decoded
|
||||
// file. The counter-proof right after turns Stream off: decoded, it grows by about that much.
|
||||
const long = await uploadSound(t.baseUrl, sineWav(300, 330), 'long.wav');
|
||||
const rendererMb = () =>
|
||||
execSync('ps -eo rss,args')
|
||||
.toString()
|
||||
.split('\n')
|
||||
.filter((line) => line.includes('--type=renderer'))
|
||||
.reduce((sum, line) => sum + Number(line.trim().split(/\s+/)[0]) / 1024, 0);
|
||||
const restingMb = async () => {
|
||||
await t.cdp.send('HeapProfiler.collectGarbage');
|
||||
await sleep(1500);
|
||||
return rendererMb();
|
||||
};
|
||||
// What the game runs: On start plays the long sound, or a song.
|
||||
const startWith = (played) =>
|
||||
mutate(`
|
||||
__gm.state.project.assets.find((a) => a.id === 'sn-logic').graph = ${JSON.stringify({
|
||||
nodes: [node('sn-start', 'OnStart', 0, 0), played],
|
||||
edges: [edge('sn-e1', 'sn-start', 'exec', played.id, 'exec')],
|
||||
})};`);
|
||||
const playsLong = node('sn-play', 'PlaySound', 260, 0, { sound: 'sn-long', volume: 1, loop: '' });
|
||||
await evaluate('__gm.sound.stopAll()');
|
||||
await mutate(`
|
||||
const p = __gm.state.project;
|
||||
p.assets.push(
|
||||
{ id: 'sn-long', type: 'sound', name: 'long', props: { volume: 1, loop: true, fx: '', stream: true }, file: ${JSON.stringify(long.file)} },
|
||||
{ id: 'sn-bus', type: 'bus', name: 'Music', props: { volume: 1, fx: '' } },
|
||||
);
|
||||
p.routing = {
|
||||
nodes: [
|
||||
{ id: 'sn-long', type: 'RoutedAsset', x: 0, y: 0, params: { name: 'long', kind: 'sound' } },
|
||||
{ id: 'sn-bus', type: 'RoutedAsset', x: 320, y: 0, params: { name: 'Music', kind: 'bus' } },
|
||||
{ id: '#master', type: 'RoutedMaster', x: 640, y: -80, params: {} },
|
||||
],
|
||||
edges: [
|
||||
{ id: 'sn-r1', from: { node: 'sn-long', port: 'out' }, to: { node: 'sn-bus', port: 'in' } },
|
||||
{ id: 'sn-r2', from: { node: 'sn-bus', port: 'out' }, to: { node: '#master', port: 'in' } },
|
||||
],
|
||||
};`);
|
||||
await startWith(playsLong);
|
||||
await evaluate('__gm.streams.idle()');
|
||||
const copies = await json("__gm.streams.copies('sn-long')");
|
||||
const streamed = (await json('__gm.streams.table()')).find((entry) => entry.name === long.file);
|
||||
check(
|
||||
'with Stream on the editor makes a copy of the long sound as Opus, five minutes to the sample, and a game streams it from there',
|
||||
copies?.sound?.from === long.file &&
|
||||
copies.sound.frames === 300 * 48000 &&
|
||||
copies.sound.bytes > 0 &&
|
||||
streamed?.file === copies.sound.file &&
|
||||
streamed.frames === 300 * 48000,
|
||||
JSON.stringify({ long, copies, streamed }),
|
||||
);
|
||||
|
||||
const decodedBefore = (await stats()).decoded;
|
||||
const restStreamed = await restingMb();
|
||||
await play();
|
||||
const tapsLong = JSON.stringify(['mix', 'mix:sn-bus']);
|
||||
await json(`__gm.routing.peaks(${tapsLong})`);
|
||||
const streaming = await waitFor(stats, (s) => s.streaming.playing > 0, 40);
|
||||
const heardLong = await waitFor(
|
||||
() => json(`__gm.routing.peaks(${tapsLong})`),
|
||||
(peaks) => Math.min(Math.max(...(peaks.mix ?? [0])), Math.max(...(peaks['mix:sn-bus'] ?? [0]))) > 0.01,
|
||||
);
|
||||
await sleep(2500);
|
||||
const playingStreamed = rendererMb();
|
||||
await t.clickToolbar('pause');
|
||||
await sleep(500);
|
||||
const paused = { clock: await json('__gm.sound.clock()'), state: await json('__gm.sound.clockState()') };
|
||||
await sleep(500);
|
||||
const pausedLater = await json('__gm.sound.clock()');
|
||||
await t.clickToolbar('pause');
|
||||
await sleep(1000);
|
||||
const resumedLong = await stats();
|
||||
await stop();
|
||||
const stoppedLong = await stats();
|
||||
check(
|
||||
'played, the streamed sound is heard on its bus and the master, the renderer growing by less than 20 MB, none of the file decoded',
|
||||
streaming?.last?.file === long.file &&
|
||||
streaming.last.bus === 'mix:sn-bus' &&
|
||||
streaming.decoded === decodedBefore &&
|
||||
playingStreamed - restStreamed < 20 &&
|
||||
Math.max(...(heardLong?.mix ?? [0])) > 0.01,
|
||||
JSON.stringify({ restStreamed, playingStreamed, streaming, heardLong }),
|
||||
);
|
||||
check(
|
||||
'the pause holds the stream where it is, it plays on after it without a sample lost, and stopping ends it',
|
||||
paused.state === 'suspended' &&
|
||||
pausedLater === paused.clock &&
|
||||
resumedLong.streaming.playing === 1 &&
|
||||
resumedLong.streaming.skipped === 0 &&
|
||||
stoppedLong.streaming.playing === 0 &&
|
||||
stoppedLong.playing === 0,
|
||||
JSON.stringify({ paused, pausedLater, resumedLong: resumedLong.streaming, stoppedLong }),
|
||||
);
|
||||
|
||||
// The counter-proof: Stream off, the game plays it decoded, and the renderer holds all of it.
|
||||
// Measured from before it is switched off, as from then on the editor decodes it ahead, and before
|
||||
// anything below has had large buffers made: Chrome does not hand all of that back at once.
|
||||
await evaluate('__gm.sound.stopAll()');
|
||||
const restDecoded = await restingMb();
|
||||
await mutate(`__gm.state.project.assets.find((a) => a.id === 'sn-long').props.stream = false;`);
|
||||
await play();
|
||||
const decodedNow = await waitFor(stats, (s) => s.playing > 0 && s.last?.file === long.file, 60);
|
||||
await sleep(2500);
|
||||
const playingDecoded = rendererMb();
|
||||
await stop();
|
||||
await evaluate('__gm.streams.idle()');
|
||||
check(
|
||||
'with Stream off the same sound is decoded whole, and the renderer grows by more than 40 MB',
|
||||
decodedNow?.streaming.playing === 0 &&
|
||||
decodedNow.decoded > decodedBefore &&
|
||||
playingDecoded - restDecoded > 40 &&
|
||||
(await json("__gm.streams.copies('sn-long')"))?.sound === null,
|
||||
JSON.stringify({ restDecoded, playingDecoded, decodedNow }),
|
||||
);
|
||||
|
||||
// The Sound tab points a long sound at the switch, and says once it streams.
|
||||
await evaluate("__gm.state.openAsset('sn-long')");
|
||||
await sleep(300);
|
||||
const longInfo = await json('__gm.soundTab.info()');
|
||||
await mutate(`__gm.state.project.assets.find((a) => a.id === 'sn-long').props.stream = true;`);
|
||||
await evaluate('__gm.streams.idle()');
|
||||
await sleep(200);
|
||||
const streamedInfo = await json('__gm.soundTab.info()');
|
||||
check(
|
||||
'the Sound tab suggests Stream for a sound of a minute or more, and says when it is streamed',
|
||||
longInfo === '300.00 s · long, try Stream' && streamedInfo === '300.00 s · streamed',
|
||||
JSON.stringify({ longInfo, streamedInfo }),
|
||||
);
|
||||
// On a track of a song it starts on its beat, a clip of its own streamed from a copy of its cut.
|
||||
await mutate(`
|
||||
const sound = __gm.state.project.assets.find((a) => a.id === 'sn-long');
|
||||
sound.clips = [{ id: 'sn-long-clip', name: 'Middle', props: { start: 10, end: 20, fadeIn: 5, fadeOut: 5, fadeInCurve: 'linear', fadeOutCurve: 'linear', reverse: true } }];`);
|
||||
await clickText('+');
|
||||
await pickAssetType(t, 'Song');
|
||||
await sleep(200);
|
||||
const [longSong] = await assetsOfType('song');
|
||||
await mutate(`
|
||||
const song = __gm.state.project.assets.find((a) => a.id === ${JSON.stringify(longSong.id)});
|
||||
song.items = [{ id: 'sn-i1', track: song.tracks[0].id, kind: 'sound', asset: 'sn-long-clip', tick: 0, length: 3840, offset: 0, volume: 0, transpose: 0, instrument: '', fx: '' }];`);
|
||||
await evaluate('__gm.streams.idle()');
|
||||
await startWith(
|
||||
node('sn-song', 'PlaySong', 260, 0, { song: longSong.id, volume: 1, fadeIn: 0, loop: false }),
|
||||
);
|
||||
await showTab('scene');
|
||||
await play();
|
||||
const songRun = await waitFor(
|
||||
async () => ({ runs: await json('__gm.sound.playbacks()'), stats: await stats() }),
|
||||
(got) => got.runs[0]?.clips.length > 0 && got.stats.streaming.playing > 0,
|
||||
40,
|
||||
);
|
||||
await stop();
|
||||
const clipCopy = (await json("__gm.streams.copies('sn-long')"))?.clips[0]?.stream;
|
||||
const clipName = await json("__gm.soundTab.played('sn-long-clip')");
|
||||
const songClip = songRun?.runs[0]?.clips[0];
|
||||
check(
|
||||
'a clip of the streamed sound gets a copy of its cut, and a song streams it on its track, started on its beat',
|
||||
clipCopy?.from === clipName &&
|
||||
clipCopy.frames === 10 * 48000 &&
|
||||
songClip?.sound === 'sn-long-clip' &&
|
||||
songRun.stats.last?.file === clipName &&
|
||||
Math.abs(songRun.stats.last.at - songClip.at) < 1e-6,
|
||||
JSON.stringify({ clipCopy, clipName, songClip, last: songRun?.stats.last }),
|
||||
);
|
||||
|
||||
// Offline, to the sample: from 0.25 s on, two seconds into the copy; round the seam of its loop;
|
||||
// and the reversed clip from its start – each against the copy decoded whole.
|
||||
const exactly = async (...args) =>
|
||||
JSON.parse(await evaluate(`__gm.streams.exact(...${JSON.stringify(args)}).then(JSON.stringify)`));
|
||||
const exact = {
|
||||
start: await exactly(long.file, 2, false, 0.25, 2),
|
||||
seam: await exactly(long.file, 299, true, 0, 2.5),
|
||||
clip: await exactly(clipName, 9, true, 0.1, 2),
|
||||
};
|
||||
check(
|
||||
'streamed, it starts on its sample, runs on past where it jumped in and round its loop as the copy decoded whole',
|
||||
Object.values(exact).every(
|
||||
(run) =>
|
||||
run !== null &&
|
||||
run.own < 1e-6 &&
|
||||
run.browser < 0.01 &&
|
||||
run.skipped === 0 &&
|
||||
run.decoded === run.frames,
|
||||
),
|
||||
JSON.stringify(exact),
|
||||
);
|
||||
|
||||
await mutate(`
|
||||
const p = __gm.state.project;
|
||||
p.assets = p.assets.filter((a) => !['sn-long', 'sn-bus', ${JSON.stringify(longSong.id)}].includes(a.id));
|
||||
p.routing = { nodes: [], edges: [] };`);
|
||||
|
||||
// 34i. The step takes back everything it made, so the steps after it see the fixture again.
|
||||
await mutate(`
|
||||
const gone = new Set(['sn-logic', ...${JSON.stringify(both.map((a) => a.id))}]);
|
||||
|
||||
Reference in new issue
Block a user