Stream long sounds from an Opus copy instead of decoding them (#349)
A sound was decoded whole as soon as the project opened, in the editor and in a released game: five minutes of stereo held some 115 MB for as long as the page was open. A sound with Stream on now plays from a copy as Opus in Ogg, fetched a piece at a time and decoded a few seconds ahead by WebCodecs, so a game holds a few megabytes for it. - The Stream switch is a row of the sound, off by default. The Sound tab suggests it from a minute on and says how far the copy has got. - StreamKeeper makes the copies in the browser, a moment after the last change, one at a time: the sound's own and one per clip of its cut, fades and Reverse baked in. A copy that is no longer wanted is taken off and deleted once the project is saved without it. - streamTable says which played names stream from which copy. A clip without a fitting copy, a Sample and a Reverb impulse stay decoded. A sound with Stream on is never decoded ahead. - The page fetches by Range and takes the whole file from a server that ignores it, as the CI's PHP server does. It keeps the bytes and the first two seconds decoded, the rest goes four seconds ahead. - A jump decodes 32 packets before the place first. Measured in Chrome, what follows is then the same to the bit as decoding from the start. With 4 packets it was 0.06 off. - The stream-player on the FX worklet reads the samples at the rate of the context along a Catmull-Rom curve. What comes late is silent and skipped once it is there, so a clip keeps its beat. - The opening stretch goes with the player as it is made. Sent by message it came a few milliseconds late, and 557 samples of the attack were lost. - Pitch on a streamed sound is a shifter after the player, as on a bus: the length stays and it is heard 42.7 ms late. - Bakes of patterns and songs (#340) always stream now. - The song timeline draws a streamed sound from its copy decoded at 8 kHz, keeping only the peaks. Tests: - Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3, stream-queue 6, streams 11, streaming 6, sound-library +5). - e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono, 4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by 5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and a reversed clip are the copy decoded whole to the sample (0). - Counter-checks red as they should be: no pre-roll, no catch-up of late samples at the end, no opening stretch, a streamed sound decoded ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
84cd36f902
commit
2764919bb8
30 files changed
+2443
-146
No files matched your search
@@ -1,5 +1,7 @@
|
||||
import { findAssetOf } from '../../core/model/Project';
|
||||
import { clipsOf, findSound } from '../../core/model/sound';
|
||||
import { STREAM_RATE } from '../../core/music/streamQueue';
|
||||
import { playedPlan, streamTable } from '../../core/music/streams';
|
||||
import { playedSoundOf } from '../../core/properties/playedSound';
|
||||
import { soundLibrary } from '../../viewport/SoundLibrary';
|
||||
import { centreOf, type HookContext } from './hookHelpers';
|
||||
@@ -72,5 +74,62 @@ export function soundHooks({ state, editor }: HookContext) {
|
||||
};
|
||||
},
|
||||
},
|
||||
/** Streamed sounds (#349). */
|
||||
streams: {
|
||||
/** Done once every copy wanted has been made. */
|
||||
idle: () => editor.streamKeeper.idle(),
|
||||
/** Every name a game streams, with its copy. */
|
||||
table: () => [...streamTable(state.project)].map(([name, copy]) => ({ name, ...copy })),
|
||||
/**
|
||||
* Streams `seconds` of a name offline from `offset` seconds in, heard from `at`, looped or not,
|
||||
* and holds it against its copy decoded whole: the largest difference of any sample from what
|
||||
* should be heard there – silence before `at` – against the feed's own decoder and against the
|
||||
* browser's `decodeAudioData`, and how many samples came too late.
|
||||
*/
|
||||
exact: async (name: string, offset: number, loop: boolean, at: number, seconds: number) => {
|
||||
const entry = streamTable(state.project).get(name);
|
||||
if (!entry) return null;
|
||||
const plan = playedPlan(entry.frames, offset, loop);
|
||||
const frames = Math.round((at + seconds) * STREAM_RATE);
|
||||
const [played, whole, browser] = await Promise.all([
|
||||
soundLibrary.renderStream(name, plan, at, frames),
|
||||
soundLibrary.streamedWhole(name),
|
||||
soundLibrary.decodeApart(entry.file, STREAM_RATE),
|
||||
]);
|
||||
if (!played || !whole) return null;
|
||||
const first = Math.round(at * STREAM_RATE);
|
||||
const placeOf = (i: number) =>
|
||||
loop ? (plan.from + i - first) % entry.frames : plan.from + i - first;
|
||||
const worst = (reference: (c: number, place: number) => number) => {
|
||||
let most = 0;
|
||||
played.channels.forEach((side, c) => {
|
||||
for (let i = 0; i < frames; i++) {
|
||||
const place = placeOf(i);
|
||||
const expected = i < first || place >= entry.frames ? 0 : reference(c, place);
|
||||
most = Math.max(most, Math.abs(side[i]! - expected));
|
||||
}
|
||||
});
|
||||
return most;
|
||||
};
|
||||
return {
|
||||
frames: entry.frames,
|
||||
decoded: browser.length,
|
||||
own: worst((c, place) => whole[c]![place]!),
|
||||
browser: worst(
|
||||
(c, place) => browser.getChannelData(Math.min(c, browser.numberOfChannels - 1))[place]!,
|
||||
),
|
||||
skipped: played.skipped,
|
||||
};
|
||||
},
|
||||
/** What a sound and each of its clips keep of their copies. */
|
||||
copies: (id: string) => {
|
||||
const asset = findAssetOf(state.project, id, 'sound');
|
||||
if (!asset) return null;
|
||||
return {
|
||||
sound: asset.stream ?? null,
|
||||
clips: clipsOf(asset).map((clip) => ({ id: clip.id, stream: clip.stream ?? null })),
|
||||
};
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user