Stream long sounds from an Opus copy instead of decoding them (#349)

A sound was decoded whole as soon as the project opened, in the editor
and in a released game: five minutes of stereo held some 115 MB for as
long as the page was open. A sound with Stream on now plays from a copy
as Opus in Ogg, fetched a piece at a time and decoded a few seconds
ahead by WebCodecs, so a game holds a few megabytes for it.

- The Stream switch is a row of the sound, off by default. The Sound
  tab suggests it from a minute on and says how far the copy has got.
- StreamKeeper makes the copies in the browser, a moment after the
  last change, one at a time: the sound's own and one per clip of its
  cut, fades and Reverse baked in. A copy that is no longer wanted is
  taken off and deleted once the project is saved without it.
- streamTable says which played names stream from which copy. A clip
  without a fitting copy, a Sample and a Reverb impulse stay decoded.
  A sound with Stream on is never decoded ahead.
- The page fetches by Range and takes the whole file from a server
  that ignores it, as the CI's PHP server does. It keeps the bytes
  and the first two seconds decoded, the rest goes four seconds ahead.
- A jump decodes 32 packets before the place first. Measured in
  Chrome, what follows is then the same to the bit as decoding from
  the start. With 4 packets it was 0.06 off.
- The stream-player on the FX worklet reads the samples at the rate
  of the context along a Catmull-Rom curve. What comes late is silent
  and skipped once it is there, so a clip keeps its beat.
- The opening stretch goes with the player as it is made. Sent by
  message it came a few milliseconds late, and 557 samples of the
  attack were lost.
- Pitch on a streamed sound is a shifter after the player, as on a
  bus: the length stays and it is heard 42.7 ms late.
- Bakes of patterns and songs (#340) always stream now.
- The song timeline draws a streamed sound from its copy decoded at
  8 kHz, keeping only the peaks.

Tests:
- Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3,
  stream-queue 6, streams 11, streaming 6, sound-library +5).
- e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono,
  4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by
  5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and
  a reversed clip are the copy decoded whole to the sample (0).
- Counter-checks red as they should be: no pre-roll, no catch-up of
  late samples at the end, no opening stretch, a streamed sound
  decoded ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-09 20:27:32 +02:00
1 parent 84cd36f902
commit 2764919bb8
30 files changed
+2443 -146

No files matched your search

+59
View File
@@ -1,5 +1,7 @@
import { findAssetOf } from '../../core/model/Project';
import { clipsOf, findSound } from '../../core/model/sound';
import { STREAM_RATE } from '../../core/music/streamQueue';
import { playedPlan, streamTable } from '../../core/music/streams';
import { playedSoundOf } from '../../core/properties/playedSound';
import { soundLibrary } from '../../viewport/SoundLibrary';
import { centreOf, type HookContext } from './hookHelpers';
@@ -72,5 +74,62 @@ export function soundHooks({ state, editor }: HookContext) {
};
},
},
/** Streamed sounds (#349). */
streams: {
/** Done once every copy wanted has been made. */
idle: () => editor.streamKeeper.idle(),
/** Every name a game streams, with its copy. */
table: () => [...streamTable(state.project)].map(([name, copy]) => ({ name, ...copy })),
/**
* Streams `seconds` of a name offline from `offset` seconds in, heard from `at`, looped or not,
* and holds it against its copy decoded whole: the largest difference of any sample from what
* should be heard there – silence before `at` – against the feed's own decoder and against the
* browser's `decodeAudioData`, and how many samples came too late.
*/
exact: async (name: string, offset: number, loop: boolean, at: number, seconds: number) => {
const entry = streamTable(state.project).get(name);
if (!entry) return null;
const plan = playedPlan(entry.frames, offset, loop);
const frames = Math.round((at + seconds) * STREAM_RATE);
const [played, whole, browser] = await Promise.all([
soundLibrary.renderStream(name, plan, at, frames),
soundLibrary.streamedWhole(name),
soundLibrary.decodeApart(entry.file, STREAM_RATE),
]);
if (!played || !whole) return null;
const first = Math.round(at * STREAM_RATE);
const placeOf = (i: number) =>
loop ? (plan.from + i - first) % entry.frames : plan.from + i - first;
const worst = (reference: (c: number, place: number) => number) => {
let most = 0;
played.channels.forEach((side, c) => {
for (let i = 0; i < frames; i++) {
const place = placeOf(i);
const expected = i < first || place >= entry.frames ? 0 : reference(c, place);
most = Math.max(most, Math.abs(side[i]! - expected));
}
});
return most;
};
return {
frames: entry.frames,
decoded: browser.length,
own: worst((c, place) => whole[c]![place]!),
browser: worst(
(c, place) => browser.getChannelData(Math.min(c, browser.numberOfChannels - 1))[place]!,
),
skipped: played.skipped,
};
},
/** What a sound and each of its clips keep of their copies. */
copies: (id: string) => {
const asset = findAssetOf(state.project, id, 'sound');
if (!asset) return null;
return {
sound: asset.stream ?? null,
clips: clipsOf(asset).map((clip) => ({ id: clip.id, stream: clip.stream ?? null })),
};
},
},
};
}