Stream long sounds from an Opus copy instead of decoding them (#349)

A sound was decoded whole as soon as the project opened, in the editor
and in a released game: five minutes of stereo held some 115 MB for as
long as the page was open. A sound with Stream on now plays from a copy
as Opus in Ogg, fetched a piece at a time and decoded a few seconds
ahead by WebCodecs, so a game holds a few megabytes for it.

- The Stream switch is a row of the sound, off by default. The Sound
  tab suggests it from a minute on and says how far the copy has got.
- StreamKeeper makes the copies in the browser, a moment after the
  last change, one at a time: the sound's own and one per clip of its
  cut, fades and Reverse baked in. A copy that is no longer wanted is
  taken off and deleted once the project is saved without it.
- streamTable says which played names stream from which copy. A clip
  without a fitting copy, a Sample and a Reverb impulse stay decoded.
  A sound with Stream on is never decoded ahead.
- The page fetches by Range and takes the whole file from a server
  that ignores it, as the CI's PHP server does. It keeps the bytes
  and the first two seconds decoded, the rest goes four seconds ahead.
- A jump decodes 32 packets before the place first. Measured in
  Chrome, what follows is then the same to the bit as decoding from
  the start. With 4 packets it was 0.06 off.
- The stream-player on the FX worklet reads the samples at the rate
  of the context along a Catmull-Rom curve. What comes late is silent
  and skipped once it is there, so a clip keeps its beat.
- The opening stretch goes with the player as it is made. Sent by
  message it came a few milliseconds late, and 557 samples of the
  attack were lost.
- Pitch on a streamed sound is a shifter after the player, as on a
  bus: the length stays and it is heard 42.7 ms late.
- Bakes of patterns and songs (#340) always stream now.
- The song timeline draws a streamed sound from its copy decoded at
  8 kHz, keeping only the peaks.

Tests:
- Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3,
  stream-queue 6, streams 11, streaming 6, sound-library +5).
- e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono,
  4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by
  5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and
  a reversed clip are the copy decoded whole to the sample (0).
- Counter-checks red as they should be: no pre-roll, no catch-up of
  late samples at the end, no opening stretch, a streamed sound
  decoded ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-09 20:27:32 +02:00
1 parent 84cd36f902
commit 2764919bb8
30 files changed
+2443 -146

No files matched your search

+7 -1
View File
@@ -103,7 +103,11 @@ import {
} from './particles';
import { SHADER_KEY, shaderOf, shaderParamDefs } from './shaders';
import { SHADER_RESOLUTION_KEY, SHADER_RESOLUTION_OPTIONS } from './cameraShader';
import { LOOP_KEY as SOUND_LOOP_KEY, VOLUME_KEY as SOUND_VOLUME_KEY } from './sounds';
import {
LOOP_KEY as SOUND_LOOP_KEY,
STREAM_KEY as SOUND_STREAM_KEY,
VOLUME_KEY as SOUND_VOLUME_KEY,
} from './sounds';
import {
DEFAULT_BARS,
DEFAULT_GRID,
@@ -692,6 +696,8 @@ const ASSET_SCHEMAS: Record<AssetType, PropertySchema> = {
sound: [
{ key: SOUND_VOLUME_KEY, label: 'Volume', kind: 'number', default: 1, min: 0, max: 1, step: 0.05 },
{ key: SOUND_LOOP_KEY, label: 'Loop', kind: 'boolean', default: false },
// A long one is streamed from a copy as Opus rather than decoded whole (#349).
{ key: SOUND_STREAM_KEY, label: 'Stream', kind: 'boolean', default: false },
FX_ROW,
],
// The effects are the asset's graph (see SoundFxAsset), not a property.
+6
View File
@@ -7,6 +7,8 @@ import type { PropertyDef } from './PropertySchema';
export const VOLUME_KEY = 'volume';
/** Whether this sound is music: it repeats until something stops it. */
export const LOOP_KEY = 'loop';
/** Whether a game streams it from a copy as Opus rather than decoding all of it first (#349). */
export const STREAM_KEY = 'stream';
/** Where in its file a clip starts and ends, in seconds; an end of 0 is the end of the file (#343). */
export const START_KEY = 'start';
export const END_KEY = 'end';
@@ -47,6 +49,10 @@ export function loopsOf(bag: PropertyBag): boolean {
return booleanOf(bag[LOOP_KEY]);
}
export function streamsOf(bag: PropertyBag): boolean {
return booleanOf(bag[STREAM_KEY]);
}
/** The stretch of its file a clip is, its fades in seconds. */
export function cutOf(bag: PropertyBag): SoundCut {
const seconds = (key: string, fallback: number) => Math.max(0, coerceAs(bag[key], 'number', fallback));