Describe streaming long sounds from an Opus copy (#349)

- Sounds: the Stream row, a new section on what streams, what the
  copy is, what stays decoded, and the measured memory.
- Songs: a bake is streamed now, not decoded with the project.
- Architecture: streams, streamQueue, streaming and StreamKeeper.
- Development: 1935 e2e checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adminandClaude Opus 5.5 committed 2026-10-09 20:27:32 +02:00
1 parent 26f23debb8
commit 5515c67008
4 files changed
+59 -7

No files matched your search

+11 -3
@@ -173,7 +173,10 @@ src/core/music/ time (ticks, 480 to the quarter, tempo and meter, a posit
pattern item), take (#342: a recorded take as 16-bit WAV, cut to where the song started),
bake (#340: what a bake is made from as a print, where the parts of its file lie, how they
are put together from the playing looped and the one played once, and played back) +
ogg (#340: Opus in Ogg, the pages around an encoder's packets and back)
ogg (#340: Opus in Ogg, the pages around an encoder's packets and back, also read as
the bytes come in, #349), streams (#349: which names stream from which copy, what
copies are missing, how a played sound walks its copy) + streamQueue (#349: the
samples a streamed sound plays from on the worklet, read at the rate of the context)
src/core/soundfx/ the sound FX graph (#311): nodes (every effect and what it adds to a plan), compile (the
graph walked back from the Output to a plan; a parameter stays open until a voice
starts; a tapped node gets a `tap` in front and behind, #374), parametricEq (#367:
@@ -252,7 +255,10 @@ src/viewport/ WorldViewport (the drawing of a world or a holder, the ba
sound until its end, a bus per track of a song with its level and pan, #334; the buses of
the mix and the Master routing, #325/#351; a bake as one voice on the master of its
playing, and a library of its own on an offline context to bake with, #340; a channel
probe that says whether a bake is mono), bakeRender (#340: a pattern or song played
probe that says whether a bake is mono; a sound or a bake streamed from its copy by a
`stream-player`, #349), streaming (#349: a copy fetched by Range, its head decoded
ahead, the rest decoded by WebCodecs a few seconds before it is heard),
bakeRender (#340: a pattern or song played
offline, what reaches its master taken, encoded as Opus with WebCodecs), music (the scheduler on
the library's clock, its notes played by the project's instruments and its clips by
the library: the player of the Pattern and Song tabs, #333/#334, and `__gm.music`) + musicTimer
@@ -405,7 +411,9 @@ src/editor/song/ SongEditor (header: Play, Grid, Bake; plays through the s
ownFx (#355: a sound FX the song keeps for a song, a track or an item)
src/editor/sound/ SoundEditor (#343: header with Play, the clip's direction and fades), WaveCanvas (the
waveform, the clips as bands, the cut) + waveView (its time axis), SoundPreview (play
and length in the inspector), SoundPicker (the "+ → Sound" rows with their play mark)
and length in the inspector), SoundPicker (the "+ → Sound" rows with their play mark),
StreamKeeper (#349: the copies streamed sounds and their clips play from, made in the
browser and let go once no longer wanted)
src/editor/soundfx/ SoundFxEditor (#311: the graph, a sound to try it on, Play and Stop), PreviewParams (the
header's fields for the parameters a tab plays with, shared with the Instrument tab),
EqEditor + eqView (#367/#370: the Parametric EQ in a tab of its own, its handles on
+1 -1
@@ -19,7 +19,7 @@ php tests/php/accounts.test.php # the API's accoun
php tests/php/snapshots.test.php # the snapshot store (#231), the same way
php tests/php/releases.test.php # the release store (#266), the same way
php tests/php/files.test.php # uploaded files in a folder per account (#344), the same way
node tools/seed-e2e.mjs && node tools/e2e.mjs # 1928 editor checks, screenshots in /tmp/gm-e2e-*.png
node tools/seed-e2e.mjs && node tools/e2e.mjs # 1935 editor checks, screenshots in /tmp/gm-e2e-*.png
GM_STEPS=20 node tools/e2e.mjs # or 1-12, or 6,20 — a subset while working on one step
./e2e.sh start [GM_STEPS] # the same run detached, see below
./e2e.sh status # live progress, or the last run's result
+4 -2
@@ -385,8 +385,10 @@ edit is heard at once.
level it had. It goes up into the sounds of your account under *[bake] name*, and the sound picker
leaves it out. The old file of a bake made again or removed is deleted once the project is saved
without it, unless a snapshot or a release still names it.
- **Memory:** a bake that holds is loaded and decoded with the project like a sound, at 48 kHz:
about 23 MB a minute in stereo. An outdated one is not loaded. Streaming it instead is #349.
- **Memory:** a bake is [streamed](Sounds#streaming-long-sounds) (#349): only its compressed bytes
and its first two seconds are held, never the whole of it decoded. An outdated one is not
loaded. Where a browser cannot stream it, it is decoded at 48 kHz as before: about 23 MB a minute
in stereo.
Measured (2026-10-09, e2e 64 on the CI's Windows machine): a bar of quarters at 150 BPM with one
more note just before its end, baked: 76800 frames for the bar and 76800 for the second round,
+43 -1
@@ -30,6 +30,7 @@ signature, so nothing a request claims can decide what is written.
| --- | --- |
| **Volume** | How loud this sound is by itself, 0-1. |
| **Loop** | Whether it is music: it repeats until something stops it. |
| **Stream** | Whether a game plays it from a copy as Opus, a few seconds at a time, rather than decoding all of it first (#349). Off by default. See [Streaming long sounds](#streaming-long-sounds). |
| **Sound FX** | The [sound FX](#sound-fx) it goes through wherever it plays, none while none is picked (#352). See [The sound FX of an asset](#the-sound-fx-of-an-asset). |
The preview plays through the very same library a running game uses — same decode, same device —
@@ -203,7 +204,8 @@ sounds as well: playing a loop twice gives two loops, and it is `Stop sound` tha
them at once.
**It is loaded ahead.** Every sound of the project is fetched and decoded as soon as the project is
open, the way images and fonts are, so the first play starts in the frame its node ran. Decoding
open, the way images and fonts are, so the first play starts in the frame its node ran. A sound
that [streams](#streaming-long-sounds) has only the start of its copy fetched. Decoding
ahead opens no audio device: that only happens on the first play. A sound that is still loading
when it is played — a run started the moment the project opened — starts a moment late instead.
`Play sound` never waits on the network, and a `Stop sound` that happens while the bytes are still
@@ -233,6 +235,46 @@ any edit, so a loop cannot pile up one voice per keystroke.
normal speed rather than sounding like a tape being slowed — the global clock scales what the
world does, not how the device plays back.
## Streaming long sounds
A sound is decoded whole before it plays: 48000 samples a second for each side, four bytes each.
That is nothing for a jump, but five minutes of music in stereo are about 115 MB, held for as long
as the page is open. **Stream** (#349) plays such a sound from a copy, a few seconds at a time.
- **The copy.** Switched on, the editor decodes the file once, encodes it as Opus in Ogg (160 kbit/s
in stereo, 96 kbit/s in mono, as a [bake](Songs#baking) is) and uploads it as *[stream] name*.
The sound picker leaves it out. The original stays as it is: the Sound tab shows and cuts that one.
Switched off, the copy is taken off the sound and deleted once the project is saved without it.
- **Each clip gets a copy of its own cut**, its fades and Reverse in it, so a clip never needs the
whole file. A changed clip gets a new copy a moment after the last change. A clip that is the whole
file plays the sound's copy.
- **Until a copy is there**, the sound or clip plays decoded, as without Stream. A sound with Stream
on is never decoded ahead when the project opens, though: only once it is played without a copy.
- **What a game holds** is the compressed copy as it is fetched, and its first two seconds decoded,
so it starts at once. The rest is decoded about four seconds ahead of what is heard and let go
once it has been played.
- **Everything after it stays the same:** volume, Loop, its sound FX, its bus and the master. A
`Play sound`, a song item and Stop sound treat it as any other sound.
- **It starts on its sample**, also in a song, and runs round its loop without a gap: the copy is
decoded straight on, and where it jumps — a loop, a song item that starts mid-way — the decoder
is started 640 ms earlier, after which it gives back what decoding from the start would, to the bit.
- **Pitch** moves what is streamed with a shifter, as on a [bus](#a-pitch-on-a-bus): the length
stays, and it is heard a frame later, 42.7 ms at 48 kHz.
- **The Sound tab** says *long, try Stream* for a sound of a minute or more that does not stream,
and *streamed*, *copy waiting* or *copy 40%* for one that does.
- **What always stays decoded:** the Sample of an [instrument](Instruments) and the impulse of a
Reverb, which read the samples themselves, and the original in the Sound tab while it is edited.
- **Browsers.** Streaming needs Opus in WebCodecs, which Chrome has. Elsewhere the sound is decoded
as before. The copy is made in the browser too, so making one needs Chrome as well.
- **Bakes** of [patterns and songs](Songs#baking) always stream.
Measured (2026-10-09, e2e 34 on the CI's Windows machine): five minutes of a sine at 8 kHz in mono,
4.8 MB as WAV, 3.6 MB as its copy. Streamed while a game plays it, the renderer grows by 5.8 MB;
with Stream off the same sound decoded grows it by 69.6 MB. Streamed offline at 48 kHz, every
sample is the one the copy decoded whole has there, by the feed's decoder and by the browser's
`decodeAudioData` alike: from a start at 0.25 s two seconds in, round the seam of its loop, and for
a reversed clip of it round its loop (largest difference 0, no sample late).
## Buses and the master
Every sound ends in the **master** (#325). Between a sound and the master there can be **buses**: