Describe streaming long sounds from an Opus copy (#349)
- Sounds: the Stream row, a new section on what streams, what the copy is, what stays decoded, and the measured memory. - Songs: a bake is streamed now, not decoded with the project. - Architecture: streams, streamQueue, streaming and StreamKeeper. - Development: 1935 e2e checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent
26f23debb8
commit
5515c67008
4 files changed
+59
-7
No files matched your search
+11
-3
@@ -173,7 +173,10 @@ src/core/music/ time (ticks, 480 to the quarter, tempo and meter, a posit
|
||||
pattern item), take (#342: a recorded take as 16-bit WAV, cut to where the song started),
|
||||
bake (#340: what a bake is made from as a print, where the parts of its file lie, how they
|
||||
are put together from the playing looped and the one played once, and played back) +
|
||||
ogg (#340: Opus in Ogg, the pages around an encoder's packets and back)
|
||||
ogg (#340: Opus in Ogg, the pages around an encoder's packets and back, also read as
|
||||
the bytes come in, #349), streams (#349: which names stream from which copy, what
|
||||
copies are missing, how a played sound walks its copy) + streamQueue (#349: the
|
||||
samples a streamed sound plays from on the worklet, read at the rate of the context)
|
||||
src/core/soundfx/ the sound FX graph (#311): nodes (every effect and what it adds to a plan), compile (the
|
||||
graph walked back from the Output to a plan; a parameter stays open until a voice
|
||||
starts; a tapped node gets a `tap` in front and behind, #374), parametricEq (#367:
|
||||
@@ -252,7 +255,10 @@ src/viewport/ WorldViewport (the drawing of a world or a holder, the ba
|
||||
sound until its end, a bus per track of a song with its level and pan, #334; the buses of
|
||||
the mix and the Master routing, #325/#351; a bake as one voice on the master of its
|
||||
playing, and a library of its own on an offline context to bake with, #340; a channel
|
||||
probe that says whether a bake is mono), bakeRender (#340: a pattern or song played
|
||||
probe that says whether a bake is mono; a sound or a bake streamed from its copy by a
|
||||
`stream-player`, #349), streaming (#349: a copy fetched by Range, its head decoded
|
||||
ahead, the rest decoded by WebCodecs a few seconds before it is heard),
|
||||
bakeRender (#340: a pattern or song played
|
||||
offline, what reaches its master taken, encoded as Opus with WebCodecs), music (the scheduler on
|
||||
the library's clock, its notes played by the project's instruments and its clips by
|
||||
the library: the player of the Pattern and Song tabs, #333/#334, and `__gm.music`) + musicTimer
|
||||
@@ -405,7 +411,9 @@ src/editor/song/ SongEditor (header: Play, Grid, Bake; plays through the s
|
||||
ownFx (#355: a sound FX the song keeps for a song, a track or an item)
|
||||
src/editor/sound/ SoundEditor (#343: header with Play, the clip's direction and fades), WaveCanvas (the
|
||||
waveform, the clips as bands, the cut) + waveView (its time axis), SoundPreview (play
|
||||
and length in the inspector), SoundPicker (the "+ → Sound" rows with their play mark)
|
||||
and length in the inspector), SoundPicker (the "+ → Sound" rows with their play mark),
|
||||
StreamKeeper (#349: the copies streamed sounds and their clips play from, made in the
|
||||
browser and let go once no longer wanted)
|
||||
src/editor/soundfx/ SoundFxEditor (#311: the graph, a sound to try it on, Play and Stop), PreviewParams (the
|
||||
header's fields for the parameters a tab plays with, shared with the Instrument tab),
|
||||
EqEditor + eqView (#367/#370: the Parametric EQ in a tab of its own, its handles on
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ php tests/php/accounts.test.php # the API's accoun
|
||||
php tests/php/snapshots.test.php # the snapshot store (#231), the same way
|
||||
php tests/php/releases.test.php # the release store (#266), the same way
|
||||
php tests/php/files.test.php # uploaded files in a folder per account (#344), the same way
|
||||
node tools/seed-e2e.mjs && node tools/e2e.mjs # 1928 editor checks, screenshots in /tmp/gm-e2e-*.png
|
||||
node tools/seed-e2e.mjs && node tools/e2e.mjs # 1935 editor checks, screenshots in /tmp/gm-e2e-*.png
|
||||
GM_STEPS=20 node tools/e2e.mjs # or 1-12, or 6,20 — a subset while working on one step
|
||||
./e2e.sh start [GM_STEPS] # the same run detached, see below
|
||||
./e2e.sh status # live progress, or the last run's result
|
||||
|
||||
+4
-2
@@ -385,8 +385,10 @@ edit is heard at once.
|
||||
level it had. It goes up into the sounds of your account under *[bake] name*, and the sound picker
|
||||
leaves it out. The old file of a bake made again or removed is deleted once the project is saved
|
||||
without it, unless a snapshot or a release still names it.
|
||||
- **Memory:** a bake that holds is loaded and decoded with the project like a sound, at 48 kHz:
|
||||
about 23 MB a minute in stereo. An outdated one is not loaded. Streaming it instead is #349.
|
||||
- **Memory:** a bake is [streamed](Sounds#streaming-long-sounds) (#349): only its compressed bytes
|
||||
and its first two seconds are held, never the whole of it decoded. An outdated one is not
|
||||
loaded. Where a browser cannot stream it, it is decoded at 48 kHz as before: about 23 MB a minute
|
||||
in stereo.
|
||||
|
||||
Measured (2026-10-09, e2e 64 on the CI's Windows machine): a bar of quarters at 150 BPM with one
|
||||
more note just before its end, baked: 76800 frames for the bar and 76800 for the second round,
|
||||
|
||||
+43
-1
@@ -30,6 +30,7 @@ signature, so nothing a request claims can decide what is written.
|
||||
| --- | --- |
|
||||
| **Volume** | How loud this sound is by itself, 0-1. |
|
||||
| **Loop** | Whether it is music: it repeats until something stops it. |
|
||||
| **Stream** | Whether a game plays it from a copy as Opus, a few seconds at a time, rather than decoding all of it first (#349). Off by default. See [Streaming long sounds](#streaming-long-sounds). |
|
||||
| **Sound FX** | The [sound FX](#sound-fx) it goes through wherever it plays, none while none is picked (#352). See [The sound FX of an asset](#the-sound-fx-of-an-asset). |
|
||||
|
||||
The preview plays through the very same library a running game uses — same decode, same device —
|
||||
@@ -203,7 +204,8 @@ sounds as well: playing a loop twice gives two loops, and it is `Stop sound` tha
|
||||
them at once.
|
||||
|
||||
**It is loaded ahead.** Every sound of the project is fetched and decoded as soon as the project is
|
||||
open, the way images and fonts are, so the first play starts in the frame its node ran. Decoding
|
||||
open, the way images and fonts are, so the first play starts in the frame its node ran. A sound
|
||||
that [streams](#streaming-long-sounds) has only the start of its copy fetched. Decoding
|
||||
ahead opens no audio device: that only happens on the first play. A sound that is still loading
|
||||
when it is played — a run started the moment the project opened — starts a moment late instead.
|
||||
`Play sound` never waits on the network, and a `Stop sound` that happens while the bytes are still
|
||||
@@ -233,6 +235,46 @@ any edit, so a loop cannot pile up one voice per keystroke.
|
||||
normal speed rather than sounding like a tape being slowed — the global clock scales what the
|
||||
world does, not how the device plays back.
|
||||
|
||||
## Streaming long sounds
|
||||
|
||||
A sound is decoded whole before it plays: 48000 samples a second for each side, four bytes each.
|
||||
That is nothing for a jump, but five minutes of music in stereo are about 115 MB, held for as long
|
||||
as the page is open. **Stream** (#349) plays such a sound from a copy, a few seconds at a time.
|
||||
|
||||
- **The copy.** Switched on, the editor decodes the file once, encodes it as Opus in Ogg (160 kbit/s
|
||||
in stereo, 96 kbit/s in mono, as a [bake](Songs#baking) is) and uploads it as *[stream] name*.
|
||||
The sound picker leaves it out. The original stays as it is: the Sound tab shows and cuts that one.
|
||||
Switched off, the copy is taken off the sound and deleted once the project is saved without it.
|
||||
- **Each clip gets a copy of its own cut**, its fades and Reverse in it, so a clip never needs the
|
||||
whole file. A changed clip gets a new copy a moment after the last change. A clip that is the whole
|
||||
file plays the sound's copy.
|
||||
- **Until a copy is there**, the sound or clip plays decoded, as without Stream. A sound with Stream
|
||||
on is never decoded ahead when the project opens, though: only once it is played without a copy.
|
||||
- **What a game holds** is the compressed copy as it is fetched, and its first two seconds decoded,
|
||||
so it starts at once. The rest is decoded about four seconds ahead of what is heard and let go
|
||||
once it has been played.
|
||||
- **Everything after it stays the same:** volume, Loop, its sound FX, its bus and the master. A
|
||||
`Play sound`, a song item and Stop sound treat it as any other sound.
|
||||
- **It starts on its sample**, also in a song, and runs round its loop without a gap: the copy is
|
||||
decoded straight on, and where it jumps — a loop, a song item that starts mid-way — the decoder
|
||||
is started 640 ms earlier, after which it gives back what decoding from the start would, to the bit.
|
||||
- **Pitch** moves what is streamed with a shifter, as on a [bus](#a-pitch-on-a-bus): the length
|
||||
stays, and it is heard a frame later, 42.7 ms at 48 kHz.
|
||||
- **The Sound tab** says *long, try Stream* for a sound of a minute or more that does not stream,
|
||||
and *streamed*, *copy waiting* or *copy 40%* for one that does.
|
||||
- **What always stays decoded:** the Sample of an [instrument](Instruments) and the impulse of a
|
||||
Reverb, which read the samples themselves, and the original in the Sound tab while it is edited.
|
||||
- **Browsers.** Streaming needs Opus in WebCodecs, which Chrome has. Elsewhere the sound is decoded
|
||||
as before. The copy is made in the browser too, so making one needs Chrome as well.
|
||||
- **Bakes** of [patterns and songs](Songs#baking) always stream.
|
||||
|
||||
Measured (2026-10-09, e2e 34 on the CI's Windows machine): five minutes of a sine at 8 kHz in mono,
|
||||
4.8 MB as WAV, 3.6 MB as its copy. Streamed while a game plays it, the renderer grows by 5.8 MB;
|
||||
with Stream off the same sound decoded grows it by 69.6 MB. Streamed offline at 48 kHz, every
|
||||
sample is the one the copy decoded whole has there, by the feed's decoder and by the browser's
|
||||
`decodeAudioData` alike: from a start at 0.25 s two seconds in, round the seam of its loop, and for
|
||||
a reversed clip of it round its loop (largest difference 0, no sample late).
|
||||
|
||||
## Buses and the master
|
||||
|
||||
Every sound ends in the **master** (#325). Between a sound and the master there can be **buses**:
|
||||
|
||||
Reference in new issue
Block a user