diff --git a/Architecture.md b/Architecture.md index 08e4837..4eb7dec 100644 --- a/Architecture.md +++ b/Architecture.md @@ -173,7 +173,10 @@ src/core/music/ time (ticks, 480 to the quarter, tempo and meter, a posit pattern item), take (#342: a recorded take as 16-bit WAV, cut to where the song started), bake (#340: what a bake is made from as a print, where the parts of its file lie, how they are put together from the playing looped and the one played once, and played back) + - ogg (#340: Opus in Ogg, the pages around an encoder's packets and back) + ogg (#340: Opus in Ogg, the pages around an encoder's packets and back, also read as + the bytes come in, #349), streams (#349: which names stream from which copy, what + copies are missing, how a played sound walks its copy) + streamQueue (#349: the + samples a streamed sound plays from on the worklet, read at the rate of the context) src/core/soundfx/ the sound FX graph (#311): nodes (every effect and what it adds to a plan), compile (the graph walked back from the Output to a plan; a parameter stays open until a voice starts; a tapped node gets a `tap` in front and behind, #374), parametricEq (#367: @@ -252,7 +255,10 @@ src/viewport/ WorldViewport (the drawing of a world or a holder, the ba sound until its end, a bus per track of a song with its level and pan, #334; the buses of the mix and the Master routing, #325/#351; a bake as one voice on the master of its playing, and a library of its own on an offline context to bake with, #340; a channel - probe that says whether a bake is mono), bakeRender (#340: a pattern or song played + probe that says whether a bake is mono; a sound or a bake streamed from its copy by a + `stream-player`, #349), streaming (#349: a copy fetched by Range, its head decoded + ahead, the rest decoded by WebCodecs a few seconds before it is heard), + bakeRender (#340: a pattern or song played offline, what reaches its master taken, encoded as Opus with WebCodecs), music (the scheduler on the library's clock, its notes played by the project's instruments and its clips by the library: the player of the Pattern and Song tabs, #333/#334, and `__gm.music`) + musicTimer @@ -405,7 +411,9 @@ src/editor/song/ SongEditor (header: Play, Grid, Bake; plays through the s ownFx (#355: a sound FX the song keeps for a song, a track or an item) src/editor/sound/ SoundEditor (#343: header with Play, the clip's direction and fades), WaveCanvas (the waveform, the clips as bands, the cut) + waveView (its time axis), SoundPreview (play - and length in the inspector), SoundPicker (the "+ → Sound" rows with their play mark) + and length in the inspector), SoundPicker (the "+ → Sound" rows with their play mark), + StreamKeeper (#349: the copies streamed sounds and their clips play from, made in the + browser and let go once no longer wanted) src/editor/soundfx/ SoundFxEditor (#311: the graph, a sound to try it on, Play and Stop), PreviewParams (the header's fields for the parameters a tab plays with, shared with the Instrument tab), EqEditor + eqView (#367/#370: the Parametric EQ in a tab of its own, its handles on diff --git a/Development.md b/Development.md index 34bdecb..98ca621 100644 --- a/Development.md +++ b/Development.md @@ -19,7 +19,7 @@ php tests/php/accounts.test.php # the API's accoun php tests/php/snapshots.test.php # the snapshot store (#231), the same way php tests/php/releases.test.php # the release store (#266), the same way php tests/php/files.test.php # uploaded files in a folder per account (#344), the same way -node tools/seed-e2e.mjs && node tools/e2e.mjs # 1928 editor checks, screenshots in /tmp/gm-e2e-*.png +node tools/seed-e2e.mjs && node tools/e2e.mjs # 1935 editor checks, screenshots in /tmp/gm-e2e-*.png GM_STEPS=20 node tools/e2e.mjs # or 1-12, or 6,20 — a subset while working on one step ./e2e.sh start [GM_STEPS] # the same run detached, see below ./e2e.sh status # live progress, or the last run's result diff --git a/Songs.md b/Songs.md index 99e75a6..61bbf1b 100644 --- a/Songs.md +++ b/Songs.md @@ -385,8 +385,10 @@ edit is heard at once. level it had. It goes up into the sounds of your account under *[bake] name*, and the sound picker leaves it out. The old file of a bake made again or removed is deleted once the project is saved without it, unless a snapshot or a release still names it. -- **Memory:** a bake that holds is loaded and decoded with the project like a sound, at 48 kHz: - about 23 MB a minute in stereo. An outdated one is not loaded. Streaming it instead is #349. +- **Memory:** a bake is [streamed](Sounds#streaming-long-sounds) (#349): only its compressed bytes + and its first two seconds are held, never the whole of it decoded. An outdated one is not + loaded. Where a browser cannot stream it, it is decoded at 48 kHz as before: about 23 MB a minute + in stereo. Measured (2026-10-09, e2e 64 on the CI's Windows machine): a bar of quarters at 150 BPM with one more note just before its end, baked: 76800 frames for the bar and 76800 for the second round, diff --git a/Sounds.md b/Sounds.md index 2b9de45..9b77eda 100644 --- a/Sounds.md +++ b/Sounds.md @@ -30,6 +30,7 @@ signature, so nothing a request claims can decide what is written. | --- | --- | | **Volume** | How loud this sound is by itself, 0-1. | | **Loop** | Whether it is music: it repeats until something stops it. | +| **Stream** | Whether a game plays it from a copy as Opus, a few seconds at a time, rather than decoding all of it first (#349). Off by default. See [Streaming long sounds](#streaming-long-sounds). | | **Sound FX** | The [sound FX](#sound-fx) it goes through wherever it plays, none while none is picked (#352). See [The sound FX of an asset](#the-sound-fx-of-an-asset). | The preview plays through the very same library a running game uses — same decode, same device — @@ -203,7 +204,8 @@ sounds as well: playing a loop twice gives two loops, and it is `Stop sound` tha them at once. **It is loaded ahead.** Every sound of the project is fetched and decoded as soon as the project is -open, the way images and fonts are, so the first play starts in the frame its node ran. Decoding +open, the way images and fonts are, so the first play starts in the frame its node ran. A sound +that [streams](#streaming-long-sounds) has only the start of its copy fetched. Decoding ahead opens no audio device: that only happens on the first play. A sound that is still loading when it is played — a run started the moment the project opened — starts a moment late instead. `Play sound` never waits on the network, and a `Stop sound` that happens while the bytes are still @@ -233,6 +235,46 @@ any edit, so a loop cannot pile up one voice per keystroke. normal speed rather than sounding like a tape being slowed — the global clock scales what the world does, not how the device plays back. +## Streaming long sounds + +A sound is decoded whole before it plays: 48000 samples a second for each side, four bytes each. +That is nothing for a jump, but five minutes of music in stereo are about 115 MB, held for as long +as the page is open. **Stream** (#349) plays such a sound from a copy, a few seconds at a time. + +- **The copy.** Switched on, the editor decodes the file once, encodes it as Opus in Ogg (160 kbit/s + in stereo, 96 kbit/s in mono, as a [bake](Songs#baking) is) and uploads it as *[stream] name*. + The sound picker leaves it out. The original stays as it is: the Sound tab shows and cuts that one. + Switched off, the copy is taken off the sound and deleted once the project is saved without it. +- **Each clip gets a copy of its own cut**, its fades and Reverse in it, so a clip never needs the + whole file. A changed clip gets a new copy a moment after the last change. A clip that is the whole + file plays the sound's copy. +- **Until a copy is there**, the sound or clip plays decoded, as without Stream. A sound with Stream + on is never decoded ahead when the project opens, though: only once it is played without a copy. +- **What a game holds** is the compressed copy as it is fetched, and its first two seconds decoded, + so it starts at once. The rest is decoded about four seconds ahead of what is heard and let go + once it has been played. +- **Everything after it stays the same:** volume, Loop, its sound FX, its bus and the master. A + `Play sound`, a song item and Stop sound treat it as any other sound. +- **It starts on its sample**, also in a song, and runs round its loop without a gap: the copy is + decoded straight on, and where it jumps — a loop, a song item that starts mid-way — the decoder + is started 640 ms earlier, after which it gives back what decoding from the start would, to the bit. +- **Pitch** moves what is streamed with a shifter, as on a [bus](#a-pitch-on-a-bus): the length + stays, and it is heard a frame later, 42.7 ms at 48 kHz. +- **The Sound tab** says *long, try Stream* for a sound of a minute or more that does not stream, + and *streamed*, *copy waiting* or *copy 40%* for one that does. +- **What always stays decoded:** the Sample of an [instrument](Instruments) and the impulse of a + Reverb, which read the samples themselves, and the original in the Sound tab while it is edited. +- **Browsers.** Streaming needs Opus in WebCodecs, which Chrome has. Elsewhere the sound is decoded + as before. The copy is made in the browser too, so making one needs Chrome as well. +- **Bakes** of [patterns and songs](Songs#baking) always stream. + +Measured (2026-10-09, e2e 34 on the CI's Windows machine): five minutes of a sine at 8 kHz in mono, +4.8 MB as WAV, 3.6 MB as its copy. Streamed while a game plays it, the renderer grows by 5.8 MB; +with Stream off the same sound decoded grows it by 69.6 MB. Streamed offline at 48 kHz, every +sample is the one the copy decoded whole has there, by the feed's decoder and by the browser's +`decodeAudioData` alike: from a start at 0.25 s two seconds in, round the seam of its loop, and for +a reversed clip of it round its loop (largest difference 0, no sample late). + ## Buses and the master Every sound ends in the **master** (#325). Between a sound and the master there can be **buses**: