diff --git a/Architecture.md b/Architecture.md index c94d2bb..cf70802 100644 --- a/Architecture.md +++ b/Architecture.md @@ -198,7 +198,8 @@ src/viewport/ WorldViewport (the drawing of a world or a holder, the ba PixelCels (the pixels of pixel art in memory: a painted cel is named at once, the ones the server lacks are uploaded before a save, #130) + pixelArtPicture (every frame composed on the CPU and laid side by side onto one canvas, so a frame is cut out like a texture frame), - SoundLibrary, ShaderLibrary, ParticleLibrary, + SoundLibrary + pitchWorklet (the AudioWorklet that plays a way with a Pitch through + core/soundfx/pitchPlayer, a phase vocoder, #317), ShaderLibrary, ParticleLibrary, particles (one ParticleContainer per source), tilemap (one container per layer), shaderPicture (the picture a shader is laid over when it shades a camera's frame, #299, or a particle field, #170: drawn anew each time into a render texture kept while its size diff --git a/Sounds.md b/Sounds.md index 0d34fd5..4e4693d 100644 --- a/Sounds.md +++ b/Sounds.md @@ -102,7 +102,7 @@ Input and the Output once each; their pins carry sound and fit nothing but each | Node | What it does | | --- | --- | | **Gain** | Louder or quieter, in dB (`Gain (dB)`): 0 leaves it alone, -6 is about half as loud, +6 twice. Above 0 the sound can clip. | -| **Pitch** | Higher or lower, in semitones (12 is an octave, 0.5 a quarter tone). The sound **keeps its length**. The pitch is worked out on the sound as it starts, so an effect before the Pitch acts as if it came after it. | +| **Pitch** | Higher or lower, in semitones (12 is an octave, 0.5 a quarter tone). The sound **keeps its length**, also when the pitch changes while it plays. The pitch is made where the sound is read, so an effect before the Pitch acts as if it came after it. | | **5-Band-EQ** | Five bands in dB: 80 Hz and 12 kHz as shelves, 250 Hz, 1 kHz and 4 kHz as peaks. A band at 0 does nothing. | | **Mix** | Blends the sounds at `a` and `b`: at 0 only `a` is heard, at 1 only `b`, at 0.5 half of each (0.5 is the default). | @@ -110,13 +110,34 @@ Every number of these nodes is an **input**: typed in while nothing is connected parameter of the Input. Each is held to its range (Gain -120 to +24 dB, Pitch -64 to +64, the EQ bands -60 to +24 dB, Mix 0 to 1), whichever way the value arrives. -**How Pitch keeps the length.** The sound is first read faster or slower, which sets the pitch, -and then stretched back to its length by WSOLA: short frames (40 ms) laid over each other, each -moved a little so a tone carries on without a jump. The pitched copy is worked out when a voice -starts — about 15 ms per second of a 48 kHz channel — and kept for the next voice at the same pitch -(the last 32 are kept). A looping sound is stretched across its seam, so the loop stays clean. -Far from 0 it gets rough: a pitch that lands above half the sample rate folds back down, and a -very low one smears, since a frame is shorter than one period of what is left. +**How Pitch keeps the length** (#317). A way with a Pitch is played by an `AudioWorklet` on the +audio thread, a **phase vocoder**: the sound is cut into frames of about 43 ms (2048 samples at +48 kHz, four overlapping), and the spectrum of each is moved up or down, peak by peak with the +bins around it, so a tone stays one tone (phase locking after Laroche and Dolson). The worklet +reads the sound itself, so it can look ahead and starts with the very first sample, with no +latency, and every way of a Mix stays in step. The pitch is an `AudioParam` of the worklet, in +semitones: it glides, ramps and eases like a Gain, and the sound never gets longer or shorter. A +looping sound is read on across its seam, so the loop stays clean. + +- **At exactly 0** a frame is the sound itself, so a Pitch fed by a parameter that sits at 0 + plays the sound untouched and costs next to nothing. Coming back to 0 from another pitch, the + phases are pulled round to the sound's for about half a second (the pitch wavers by a fifth of + a semitone at most) and then it switches to the sound itself, without a dip. +- **A change of pitch is heard about 20 ms late**, half a frame: a frame takes the pitch at the + sample it starts on. A sweep over the whole sound misses the last 20 ms of its way. +- **Far from 0 it gets rough**, as every phase vocoder: a hard attack is smeared over a frame, + everything moved above half the sample rate is dropped, and far down the peaks crowd into the + lowest bins until it is a hum. The range is not limited further than the -64 to +64 of the + node. +- **Cost**: a voice with a Pitch other than 0 works out two FFTs per frame and channel while it + plays. Measured (2026-10-06): in Node on an i5-7500T, one stereo voice at +5 takes about 53 ms + per second (5 % of a core), at 0 about 5 ms. In e2e 34, headless Chrome on the CI's Windows + machine rendered 16 stereo voices at +5 for 2 s in 1.3 s, so about **24 such voices fill the + audio thread**; fewer leave room for the rest of the mix. There is no limit on how many play at + once: past that the sound starts to break up. +- Without `AudioWorklet` (or if it cannot be loaded) a way with a Pitch plays at its own pitch, + and the console says why. The worklet is loaded with the first play that has a Pitch, which + waits for it as it waits for its bytes. **Ways and the Mix.** An input takes one edge, but an output feeds as many as you like: the Input can start two ways with effects of their own, and a **Mix** joins them again. Only a Mix joins @@ -124,12 +145,11 @@ ways, and a node whose sound never reaches the Output is simply not heard. A nod nothing coming in (a Mix needs both `a` and `b`), or a graph without an Output, does not compile: the node says why, and the tab's header says it too (otherwise it counts the effects on the way). -**A Pitch on one way only** works: the voice starts one copy of the sound per pitch, all at the -same moment and all of the same length, and each runs through its own way. Two ways at the same -pitch play the same copy, each from a source of its own, so each can move on its own later. A Pitch -behind the Mix goes to both of its sounds. A Mix typed in at exactly 0 or 1 leaves the silent way -out, so that one is not even worked out. A Mix fed by a parameter keeps both ways, because the -value may still change while the voice plays. +**A Pitch on one way only** works: every way has a source of its own, all started at the same +moment, a way with a Pitch its own worklet and one without the plain sound. A Pitch behind the Mix +goes to both of its sounds. A Mix typed in at exactly 0 or 1 leaves the silent way out, so that +one is not even played. A Mix fed by a parameter keeps both ways, because the value may still +change while the voice plays. **Listening in the tab.** The header picks one of the project's sounds and plays it through the graph as it stands — the same library as a running game, so it sounds the way the game will. @@ -189,15 +209,11 @@ speed, or a loop that gets quieter. `Play sound`, `pitch` -64, `time` 1.28) falls by 50 semitones a second. Counting a value down in `On update` or `On fixed update` instead runs on game time, and the sound drifts away from it. - **To span the whole sound**, wire `length` of `Play sound` into `time`: a fade or a filter sweep - then ends with the last sample. `length` is one run of the sound (one pass of a loop), the same - with a pitch it starts on. It is 0 while the sound has not loaded yet, which only happens right - after a project is opened, and `time` 0 is the quick glide. A pitch changed over `length` does - not end with the sound: a lower pitch plays slower (see the next point), so the rest of the sound - is heard at the target pitch, and a higher one ends before it gets there. + then ends with the last sample. `length` is one run of the sound (one pass of a loop), whatever + the pitch does. It is 0 while the sound has not loaded yet, which only happens right after a + project is opened, and `time` 0 is the quick glide. A pitch sweep over `length` ends with the + sound too, only heard about 20 ms behind (see *How Pitch keeps the length*). - A new change starts from where the voice is right now; whatever was still under way is dropped. -- **A pitch changes by speed here.** A pitched copy cannot be worked out again in the middle of a - sound, so the voice plays faster or slower to reach the new pitch: it gets that much shorter or - longer. On a loop that is not heard; the next play gets its own length again. - A sound that has ended and the `voice` 0 of a play that played nothing are not reached. A detached voice is reached like any other. A voice still waiting for its bytes starts with the new values, and a change with a `time` starts with the voice, so the whole of it is heard.