e191dc4b2ed48391bf61c0425bb1f4b2a929b14e
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  adminandClaude Opus 5.5
|
e191dc4b2e |
Record the game view from a behavior into a video asset (#405)
A game can now film itself. A behavior starts and stops a take, and the editor turns it into a video asset with the run's picture and the sound the player hears. The exported game hands the file over as a download, since there is no project to keep it in. - New action nodes `Start recording` (Name, Resolution, FPS) and `Stop recording`. An empty name becomes `Recording <timestamp>`. - Resolution is `Game view`, `720p` or `1080p`. For the fixed sizes the height is set and the width follows the camera. - The picture is the camera only, without hints, overlays or the letterbox. It is drawn into a render texture of the target size, so a smaller window still gives full 1080p. - `PictureReader` reads the texture back through pixel buffers with a fence. The editor never waits on the GPU, and a frame with all three buffers busy is dropped and counted. - Nothing of the take reaches the visible canvas. A first version drew tiles into it and copied them out, and with the low-latency canvas that midway state flickered on screen. - The sound is tapped behind the master limiter in the FX worklet and encoded as Opus. - H.264 is chosen when the browser has it, VP9 otherwise. A small own Matroska writer packs both tracks, no new dependency. It names the frame rate and BT.709 so ffmpeg keeps both. - A second `Start recording` is ignored with a warning in the log. A take goes on through `Load scene` and a restart, pauses with the editor and is saved when the run ends. It stops at 2 GB. - The editor uploads the take on the existing video route and makes the asset once the run is over. The video tab gets a Download button for the converted MP4. - `downloadBlob` in `src/ui` replaces the song editor's own copy. Findings: - `On start` runs before the game screen knows the world. A take therefore fixes its size on the first frame that has a camera. - Without a frame rate in the file, ffmpeg turned 30 fps into 59.94. - The worker's poster JPEG is off in colour for every upload (red 255 comes out as 229, BT.601). Not touched here. Tests: - Unit recording + matroska: 21 green, plus the node catalog cases. The Matroska test reads the file back with its own EBML reader. - e2e 67 (new) on melektaus-atx: ALL PASSED. 1080p30 from a 1280x800 window gives a 1920x1080 MP4 with sound and no dropped frame. Every 4th pixel is checked, a green quarter catches an upside-down picture. A take ended by Esc is saved with the timestamp name. - e2e 60j: the exported game downloads an `.mkv` with EBML magic. - The flicker is gone on the user's machine without `?vsync`. - Gate on melektaus-atx: CHECK EXIT 0, 299 files, 4586 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
2764919bb8 |
Stream long sounds from an Opus copy instead of decoding them (#349)
A sound was decoded whole as soon as the project opened, in the editor and in a released game: five minutes of stereo held some 115 MB for as long as the page was open. A sound with Stream on now plays from a copy as Opus in Ogg, fetched a piece at a time and decoded a few seconds ahead by WebCodecs, so a game holds a few megabytes for it. - The Stream switch is a row of the sound, off by default. The Sound tab suggests it from a minute on and says how far the copy has got. - StreamKeeper makes the copies in the browser, a moment after the last change, one at a time: the sound's own and one per clip of its cut, fades and Reverse baked in. A copy that is no longer wanted is taken off and deleted once the project is saved without it. - streamTable says which played names stream from which copy. A clip without a fitting copy, a Sample and a Reverb impulse stay decoded. A sound with Stream on is never decoded ahead. - The page fetches by Range and takes the whole file from a server that ignores it, as the CI's PHP server does. It keeps the bytes and the first two seconds decoded, the rest goes four seconds ahead. - A jump decodes 32 packets before the place first. Measured in Chrome, what follows is then the same to the bit as decoding from the start. With 4 packets it was 0.06 off. - The stream-player on the FX worklet reads the samples at the rate of the context along a Catmull-Rom curve. What comes late is silent and skipped once it is there, so a clip keeps its beat. - The opening stretch goes with the player as it is made. Sent by message it came a few milliseconds late, and 557 samples of the attack were lost. - Pitch on a streamed sound is a shifter after the player, as on a bus: the length stays and it is heard 42.7 ms late. - Bakes of patterns and songs (#340) always stream now. - The song timeline draws a streamed sound from its copy decoded at 8 kHz, keeping only the peaks. Tests: - Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3, stream-queue 6, streams 11, streaming 6, sound-library +5). - e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono, 4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by 5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and a reversed clip are the copy decoded whole to the sample (0). - Counter-checks red as they should be: no pre-roll, no catch-up of late samples at the end, no opening stretch, a streamed sound decoded ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
84cd36f902 |
Bake a pattern or song to Opus, played in its place by a game (#340)
A pattern or song played by a game runs every note through the scheduler, the instruments and their FX, live. Bake works that out once, keeps it as a file on the asset, and Play pattern and Play song play the file instead, as one voice, for as long as nothing it is made of has changed. The tabs always play live. - Bake is an icon in the header of the Pattern and the Song tab, lit while the bake holds. A click bakes, cancels a bake that runs, or offers Bake again and Remove bake. The toolbar shows the progress. The info of the tab says "baked" or "bake outdated". - The bake holds everything up to the output of the pattern or song. The sound FX of the node, fx params, volume, fade in, bus, Master routing and stopping with a fade stay live. - A print of the worked-out plan decides whether it holds: notes and clips, tempo, length, loop, tracks, the build of each instrument for each note, sound FX, files, cuts and levels. A rename keeps it, an undo back to the baked state makes it hold again. - The file holds the straight run and then one more round of the loop as heard the second time, tails included. Looped, it plays to the loop end and then round that round, to the sample. Once, it fades out over its last 10 ms. Let ring fades it out over 0.3 s. - It is worked out by a second SoundLibrary on an OfflineAudioContext, its scheduler woken by suspends of the offline clock, and what reaches the master is taken. A worklet probe tells whether that was mono, so a mono bake goes on exactly as the notes did. - Opus in Ogg, 160 kbit/s stereo or 96 kbit/s mono, encoded with WebCodecs, the Ogg pages written by hand. Normalized to full scale, the peak kept and played back. - The file goes up into the sounds of the account as "[bake] name", which the sound picker leaves out. An old bake file is deleted once the project is saved without it. Only a bake that holds is loaded. - Ogg files are decoded at 48 kHz: resampled to 44.1 kHz, the loop of a bake lost a sample a round. - Baking and removing a bake are no undo steps, like the EQ's zoom. Findings: - Chrome's AudioEncoder refuses opus.format 'ogg' (NotSupportedError) but hands out the OpusHead with a pre-skip of 312 for 'opus'. - The Song tab's header is full at 1280 px: a captioned Bake pushed Fit off the tab. Hence the icon. Tests: - Unit: tests/core/ogg.test.ts (6), tests/core/bake.test.ts (16), four new in sound-library.test.ts, one in song-play.test.ts. One test cuts, reverses and fades a clip on a song's track: each change outdates the bake, each undo makes it hold again. - e2e 64 (melektaus-atx): ALL PASSED, 36 checks (6 new). Live and baked rendered offline differ by at most 0.06 dB over 114 windows of 20 ms, two loop ends among them. e2e 65 had no red check with the icon in its header. - Gate (melektaus-atx): CHECK EXIT 0, 277 files, 4281 tests. - Counter-checks: print without instrument builds, print taking the sound's file instead of the clip's cut, round taken from the first pass, bake voice without its looping source, Play pattern or Play song ignoring the bake: one unit test red each. Pre-skip set to 0: the e2e comparison 4.8 dB off, only that check red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
25084f8576 |
Refactoring round 10: split SoundLibrary, fold sound duplicates (#85)
Refactoring round 10 covers everything since round 9 ( |
||
|
adminandClaude Opus 5.5
|
25c1d02b06 |
Keep the audio thread free of ended voices and too many vocoders (#356)
A song that had been played a few times in a long session stuttered at regular intervals until the page was reloaded. A quick run up the keys of a sampled instrument made every sound drop out for a second or two. Both came from work the audio thread should not have had. - Every worklet node of a voice listened on its port. Chrome never collects a port that is listened to, and through its listener not the voice either. Every node of the voice stayed on its bus and was worked on in every block. A way or stage on the worklet now closes its port once it ends or is stopped, and the processor closes its own side when it is told to stop. - A voice without echoes now leaves its bus as soon as its sound and what its stages hold back are through. Before, only a voice with echoes did. - At most 12 ways play through the pitch worklet at once. One more cuts the oldest in 10 ms, a note already let go before one still held. Each runs a phase vocoder, whose first block alone took 2.7 ms of the 2.67 ms a block has on stein2. - Play for the game ends everything the tabs play: the piece holding the transport and every key or preview (`stopPreviews`). - Closing a Sound, Sound FX, Instrument, Pattern or Song tab stops what it plays of that asset (`AssetTabEditor.closed`). - A preset in the Instrument tab plays nothing any more. With Audition on it is heard like any other change. - Test hooks: `__gm.instrument.play`, `__gm.sound.playing`, and `vocoders` in the stats of the sound library. Measured: - A copy of the project, its Song 2 played for 12 s and stopped, six times (Chrome on stein2, CDP WebAudio): before, 746 nodes were left after the first time and 4469 after the sixth, the load at rest grew from 0.01 to 0.06. After, 7 nodes are left every time (the master and one bus) and the load at rest stays at 0.01 to 0.02. - Pitch-player voices live in Chrome on melektaus-atx: 12 take 0.50 of the audio thread, 16 take 0.71, 24 take 0.99. 30 struck 40 ms apart held it at 0.99 from the first second on. Tests: - Unit sound-library: 110 green (3 new), 4 counter-probes each red - e2e 34 (melektaus-atx): ALL PASSED, 86 checks (3 new, 2 changed) - e2e counter-probe with game start and preset undone: their 3 checks red, and the old check of Play note in the game as well, which now counts on the preview being gone - e2e counter-probe with tab close undone: exactly its check red. A first form of it held a key and stayed green, since the keys let go of a held key by themselves once the tab shows no asset - Gate (melektaus-atx): 270 files, 4181 tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
071f194326 |
Swing the volume or the pitch with Tremolo and Vibrato (#320)
Alarms, sirens and wobbly retro sounds need a volume or a pitch that swings. Two new nodes do that, on a voice and on a bus alike. - Tremolo: Rate (0 to 20 Hz), Depth (0 to 100 %), Shape (Sine, Triangle, Square, Saw). The volume swings between full and 1 - depth. - Vibrato: Rate (0.5 to 20 Hz), Depth (0 to 4 semitones), Shape. The shape is the curve of the pitch, a Square jumps between two pitches. - As the issue decided: two nodes, phase 0 on the first sample of the voice, no fade-in time. Both run on the FX worklet, the classes are in lfo.ts and tested in Node. - Deviation from the issue text, as in the plan: the Vibrato is a delay that swings, not playbackRate. A bus has no playbackRate, and the sound keeps its length. - The delay swings around a center taken from the start values. The center counts as the latency of the node, so a voice starts that much earlier and a Mix and a bus make up for it. - Known limit, in the node text and the wiki: a depth turned up or a rate turned down after the start asks for a delay below 0. The lower half of the swing then flattens. Finding: the first e2e run was red at the edge of a window. The phase is summed up, so the square flips one sample later than 2400 steps of 10/48000 suggest. The check keeps 10 samples away from the edges. Tests: - Unit lfo (new) 5, soundfx 53 (2 new), sound-library 102 (1 new), green. - Counter-checks, each turns one test red: the swing turned over (up and down swapped), the start time not passed to the worklet. - e2e 34 (melektaus-atx): ALL PASSED, 80 checks (1 new). Chrome offline: Tremolo square silent before its voice, then 0.5 and 0 for 50 ms each. Vibrato square at 5 Hz, 1 semitone: 1059.5 Hz and 940.5 Hz. - The gate runs once on the last commit of this series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
a645026568 |
Tame the loudness with a Compressor and a Limiter node (#326)
A sound FX could not control dynamics. A Compressor squeezes the loudness together, and a Limiter keeps every peak below a line. Both work on a voice and on a bus, where they tame everything at once. - Compressor: DynamicsCompressorNode with Threshold, Ratio, Knee, Attack and Release, then Makeup and Auto-gain from 0 to 1. After the node the sound gets (1 - w) * makeup + w * A dB. A is what the curve takes off at 0 dBFS. - The node adds 0.6 of A by itself, as the spec says. That gain is taken back out, so 0 really means no makeup. - The knee shape is left to the browser by the spec. dynamics.ts ports Chromium's curve. It gives the gain Chrome was measured to add at four settings to three places. - Limiter: a class on the FX worklet with Threshold, Ceiling, Release and Auto-gain. It looks 5 ms ahead: the smallest gain of the window ahead, averaged over the window, then a release. No sample passes the lower of threshold and ceiling. Auto-gain raises it toward the ceiling. - Deviation from the plan in the issue: the Limiter has a Threshold as well. Without it Auto-gain has nothing to measure against. - Latency is made up on voices now too. A voice starts as much earlier as its stages are late (Compressor 6 ms, Limiter 5 ms). The other way of a Mix and the other layers of a note are held back to the latest, and a bus counts it into its latency. - A voice with a Limiter counts as ringing, so the worklet is stopped only after what it still holds is out. Tests: - Unit dynamics (new) 8, soundfx 49 (3 new), sound-library 100 (4 new), green. - Counter-checks, each turns tests red: no look-ahead in the Limiter (3), no taking back of the node's gain (1), a voice not started earlier (1), no stage latency on a bus (1), no hold-back in a Mix (1). - e2e 34 (melektaus-atx): ALL PASSED, 78 checks (1 new). Chrome offline: Compressor 288 samples late, Limiter 240. Full scale through -12 dB 4:1 hard knee: +0.19 dB with full auto-gain, -8.81 dB without. A sine at 2 through the Limiter at -6 dB peaks at -6.00 dB, with auto-gain 1 at -1.00 dB. - The gate runs once on the last commit of this series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
b519cd192f |
Send a track, a pattern item or a whole song through a sound FX (#346)
A sound FX could only be put on a single sound. Now a track, a pattern item, a Play pattern and a Play song send everything they play through one chain of it, as an insert in a DAW. A Play note gets one of its own. - Track: an FX box in the head next to M and S opens a menu of the project's sound FX, and the track's inspector has the row too. Order: items, track FX, volume, pan. - Pattern item: Sound FX in its inspector. Its notes play on a bus of their own into the track's bus. - Play pattern and Play song take fx and fx params. The chain sits on the master of the playing, after the tracks and before the level and the fades. Set sound FX parameters reaches it by the playback. - Play note takes fx and fx params next to the instrument's params. The FX parameters go by "fx:" so the two sets stay apart. Set sound FX parameters on a note changes a name both have in both. - A Pitch on a bus is a new PitchShifter: the phase vocoder of the PitchPlayer, fed while it plays. It cannot look ahead and is exactly one frame late at every pitch, 2048 samples. - That latency is made up. The other way of a Mix is held back by a delay of the same length. Every note and clip starts as much earlier as the buses it passes are late. The scheduler plans that much further ahead, and the first note waits that much after Play. - The Song tab waits for the chains before it plays and lets its buses go a moment after Stop. A playing in a game lets them go after its echoes and what the shifters still hold. - songBuses moved to its own file. Kept in song.ts it pulled the FX compiler into the node registry through an import cycle. Decisions of the user: one chain for the whole (1a), FX everywhere asked for (2a), Set sound FX parameters on playbacks (3a), and a pitch shifter of its own with exact latency compensation for a Pitch on a bus. Tests: - Unit pitch-shifter (new): 11 green. At 0 semitones the output is the input 2048 samples later, off by at most 1.2e-7. - Unit sound-library: 84 green, 8 new. Counter-checks (no early start, no Mix delay, chain made every time, tune misses the master) each turned their test red. - Unit bus-fx (new): 6 green, 4 counter-checks red. Scheduler ahead: 1 new, its counter-check red. - e2e 65 (melektaus-atx): ALL PASSED, 35 checks, 4 new, first run. Chrome at 44.1 kHz: the bus was 0.046440 s late, the note started exactly that much early. Offline the shifter moved 1 kHz to 2000 Hz and passed a click 2048 samples later. - Gate (melektaus-atx): CHECK EXIT 0, 259 files, 4021 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
01fd6e6464 |
Swirl and comb a sound with a Phaser and a Flanger node (#341)
sfxr's "phaser" is a comb filter, the sound plus a copy delayed by up to 2.9 ms. That is a flanger, not the phaser a guitarist knows. Both effects come in as nodes of their own, so each name fits its sound and sfxr is covered in full. Both run on the FX worklet like the Bitcrusher, because a feedback loop in Web Audio costs a render quantum per round. - Phaser: first-order allpasses (Stages 2, 4, 6, 8, 12) around Frequency, moved by a sine LFO Depth octaves wide at Rate Hz, with Feedback and Mix. An Envelope can move Frequency and Mix. - Flanger: a delay line of Offset ms. Sweep moves it on in ms/s from the start of the voice, an LFO adds up to Depth ms. As in sfxr the delay is mirrored at 0, and it never goes past 20 ms. Feedback comes back exactly one delay later. An Envelope can move Offset and Mix. - Mix 0 leaves the sound as it is, 1 is half sound and half copy. The notches are deepest there and the peaks stay at the sound's level. sfxr adds both at full, so its sound comes out 6 dB louder. - The Flanger's processor gets the start of its voice and begins the sweep on that sample. stageNodes takes it as a new argument `at`. - Explosion gets a Flanger in one draw out of two, with sfxr's offset and sweep and no LFO. Offset is offset^2 * 2.89 ms, Sweep is ramp^2 * 125 ms/s, signed so the direction holds. - Randomize draws a Flanger's offset and sweep like sfxr. Its LFO and feedback and all of a Phaser are drawn within bounds of their own. Mix and Stages stay, as a Bitcrusher's Mix does. Mutate moves them too, and a 0 stays 0. - New instrument preset Phased Pad: saw, Envelope, low-pass, Phaser of six stages. The other 14 are unchanged. - The Bitcrusher's worklet setup is now shared with both nodes (workletNodes), its behaviour is unchanged. Tests: - Unit phaser (new): 8 green. Counter-check with clamping at 0 instead of mirroring and a flipped allpass coefficient: 3 red. - Unit soundfx, sound-library, chip-presets: 45 / 76 / 14 green. Counter-checks: start time not passed on, phaser frequency eased linearly, sweep sign dropped, Phaser left out of Mutate, each red. - Unit core, viewport, graph: 179 files, 2979 tests green. - e2e 34 (melektaus-atx): ALL PASSED, 70 checks (1 new), first run. Chrome offline: 1 kHz notch RMS 0.0000 with two stages, 0.5657 with four. Flanger echo after 48 samples, swept echo at 533/534. About 47 Phasers (12 stages) or 63 Flangers fill the audio thread. - Gate (melektaus-atx): CHECK EXIT 0, 257 files, 3991 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
f46b2316f2 |
Make 8-bit sounds without a file with a Chip node in instruments (#329)
An instrument could only play sound files so far. The Chip makes the sound of a note itself, the way old game consoles and sfxr did: a raw wave and what its frequency does over the note. A laser, a jump or a pickup is now an instrument and ends by itself. - Chip node with wave (square, triangle, saw, sine, noise and the periodic noise of the NES), follow note, frequency, duty, duty sweep, slide, delta slide, min frequency, vibrato, arpeggio and repeat. It counts in Hz, semitones and seconds, not sfxr's 0 to 1. - It runs as a third processor in the FX worklet. The math is a plain class, ChipPlayer, tested in Node. Every sample is worked out 8 times and averaged, as sfxr does. - Frequency and duty are AudioParams, so an Envelope can move them and a parameter can tune them while the note plays. A Pitch behind a Chip moves its frequency directly, without the phase vocoder. - Below its min frequency a Chip ends and tells the page. Without the worklet it is silent and its note ends at once. - An Envelope with sustain 0 now ends its note once the decay is over, for every source under it, Samples too. Before, a note with duration 0 ran on silently until a Stop sound. - A note made of Chips alone needs no file. playNote and the wait for the bytes no longer drop it for having none. - Inputs that do nothing as things stand are grey in the inspector. idleParams is told which inputs are connected, so a connected one always counts. The credit names sfxr (Tomas Pettersson, MIT) and jsfxr, which turned out to be public domain, not MIT as the plan in #329 said. Tests: - Unit chip (new) 12, sound-library +4. All of tests/core, tests/viewport, tests/editor/graph and tests/editor/inspector: 178 files, 2953 green. - Counter-checks: semitones read as /24 turn 4 chip tests red. Without the sustain rule and the file check 4 library tests fail. - e2e 34 (melektaus-atx): ALL PASSED, 60 checks (2 new). Offline in Chrome a square at 220 Hz plus 12 semitones and duty 0.25 gave 329 periods in 0.75 s, up 0.249. 64 noise voices for 2 s rendered in 2.0 s, about 63 at once. Live a 0.2 s note with sustain 0 was gone after 0.26 s. - Gate (melektaus-atx): CHECK EXIT 0, 255 files, 3962 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
adminandClaude Opus 5.5
|
165f022fc0 |
Crush a sound to fewer bits and a lower rate with a Bitcrusher (#322)
A new effect for sound FX and instruments makes a sound rough like an
old console. It rounds the sound to fewer steps and holds it at a lower
sample rate, and all three of its numbers move while the sound plays.
- Bitcrusher node with bits (1 to 16, not only whole), rate (100 Hz to
48 kHz, sample-and-hold) and mix. A mix typed in at 0 leaves it out.
- It runs on the audio thread as a second processor next to the pitch
player. The module is now fxWorklet.ts and the test hook is called
fxWorkletUrl.
- A play that needs the worklet waits for it, for a Pitch or a
Bitcrusher alike. Without the worklet the sound goes through as is.
- Bits and mix move in what they hold, the rate in octaves. An
Envelope can move all three in an instrument.
- A stage that runs on the worklet is stopped with its voice, like the
drivers of an Envelope or a Mix.
- The math is a plain class, Crusher, so it is tested in Node.
The cleanup check of e2e 64 from
|