Bake a pattern or song to Opus, played in its place by a game (#340)

A pattern or song played by a game runs every note through the
scheduler, the instruments and their FX, live. Bake works that out
once, keeps it as a file on the asset, and Play pattern and Play song
play the file instead, as one voice, for as long as nothing it is made
of has changed. The tabs always play live.

- Bake is an icon in the header of the Pattern and the Song tab, lit
  while the bake holds. A click bakes, cancels a bake that runs, or
  offers Bake again and Remove bake. The toolbar shows the progress.
  The info of the tab says "baked" or "bake outdated".
- The bake holds everything up to the output of the pattern or song.
  The sound FX of the node, fx params, volume, fade in, bus, Master
  routing and stopping with a fade stay live.
- A print of the worked-out plan decides whether it holds: notes and
  clips, tempo, length, loop, tracks, the build of each instrument for
  each note, sound FX, files, cuts and levels. A rename keeps it, an
  undo back to the baked state makes it hold again.
- The file holds the straight run and then one more round of the loop
  as heard the second time, tails included. Looped, it plays to the
  loop end and then round that round, to the sample. Once, it fades
  out over its last 10 ms. Let ring fades it out over 0.3 s.
- It is worked out by a second SoundLibrary on an OfflineAudioContext,
  its scheduler woken by suspends of the offline clock, and what
  reaches the master is taken. A worklet probe tells whether that was
  mono, so a mono bake goes on exactly as the notes did.
- Opus in Ogg, 160 kbit/s stereo or 96 kbit/s mono, encoded with
  WebCodecs, the Ogg pages written by hand. Normalized to full scale,
  the peak kept and played back.
- The file goes up into the sounds of the account as "[bake] name",
  which the sound picker leaves out. An old bake file is deleted once
  the project is saved without it. Only a bake that holds is loaded.
- Ogg files are decoded at 48 kHz: resampled to 44.1 kHz, the loop of
  a bake lost a sample a round.
- Baking and removing a bake are no undo steps, like the EQ's zoom.

Findings:
- Chrome's AudioEncoder refuses opus.format 'ogg' (NotSupportedError)
  but hands out the OpusHead with a pre-skip of 312 for 'opus'.
- The Song tab's header is full at 1280 px: a captioned Bake pushed
  Fit off the tab. Hence the icon.

Tests:
- Unit: tests/core/ogg.test.ts (6), tests/core/bake.test.ts (16),
  four new in sound-library.test.ts, one in song-play.test.ts. One
  test cuts, reverses and fades a clip on a song's track: each change
  outdates the bake, each undo makes it hold again.
- e2e 64 (melektaus-atx): ALL PASSED, 36 checks (6 new). Live and baked
  rendered offline differ by at most 0.06 dB over 114 windows of
  20 ms, two loop ends among them. e2e 65 had no red check with the
  icon in its header.
- Gate (melektaus-atx): CHECK EXIT 0, 277 files, 4281 tests.
- Counter-checks: print without instrument builds, print taking the
  sound's file instead of the clip's cut, round taken from the first
  pass, bake voice without its looping source, Play pattern or Play
  song ignoring the bake: one unit test red each. Pre-skip set to 0:
  the e2e comparison 4.8 dB off, only that check red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/viewport/SoundLibrary.ts

@@ -16,7 +16,14 @@ import {
 } from '../core/soundfx/nodes';
 import { releaseTime } from '../core/instrument/envelope';
 import type { EaseShare } from '../core/model/animation';
-import { type BusLevel, Playbacks, type SequenceNeeds, type SequencePlay } from '../core/music/playbacks';
+import { BAKE_RATE } from '../core/music/bake';
+import {
+  type BakePlay,
+  type BusLevel,
+  Playbacks,
+  type SequenceNeeds,
+  type SequencePlay,
+} from '../core/music/playbacks';
 import { arrivalOf, sendDelays, type TimedBus } from '../core/music/routing';
 import type { SendTap } from '../core/graph/Graph';
 import { MASTER_MIX, type MixBus, type MixSend, sameMix } from '../core/music/mixer';
@@ -160,6 +167,8 @@ interface Voice {
   /** How many of its ways play through the pitch worklet, and whether its note has been let go. */
   vocoders: number;
   released: boolean;
+  /** Seconds a stop that lets it ring fades it out over: a bake (#340), whose notes are one sound. */
+  ringFade: number;
 }
 
 /**
@@ -180,7 +189,7 @@ type Way = Exclude<FxChain, { mix: number }>;
  * its own plays (the sound a `play` was given), and what has been made for it so far.
  */
 interface Wiring {
-  context: AudioContext;
+  context: BaseAudioContext;
   /** When its sources start on the audio clock. */
   at: number;
   buffers: ReadonlyMap<string, AudioBuffer>;
@@ -256,7 +265,7 @@ interface Pending {
  */
 export class SoundLibrary implements AudioSink {
   private base = DEFAULT_BASE;
-  private context: AudioContext | null = null;
+  private context: BaseAudioContext | null = null;
   /**
    * Where the master goes out (#359): through a limiter that keeps every sample at `GUARD_DB` or
    * below, once the worklet is there; straight to the speakers until then. Late by the limiter.
@@ -264,9 +273,22 @@ export class SoundLibrary implements AudioSink {
   private guard: { input: GainNode; limiter: AudioWorkletNode | null; latency: number } | null = null;
   private guardMeter: BusMeter | null = null;
 
-  /** `guarded` false leaves the limiter out: for checks that follow every node of a voice. */
-  constructor(private readonly guarded = true) {}
-  private offline: OfflineAudioContext | null = null;
+  /**
+   * `guarded` false leaves the limiter out: for checks that follow every node of a voice. With
+   * `render` it plays into an offline context to bake (#340), never resumed by itself.
+   */
+  constructor(
+    private readonly guarded = true,
+    private readonly render: OfflineRender | null = null,
+  ) {
+    if (render) {
+      this.context = render.context;
+      this.held = true;
+    }
+  }
+  private decodeContext: OfflineAudioContext | null = null;
+  /** Where a bake (#340) is decoded: always at the rate it was made at, so its loop lands on the sample. */
+  private opusContext: OfflineAudioContext | null = null;
   private readonly buffers = new Map<string, AudioBuffer>();
   /**
    * The stretches of files that sound assets are cut to (#343), made from the whole file on first
@@ -320,12 +342,13 @@ export class SoundLibrary implements AudioSink {
     stop: (voices, letRing) => {
       for (const id of voices) this.stopVoice(id, letRing);
     },
-    wake,
+    wake: (pump) => (this.render?.wake ?? wake)(pump),
     prepare: (needs, id) => this.ready(needs, id),
     ahead: (id) => this.groupAhead(id),
     level: (id, gain, at, seconds) => this.level(id, gain, at, seconds),
     after: (at, then) => void this.after(at, then),
     closed: () => this.sweep(),
+    playBaked: (id, bake, at) => this.playBaked(id, bake, at),
   });
   /** The playings whose buses are still there: a master each, and the tracks of a song into it. */
   private readonly groups = new Set<number>();
@@ -833,7 +856,7 @@ export class SoundLibrary implements AudioSink {
    * a Mix is held back as far, so the two stay together. Says where it comes out and how late.
    */
   private chainOf(
-    context: AudioContext,
+    context: BaseAudioContext,
     chain: FxChain,
     input: AudioNode,
     made: BusChain,
@@ -897,7 +920,7 @@ export class SoundLibrary implements AudioSink {
 
   /** The Pitch of a way of a bus (#346); without the worklet the sound goes by as it is, on time. */
   private shifter(
-    context: AudioContext,
+    context: BaseAudioContext,
     pitch: number,
     input: AudioNode,
     made: BusChain,
@@ -953,15 +976,20 @@ export class SoundLibrary implements AudioSink {
   playSequence(name: string, play: SequencePlay, owner = ''): number {
     if (play.sequence.notes.length === 0 && !play.sequence.clips?.length) return 0;
     const id = this.nextId++;
-    const buses = play.buses ?? [];
+    // A bake whose file could not be had is no bake: the notes are played instead (#340).
+    const { bake: asked, ...rest } = play;
+    const bake = asked && !this.hasFailed(asked.file) ? asked : null;
+    const buses = bake ? [] : (play.buses ?? []);
     this.open(id, buses, play.fx ?? null, play.bus ?? MASTER_MIX, play.sends);
     const tracks = new Set(buses.map((bus) => bus.name));
     const into = (bus: string | undefined) => groupBus(id, bus !== undefined && tracks.has(bus) ? bus : '');
     const { strike, strikeClip } = play;
     const chains = [...buses.flatMap((bus) => bus.fx ?? []), ...(play.fx ? [play.fx] : [])];
+    const needs = bake ? { layers: [], files: [bake.file] } : play.needs;
     const routed: SequencePlay = {
-      ...play,
-      needs: { ...play.needs, layers: [...play.needs.layers, ...chains.map((fx) => fx.plan)] },
+      ...rest,
+      ...(bake && { bake }),
+      needs: { ...needs, layers: [...needs.layers, ...chains.map((fx) => fx.plan)] },
       strike: (note) => strike({ ...note, bus: into(note.bus) }),
       ...(strikeClip && {
         strikeClip: (clip: ScheduledClip) => strikeClip({ ...clip, bus: into(clip.bus) }),
@@ -980,7 +1008,10 @@ export class SoundLibrary implements AudioSink {
     else this.stopVoice(id);
   }
 
-  /** The patterns and songs playing, oldest first, with the notes and clips last planned of each (test hook). */
+  /**
+   * The patterns and songs playing, oldest first, with the notes and clips last planned of each, and
+   * whether it plays its bake (test hook).
+   */
   playbacks(): {
     id: number;
     name: string;
@@ -989,18 +1020,128 @@ export class SoundLibrary implements AudioSink {
     fading: number | null;
     planned: ScheduledNote[];
     clips: ScheduledClip[];
+    baked: boolean;
+    startedAt: number | null;
   }[] {
-    return this.sequences.current().map(({ id, name, owner, scheduler, fading, planned, clips }) => ({
+    return this.sequences.current().map(({ id, name, owner, runner, fading, planned, clips, baked }) => ({
       id,
       name,
       owner,
-      playing: scheduler.playing,
+      playing: runner.playing,
+      startedAt: runner.startedAt,
       fading,
       planned: [...planned],
       clips: [...clips],
+      baked,
     }));
   }
 
+  /**
+   * What goes into the master of a playing, taken off before its level into `to` (#340): what a bake
+   * keeps. Says whether there was such a playing.
+   */
+  tapPlaying(id: number, to: AudioNode): boolean {
+    const master = this.buses.get(groupBus(id, ''));
+    if (!master) return false;
+    master.input.disconnect();
+    master.input.connect(to);
+    return true;
+  }
+
+  /**
+   * The bake of a playing (#340) as one voice on its master, heard from `heard`: its file once to its
+   * end, faded out there, or up to the loop end and then round its second round – in one source where
+   * the round follows at once, else in a second that takes over to the sample. 0 without its bytes.
+   */
+  private playBaked(id: number, bake: BakePlay, heard: number): number {
+    const buffer = this.buffer(bake.file);
+    if (!buffer) return 0;
+    const context = this.audio();
+    const bus = groupBus(id, '');
+    const at = Math.max(context.currentTime, heard - this.busLatency(bus));
+    const gain = context.createGain();
+    gain.gain.value = Math.max(0, Math.min(MAX_VOLUME, bake.gain));
+    gain.connect(this.busInput(bus));
+    const { end, until, round } = bake.playback;
+    const made = (from: number, loop: { start: number; end: number } | null) => {
+      const node = context.createBufferSource();
+      node.buffer = buffer;
+      if (loop) {
+        node.loop = true;
+        node.loopStart = loop.start;
+        node.loopEnd = loop.end;
+      }
+      node.connect(gain);
+      const source: Source = {
+        start: () => {},
+        stop: () => node.stop(),
+        ended: (then) => {
+          node.onended = then;
+        },
+        fade: null,
+        gate: false,
+        leave: null,
+      };
+      return { node, source, from };
+    };
+    const parts =
+      round && round.start === until
+        ? [made(0, round)]
+        : round
+          ? [made(0, null), made(round.start, round)]
+          : [made(0, null)];
+    const voice: Voice = {
+      id: this.nextId++,
+      file: bake.file,
+      owner: '',
+      bus,
+      at,
+      sources: parts.map((part) => part.source),
+      ending: new Set(),
+      drivers: [],
+      envelopes: [],
+      finish: null,
+      gain,
+      fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO },
+      knobs: [],
+      rings: false,
+      ringing: false,
+      quiet: null,
+      vocoders: 0,
+      released: false,
+      ringFade: BAKE_LET_RING,
+    };
+    const going = new Set(voice.sources);
+    voice.finish = (source) => {
+      if (going.delete(source) && going.size === 0) this.ended(voice);
+    };
+    for (const { source } of parts) source.ended(() => voice.finish?.(source));
+    this.voices.push(voice);
+    this.started += 1;
+    this.last = {
+      file: bake.file,
+      volume: gain.gain.value,
+      loop: round !== null,
+      layers: 0,
+      pitches: [],
+      stages: 0,
+      duration: buffer.duration,
+      at,
+      envelopes: 0,
+      bus,
+    };
+    const [head, next] = parts;
+    if (end !== null) {
+      gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE));
+      gain.gain.linearRampToValueAtTime(0, at + end);
+      head!.node.start(at, 0, end);
+    } else if (next) {
+      head!.node.start(at, 0, until);
+      next.node.start(at + until, next.from);
+    } else head!.node.start(at, 0);
+    return voice.id;
+  }
+
   /**
    * The buses of a playing, its tracks into its master and a pattern item's into its track (#346),
    * each with the sound FX it was given; its master into `into`, or along `sends` (#351). Their
@@ -1302,7 +1443,7 @@ export class SoundLibrary implements AudioSink {
    */
   setSuspended(suspended: boolean): void {
     this.held = suspended;
-    const context = this.context;
+    const context = this.live();
     if (!context) return;
     if (suspended) void context.suspend();
     else void context.resume();
@@ -1363,13 +1504,19 @@ export class SoundLibrary implements AudioSink {
   /** Seconds on the audio clock, opening the context where there is none yet: what the scheduler plans by (#332). */
   time(): number {
     const context = this.audio();
-    if (context.state === 'suspended' && !this.held) void context.resume();
+    if (context.state === 'suspended' && !this.held) void this.live()?.resume();
     return context.currentTime;
   }
 
   /** The context everything plays on, opened where there is none yet: a take is recorded on its clock (#342). */
   liveContext(): AudioContext {
-    return this.audio();
+    this.audio();
+    return this.live()!;
+  }
+
+  /** The context it plays to the speakers on; null before there is one, and for a bake (#340). */
+  private live(): AudioContext | null {
+    return this.render ? null : (this.context as AudioContext | null);
   }
 
   /**
@@ -1377,7 +1524,7 @@ export class SoundLibrary implements AudioSink {
    * device, where the browser says.
    */
   latency(): number {
-    const context = this.context;
+    const context = this.live();
     if (!context) return 0;
     const guard = this.guard?.latency ?? 0;
     // Older Safari and Firefox have no `outputLatency`, some no `baseLatency` either.
@@ -1399,6 +1546,11 @@ export class SoundLibrary implements AudioSink {
     ]);
   }
 
+  /** Loads the FX worklet into its context, for a node of it made outside (#340); says whether it is there. */
+  worklet(): Promise<boolean> {
+    return this.fxWorklet();
+  }
+
   /** Where the FX worklet is served from, for a check that runs it in an offline context (test hook). */
   fxWorkletUrl(): string {
     return fxWorkletUrl;
@@ -1475,10 +1627,8 @@ export class SoundLibrary implements AudioSink {
     if (running) return running;
     const work = (async () => {
       try {
-        const response = await fetch(`${this.base}/${file}`);
-        if (!response.ok) throw new Error(`HTTP ${response.status}`);
-        const bytes = await response.arrayBuffer();
-        const buffer = await this.decoder().decodeAudioData(bytes);
+        const buffer = this.render ? await this.render.files.load(file) : await this.fetched(file);
+        if (!buffer) throw new Error('not there where it is baked from');
         this.buffers.set(file, buffer);
         return buffer;
       } catch (error: unknown) {
@@ -1493,6 +1643,12 @@ export class SoundLibrary implements AudioSink {
     return work;
   }
 
+  private async fetched(file: string): Promise<AudioBuffer> {
+    const response = await fetch(`${this.base}/${file}`);
+    if (!response.ok) throw new Error(`HTTP ${response.status}`);
+    return this.decoder(file).decodeAudioData(await response.arrayBuffer());
+  }
+
   /**
    * Ends these voices now: `onended` only fires next frame and the counts have to be right here. A
    * voice is faded out rather than cut, so it does not click. With `letRing` one with echoes only
@@ -1502,6 +1658,10 @@ export class SoundLibrary implements AudioSink {
   private silence(going: Voice[], letRing = false): void {
     this.stopped += going.filter((voice) => !(letRing && voice.ringing)).length;
     for (const voice of going) {
+      if (letRing && voice.ringFade > 0) {
+        this.cut(voice, voice.ringFade);
+        continue;
+      }
       if (letRing && voice.envelopes.length > 0) {
         this.letGo(voice, this.audio().currentTime, true);
         continue;
@@ -1530,8 +1690,8 @@ export class SoundLibrary implements AudioSink {
     }
   }
 
-  /** Fades the whole voice out and lets it go. */
-  private cut(voice: Voice): void {
+  /** Fades the whole voice out over `seconds` and lets it go. */
+  private cut(voice: Voice, seconds = FADE): void {
     this.forget(voice);
     const context = this.audio();
     const done = () => {
@@ -1539,10 +1699,11 @@ export class SoundLibrary implements AudioSink {
       for (const driver of voice.drivers) driver.stop();
       voice.gain.disconnect();
     };
-    // A suspended clock would hold the fade – and the sound – until the next run resumes it.
-    if (context.state !== 'running') return done();
-    fadeOut(voice.gain.gain, context.currentTime);
-    this.after(context.currentTime + FADE, done);
+    // A suspended clock would hold the fade – and the sound – until the next run resumes it. An
+    // offline one is suspended whenever the page gets to plan, and runs on after.
+    if (context.state !== 'running' && !this.render) return done();
+    fadeOut(voice.gain.gain, context.currentTime, seconds);
+    this.after(context.currentTime + seconds, done);
   }
 
   /** Fades the sources of a voice with echoes out and lets the echoes ring on (#324). */
@@ -1649,7 +1810,7 @@ export class SoundLibrary implements AudioSink {
     const context = this.audio();
     // A context starts suspended where the page has not been interacted with; without this the
     // voice would run silently and end as if it had played.
-    if (context.state === 'suspended' && !this.held) void context.resume();
+    if (context.state === 'suspended' && !this.held) void this.live()?.resume();
     // As much earlier as its bus and its own stages are late, so it is heard when it was planned
     // (#346, #326). A layer that is less late is held back to the latest.
     const chains = fx.layers.map((plan) => fxChainOf(plan, fx.params, fx.tempo));
@@ -1721,6 +1882,7 @@ export class SoundLibrary implements AudioSink {
       quiet: null,
       vocoders: wiring.vocoders,
       released: false,
+      ringFade: 0,
     };
     const going = new Set(sources);
     voice.finish = (source) => {
@@ -1967,9 +2129,10 @@ export class SoundLibrary implements AudioSink {
     return glideKnobs(voice.knobs, levels, this.audio().currentTime, time, ease);
   }
 
-  private audio(): AudioContext {
+  private audio(): BaseAudioContext {
     if (this.context) return this.context;
-    const context = (this.context = new AudioContext());
+    const context = new AudioContext();
+    this.context = context;
     if (this.guarded) this.guardOut(context);
     if (this.mix.length > 0) void this.applyMix();
     return context;
@@ -2015,7 +2178,7 @@ export class SoundLibrary implements AudioSink {
     }
     const range = this.rangeOf(name);
     if (!range) return null;
-    const made = this.decoder().createBuffer(
+    const made = this.decoder(name).createBuffer(
       range.whole.numberOfChannels,
       range.to - range.from,
       range.whole.sampleRate,
@@ -2073,10 +2236,16 @@ export class SoundLibrary implements AudioSink {
     })());
   }
 
-  /** The context that is already playing, or else one that decodes without opening a device. */
-  private decoder(): BaseAudioContext {
-    if (this.context || typeof OfflineAudioContext === 'undefined') return this.audio();
-    return (this.offline ??= new OfflineAudioContext(1, 1, DECODE_RATE));
+  /**
+   * The context that is already playing, or else one that decodes without opening a device. An Ogg
+   * file, which a bake (#340) is, is decoded at the rate Opus is made at, whatever the device runs
+   * at: resampled, the loop of a bake would lose a sample a round.
+   */
+  private decoder(file: string): BaseAudioContext {
+    if (typeof OfflineAudioContext === 'undefined') return this.audio();
+    if (file.endsWith('.ogg')) return (this.opusContext ??= new OfflineAudioContext(1, 1, BAKE_RATE));
+    if (this.context) return this.audio();
+    return (this.decodeContext ??= new OfflineAudioContext(1, 1, DECODE_RATE));
   }
 
   /** Everything that would have silenced this file for this owner, counted rather than flagged. */
@@ -2085,6 +2254,19 @@ export class SoundLibrary implements AudioSink {
   }
 }
 
+/**
+ * What a library that bakes (#340) runs on: an offline context, the library whose decoded files it
+ * plays, and what wakes its schedulers – the offline clock, not the page's timers.
+ */
+export interface OfflineRender {
+  context: OfflineAudioContext;
+  files: SoundLibrary;
+  wake(pump: () => void): () => void;
+}
+
+/** Seconds a bake fades out over where its notes would have rung on (#340). */
+const BAKE_LET_RING = 0.3;
+
 /** The bus of a game's playing (#336): its master for `''`, else the track of that name inside it. */
 function groupBus(id: number, track: string): string {
   return track === '' ? `play:${id}` : `play:${id}:${track}`;