Play notes in time on the audio clock with a scheduler #332

Patterns and songs (#333, #334) have to play sample-accurate in time,
whatever the frame rate does. What is counted in game time runs away
from the sound. This lays the ground: musical time, a scheduler that
plans notes ahead on the audio clock, and a player that starts and ends
a note at a given second. Nothing in the editor plays a sequence yet,
the check scripts drive it through `__gm.music`.

- Ticks, 480 to the quarter as in MIDI files (decided in #332).
  Triplets and quintuplets land on whole ticks. Any meter, a beat being
  the note value under the line. The tempo always counts quarters.
- `core/music/sequence.ts` is a pure rule: the notes starting in a
  window of playing time. Across the end of a loop it looks into the
  next round, so the seam has no gap. A note past the loop end ends
  there. Windows are whole ticks, and the seam is stepped exactly, so
  no rounding can drop the first note of a round.
- `core/music/Scheduler.ts` plans the next 100 ms every 25 ms through a
  host, so it is tested with a clock of its own. The first note sounds
  50 ms after the start. Changes to the sequence are heard from the
  next window on. Stop and seek fade what it started. A new tempo or
  loop counts from what is planned, and the playhead keeps its old
  pace until the clock gets there.
- The position is the audio clock less `outputLatency`, as bar, beat
  and tick.
- The beat comes from a Web Worker (`viewport/musicTimer.ts`), which
  keeps its pace in a hidden tab.
- The sound library starts a note at `timing.at`: a buffer source by
  `start(when)`, the pitch worklet by counting down inside the render
  block it falls in. A note's end is set on the clock at once, its
  fade to the sample. Leaving a loop still waits for the page.
- Delay synced to beats (decided in #332, noted from #327): a setting
  `Sync` (Seconds / Beats), `time` then in quarter notes, held to the
  2 s of the line. Play sound and Play note have an input `tempo`,
  120 BPM where nothing is connected. A scheduled note carries the
  tempo of its sequence.
- Swing is left for #333/#334, as decided.

Also fixes the red suite of 6f1f315 (task 526). Three checks in e2e 34
waited on the page's clock for things that run on the audio clock. In
CI the voice was already 0.7 s old when first seen, and the 0.3 s note
of Play note was over. `stats().last` now says when a voice started on
the audio clock, and the Quad, Step and Play note checks hold every
reading against it.

Tests:
- music.test.ts (new): 17 tests, time, windows, loop seam, scheduler
  with a jittery timer, stop, seek, tempo change
- sound-library +4, soundfx +1, node-catalog +2 (tempo inputs)
- Counter-checks: an unclamped note length, a window that never moves
  on, a buffer source ignoring its start time and a Delay ignoring
  Beats each turn 1, 6, 1 and 2 tests red
- e2e 34 on melektaus-atx: ALL PASSED, 47 checks (+2). Nine notes of a
  looped bar at 150 BPM exactly 0.4 s apart, each planned 89 to 91 ms
  ahead. The position matched the clock less 32 ms latency to the
  tick. Offline the worklet started on samples 590, 12000 and 15200.
- Gate on melektaus-atx: CHECK EXIT 0, 244 test files, 3786 tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/viewport/SoundLibrary.ts

@@ -12,7 +12,8 @@ import {
   type FxVoice,
 } from '../core/soundfx/nodes';
 import type { EaseShare } from '../core/model/animation';
-import type { AudioSink } from '../runtime/systems/Audio';
+import { DEFAULT_TEMPO } from '../core/music/time';
+import type { AudioSink, NoteTiming } from '../runtime/systems/Audio';
 import type { PitchOptions } from './pitchWorklet';
 import pitchWorklet from './pitchWorklet.ts?worker&url';
 
@@ -44,7 +45,8 @@ interface Knob {
  * fall silent without a click while the echoes carry on (#324).
  */
 interface Source {
-  start(): void;
+  /** At that second of the audio clock (#332); a worklet was told when it was made. */
+  start(at: number): void;
   stop(): void;
   ended(then: () => void): void;
   fade: GainNode | null;
@@ -82,11 +84,12 @@ interface Voice {
 
 /**
  * The plans of a voice and the values of their parameters: one plan for a sound played through its
- * FX, one per layer for a note (#331).
+ * FX, one per layer for a note (#331), and the tempo a Delay synced to beats counts by (#332).
  */
 interface Layered {
   layers: readonly FxPlan[];
   params: FxParams;
+  tempo: number;
 }
 
 /** One way of a chain: what a source plays. */
@@ -98,6 +101,8 @@ type Way = Exclude<FxChain, { mix: number }>;
  */
 interface Wiring {
   context: AudioContext;
+  /** When its sources start on the audio clock. */
+  at: number;
   buffers: ReadonlyMap<string, AudioBuffer>;
   file: string;
   loop: boolean;
@@ -120,6 +125,8 @@ interface Start {
   fx: Layered;
   /** Seconds until a note is released (#331); 0 holds it until it is released or stopped. */
   duration: number;
+  /** When it starts on the audio clock (#332); one already past starts now. */
+  at: number;
 }
 
 /**
@@ -198,14 +205,15 @@ export class SoundLibrary implements AudioSink {
    */
   play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE): number {
     if (file === '' || this.failed.has(file)) return 0;
-    const layered = { layers: [fx.plan], params: fx.params };
-    return this.begin({ id: 0, file, files: [file], volume, loop, owner, fx: layered, duration: 0 });
+    const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO };
+    return this.begin({ id: 0, file, files: [file], volume, loop, owner, fx: layered, duration: 0, at: 0 });
   }
 
   /**
    * One note of an instrument (#331): its layers played side by side as one voice, each way with its
    * own file. `name` stands for the instrument where a file would stand – what `levels` finds it by.
-   * A `duration` above 0 releases it that many seconds after it starts.
+   * A `duration` above 0 releases it that many seconds after it starts. `timing.at` starts it at that
+   * second of the audio clock, as the scheduler plans a note (#332).
    */
   playNote(
     name: string,
@@ -214,31 +222,42 @@ export class SoundLibrary implements AudioSink {
     volume: number,
     owner = '',
     duration = 0,
+    timing: NoteTiming = {},
   ): number {
     const files = [...new Set(layers.flatMap(filesOf))].filter(
       (file) => file !== '' && !this.failed.has(file),
     );
     if (files.length === 0) return 0;
-    const fx = { layers, params };
-    return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration });
+    const fx = { layers, params, tempo: timing.tempo ?? DEFAULT_TEMPO };
+    const at = timing.at ?? 0;
+    return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration, at });
   }
 
   /**
    * The end of a note (#331): the ways that end with it fade out, a looping one that does not leaves
    * its loop, the others play on, and the voice goes once they have all ended – or rings on first
-   * where it has echoes.
+   * where it has echoes. `at` is the second of the audio clock it ends at (#332): the fade is set on
+   * the clock to the sample, leaving a loop waits for the page.
    */
-  release(id: number): void {
+  release(id: number, at = 0): void {
     const waiting = this.pending.find((other) => other.id === id);
     if (waiting) waiting.released = true;
     const voice = this.voices.find((other) => other.id === id);
     if (!voice) return;
-    for (const source of voice.sources) if (!source.gate) source.leave?.();
     const context = this.audio();
+    const when = Math.max(at, context.currentTime);
+    const leaving = voice.sources.filter((source) => !source.gate && source.leave);
+    const leave = () => {
+      for (const source of leaving) source.leave!();
+    };
+    if (leaving.length > 0 && when > context.currentTime)
+      this.after(when, () => this.voices.includes(voice) && leave());
+    else leave();
     const gated = voice.sources.filter((source) => source.gate);
-    for (const source of gated) if (source.fade) fadeOut(source.fade.gain, context.currentTime);
+    for (const source of gated) if (source.fade) fadeOut(source.fade.gain, when);
     if (gated.length === 0) return;
-    this.after(context.currentTime + FADE, () => {
+    this.after(when + FADE, () => {
+      if (!this.voices.includes(voice)) return;
       for (const source of gated) {
         source.stop();
         voice.finish?.(source);
@@ -258,7 +277,7 @@ export class SoundLibrary implements AudioSink {
     const askedAt = this.cancelsFor(start.file, start.owner);
     const waiting: Pending = {
       id,
-      fx: { layers: start.fx.layers, params: { ...start.fx.params } },
+      fx: { ...start.fx, params: { ...start.fx.params } },
       timed: [],
       stopped: false,
       released: false,
@@ -286,12 +305,12 @@ export class SoundLibrary implements AudioSink {
       // A quick change is where the voice starts; a timed one – and anything after it, so the order
       // holds – runs from the moment it is heard.
       if (time > 0 || waiting.timed.length > 0) waiting.timed.push({ params, time, ease });
-      else waiting.fx = { layers: waiting.fx.layers, params: { ...waiting.fx.params, ...params } };
+      else waiting.fx = { ...waiting.fx, params: { ...waiting.fx.params, ...params } };
       return true;
     }
     const voice = this.voices.find((other) => other.id === id);
     if (!voice) return false;
-    voice.fx = { layers: voice.fx.layers, params: { ...voice.fx.params, ...params } };
+    voice.fx = { ...voice.fx, params: { ...voice.fx.params, ...params } };
     this.glide(voice, time, ease);
     // A longer feedback makes for a longer tail: it is counted again from where the change ends.
     if (voice.ringing) this.ringFor(voice, Math.max(0, time) + this.tailOf(voice.fx));
@@ -396,6 +415,34 @@ export class SoundLibrary implements AudioSink {
     return this.context?.currentTime ?? 0;
   }
 
+  /** Seconds on the audio clock, opening the context where there is none yet: what the scheduler plans by (#332). */
+  time(): number {
+    const context = this.audio();
+    if (context.state === 'suspended') void context.resume();
+    return context.currentTime;
+  }
+
+  /** Seconds between the audio clock and what is heard; 0 where the browser does not say. */
+  latency(): number {
+    const context = this.context;
+    if (!context) return 0;
+    // Older Safari and Firefox have no `outputLatency`, some no `baseLatency` either.
+    const out = context.outputLatency;
+    if (Number.isFinite(out) && out > 0) return out;
+    return Number.isFinite(context.baseLatency) ? context.baseLatency : 0;
+  }
+
+  /**
+   * Loads every file and the worklet these notes need, so the first note the scheduler plans is
+   * not late waiting on its bytes (#332).
+   */
+  async prepare(layers: readonly FxPlan[]): Promise<void> {
+    await Promise.all([
+      ...[...new Set(layers.flatMap(filesOf))].filter((file) => file !== '').map((file) => this.load(file)),
+      layers.some(pitches) ? this.pitchWorklet() : null,
+    ]);
+  }
+
   /** Where the pitch worklet is served from, for a check that runs it in an offline context (test hook). */
   pitchWorkletUrl(): string {
     return pitchWorklet;
@@ -520,7 +567,8 @@ export class SoundLibrary implements AudioSink {
   /** The longest tail of its layers: they are heard side by side. */
   private tailOf(fx: Layered): number {
     return fx.layers.reduce(
-      (longest, plan) => (fxRings(plan) ? Math.max(longest, fxTailOf(fxChainOf(plan, fx.params))) : longest),
+      (longest, plan) =>
+        fxRings(plan) ? Math.max(longest, fxTailOf(fxChainOf(plan, fx.params, fx.tempo))) : longest,
       0,
     );
   }
@@ -547,16 +595,18 @@ export class SoundLibrary implements AudioSink {
     };
   }
 
-  private start({ id, file, files, volume, loop, owner, fx, duration }: Start): void {
+  private start({ id, file, files, volume, loop, owner, fx, duration, at }: Start): void {
     const context = this.audio();
     // A context starts suspended where the page has not been interacted with; without this the
     // voice would run silently and end as if it had played.
     if (context.state === 'suspended') void context.resume();
+    const when = Math.max(at, context.currentTime);
     const gain = context.createGain();
     gain.gain.value = Math.max(0, Math.min(1, volume));
     const rings = fx.layers.some(fxRings);
     const wiring: Wiring = {
       context,
+      at: when,
       buffers: this.buffers,
       file,
       loop,
@@ -566,7 +616,7 @@ export class SoundLibrary implements AudioSink {
       knobs: [],
       stages: 0,
     };
-    for (const plan of fx.layers) this.wire(wiring, fxChainOf(plan, fx.params)).connect(gain);
+    for (const plan of fx.layers) this.wire(wiring, fxChainOf(plan, fx.params, fx.tempo)).connect(gain);
     gain.connect(context.destination);
     const { sources } = wiring;
     this.last = {
@@ -577,6 +627,7 @@ export class SoundLibrary implements AudioSink {
       pitches: wiring.pitches,
       stages: wiring.stages,
       duration: Math.max(0, ...files.map((name) => this.buffers.get(name)?.duration ?? 0)),
+      at: when,
     };
     const voice: Voice = {
       id,
@@ -598,8 +649,8 @@ export class SoundLibrary implements AudioSink {
     for (const source of sources) source.ended(() => voice.finish?.(source));
     this.voices.push(voice);
     this.started += 1;
-    for (const source of sources) source.start();
-    if (duration > 0) this.after(context.currentTime + duration, () => this.release(id));
+    for (const source of sources) source.start(when);
+    if (duration > 0) this.release(id, when + duration);
   }
 
   /**
@@ -655,7 +706,12 @@ export class SoundLibrary implements AudioSink {
     const buffer = wiring.buffers.get(file) ?? context.createBuffer(1, 1, context.sampleRate);
     wiring.pitches.push(pitch);
     if (shifted && this.canPitch) {
-      const options: PitchOptions = { file, step: buffer.sampleRate / context.sampleRate, loop };
+      const options: PitchOptions = {
+        file,
+        step: buffer.sampleRate / context.sampleRate,
+        loop,
+        startAt: wiring.at,
+      };
       if (own && loop) Object.assign(options, loopIn(buffer, own.loopStart, own.loopEnd));
       if (!this.sent.has(file)) {
         options.channels = Array.from({ length: buffer.numberOfChannels }, (_, c) =>
@@ -695,7 +751,7 @@ export class SoundLibrary implements AudioSink {
       source.loopEnd = own.loopEnd;
     }
     wiring.sources.push({
-      start: () => source.start(),
+      start: (at) => source.start(at),
       stop: () => source.stop(),
       ended: (then) => {
         source.onended = then;
@@ -720,7 +776,7 @@ export class SoundLibrary implements AudioSink {
    */
   private glide(voice: Voice, time: number, ease: EaseShare): void {
     const levels: number[] = [];
-    for (const plan of voice.fx.layers) levelsOf(fxChainOf(plan, voice.fx.params), levels);
+    for (const plan of voice.fx.layers) levelsOf(fxChainOf(plan, voice.fx.params, voice.fx.tempo), levels);
     const now = this.audio().currentTime;
     voice.knobs.forEach((knob, i) => {
       const level = levels[i];
@@ -771,6 +827,8 @@ interface LastVoice {
   stages: number;
   /** Seconds of what was played: a pitched voice is as long as the sound itself (#312, #317). */
   duration: number;
+  /** When it started on the audio clock, what a check times it from. */
+  at: number;
 }
 
 /** Whether any way of a plan has a Pitch, and so needs the worklet. */