Play sounds, patterns and songs on the tracks of a composition (#418)

A trailer needs its music and effects on the same timeline as its
pictures. Until now a composition could only place objects, so sound
had to be started by hand from a behavior and drifted against the
picture.

- Audio items: a sound, a clip of a sound, a pattern or a song dragged
  from the asset panel onto a track becomes an item as long as one
  pass of it. It may share a track with object and video items and
  overlap them.
- Editing works as for other items: move, resize, trim on the left
  (moves the offset), fit to media, delete, copy and paste, box
  selection. Sounds draw their waveform, the colour tells the kind.
- The inspector shows what it plays, start, length, offset, volume
  (-60 to +12 dB) and bus. The bus is "From asset", the master or one
  of the buses, as asked for in the issue.
- An item longer than its material plays in rounds. Sounds and
  patterns start over, a song runs on in its loop. Loop marks on the
  item follow the same rule, a song's first mark sits at its loop end.
- compositionAudio.ts turns the items into sequences, each with its
  own tempo. CompositionSound plays them, in the tab and in a game.
- In the tab the items are heard from the playhead on, straight into
  the master like every tab. A scrub is silent. Playing a song or
  pattern in its own tab stops the composition, and the other way
  round.
- In a game they are planned on the audio clock for the moment the
  composition clock reaches them, with the scheduler's LEAD made up.
  While the clock stands (waiting for a video, paused, placement off)
  they are silent and resume at the right spot.

Found along the way:

- The asset panel never let a song out of its list, so a song could
  not be dragged anywhere. Song and scene tab reject it as they reject
  a pattern.
- The asset panel scrolled to the selected asset on every rebuild of
  its list, so folding a group far below the selection jumped back up.
  Only a new selection scrolls now.
- At the end of the list, folding a group pulled every row down by the
  rows that went away. ScrollContainer has a new opt-in holdOffset,
  set only in the asset panel, that keeps the offset until the user
  scrolls up.

Limits: a time scale other than 1 or a frame longer than 0.1 s moves
the picture and not the sound. A baked song plays its notes here, not
the bake.

Tests:
- tsc, eslint, Prettier clean. Gate on melektaus-atx green,
  309 files, 4788 tests (4772 plus 16 new).
- Unit: composition-audio.test.ts 13 new, scroll-container 3 new.
  Counter-checks: without catching up on load time or without the
  silence on pause the unit tests go red.
- e2e 68h: the sound is planned 1.2 ms after the composition reaches
  0.5 s. Without the LEAD make-up it is 54 ms late and red.
- e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a
  song onto a track, 20 frames each.
- e2e 04: folding Sound at the end of the list keeps its header at
  y 234. With the old panel it jumps to 984 and the check is red.
- Not run in a browser: the loop marks of a song item, copy and paste
  of audio items, the bus choice.
- README and wiki count 2072 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tests/core/composition-audio.test.ts

@@ -0,0 +1,360 @@
+/**
+ * Sounds, patterns and songs as items of a composition (#418): kept without an object, laid out as
+ * sequences in milliseconds on buses of their own, and heard in a game while the composition's
+ * clock runs.
+ */
+import { describe, expect, it } from 'vitest';
+import {
+  CreateAssetCommand,
+  CreateSoundCommand,
+  SetAssetPropertyCommand,
+} from '../../src/core/commands/assetCommands';
+import { SetCompositionCommand } from '../../src/core/commands/compositionCommands';
+import { SetLoopCommand, SetPatternNotesCommand } from '../../src/core/commands/patternCommands';
+import { PlaceAssetCommand } from '../../src/core/commands/sceneObjectCommands';
+import { SetSongCommand } from '../../src/core/commands/songCommands';
+import type {
+  CompositionAsset,
+  CompositionItem,
+  InstrumentAsset,
+  SongAsset,
+} from '../../src/core/model/Asset';
+import { compositionLength, fittedLength, setSoundMeasure } from '../../src/core/model/composition';
+import { carriedClips, itemMedia, mediaMarks } from '../../src/core/model/compositionEdit';
+import { findAssetOf } from '../../src/core/model/Project';
+import { audioItemBus, compositionPlayings, compositionSpan } from '../../src/core/music/compositionAudio';
+import { MASTER_MIX, mixBusName } from '../../src/core/music/mixer';
+import { Playbacks, type SequencePlay } from '../../src/core/music/playbacks';
+import { LEAD } from '../../src/core/music/Scheduler';
+import { newItem } from '../../src/core/music/song';
+import { MASTER_BUS } from '../../src/core/properties/buses';
+import { SILENT_AUDIO } from '../../src/runtime/systems/Audio';
+import { addGameSystems } from '../../src/runtime/systems/BehaviorSystem';
+import { Input } from '../../src/runtime/systems/Input';
+import { World } from '../../src/runtime/World';
+import { graphEditor, makeBehavior, makeProject } from '../helpers';
+
+const FILE = `${'e5'.repeat(32)}.wav`;
+const Q = 480;
+
+/**
+ * A composition "Cut" at 10 fps placed in a scene, a sound "Blip", a one-bar pattern "Riff" at
+ * 120 BPM (two seconds) with notes on beats 1 and 3, and a song "Theme" playing it from its start.
+ */
+function makeCut() {
+  const f = makeProject();
+  const scene = new CreateAssetCommand('scene', 'Level');
+  const comp = new CreateAssetCommand('composition', 'Cut', { fps: 10 });
+  const sound = new CreateSoundCommand('Blip', FILE);
+  const ins = new CreateAssetCommand('instrument', 'Lead');
+  const pattern = new CreateAssetCommand('pattern', 'Riff');
+  const song = new CreateAssetCommand('song', 'Theme');
+  const bus = new CreateAssetCommand('bus', 'Music');
+  for (const c of [scene, comp, sound, ins, pattern, song, bus]) f.stack.execute(c);
+  (ins.asset as InstrumentAsset).graph.nodes.find((n) => n.type === 'InstrumentSample')!.params.sound =
+    sound.asset.id;
+  f.stack.execute(new SetAssetPropertyCommand(pattern.asset.id, 'instrument', ins.asset.id));
+  f.stack.execute(new SetAssetPropertyCommand(pattern.asset.id, 'bars', 1));
+  f.stack.execute(
+    new SetPatternNotesCommand(pattern.asset.id, [
+      { id: 'n1', tick: 0, length: Q, note: 60, velocity: 1 },
+      { id: 'n2', tick: 2 * Q, length: Q, note: 64, velocity: 1 },
+    ]),
+  );
+  const theme = () => findAssetOf(f.project, song.asset.id, 'song') as SongAsset;
+  const track = theme().tracks[0]!;
+  f.stack.execute(
+    new SetSongCommand(song.asset.id, {
+      items: [newItem({ track: track.id, kind: 'pattern', asset: pattern.asset.id, tick: 0, length: 4 * Q })],
+    }),
+  );
+  const place = new PlaceAssetCommand(scene.asset.id, comp.asset.id, { x: 0, y: 0 });
+  f.stack.execute(place);
+  const cut = () => findAssetOf(f.project, comp.asset.id, 'composition') as CompositionAsset;
+  const lay = (...items: CompositionItem[]) =>
+    f.stack.execute(new SetCompositionCommand(cut().id, { items }));
+  const audioItem = (
+    id: string,
+    kind: 'sound' | 'pattern' | 'song',
+    start: number,
+    length: number,
+    more: Partial<CompositionItem> & { volume?: number; bus?: string } = {},
+  ): CompositionItem => ({
+    id,
+    track: cut().tracks[0]!.id,
+    start,
+    length,
+    offset: more.offset ?? 0,
+    audio: {
+      kind,
+      asset: { sound: sound.asset.id, pattern: pattern.asset.id, song: song.asset.id }[kind],
+      volume: more.volume ?? 0,
+      bus: more.bus ?? '',
+    },
+  });
+  return {
+    ...f,
+    sceneId: scene.asset.id,
+    placement: place.objectId,
+    soundId: sound.asset.id,
+    patternId: pattern.asset.id,
+    songId: song.asset.id,
+    busId: bus.asset.id,
+    cut,
+    lay,
+    audioItem,
+  };
+}
+
+describe('audio items of a composition', () => {
+  it('are kept without an object, and count to its length', () => {
+    const f = makeCut();
+    f.lay(f.audioItem('a', 'sound', 5, 20, { volume: 40 }), f.audioItem('b', 'pattern', 0, 10));
+    expect(f.cut().items.map((i) => i.id)).toEqual(['a', 'b']);
+    // Held to the most a sound item may be turned up by.
+    expect(f.cut().items[0]!.audio).toEqual({ kind: 'sound', asset: f.soundId, volume: 12, bus: '' });
+    expect(f.cut().objects).toEqual([]);
+    expect(compositionLength(f.project, f.cut())).toBe(25);
+  });
+
+  it('drops one that plays nothing or an unknown kind', () => {
+    const f = makeCut();
+    const bad = {
+      ...f.audioItem('a', 'sound', 0, 10),
+      audio: { kind: 'video', asset: 'x', volume: 0, bus: '' },
+    };
+    const empty = {
+      ...f.audioItem('b', 'sound', 0, 10),
+      audio: { kind: 'sound', asset: '', volume: 0, bus: '' },
+    };
+    f.lay(bad as CompositionItem, empty as CompositionItem);
+    expect(f.cut().items).toEqual([]);
+  });
+
+  it('take the length of a pattern or song, of a sound once it is measured', () => {
+    const f = makeCut();
+    f.lay(f.audioItem('p', 'pattern', 0, 1, { offset: 0.5 }), f.audioItem('s', 'sound', 0, 1));
+    const [pattern, sound] = f.cut().items;
+    expect(fittedLength(f.project, f.cut(), pattern!)).toBe(15);
+    expect(fittedLength(f.project, f.cut(), sound!)).toBe(30);
+    setSoundMeasure((file) => (file === FILE ? 1.25 : null));
+    try {
+      expect(fittedLength(f.project, f.cut(), sound!)).toBe(13);
+      expect(itemMedia(f.project, f.cut(), sound!)).toEqual({ begins: 0, length: 12.5, loops: true });
+    } finally {
+      setSoundMeasure(() => null);
+    }
+  });
+
+  it('mark a song to the end of its loop, then every time round it', () => {
+    const f = makeCut();
+    f.stack.execute(new SetLoopCommand('song', f.songId, { start: Q, end: 3 * Q }));
+    f.lay(f.audioItem('s', 'song', 0, 40));
+    const song = f.cut().items[0]!;
+    expect(fittedLength(f.project, f.cut(), song)).toBe(15);
+    const media = itemMedia(f.project, f.cut(), song)!;
+    expect(media).toEqual({ begins: 0, length: 15, loops: true, round: 10 });
+    expect(mediaMarks(song, media).seams).toEqual([15, 25, 35]);
+  });
+
+  it('own no clips: a clip on one belongs to the object item it starts on', () => {
+    const f = makeCut();
+    const sound = f.audioItem('a', 'sound', 0, 20);
+    const object: CompositionItem = { id: 'o', track: sound.track, start: 0, length: 20, offset: 0 };
+    const clip = { id: 'c', track: sound.track, animation: 'x', start: 5 };
+    expect([...carriedClips([object, sound], [clip], new Set(['a']))]).toEqual([]);
+    expect([...carriedClips([object, sound], [clip], new Set(['o']))]).toEqual(['c']);
+  });
+});
+
+describe('the sequences of a composition', () => {
+  it('lay a sound in milliseconds on a bus of its own at its volume, late by an offset below 0', () => {
+    const f = makeCut();
+    f.lay(
+      f.audioItem('a', 'sound', 5, 20, { volume: -6, offset: 0.25 }),
+      f.audioItem('b', 'sound', 30, 10, { offset: -0.3 }),
+    );
+    const [playing] = compositionPlayings(f.project, f.cut());
+    expect(playing!.route).toBe(MASTER_MIX);
+    expect(playing!.sequence.clips).toEqual([
+      {
+        tick: 500,
+        length: 2000,
+        sound: f.soundId,
+        fx: '',
+        offset: 0.25,
+        volume: 1,
+        bus: audioItemBus(f.cut().id, 'a'),
+      },
+      {
+        tick: 3300,
+        length: 700,
+        sound: f.soundId,
+        fx: '',
+        offset: 0,
+        volume: 1,
+        bus: audioItemBus(f.cut().id, 'b'),
+      },
+    ]);
+    expect(playing!.buses.map((b) => [b.name, Number(b.gain.toFixed(3))])).toEqual([
+      [audioItemBus(f.cut().id, 'a'), 0.501],
+      [audioItemBus(f.cut().id, 'b'), 1],
+    ]);
+  });
+
+  it('play a pattern round and round at its own tempo, cut where the item ends', () => {
+    const f = makeCut();
+    // 3.5 s from half a second in: beat 3 (1 s), then round two (1.5 s, 2.5 s), then the next (3.5 s) is cut.
+    f.lay(f.audioItem('p', 'pattern', 10, 35, { offset: 0.5 }));
+    const notes = compositionPlayings(f.project, f.cut())[0]!.sequence.notes;
+    expect(notes.map((n) => [n.tick, n.length, n.note, n.tempo])).toEqual([
+      [1500, 500, 64, 120],
+      [2500, 500, 60, 120],
+      [3500, 500, 64, 120],
+    ]);
+  });
+
+  it('give a song item its own copy of the song’s buses, sending into the item’s', () => {
+    const f = makeCut();
+    f.lay(f.audioItem('s', 'song', 0, 20), f.audioItem('t', 'song', 0, 20));
+    const [playing] = compositionPlayings(f.project, f.cut());
+    const bus = audioItemBus(f.cut().id, 's');
+    const track =
+      f.cut().tracks.length && (findAssetOf(f.project, f.songId, 'song') as SongAsset).tracks[0]!.id;
+    const names = playing!.buses.map((b) => b.name);
+    expect(names).toContain(`${bus}:${track}`);
+    expect(names).toContain(`${audioItemBus(f.cut().id, 't')}:${track}`);
+    expect(playing!.buses.find((b) => b.name === `${bus}:${track}`)!.sends).toEqual([
+      expect.objectContaining({ to: bus }),
+    ]);
+    const own = playing!.sequence.notes.filter((n) => n.bus?.startsWith(`${bus}:`));
+    expect(own.map((n) => [n.tick, n.note])).toEqual([
+      [0, 60],
+      [1000, 64],
+    ]);
+  });
+
+  it('go one to each place of the mix their items are sent to, or all to the master unrouted', () => {
+    const f = makeCut();
+    f.lay(
+      f.audioItem('a', 'sound', 0, 10),
+      f.audioItem('b', 'sound', 0, 10, { bus: f.busId }),
+      f.audioItem('c', 'pattern', 0, 10, { bus: MASTER_BUS }),
+    );
+    const routes = compositionPlayings(f.project, f.cut()).map((p) => [p.route, p.buses.length]);
+    expect(routes).toEqual([
+      [MASTER_MIX, 2],
+      [mixBusName(f.busId), 1],
+    ]);
+    expect(compositionPlayings(f.project, f.cut(), false).map((p) => p.route)).toEqual([MASTER_MIX]);
+  });
+
+  it('run as long as the composition, round it where it loops', () => {
+    const f = makeCut();
+    f.lay(f.audioItem('a', 'sound', 5, 20));
+    expect(compositionSpan(f.project, f.cut())).toEqual({ length: 2500, loop: null });
+    f.stack.execute(new SetAssetPropertyCommand(f.cut().id, 'loop', true));
+    expect(compositionSpan(f.project, f.cut())).toEqual({ length: 2500, loop: { start: 0, end: 2500 } });
+  });
+});
+
+describe('a playing started from a tick (#418)', () => {
+  it('starts as much further in as it waited for its files', async () => {
+    let time = 0;
+    let loaded = () => undefined as void;
+    const started: number[] = [];
+    const playbacks = new Playbacks({
+      now: () => time,
+      latency: () => 0,
+      playing: () => false,
+      stop: () => undefined,
+      wake: () => () => undefined,
+      prepare: () => new Promise<void>((resolve) => (loaded = resolve)),
+      level: () => undefined,
+      after: () => undefined,
+    });
+    const play: SequencePlay = {
+      sequence: { notes: [{ tick: 0, length: 1, note: 60, velocity: 1, instrument: 'x' }] },
+      tempo: 125,
+      length: 10000,
+      loop: null,
+      needs: { layers: [], files: [] },
+      from: 1000,
+      strike: (note) => (started.push(note.at), 1),
+    };
+    playbacks.start(1, 'composition:x', play, '');
+    time = 0.25;
+    loaded();
+    await Promise.resolve();
+    const runner = playbacks.current()[0]!.runner as unknown as { position(): { ticks: number } };
+    expect(runner.position().ticks).toBe(1250);
+  });
+});
+
+/** A sink that writes down what it is asked to play and stop. */
+function recordingAudio() {
+  const sink = {
+    ...SILENT_AUDIO,
+    started: [] as { id: number; name: string; play: SequencePlay }[],
+    stopped: [] as number[],
+    playSequence: (name: string, play: SequencePlay) => {
+      const id = 100 + sink.started.length;
+      sink.started.push({ id, name, play });
+      return id;
+    },
+    stopPlayback: (id: number) => {
+      sink.stopped.push(id);
+    },
+  };
+  return sink;
+}
+
+describe('a composition played in a game', () => {
+  function run(f: ReturnType<typeof makeCut>) {
+    const behavior = makeBehavior(f, 'Director');
+    f.stack.execute(new PlaceAssetCommand(f.sceneId, behavior.id, { x: 0, y: 0 }, f.placement));
+    const edit = graphEditor(f, behavior);
+    const start = edit.addNode('OnStart');
+    const owner = edit.addNode('Owner');
+    const play = edit.addNode('PlayComposition');
+    edit.connect(start, 'exec', play, 'exec');
+    edit.connect(owner, 'object', play, 'object');
+    const audio = recordingAudio();
+    const world = World.instantiate(f.project, f.sceneId);
+    addGameSystems(world, f.project, new Input(), { audio });
+    return { audio, world, frame: (times = 1) => Array.from({ length: times }, () => world.tick(0.1)) };
+  }
+
+  it('is heard from where its clock stands once it runs, and stops with its end', () => {
+    const f = makeCut();
+    f.lay(f.audioItem('a', 'sound', 0, 5));
+    const game = run(f);
+    game.frame();
+    // The first tick shows frame 0; the clock and the sound run from the next.
+    expect(game.audio.started).toEqual([]);
+    game.frame();
+    expect(game.audio.started).toHaveLength(1);
+    expect(game.audio.started[0]!.name).toBe(`composition:${f.cut().id}`);
+    expect(game.audio.started[0]!.play.from).toBe(Math.round((0.1 + LEAD) * 1000));
+    game.frame(2);
+    expect(game.audio.started).toHaveLength(1);
+    game.frame(3);
+    expect(game.audio.stopped).toEqual([100]);
+  });
+
+  it('stands silent while the game is paused, and goes on from where its clock stood', () => {
+    const f = makeCut();
+    f.lay(f.audioItem('a', 'sound', 0, 50));
+    const game = run(f);
+    game.frame(3);
+    expect(game.audio.started).toHaveLength(1);
+    game.world.paused = true;
+    game.frame(2);
+    expect(game.audio.stopped).toEqual([100]);
+    expect(game.audio.started).toHaveLength(1);
+    game.world.paused = false;
+    game.frame();
+    expect(game.audio.started).toHaveLength(2);
+    expect(game.audio.started[1]!.play.from).toBe(Math.round((0.3 + LEAD) * 1000));
+  });
+});

tests/ui/scroll-container.test.ts

@@ -67,6 +67,44 @@ describe('a scroll container with tail space', () => {
   });
 });
 
+describe('a scroll container holding its offset', () => {
+  const scrolledToEnd = (holdOffset: boolean) => {
+    const scroll = new ScrollContainer();
+    scroll.holdOffset = holdOffset;
+    scroll.resize(200, 100);
+    const rows = new VStack();
+    for (let i = 0; i < 10; i++) rows.add(new Spacer(20));
+    scroll.setContent(rows);
+    scroll.scrollTo(100);
+    // Three rows go, as a folded group's would.
+    rows.removeChildren(7);
+    scroll.relayoutContent();
+    return { scroll, rows };
+  };
+
+  it('keeps the offset when the content shrinks, the room left below', () => {
+    const { scroll, rows } = scrolledToEnd(true);
+    expect(scroll.offset).toBe(100);
+    expect(rows.h).toBe(200);
+  });
+
+  it('lets the room go as the view is scrolled up off it', () => {
+    const { scroll, rows } = scrolledToEnd(true);
+    scroll.scrollTo(70);
+    expect(rows.h).toBe(170);
+    expect(scroll.maxScroll).toBe(70);
+    scroll.scrollTo(20);
+    expect(rows.h).toBe(140);
+    expect(scroll.maxScroll).toBe(40);
+  });
+
+  it('without it clamps the offset, sliding every row down', () => {
+    const { scroll, rows } = scrolledToEnd(false);
+    expect(scroll.offset).toBe(40);
+    expect(rows.h).toBe(140);
+  });
+});
+
 describe('a scroll container and its scrollbar', () => {
   const filled = (viewportH: number, rowCount: number) => {
     const scroll = new ScrollContainer();