Lay patterns and sounds out on tracks and play them as a song #334

Patterns could only be heard one at a time, in their own tab or from
a behavior. A song lays patterns and sounds out on tracks along one
time axis and plays them through the scheduler of #332, the third
piece of the music tools of #330.

- New asset type song under the heading Songs, with four empty tracks,
  120 BPM in 4/4 and a grid of a bar. Tempo, meter and grid are rows
  in the inspector. Tracks, items and the loop are the asset's data,
  held to their rules on load and in every command.
- A track has a name, volume, pan, mute and solo. Each is heard on a
  bus of its own, so a change reaches what is sounding at once.
- A pattern item repeats its pattern for as long as it is, with a
  transpose and another instrument if wanted. A sound item plays its
  sound as a clip at its own speed, round and round, through a sound
  FX if wanted. Both have a volume on top of their track's.
- Cut back at the left end or split, an item starts as far into what
  it plays as the cut is: ticks into a pattern, seconds into a sound.
- The tab shows the heads with M, S, volume and pan, the ruler of the
  Pattern tab and the items, a pattern with its notes, a sound with
  its waveform, a missing asset red. Patterns and sounds are dragged
  in from the Assets panel. Body, right and left end move, lengthen
  and cut back, Alt leaves the grid. Ctrl+C/V/D/A, Ctrl+E splits
  where playing starts, Delete, arrows. A double-click opens a
  pattern item in its tab.
- What is picked is a new selection of its own, so the inspector
  shows the item or the track, with a button to delete it.
- The scheduler plans clips beside notes. Where the playhead starts
  or jumps back inside a clip, it is taken up that far into its
  sound. A clip fades out at the end of its item or of the loop.
- The editor plays one piece at a time: a pattern started stops the
  song and the other way round.
- The rules are pure functions in core/music/song, editor/song/
  songView and songEdit.

Findings:
- The suites of 4ea752f and 2a3b90b were red at e2e 54: the Pattern
  pushed Struct below the fold of the + menu. The module now scrolls
  to it, as e2e 47 does for Vector art. It also left the menu open,
  which made e2e 55 fail behind it.
- The Assets panel only dragged assets a scene can place. Patterns
  and sounds are dragged now as well, and a scene says why it does
  not take one instead of calling every asset a scene.

Tests:
- song.test.ts (new, 14) and song-edit.test.ts (new, 9): model,
  round trip, sequence of patterns and clips, mute and solo, cuts,
  clips on the scheduler, moving, sizing, trimming, splitting,
  duplicating, hits, head parts. sound-library.test.ts +5: clip
  offset, end fade, pitched clip, track bus, peaks.
- e2e 65 (new) on melektaus-atx: ALL PASSED, 18 checks. e2e 54 and
  55 with the fix: ALL PASSED, 33 checks. e2e 4 with 30 asset
  types: ALL PASSED, 62 checks.
- Gate on melektaus-atx: CHECK EXIT 0, 250 test files, 3884 tests.
- Counter-checks: no taking up of a clip, no pattern offset, no clip
  end each fail 3, 1 and 1 tests.
- Probe on stein2: the tab with four tracks, before and while
  playing, looked at in two screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tools/e2e/04-assets.mjs

@@ -2,7 +2,7 @@ import { colourDistance } from '../lib/harness.mjs';
 import { inView } from './lib/scaffold.mjs';
 
 /** One per entry of `ASSET_TYPES` – a new asset type counts here too. */
-const ASSET_TYPE_COUNT = 29;
+const ASSET_TYPE_COUNT = 30;
 
 /** The asset panel: placing by dragging into the view, creating through "+", and resetting an asset value. */
 export async function assets(t) {

tools/e2e/54-struct-types.mjs

@@ -7,7 +7,7 @@
  *
  * Everything it adds carries an `st54` id or is an asset it made, and goes again at the end.
  */
-import { MAKE_OBJECT } from './lib/scaffold.mjs';
+import { inView, MAKE_OBJECT } from './lib/scaffold.mjs';
 
 export async function structTypes(t) {
   const {
@@ -93,6 +93,19 @@ export async function structTypes(t) {
   await t.showTab('scene');
   await clickText('+');
   const offered = (await json('__gm.menuRows()')) ?? [];
+  // Struct sits below the fold of the menu since the Pattern (#333); the lowest hit between the
+  // box's sides is the entry, not the tab above it.
+  await inView(
+    t,
+    '__gm.overlayRect()',
+    `(() => {
+      const box = __gm.overlayRect();
+      const hits = (__gm.locateAll('Struct') ?? []).filter(
+        (hit) => hit.x >= box.x && hit.x <= box.x + box.w && hit.y >= box.y,
+      );
+      return hits.sort((a, b) => a.y - b.y).at(-1) ?? null;
+    })()`,
+  );
   const entry = await inOverlay('Struct');
   check('the + menu offers Struct', offered.includes('Struct') && Boolean(entry), JSON.stringify(offered));
   if (!entry) return;

tools/e2e/65-song.mjs

@@ -0,0 +1,363 @@
+import { edge, node } from '../lib/builders.mjs';
+import { pickAssetType } from './lib/scaffold.mjs';
+
+/**
+ * Songs and their tracks (#334). "+ → Song" opens the tab on four empty tracks. A pattern and a
+ * sound dragged in from the Assets panel land as items where the preview showed them; an item's
+ * right end lengthens it, its body moves it across tracks, its left end cuts it back, Ctrl+D puts a
+ * copy behind it, Ctrl+E splits it where playing starts, Delete takes it away. A head mutes, solos
+ * and sets the volume of its track, "+ Track" adds one. Space plays the song through the scheduler:
+ * the pattern round and round at the song's tempo, the sound as a clip, each track on its own bus.
+ * A pattern started meanwhile takes over, and a double-click opens a pattern item in its tab.
+ * Nothing is heard in CI (muted browser): the witnesses are the model, the planned notes and clips
+ * and what the buses were set to, as in e2e 64.
+ *
+ * Everything it adds carries a `sg65` id or is the song it made, and goes again at the end.
+ */
+const BLIP_BYTES = 4044;
+const BAR = 1920;
+const PATTERN = 'sg65-pat';
+const SOUND = 'sg65-snd';
+
+export async function song(t) {
+  const {
+    check,
+    sleep,
+    json,
+    evaluate,
+    mutate,
+    mouse,
+    click,
+    clickText,
+    doubleClick,
+    press,
+    drag,
+    shot,
+    showTab,
+  } = t;
+  const waitFor = async (read, done, tries = 25) => {
+    let value = await read();
+    for (let i = 0; i < tries && !done(value); i++) {
+      await sleep(150);
+      value = await read();
+    }
+    return value;
+  };
+  /** Drags a row of the Assets panel to a point step by step: the drop only exists while the button is down. */
+  const dragAsset = async (assetId, to) => {
+    const from = await json(`__gm.assetRow(${JSON.stringify(assetId)})`);
+    await mouse('mouseMoved', from.x, from.y);
+    await mouse('mousePressed', from.x, from.y);
+    for (let i = 1; i <= 8; i++) {
+      await mouse(
+        'mouseMoved',
+        from.x + ((to.x - from.x) * i) / 8,
+        from.y + ((to.y - from.y) * i) / 8,
+        'left',
+        {
+          buttons: 1,
+        },
+      );
+      await sleep(16);
+    }
+    const drop = (await json('__gm.song.view()')).drop;
+    await mouse('mouseReleased', to.x, to.y);
+    await sleep(200);
+    return drop;
+  };
+
+  await press('Escape');
+  await evaluate('__gm.state.select(null)');
+  await showTab('scene');
+  const stored = await evaluate(`fetch('/game-maker/api/index.php?r=sounds').then((r) => r.json())`);
+  const blip = stored.find((item) => item.bytes === BLIP_BYTES);
+  await mutate(`
+    __gm.state.project.assets.push(
+      { id: ${JSON.stringify(SOUND)}, type: 'sound', name: 'Blip 65', props: {}, file: ${JSON.stringify(blip?.file ?? '')} },
+      {
+        id: 'sg65-ins',
+        type: 'instrument',
+        name: 'Lead 65',
+        props: {},
+        graph: {
+          nodes: ${JSON.stringify([
+            node('sg65-notes', 'InstrumentNotes', 0, 0, { from: 48, to: 84 }),
+            node('sg65-sample', 'InstrumentSample', 200, 0, { sound: SOUND }),
+            node('sg65-out', 'InstrumentOutput', 420, 0),
+          ])},
+          edges: ${JSON.stringify([
+            edge('sg65-e1', 'sg65-notes', 'any', 'sg65-sample', 'trigger'),
+            edge('sg65-e2', 'sg65-sample', 'out', 'sg65-out', 'in'),
+          ])},
+        },
+      },
+      {
+        id: ${JSON.stringify(PATTERN)},
+        type: 'pattern',
+        name: 'Riff 65',
+        props: { instrument: 'sg65-ins', bars: 1 },
+        notes: [
+          { id: 'sg65-n1', tick: 0, length: 480, note: 60, velocity: 1 },
+          { id: 'sg65-n2', tick: 960, length: 480, note: 64, velocity: 0.5 },
+        ],
+      },
+    );`);
+  const seconds = await evaluate(`__gm.sound.load(${JSON.stringify(blip?.file ?? '')})`);
+
+  // 65a. "+ → Song" makes a song of four empty tracks and opens its tab.
+  await clickText('+');
+  await pickAssetType(t, 'Song');
+  const created = await json(`{
+    tab: __gm.tab(),
+    current: __gm.song.current(),
+    info: __gm.song.info(),
+    songs: __gm.state.project.assets.filter((a) => a.type === 'song').map((a) => ({ id: a.id, tracks: a.tracks.length })),
+  }`);
+  check(
+    'the + menu creates a song of four empty tracks and opens its tab',
+    created.tab === 'song' &&
+      created.songs.length === 1 &&
+      created.current === created.songs[0].id &&
+      created.songs[0].tracks === 4 &&
+      created.info === '4 tracks · 0 items',
+    JSON.stringify(created),
+  );
+  const id = created.songs[0]?.id;
+  const model = () => json(`__gm.song.model(${JSON.stringify(id)})`);
+  const tracks = (await model()).tracks.map((track) => track.id);
+
+  // 65b. A pattern dragged onto the second track lands on the bar before the pointer, a bar long.
+  const preview = await dragAsset(PATTERN, await json(`__gm.song.laneAt(${BAR + 400}, 1)`));
+  let items = (await model()).items;
+  const riff = items[0];
+  check(
+    'a pattern dragged in from the Assets panel shows where it lands and lands there, picked',
+    preview?.kind === 'pattern' &&
+      preview.track === tracks[1] &&
+      preview.tick === BAR &&
+      items.length === 1 &&
+      riff.kind === 'pattern' &&
+      riff.asset === PATTERN &&
+      riff.track === tracks[1] &&
+      riff.tick === BAR &&
+      riff.length === BAR &&
+      (await json('__gm.song.picked()')).join() === riff.id &&
+      (await json('__gm.state.selection?.kind')) === 'song',
+    JSON.stringify({ preview, items }),
+  );
+
+  // 65c. A sound dragged onto the first track is as long as it plays at 120 BPM.
+  await dragAsset(SOUND, await json('__gm.song.laneAt(300, 0)'));
+  items = (await model()).items;
+  const clip = items.find((item) => item.kind === 'sound');
+  const drawn = await waitFor(
+    () => json('__gm.song.items()'),
+    (list) => list.find((item) => item.id === clip?.id)?.content === 'wave',
+  );
+  check(
+    'a sound dropped on a track lasts as long as the sound at the tempo, and shows its waveform',
+    clip?.track === tracks[0] &&
+      clip.tick === 0 &&
+      clip.length === Math.max(1, Math.round(seconds * 960)) &&
+      drawn.find((item) => item.id === riff.id)?.content === 'notes' &&
+      drawn.find((item) => item.id === clip.id)?.label === 'Blip 65',
+    JSON.stringify({ seconds, clip, drawn }),
+  );
+  await shot('01-dropped');
+
+  // 65d. The pattern's right end takes it to three bars, its body to the third track a bar later.
+  let p = await json(`__gm.song.itemAt(${JSON.stringify(riff.id)}, 'end')`);
+  let to = await json(`__gm.song.laneAt(${4 * BAR}, 1)`);
+  await drag(p.x, p.y, to.x, to.y);
+  await sleep(200);
+  const lengthened = (await model()).items.find((item) => item.id === riff.id);
+  p = await json(`__gm.song.itemAt(${JSON.stringify(riff.id)}, 'body')`);
+  const { pxPerTick } = await json('__gm.song.view()');
+  to = await json(`__gm.song.laneAt(0, 2)`);
+  await drag(p.x, p.y, p.x + BAR * pxPerTick, to.y);
+  await sleep(200);
+  const movedRiff = (await model()).items.find((item) => item.id === riff.id);
+  check(
+    'its right end lengthens the pattern to three bars, its body takes it a bar on and a track down',
+    lengthened?.length === 3 * BAR && movedRiff?.tick === 2 * BAR && movedRiff.track === tracks[2],
+    JSON.stringify({ lengthened, movedRiff }),
+  );
+
+  // 65e. Its left end cuts a bar off the front: it then starts a bar into what it repeats.
+  p = await json(`__gm.song.itemAt(${JSON.stringify(riff.id)}, 'start')`);
+  to = await json(`__gm.song.laneAt(${3 * BAR}, 2)`);
+  await drag(p.x, p.y, to.x, to.y);
+  await sleep(200);
+  const trimmedRiff = (await model()).items.find((item) => item.id === riff.id);
+  check(
+    'its left end cuts the front off, and what it plays starts that far in',
+    trimmedRiff?.tick === 3 * BAR && trimmedRiff.length === 2 * BAR && trimmedRiff.offset === BAR,
+    JSON.stringify(trimmedRiff),
+  );
+  await press('z', { ctrl: true });
+  check(
+    'one undo takes the cut back',
+    JSON.stringify((await model()).items.find((item) => item.id === riff.id)) === JSON.stringify(movedRiff),
+  );
+
+  // 65f. Ctrl+D puts a copy right behind it; a click on the ruler and Ctrl+E split the copy there.
+  p = await json(`__gm.song.itemAt(${JSON.stringify(riff.id)}, 'body')`);
+  await click(p.x, p.y);
+  await press('d', { ctrl: true });
+  items = (await model()).items;
+  const copy = items.find((item) => item.id !== riff.id && item.kind === 'pattern');
+  check(
+    'Ctrl+D puts a copy right behind the item on its track, picked',
+    items.length === 3 &&
+      copy?.tick === 5 * BAR &&
+      copy.length === 3 * BAR &&
+      copy.track === tracks[2] &&
+      (await json('__gm.song.picked()')).join() === copy.id,
+    JSON.stringify(items),
+  );
+  p = await json('__gm.song.fitAt()');
+  await click(p.x, p.y);
+  p = await json(`__gm.song.rulerAt(${6 * BAR + 300})`);
+  await click(p.x, p.y);
+  await press('e', { ctrl: true });
+  items = (await model()).items;
+  const parts = items.filter((item) => item.tick >= 5 * BAR).sort((a, b) => a.tick - b.tick);
+  check(
+    'a click on the ruler sets where playing starts, and Ctrl+E splits the picked item there',
+    (await json('__gm.song.view()')).start === 6 * BAR &&
+      parts.length === 2 &&
+      parts[0].length === BAR &&
+      parts[1].tick === 6 * BAR &&
+      parts[1].length === 2 * BAR &&
+      parts[1].offset === BAR &&
+      (await json('__gm.song.picked()')).join() === parts[1].id,
+    JSON.stringify(parts),
+  );
+  await press('Delete');
+  p = await json(`__gm.song.itemAt(${JSON.stringify(parts[0].id)}, 'body')`);
+  await click(p.x, p.y, 'right');
+  await sleep(150);
+  items = (await model()).items;
+  check(
+    'Delete and a right-click take items away',
+    items.length === 2 && items.every((item) => item.id === riff.id || item.id === clip.id),
+    JSON.stringify(items),
+  );
+
+  // 65g. The heads: mute and solo toggle, the volume slider sets decibels, "+ Track" adds a track.
+  await click(...Object.values(await json("__gm.song.headAt(0, 'mute')")));
+  await click(...Object.values(await json("__gm.song.headAt(2, 'solo')")));
+  p = await json("__gm.song.headAt(2, 'volume', 0.5)");
+  to = await json("__gm.song.headAt(2, 'volume', 0)");
+  await drag(p.x, p.y, to.x + 1, to.y);
+  await sleep(150);
+  p = await json('__gm.song.addTrackAt()');
+  await click(p.x, p.y);
+  const heads = await model();
+  check(
+    'a head mutes, solos and turns down its track, "+ Track" adds a fifth and picks it',
+    heads.tracks.length === 5 &&
+      heads.tracks[0].mute &&
+      heads.tracks[2].solo &&
+      heads.tracks[2].volume < -55 &&
+      (await json('__gm.song.pickedTrack()')) === heads.tracks[4].id &&
+      (await json('__gm.song.info()')) === '5 tracks · 2 items',
+    JSON.stringify(heads.tracks),
+  );
+  await mutate(`
+    const song = __gm.state.project.assets.find((a) => a.id === ${JSON.stringify(id)});
+    song.tracks[2].volume = -6;
+    song.tracks[2].solo = false;
+    song.tracks[0].mute = false;`);
+
+  // 65h. Space plays from the start: the pattern a bar into itself at 2 s a bar, the sound as a
+  //      clip round and round, each on the bus of its track.
+  p = await json('__gm.song.rulerAt(0)');
+  await click(p.x, p.y);
+  await mutate(`
+    const song = __gm.state.project.assets.find((a) => a.id === ${JSON.stringify(id)});
+    song.items.find((i) => i.kind === 'sound').length = ${BAR};`);
+  await press('Space');
+  const played = await waitFor(
+    () => json('{ planned: __gm.song.planned(), clips: __gm.song.clips(), playing: __gm.song.playing() }'),
+    (s) => s.planned.length >= 2 && s.clips.length >= 1,
+    60,
+  );
+  const busOf = (track) => `song:${id}:${track}`;
+  const firstClip = played.clips[0];
+  const notes = played.planned.slice(0, 2);
+  check(
+    'Space plays the song: the clip at once, the pattern from bar 3 with its notes a second apart, each on its track',
+    played.playing &&
+      Boolean(firstClip) &&
+      firstClip.sound === SOUND &&
+      firstClip.bus === busOf(tracks[0]) &&
+      Math.abs(firstClip.duration - 2) < 1e-9 &&
+      notes.length === 2 &&
+      notes.every((n) => n.instrument === 'sg65-ins' && n.bus === busOf(tracks[2]) && n.tempo === 120) &&
+      Math.abs(notes[0].at - firstClip.at - 4) < 1e-6 &&
+      Math.abs(notes[1].at - notes[0].at - 1) < 1e-6 &&
+      notes[0].note === 60,
+    JSON.stringify(played),
+  );
+  const buses = await json('__gm.song.buses()');
+  const bus = (track) => buses.find((b) => b.name === busOf(track));
+  check(
+    'each track is set on its bus: the turned-down one at -6 dB, the others at unity',
+    Math.abs(bus(tracks[2])?.gain - 10 ** (-6 / 20)) < 1e-6 &&
+      bus(tracks[0])?.gain === 1 &&
+      bus(tracks[1])?.gain === 1,
+    JSON.stringify(buses),
+  );
+  await click(...Object.values(await json("__gm.song.headAt(0, 'mute')")));
+  const muted = await waitFor(
+    () => json('__gm.song.buses()'),
+    (list) => list.find((b) => b.name === busOf(tracks[0]))?.gain === 0,
+  );
+  check(
+    'muting a track while it plays silences its bus at once',
+    muted.find((b) => b.name === busOf(tracks[0]))?.gain === 0 && (await json('__gm.song.playing()')),
+    JSON.stringify(muted),
+  );
+  await shot('02-playing');
+
+  // 65i. A pattern played in its tab takes over; a double-click on a pattern item opens it there.
+  await evaluate(`__gm.state.openAsset(${JSON.stringify(PATTERN)})`);
+  await sleep(200);
+  p = await json('__gm.pattern.playAt()');
+  await click(p.x, p.y);
+  await sleep(200);
+  const swapped = await json('{ song: __gm.song.playing(), pattern: __gm.pattern.playing() }');
+  await click(p.x, p.y);
+  check(
+    'a pattern started in its tab stops the song',
+    swapped.pattern && !swapped.song,
+    JSON.stringify(swapped),
+  );
+  await evaluate(`__gm.state.openAsset(${JSON.stringify(id)})`);
+  await sleep(200);
+  p = await json(`__gm.song.itemAt(${JSON.stringify(riff.id)}, 'body')`);
+  await doubleClick(p.x, p.y);
+  await sleep(200);
+  const opened = await json('{ tab: __gm.tab(), pattern: __gm.pattern.current() }');
+  check(
+    'a double-click on a pattern item opens the pattern in its tab',
+    opened.tab === 'pattern' && opened.pattern === PATTERN,
+    JSON.stringify(opened),
+  );
+
+  // 65j. The step takes back everything it made.
+  await evaluate('__gm.sound.stopAll()');
+  await mutate(`
+    const gone = new Set([${JSON.stringify(SOUND)}, 'sg65-ins', ${JSON.stringify(PATTERN)}, ${JSON.stringify(id)}]);
+    const project = __gm.state.project;
+    project.assets = project.assets.filter((a) => !gone.has(a.id));
+    __gm.state.select(null);`);
+  await showTab('scene');
+  check(
+    'the step leaves nothing of its own behind',
+    (await json(
+      `__gm.state.project.assets.filter((a) => a.id.startsWith('sg65') || a.type === 'song').length`,
+    )) === 0,
+  );
+}

tools/e2e/index.mjs

@@ -67,6 +67,7 @@ import { fullscreen } from './61-fullscreen.mjs';
 import { devicePixels } from './62-device-pixels.mjs';
 import { renderLayers } from './63-render-layers.mjs';
 import { pattern } from './64-pattern.mjs';
+import { song } from './65-song.mjs';
 
 export const STEPS = [
   hierarchy,
@@ -133,4 +134,5 @@ export const STEPS = [
   devicePixels,
   renderLayers,
   pattern,
+  song,
 ];

tools/lib/harness.mjs

@@ -49,6 +49,7 @@ const KEYS = {
   a: { key: 'a', code: 'KeyA', windowsVirtualKeyCode: 65 },
   c: { key: 'c', code: 'KeyC', windowsVirtualKeyCode: 67 },
   d: { key: 'd', code: 'KeyD', windowsVirtualKeyCode: 68 },
+  e: { key: 'e', code: 'KeyE', windowsVirtualKeyCode: 69 },
   g: { key: 'g', code: 'KeyG', windowsVirtualKeyCode: 71 },
   p: { key: 'p', code: 'KeyP', windowsVirtualKeyCode: 80 },
   s: { key: 's', code: 'KeyS', windowsVirtualKeyCode: 83 },