Draw notes for an instrument in a piano roll and loop them #333

Notes could only be played one at a time, by a key or a behavior. A
pattern is a stretch of notes for one instrument, drawn in a piano
roll and played in time through the scheduler of #332. Songs (#334)
will lay patterns out on tracks.

- New asset type pattern under the heading Patterns. Instruments are
  listed under Instruments now, since a heading is one per type.
- Notes are a start and a length in ticks, a note number and a
  velocity, kept to their rules on load and in every command. The
  instrument, the length in bars, the grid, the meter and the tempo
  are rows in the inspector. The loop is the asset's own data, the
  whole pattern while none is set.
- Grids from 1/1 to 1/32 and the triplets 1/4T to 1/32T, all whole
  ticks. The meter only moves the bar lines, the notes stay put.
- The tab shows the keys down the left, a ruler with bars, beats,
  loop and start along the top, the notes between them and a
  velocity lane below. Keys the instrument plays nothing for are
  grey, the names of a drum kit stand on its keys.
- A click draws a note on the grid line before the pointer, as long
  as the last one drawn or lengthened and as hard as the last
  velocity set. The body moves a note, the right end lengthens it,
  Alt leaves the grid. A drag on empty grid picks with a band, Shift
  adds, a right-click deletes.
- Keys: Space plays, Delete, Ctrl+C/V/D/A, the arrows by a semitone
  or a grid step and with Shift by an octave or a bar. A paste lands
  where playing starts. Every gesture is one undo step.
- A click on the ruler sets where playing starts, a drag the loop or
  one of its ends, its menu loops a bar or the whole pattern.
- The wheel scrolls, Shift across, on the ruler or with Alt it zooms.
  Ctrl+wheel stays with the browser, Pixi listens passively.
- Play runs the loop through the scheduler. An edit goes to it on
  the next window without a restart, a new loop or tempo counts from
  what is planned. Another pattern shown or a project opened stops it.
- The time axis and the ruler are their own pieces in editor/music
  for the song tab. The rules of the roll are pure functions in
  rollView and rollEdit.

Findings:
- Music.play could start after a stop that came while it was still
  loading its files. A run counter makes the late start give up.
  Music.edit now loads what a new note needs as well.
- A click on an end of the loop would have set the loop again. A
  press there only moves the end once it is dragged.

Tests:
- pattern.test.ts (new, 9) and piano-roll.test.ts (new, 16): model,
  grids, meter, held notes, sequence, commands, loop, round trip,
  drum names, time lines, rows, hits, bands, lane, edit rules.
- e2e 64 (new) on melektaus-atx: ALL PASSED, 20 checks. e2e 4 with
  29 asset types: ALL PASSED, 62 checks.
- Gate on melektaus-atx: CHECK EXIT 0, 247 test files, 3839 tests.
- Counter-checks: the move without its clamp, the sequence without
  the end of the pattern, the duplicate offset without whole grid
  steps each fail 1 test. Without passing edits to the scheduler the
  e2e check for a note drawn while playing fails, the rest passes.
- Probe on stein2: the tab with a drum kit, before and while playing,
  looked at in two screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tools/e2e/04-assets.mjs

@@ -2,7 +2,7 @@ import { colourDistance } from '../lib/harness.mjs';
 import { inView } from './lib/scaffold.mjs';
 
 /** One per entry of `ASSET_TYPES` – a new asset type counts here too. */
-const ASSET_TYPE_COUNT = 28;
+const ASSET_TYPE_COUNT = 29;
 
 /** The asset panel: placing by dragging into the view, creating through "+", and resetting an asset value. */
 export async function assets(t) {

tools/e2e/64-pattern.mjs

@@ -0,0 +1,353 @@
+import { edge, node } from '../lib/builders.mjs';
+import { pickAssetType } from './lib/scaffold.mjs';
+
+/**
+ * Patterns and their piano roll (#333). "+ → Pattern" opens the tab; with an instrument picked its
+ * keys sound where the instrument plays. A click draws a note on the grid, its right end lengthens
+ * it, its body moves it – with Alt off the grid –, a band picks notes, Ctrl+D puts a copy behind
+ * them, the arrows transpose, Delete and a right-click take notes away, Ctrl+C/V copies them to the
+ * start set on the ruler, the lane sets a velocity, a drag on the ruler sets the loop. Space plays the
+ * pattern round its loop through the scheduler, and a note drawn meanwhile is planned without a
+ * restart. Nothing is heard in CI (muted browser): the witnesses are the model, the planned notes and
+ * the sound counters, as in e2e 34.
+ *
+ * Everything it adds carries a `pt64` id or is the pattern it made, and goes again at the end.
+ */
+const BLIP_BYTES = 4044;
+const INSTRUMENT = 'pt64-ins';
+
+export async function pattern(t) {
+  const { check, sleep, json, evaluate, mutate, click, clickText, press, drag, wheel, shot, showTab } = t;
+  const stats = () => json('__gm.sound.stats()');
+  const waitFor = async (read, done, tries = 25) => {
+    let value = await read();
+    for (let i = 0; i < tries && !done(value); i++) {
+      await sleep(150);
+      value = await read();
+    }
+    return value;
+  };
+
+  await press('Escape');
+  await evaluate('__gm.state.select(null)');
+  await showTab('scene');
+  const stored = await evaluate(`fetch('/game-maker/api/index.php?r=sounds').then((r) => r.json())`);
+  const blip = stored.find((item) => item.bytes === BLIP_BYTES);
+  await mutate(`
+    __gm.state.project.assets.push(
+      { id: 'pt64-snd', type: 'sound', name: 'Blip 64', props: {}, file: ${JSON.stringify(blip?.file ?? '')} },
+      {
+        id: ${JSON.stringify(INSTRUMENT)},
+        type: 'instrument',
+        name: 'Lead 64',
+        props: {},
+        graph: {
+          nodes: ${JSON.stringify([
+            node('pt64-notes', 'InstrumentNotes', 0, 0, { from: 60, to: 71 }),
+            node('pt64-sample', 'InstrumentSample', 200, 0, { sound: 'pt64-snd' }),
+            node('pt64-out', 'InstrumentOutput', 420, 0),
+          ])},
+          edges: ${JSON.stringify([
+            edge('pt64-e1', 'pt64-notes', 'any', 'pt64-sample', 'trigger'),
+            edge('pt64-e2', 'pt64-sample', 'out', 'pt64-out', 'in'),
+          ])},
+        },
+      },
+    );`);
+  await evaluate(`__gm.sound.load(${JSON.stringify(blip?.file ?? '')})`);
+
+  // 64a. "+ → Pattern" makes an empty pattern and opens its tab.
+  await clickText('+');
+  await pickAssetType(t, 'Pattern');
+  const created = await json(`{
+    tab: __gm.tab(),
+    current: __gm.pattern.current(),
+    info: __gm.pattern.info(),
+    patterns: __gm.state.project.assets.filter((a) => a.type === 'pattern').map((a) => ({ id: a.id, notes: a.notes.length })),
+  }`);
+  check(
+    'the + menu creates an empty pattern and opens its tab, which has no instrument yet',
+    created.tab === 'pattern' &&
+      created.patterns.length === 1 &&
+      created.current === created.patterns[0].id &&
+      created.patterns[0].notes === 0 &&
+      created.info === '0 notes · no instrument',
+    JSON.stringify(created),
+  );
+  const id = created.patterns[0]?.id;
+  const notes = () => json(`__gm.pattern.notes(${JSON.stringify(id)})`);
+  await mutate(
+    `__gm.state.project.assets.find((a) => a.id === ${JSON.stringify(id)}).props.instrument = ${JSON.stringify(INSTRUMENT)};`,
+  );
+  const keyed = await json('{ info: __gm.pattern.info(), keys: __gm.pattern.keys() }');
+  const sounds = (n) => keyed.keys.find((k) => k.note === n)?.sounds;
+  check(
+    'with an instrument its keys sound from C4 to B4 and the others are grey',
+    keyed.info === '0 notes · Lead 64' &&
+      sounds(60) &&
+      sounds(71) &&
+      sounds(72) === false &&
+      sounds(59) === false,
+    JSON.stringify(keyed),
+  );
+
+  // 64b. Zoomed in on the ruler, a click draws a 1/16 on the grid line before it, and sounds.
+  const before = await json('__gm.pattern.view()');
+  const r0 = await json('__gm.pattern.rulerAt(0)');
+  await wheel(r0.x, r0.y, -600);
+  const zoomed = await json('__gm.pattern.view()');
+  check(
+    'the wheel on the ruler zooms the time about the pointer',
+    zoomed.pxPerTick > before.pxPerTick * 2 && Math.abs(zoomed.panX) < 1.5,
+    JSON.stringify({ before, zoomed }),
+  );
+  const heard = await stats();
+  let p = await json('__gm.pattern.gridAt(250, 62)');
+  await click(p.x, p.y);
+  const drawn = await notes();
+  const drawnSound = await waitFor(stats, (s) => s.started > heard.started);
+  check(
+    'a click draws a 1/16 on the grid line before the pointer, picks it and lets it sound',
+    drawn.length === 1 &&
+      drawn[0].tick === 240 &&
+      drawn[0].length === 120 &&
+      drawn[0].note === 62 &&
+      drawn[0].velocity === 0.8 &&
+      (await json('__gm.pattern.picked()')).join() === drawn[0].id &&
+      drawnSound.started > heard.started,
+    JSON.stringify({ drawn, drawnSound, heard }),
+  );
+  const first = drawn[0].id;
+
+  // 64c. Its right end takes it to the grid line nearest the pointer; the next note drawn is as long.
+  const end = await json('__gm.pattern.gridAt(360, 62)');
+  p = await json('__gm.pattern.gridAt(720, 62)');
+  await drag(end.x - 2, end.y, p.x, p.y);
+  await sleep(500);
+  p = await json('__gm.pattern.gridAt(1000, 64)');
+  await click(p.x, p.y);
+  const lengths = await notes();
+  check(
+    'dragging the end lengthens it to 1/4, and the next note drawn takes that length',
+    lengths.length === 2 &&
+      lengths.find((n) => n.id === first)?.length === 480 &&
+      lengths.find((n) => n.id !== first)?.tick === 960 &&
+      lengths.find((n) => n.id !== first)?.length === 480,
+    JSON.stringify(lengths),
+  );
+  const second = lengths.find((n) => n.id !== first)?.id;
+
+  // 64d. Its body moves it onto the grid and to another key; with Alt it lands on the tick.
+  await sleep(500);
+  p = await json('__gm.pattern.gridAt(300, 62)');
+  let to = await json('__gm.pattern.gridAt(780, 60)');
+  await drag(p.x, p.y, to.x, to.y);
+  const moved = (await notes()).find((n) => n.id === first);
+  check(
+    'dragging a note moves it to the grid line and the key under the pointer',
+    moved?.tick === 720 && moved?.note === 60,
+    JSON.stringify(moved),
+  );
+  await sleep(500);
+  p = await json('__gm.pattern.gridAt(800, 60)');
+  to = await json('__gm.pattern.gridAt(850, 60)');
+  await drag(p.x, p.y, to.x, to.y, { modifiers: { alt: true } });
+  const free = (await notes()).find((n) => n.id === first);
+  await press('z', { ctrl: true });
+  const back = (await notes()).find((n) => n.id === first);
+  check(
+    'with Alt held it moves off the grid, to the tick, and Ctrl+Z takes it back',
+    free?.tick % 120 !== 0 && Math.abs(free.tick - 770) <= 6 && back?.tick === 720,
+    JSON.stringify({ free, back }),
+  );
+
+  // 64e. A band picks both; Ctrl+D puts a copy right behind them, picked; the arrows transpose it.
+  p = await json('__gm.pattern.gridAt(100, 67)');
+  to = await json('__gm.pattern.gridAt(1600, 58)');
+  await drag(p.x, p.y, to.x, to.y);
+  const banded = await json('__gm.pattern.picked()');
+  await press('d', { ctrl: true });
+  const doubled = await notes();
+  const copies = await json('__gm.pattern.picked()');
+  const copyOf = (n) => doubled.find((c) => copies.includes(c.id) && c.note === n);
+  check(
+    'a band picks the notes it touches, and Ctrl+D copies them right behind, picking the copies',
+    banded.length === 2 &&
+      banded.includes(first) &&
+      banded.includes(second) &&
+      doubled.length === 4 &&
+      copies.length === 2 &&
+      !copies.includes(first) &&
+      copyOf(60)?.tick === 1440 &&
+      copyOf(64)?.tick === 1680,
+    JSON.stringify({ banded, doubled, copies }),
+  );
+  await press('ArrowUp');
+  const up = (await notes()).filter((n) => copies.includes(n.id)).map((n) => n.note);
+  await press('ArrowUp', { shift: true });
+  const octave = (await notes()).filter((n) => copies.includes(n.id)).map((n) => n.note);
+  await press('ArrowDown', { shift: true });
+  check(
+    'an arrow transposes the picked notes a semitone, with Shift an octave',
+    up.join() === '61,65' && octave.join() === '73,77',
+    JSON.stringify({ up, octave }),
+  );
+
+  // 64f. Delete takes the picked notes, Ctrl+Z brings them back.
+  await press('Delete');
+  const deleted = await notes();
+  await press('z', { ctrl: true });
+  const restored = await notes();
+  check(
+    'Delete takes the picked notes away and Ctrl+Z brings them back',
+    deleted.length === 2 && restored.length === 4,
+    JSON.stringify({ deleted: deleted.length, restored: restored.length }),
+  );
+
+  // 64g. Ctrl+A, Ctrl+C; a click on the ruler at bar 2 sets the start; Ctrl+V lands there.
+  await press('a', { ctrl: true });
+  await press('c', { ctrl: true });
+  const bar2 = await json('__gm.pattern.rulerAt(1920)');
+  await click(bar2.x + 3, bar2.y);
+  const startSet = (await json('__gm.pattern.view()')).start;
+  await press('v', { ctrl: true });
+  const pasted = await notes();
+  const pastedIds = await json('__gm.pattern.picked()');
+  const pastedTicks = pasted.filter((n) => pastedIds.includes(n.id)).map((n) => n.tick);
+  check(
+    'a click on the ruler sets where playing starts, and Ctrl+V pastes the copied notes there',
+    startSet === 1920 && pasted.length === 8 && pastedIds.length === 4 && Math.min(...pastedTicks) === 1920,
+    JSON.stringify({ startSet, count: pasted.length, pastedTicks }),
+  );
+
+  // 64h. A right-click takes a note away; the lane sets the velocity of the one it grabs.
+  p = await json('__gm.pattern.gridAt(1980, 60)');
+  await click(p.x, p.y, 'right');
+  const erased = await notes();
+  const lane = await json('__gm.pattern.laneAt(720, 0.8)');
+  const low = await json('__gm.pattern.laneAt(720, 0.25)');
+  await drag(lane.x + 1, lane.y + 2, low.x + 1, low.y);
+  const soft = await notes();
+  check(
+    'a right-click deletes the note under it, and dragging its bar in the lane sets its velocity alone',
+    erased.length === 7 &&
+      Math.abs(soft.find((n) => n.id === first).velocity - 0.25) < 0.04 &&
+      soft.filter((n) => n.id !== first).every((n) => n.velocity === 0.8),
+    JSON.stringify({ count: erased.length, velocities: soft.map((n) => n.velocity) }),
+  );
+
+  // 64i. A drag on the ruler sets the loop; its menu takes it back to the whole pattern.
+  p = await json('__gm.pattern.rulerAt(1920)');
+  to = await json('__gm.pattern.rulerAt(2880)');
+  await drag(p.x, p.y, to.x, to.y);
+  const looped = await json(`__gm.pattern.loop(${JSON.stringify(id)})`);
+  await click(p.x + 20, p.y, 'right');
+  await clickText('Loop the whole pattern');
+  const whole = await json(
+    `{ loop: __gm.pattern.loop(${JSON.stringify(id)}), view: __gm.pattern.view().loop }`,
+  );
+  check(
+    'a drag on the ruler sets the loop on the grid, and "Loop the whole pattern" takes it away',
+    looped?.start === 1920 &&
+      looped?.end === 2880 &&
+      whole.loop === null &&
+      whole.view.start === 0 &&
+      whole.view.end === 7680,
+    JSON.stringify({ looped, whole }),
+  );
+
+  // 64j. Space plays bar 1 round its loop from the start, each note at its tick on the audio clock;
+  //      a note drawn while it plays is planned in the next round, without starting over.
+  p = await json('__gm.pattern.rulerAt(1920)');
+  to = await json('__gm.pattern.rulerAt(0)');
+  await drag(p.x, p.y, to.x, to.y);
+  const bar1 = await json(`__gm.pattern.loop(${JSON.stringify(id)})`);
+  // At the loop's start, but a click: it sets where playing starts and leaves the loop as it is.
+  await click(to.x + 2, to.y);
+  await press('Space');
+  const planned = await waitFor(
+    () => json('__gm.pattern.planned()'),
+    (list) => list.length >= 4,
+    30,
+  );
+  const playing = await json('{ playing: __gm.pattern.playing(), button: __gm.pattern.playAt().text }');
+  const at = (tick) => planned.find((n) => Math.abs(n.at - planned[0].at - (tick - 720) / 960) < 1e-6);
+  check(
+    'Space plays the loop of bar 1 from its start: every note planned at its tick on the audio clock',
+    bar1?.start === 0 &&
+      bar1?.end === 1920 &&
+      playing.playing &&
+      playing.button === 'Stop' &&
+      planned[0].note === 60 &&
+      planned[0].instrument === INSTRUMENT &&
+      at(960)?.note === 64 &&
+      at(1440)?.note === 61 &&
+      at(1680)?.note === 65,
+    JSON.stringify({
+      bar1,
+      playing,
+      planned: planned.map((n) => [n.note, +(n.at - planned[0].at).toFixed(4)]),
+    }),
+  );
+  const head = await json('__gm.pattern.position()');
+  p = await json('__gm.pattern.gridAt(1800, 69)');
+  await click(p.x, p.y);
+  const live = await waitFor(
+    () => json('__gm.pattern.planned()'),
+    (list) => list.some((n) => n.note === 69),
+    30,
+  );
+  check(
+    'a note drawn while it plays is heard in the next round, the playing goes on',
+    live.some((n) => n.note === 69) && live.length > planned.length && (await json('__gm.pattern.playing()')),
+    JSON.stringify({ head, live: live.map((n) => n.note) }),
+  );
+  await shot('64-pattern');
+  await press('Space');
+  const stopped = await waitFor(
+    async () => ({
+      playing: await json('__gm.pattern.playing()'),
+      stats: await stats(),
+      view: await json('__gm.pattern.view()'),
+    }),
+    (s) => s.stats.playing === 0,
+  );
+  check(
+    'Space again stops it, its voices fade and the playhead goes',
+    !stopped.playing &&
+      stopped.stats.playing === 0 &&
+      stopped.view.playhead === null &&
+      (await json('__gm.pattern.playAt().text')) === 'Play',
+    JSON.stringify(stopped),
+  );
+
+  // 64k. A key down the left sounds while it is held.
+  const keyBefore = await stats();
+  const key = await json('__gm.pattern.keyAt(67)');
+  await t.mouse('mouseMoved', key.x, key.y);
+  await t.mouse('mousePressed', key.x, key.y);
+  const held = await waitFor(stats, (s) => s.started > keyBefore.started);
+  const heldKey = (await json('__gm.pattern.view()')).keys;
+  await t.mouse('mouseReleased', key.x, key.y);
+  check(
+    'a key of the roll sounds while it is held and is lit',
+    held.started > keyBefore.started && heldKey === 67 && (await json('__gm.pattern.view()')).keys === null,
+    JSON.stringify({ held, heldKey }),
+  );
+
+  // 64l. The step takes back everything it made.
+  await evaluate('__gm.sound.stopAll()');
+  await mutate(`
+    const gone = new Set(['pt64-snd', ${JSON.stringify(INSTRUMENT)}, ${JSON.stringify(id)}]);
+    const project = __gm.state.project;
+    project.assets = project.assets.filter((a) => !gone.has(a.id));
+    __gm.state.select(null);`);
+  await showTab('scene');
+  check(
+    'the step leaves nothing of its own behind',
+    (await json(
+      `__gm.state.project.assets.filter((a) => a.id.startsWith('pt64') || a.type === 'pattern').length`,
+    )) === 0,
+    '',
+  );
+}

tools/e2e/index.mjs

@@ -66,6 +66,7 @@ import { release } from './60-release.mjs';
 import { fullscreen } from './61-fullscreen.mjs';
 import { devicePixels } from './62-device-pixels.mjs';
 import { renderLayers } from './63-render-layers.mjs';
+import { pattern } from './64-pattern.mjs';
 
 export const STEPS = [
   hierarchy,
@@ -131,4 +132,5 @@ export const STEPS = [
   fullscreen,
   devicePixels,
   renderLayers,
+  pattern,
 ];