Play sounds as notes of an instrument asset #331

A sound could only be played as it was, one at a time. An instrument
is a node graph that makes a note a sound: Notes nodes bring the note
in, Samples play sound assets for it, the effects of a sound FX shape
it and the Output sums the layers. Its tab plays it from a keyboard,
a behavior with Play note. It is what patterns and songs (#330) will
play.

- New asset type instrument, under the heading Music, not placeable.
  A new one starts as Notes into a Sample into the Output.
- Notes are counted as MIDI does: 60 is C4, names like C#4, sharps
  only, the lowest C-1. Notes has From and To, outputs any, note,
  frequency and velocity, and with Single notes a trigger per note,
  each with an optional name for a drum kit.
- New port type trigger, only into the trigger of a source. Inputs
  can take any number of edges now (PortDef.many): a trigger, and
  the Output, which plays everything into it side by side.
- Sample plays a sound asset. Follow note shifts it by note - root
  through the pitch worklet of #317, so it keeps its length. Gate
  fades it out at the end of the note, One-shot plays through. Loop
  runs between Loop start and Loop end, also in the worklet.
- Parameters declares the instrument's own parameters with a
  default. Sound FX sends the sound through a sound FX asset, its
  parameters as inputs, the sound set in where its Input was.
- compileInstrument walks the graph once per note. A trigger is on
  only for the notes it stands for, a silent source silences what it
  alone feeds, a Mix with a silent side is the other side. velocity
  and the parameters stay open for the voice. Builds are cached per
  note until the graph or a sound or FX it names changes.
- The player takes a note as one voice of several layers, each way
  with its own file and volume. release fades the gated ways out,
  one-shots play on. playNote can release after a duration.
- Play note plays one note with velocity, volume, duration and
  detached, and hands out voice for Set sound FX parameters and
  Stop sound. Velocity is always loudness. Instrument parameters
  picks values for the instrument's parameters like Sound FX
  parameters does for an FX, renames follow.
- The tab has the graph, a header field per parameter, Stop, and
  128 keys below. Keys without sound are grey. A key plays while it
  is held. Focused, the computer keyboard plays as in a tracker by
  KeyboardEvent.code, the arrows move the octave.

Findings:
- A label of the keys set fill on the shared small text style and
  darkened every small label of the editor. Caught in a screenshot
  before the commit, the keys have a style of their own now.
- Suite task 523 on cf6799a failed two checks of e2e 34 that timed
  audio by the page's clock: a 2.95 s echo tail was gone after 1 s,
  and Quad in was further along than 0.7 s allow. Both time on the
  audio clock now through a new hook __gm.sound.clock().

Tests:
- instrument.test.ts (new, 22): notes, asset, palette, round trip,
  compile per note, drum kit, zones, Mix, velocity and parameters,
  Sound FX node with rename, problems, cache, Play note, the keys
- sound-library +7 (layers, release, duration, early release, loop
  region in buffer and worklet), pitch-player +2 (loop region),
  node-catalog +4, graph adjusted. 582 tests in the touched files.
- e2e 4 and 34 on melektaus-atx: ALL PASSED, 34 with 44 checks (+7)
- Gate on melektaus-atx: CHECK EXIT 0, 243 test files, 3750 tests
- Counter-checks: any always on, release ending one-shots, Sound FX
  inputs ignored, loop region ignored. 3, 2, 1 and 1 tests failed.
- Browser probe on stein2: keys, held key, Single notes names and
  the header looked at in screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tools/e2e/04-assets.mjs

@@ -2,7 +2,7 @@ import { colourDistance } from '../lib/harness.mjs';
 import { inView } from './lib/scaffold.mjs';
 
 /** One per entry of `ASSET_TYPES` – a new asset type counts here too. */
-const ASSET_TYPE_COUNT = 27;
+const ASSET_TYPE_COUNT = 28;
 
 /** The asset panel: placing by dragging into the view, creating through "+", and resetting an asset value. */
 export async function assets(t) {

tools/e2e/34-sound.mjs

@@ -404,12 +404,13 @@ export async function sound(t) {
   const beforeEcho = await stats();
   await click(fxPlayAt.x, fxPlayAt.y);
   const echoStarted = await waitFor(stats, (s) => s.started > beforeEcho.started);
-  const startedAt = Date.now();
+  // Timed on the audio clock: the page's can fall behind it (task 523 saw a 2.95 s tail end 1 s in).
+  const audioNow = () => json('({ playing: __gm.sound.stats().playing, clock: __gm.sound.clock() })');
+  const startedAt = (await audioNow()).clock;
   const echoLevels = await json(`__gm.sound.levels(${JSON.stringify(blip.file)})`);
-  await sleep(1000);
-  const ringing = await stats();
-  const quiet = await waitFor(stats, (s) => s.playing === 0, 40);
-  const rangFor = (Date.now() - startedAt) / 1000;
+  const ringing = await waitFor(audioNow, (s) => s.clock - startedAt >= 1);
+  const quiet = await waitFor(audioNow, (s) => s.playing === 0, 40);
+  const rangFor = quiet.clock - startedAt;
   const echoKnobs = echoLevels?.[0]?.slice(2) ?? [];
   check(
     'a Delay keeps the voice after its sound has ended, until its echoes have faded',
@@ -421,10 +422,11 @@ export async function sound(t) {
       Math.abs(echoKnobs[2] / (20000 * 0.025 ** 0.3) - 1) < 1e-3 &&
       Math.abs(echoKnobs[3] - 0.5) < 1e-6 &&
       ringing.playing === 1 &&
+      ringing.clock - startedAt < 2.4 &&
       quiet.playing === 0 &&
       rangFor > 2.4 &&
       rangFor < 4.5,
-    JSON.stringify({ delayInfo, last: echoStarted.last, echoLevels, ringing: ringing.playing, rangFor }),
+    JSON.stringify({ delayInfo, last: echoStarted.last, echoLevels, ringing, startedAt, rangFor }),
   );
   await click(fxPlayAt.x, fxPlayAt.y);
   await waitFor(stats, (s) => s.playing === 1);
@@ -557,20 +559,23 @@ export async function sound(t) {
   );
   await stop();
   // The easings of an animation key: Quad in hands the clock a curve, which is still near the start
-  // 0.7 s in and only then falls to the end (-5). Read a little later than that for the round
-  // trips, it is above 3.1 until 1.1 s; linear is below 2.8 from 0.7 s on.
+  // after 0.8 s and only then falls to the end (-5): 7 - 12 * (t / 2)^2, where linear is 7 - 6 * t.
+  // Timed on the audio clock from the moment the voice is first seen, which is no earlier than its
+  // start: the curve is then further along than `into` says, and still well above linear.
   await mutate(`
     const tune = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph.nodes.find((n) => n.id === 'sn-tune');
     Object.assign(tune.params, { ease: 'quad', direction: 'in' });`);
   await play();
-  await waitFor(pitch, (s) => typeof s === 'number');
-  await new Promise((resolve) => setTimeout(resolve, 700));
-  const easedEarly = await pitch();
+  const pitchNow = () =>
+    json(`({ pitch: __gm.sound.levels(${JSON.stringify(blip.file)})?.[0]?.[0], clock: __gm.sound.clock() })`);
+  const sweepSeen = await waitFor(pitchNow, (s) => typeof s.pitch === 'number');
+  const eased = await waitFor(pitchNow, (s) => s.clock - sweepSeen.clock >= 0.8);
+  const into = eased.clock - sweepSeen.clock;
   const easedEnd = await waitFor(pitch, (s) => typeof s === 'number' && s < -4.3);
   check(
     'Set sound FX parameters with Quad in ramps slowly first, then on to the end',
-    easedEarly > 3.1 && easedEarly < 6.95 && easedEnd < -4.3,
-    JSON.stringify({ easedEarly, easedEnd }),
+    eased.pitch < 6.95 && eased.pitch > 7 - 6 * into + 1 && easedEnd < -4.3,
+    JSON.stringify({ into, eased: eased.pitch, linearThere: 7 - 6 * into, easedEnd }),
   );
   await stop();
   // Step holds the pitch for the whole time and then jumps there.
@@ -738,6 +743,138 @@ export async function sound(t) {
     const project = __gm.state.project;
     project.assets = project.assets.filter((a) => a.id !== ${JSON.stringify(fxId)});`);
 
+  // 34g-5. An instrument (#331): "+ → Instrument" opens its tab on Notes → Sample → Output, which
+  //        plays nothing until the Sample has a sound. With the blip, looped, every key of C4–B4
+  //        sounds; a key held by the pointer plays until it is let go, the computer keyboard plays
+  //        as a tracker does, and a game's `Play note` releases its note after its duration.
+  await showTab('scene');
+  await clickText('+');
+  await pickAssetType(t, 'Instrument');
+  const insCreated = await json(`{
+    tab: __gm.tab(),
+    current: __gm.instrument.current(),
+    info: __gm.instrument.info(),
+    keys: __gm.instrument.keys(),
+    assets: __gm.state.project.assets.filter((a) => a.type === 'instrument')
+      .map((a) => ({ id: a.id, nodes: a.graph.nodes.map((n) => n.type) })),
+  }`);
+  check(
+    'the + menu creates an instrument, Notes → Sample → Output, and opens its tab; no key sounds yet',
+    insCreated.tab === 'instrument' &&
+      insCreated.assets.length === 1 &&
+      insCreated.current === insCreated.assets[0].id &&
+      insCreated.assets[0].nodes.join() === 'InstrumentNotes,InstrumentSample,InstrumentOutput' &&
+      insCreated.info === 'No sound picked' &&
+      insCreated.keys.sounding.length === 0,
+    JSON.stringify(insCreated),
+  );
+  const insId = insCreated.assets[0]?.id;
+  // Bottom left: the fitted graph is one row of three nodes, its links among them.
+  const insCanvas = await json('__gm.instrument.canvasRect()');
+  await click(insCanvas.x + 30, insCanvas.y + insCanvas.h - 30, 'right');
+  const insPalette = await json('__gm.overlayTexts()');
+  check(
+    'its palette offers its own nodes and the effects, but no Input and no behavior node',
+    ['Notes', 'Sample', 'Parameters', 'Sound FX', 'Gain', 'Delay'].every((label) =>
+      insPalette.includes(label),
+    ) &&
+      !insPalette.includes('Input') &&
+      !insPalette.includes('Branch'),
+    JSON.stringify(insPalette.slice(0, 14)),
+  );
+  await press('Escape');
+  await mutate(`
+    const ins = __gm.state.project.assets.find((a) => a.id === ${JSON.stringify(insId)});
+    Object.assign(ins.graph.nodes.find((n) => n.type === 'InstrumentSample').params, {
+      sound: ${JSON.stringify(soundId)},
+      loop: true,
+    });`);
+  const insReady = await json('{ info: __gm.instrument.info(), keys: __gm.instrument.keys() }');
+  check(
+    'with a sound picked the twelve keys from C4 to B4 sound, the others stay grey',
+    insReady.info === '12 notes' &&
+      insReady.keys.sounding.join() === [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71].join(),
+    JSON.stringify(insReady),
+  );
+  await shot('34-instrument');
+
+  await evaluate('__gm.sound.stopAll()');
+  const beforeKey = await stats();
+  const e4 = await json('__gm.instrument.keyAt(64)');
+  await t.mouse('mouseMoved', e4.x, e4.y);
+  await t.mouse('mousePressed', e4.x, e4.y);
+  const heldKey = await waitFor(stats, (s) => s.started > beforeKey.started);
+  // Longer than the blip: a looped note that ended with its sound would be gone by now.
+  await sleep(600);
+  const stillHeld = await json('{ stats: __gm.sound.stats(), keys: __gm.instrument.keys() }');
+  check(
+    'holding E4 plays the blip four semitones up, looped, as one layer of the instrument, and lights the key',
+    heldKey.last?.file === `instrument:${insId}` &&
+      heldKey.last.pitches.join() === '4' &&
+      heldKey.last.layers === 1 &&
+      stillHeld.stats.playing === 1 &&
+      stillHeld.keys.held.join() === '64',
+    JSON.stringify({ heldKey, stillHeld }),
+  );
+  await t.mouse('mouseReleased', e4.x, e4.y);
+  const letGo = await waitFor(stats, (s) => s.playing === 0);
+  check(
+    'letting go of the key ends the note: its sample fades out and the voice goes',
+    letGo.playing === 0 && (await json('__gm.instrument.keys()')).held.length === 0,
+    JSON.stringify(letGo),
+  );
+
+  // The click left the keys focused. Z is C3 there – no sound in this instrument –, one octave up C4.
+  const beforeZ = await stats();
+  await press('z');
+  await sleep(200);
+  const lowZ = await stats();
+  await press('ArrowRight');
+  await t.keyDown('z');
+  const typed = await waitFor(stats, (s) => s.started > lowZ.started);
+  const typedKeys = await json('__gm.instrument.keys()');
+  await t.keyUp('z');
+  const typedOff = await waitFor(stats, (s) => s.playing === 0);
+  check(
+    'the computer keyboard plays as a tracker: Z is C3 and silent here, one octave up it is C4 until let go',
+    lowZ.started === beforeZ.started &&
+      typedKeys.octave === 4 &&
+      typedKeys.held.join() === '60' &&
+      typed.last?.pitches.join() === '0' &&
+      typedOff.playing === 0,
+    JSON.stringify({ beforeZ: beforeZ.started, lowZ: lowZ.started, typed, typedKeys, typedOff }),
+  );
+  await press('ArrowLeft');
+
+  await mutate(`
+    const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph;
+    graph.nodes.push(${JSON.stringify(
+      node('sn-note', 'PlayNote', 260, 220, { instrument: insId, note: 67, duration: 0.3 }),
+    )});
+    graph.edges.find((e) => e.id === 'sn-e1').to = { node: 'sn-note', port: 'exec' };`);
+  await showTab('scene');
+  await evaluate('__gm.sound.stopAll()');
+  const beforeNote = await stats();
+  await play();
+  const noted = await waitFor(stats, (s) => s.started > beforeNote.started);
+  await sleep(900);
+  const released = await stats();
+  check(
+    'a game’s Play note plays G4 – the blip seven semitones up – and lets it go after its duration',
+    noted.last?.file === `instrument:${insId}` &&
+      noted.last.pitches.join() === '7' &&
+      noted.playing === 1 &&
+      released.playing === 0,
+    JSON.stringify({ noted, released }),
+  );
+  await stop();
+  await mutate(`
+    const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph;
+    graph.nodes = graph.nodes.filter((n) => n.id !== 'sn-note');
+    graph.edges.find((e) => e.id === 'sn-e1').to = { node: 'sn-play', port: 'exec' };
+    const project = __gm.state.project;
+    project.assets = project.assets.filter((a) => a.id !== ${JSON.stringify(insId)});`);
+
   // 34h. Dropping a sound file onto the editor uploads it and makes an asset named after the file.
   const dropped = await evaluate(`
     (async () => {