Play sounds, patterns and songs on the tracks of a composition (#418)

A trailer needs its music and effects on the same timeline as its pictures. Until now a composition could only place objects, so sound had to be started by hand from a behavior and drifted against the picture. - Audio items: a sound, a clip of a sound, a pattern or a song dragged from the asset panel onto a track becomes an item as long as one pass of it. It may share a track with object and video items and overlap them. - Editing works as for other items: move, resize, trim on the left (moves the offset), fit to media, delete, copy and paste, box selection. Sounds draw their waveform, the colour tells the kind. - The inspector shows what it plays, start, length, offset, volume (-60 to +12 dB) and bus. The bus is "From asset", the master or one of the buses, as asked for in the issue. - An item longer than its material plays in rounds. Sounds and patterns start over, a song runs on in its loop. Loop marks on the item follow the same rule, a song's first mark sits at its loop end. - compositionAudio.ts turns the items into sequences, each with its own tempo. CompositionSound plays them, in the tab and in a game. - In the tab the items are heard from the playhead on, straight into the master like every tab. A scrub is silent. Playing a song or pattern in its own tab stops the composition, and the other way round. - In a game they are planned on the audio clock for the moment the composition clock reaches them, with the scheduler's LEAD made up. While the clock stands (waiting for a video, paused, placement off) they are silent and resume at the right spot. Found along the way: - The asset panel never let a song out of its list, so a song could not be dragged anywhere. Song and scene tab reject it as they reject a pattern. - The asset panel scrolled to the selected asset on every rebuild of its list, so folding a group far below the selection jumped back up. Only a new selection scrolls now. - At the end of the list, folding a group pulled every row down by the rows that went away. ScrollContainer has a new opt-in holdOffset, set only in the asset panel, that keeps the offset until the user scrolls up. Limits: a time scale other than 1 or a frame longer than 0.1 s moves the picture and not the sound. A baked song plays its notes here, not the bake. Tests: - tsc, eslint, Prettier clean. Gate on melektaus-atx green, 309 files, 4788 tests (4772 plus 16 new). - Unit: composition-audio.test.ts 13 new, scroll-container 3 new. Counter-checks: without catching up on load time or without the silence on pause the unit tests go red. - e2e 68h: the sound is planned 1.2 ms after the composition reaches 0.5 s. Without the LEAD make-up it is 54 ms late and red. - e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a song onto a track, 20 frames each. - e2e 04: folding Sound at the end of the list keeps its header at y 234. With the old panel it jumps to 984 and the check is red. - Not run in a browser: the loop marks of a song item, copy and paste of audio items, the bus choice. - README and wiki count 2072 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:28:36 +02:00
1 parent 30b618d9a9
commit 77ac9d1aaf
@@ -57,6 +57,7 @@ export async function assets(t) {

  await addMenuOnRightClick(t);
  await emptyGroupsOnSwitch(t);
+  await foldKeepsPlace(t);

  // 4b. Drag the Crate asset from the panel into the scene view. Driven step by step rather than
  //     with `drag()`, because the preview only exists while the button is still down.
@@ -681,3 +682,56 @@ async function emptyGroupsOnSwitch(t) {
    JSON.stringify({ before: shape(before), off: shape(off) }),
  );
 }
+
+/**
+ * Folding a group leaves the list where it stood: it does not jump back to the selected asset, and
+ * at the end of the list, where the room below lets it scroll, the rows do not slide down by the
+ * ones that went away. The heading clicked stays under the pointer, folded and unfolded.
+ */
+async function foldKeepsPlace(t) {
+  const { check, sleep, json, mutate, mouse, click, select, showTab } = t;
+  const before = await json('__gm.state.selection');
+  await mutate(`
+    const p = __gm.state.project;
+    for (let i = 1; i <= 30; i++)
+      p.assets.push({ id: 'fold04-' + i, type: 'sprite', name: 'Fold ' + String(i).padStart(2, '0'),
+        props: { width: 10, height: 10, color: 0xffffff, texture: '' } });
+    p.assets.push({ id: 'fold04-sound', type: 'sound', name: 'Fold Sound', props: {}, file: '' });`);
+  // The Crate is selected far above; the Sounds group, unfolded, sits at the end of the list.
+  await select({ kind: 'asset', assetId: 'crate' });
+  await json(`__gm.assetRow('fold04-sound')`);
+  const rect = await json('__gm.assetsRect()');
+  await mouse('mouseWheel', rect.x + rect.w / 2, rect.y + rect.h / 2, 'none', { deltaX: 0, deltaY: 5000 });
+  await sleep(500);
+  const at = async () => ({
+    heading: (await json(`__gm.assetRowShown('group:sound')`))?.y,
+    crate: (await json(`__gm.assetRowShown('crate')`))?.y,
+  });
+  const start = await at();
+  const heading = await json(`__gm.assetRowShown('group:sound')`);
+  await click(heading.x, heading.y);
+  await sleep(200);
+  const folded = await at();
+  const groupFolded = !(await json('__gm.assetGroups()')).find((g) => g.type === 'sound')?.expanded;
+  await sleep(450);
+  await click(heading.x, heading.y);
+  await sleep(200);
+  const unfolded = await at();
+  check(
+    'folding a group at the end of the list leaves its heading under the pointer, the selection out of view',
+    start.heading > rect.y &&
+      start.heading < rect.y + rect.h &&
+      start.crate < rect.y &&
+      groupFolded &&
+      folded.heading === start.heading &&
+      unfolded.heading === start.heading &&
+      folded.crate === start.crate &&
+      unfolded.crate === start.crate,
+    JSON.stringify({ rect, start, folded, groupFolded, unfolded }),
+  );
+  await mutate(`
+    const p = __gm.state.project;
+    p.assets = p.assets.filter((a) => !a.id.startsWith('fold04-'));`);
+  await select(before ?? null);
+  await showTab('scene');
+}
@@ -10,7 +10,8 @@ import { MAKE_OBJECT } from './lib/scaffold.mjs';
 * item is left. Nothing of the run stays in the scene. An item's video is waited for: the clock
 * stands until the Game tab has its first picture, for a clip that starts later the picture there,
 * and then the video plays at once. A recording that runs meanwhile takes no frame. When the item
- * ends, its video stops and goes quiet with it.
+ * ends, its video stops and goes quiet with it. A sound item is planned once the clock runs, timed
+ * to the cut.
 *
 * Everything it adds carries a `cp68` id and goes again at the end.
 */
@@ -368,6 +369,59 @@ export async function composition(t) {
    }),
  );

+  // 68h (#418). A sound item half a second in, beside the video item from 68d: the cut waits for the
+  //      video, and its sound with it – planned once the clock runs, for the second of the audio
+  //      clock the cut reaches the item at, and to its end.
+  const sounds = await evaluate(`fetch('/feuerwerk/api/index.php?r=sounds').then((r) => r.json())`);
+  const blip = sounds.find((item) => item.bytes === 4044);
+  await mutate(`
+    const p = __gm.state.project;
+    p.assets.push({ id: 'cp68-blip', type: 'sound', name: 'Blip 68', props: {}, file: ${JSON.stringify(blip?.file ?? '')} });
+    p.assets.find((a) => a.id === 'cp68-screen').props.texture = 'cp68-video';
+    const c = p.assets.find((a) => a.id === 'cp68-cut');
+    c.items = [
+      { id: 'cp68-v', track: 'cp68-front', start: 0, length: 120, offset: 0 },
+      { id: 'cp68-tone', track: 'cp68-back', start: 30, length: 30, offset: 0,
+        audio: { kind: 'sound', asset: 'cp68-blip', volume: 0, bus: '' } },
+    ];`);
+  await evaluate(`{
+    window.cp68Sound = new Promise((done) => {
+      const rows = [];
+      const until = performance.now() + 20000;
+      const step = () => {
+        if (__gm.state.mode === 'play') {
+          const time = ${clock};
+          const tone = __gm.sound.playbacks().find((p) => p.name === 'composition:cp68-cut');
+          rows.push({ time, clock: __gm.sound.clock(), planned: tone?.clips.length ?? 0, clip: tone?.clips[0] ?? null });
+          if (time !== null && time >= 0.6) return done(JSON.stringify(rows));
+        }
+        if (performance.now() > until) return done(JSON.stringify(rows));
+        requestAnimationFrame(step);
+      };
+      step();
+    });
+    null;
+  }`);
+  await play();
+  const toneRows = JSON.parse(await evaluate('window.cp68Sound'));
+  await stop();
+  const ran = toneRows.findIndex((row) => row.time > 0);
+  const reached = toneRows.find((row) => row.time >= 0.5);
+  const toneClip = toneRows.at(-1)?.clip;
+  const early = reached && toneClip ? reached.clock - toneClip.at : null;
+  check(
+    'a sound item is planned only once the cut’s clock runs, for when the cut reaches it, to its end',
+    ran > 0 &&
+      toneRows.slice(0, ran).every((row) => row.planned === 0) &&
+      toneClip?.sound === 'cp68-blip' &&
+      toneClip.offset === 0 &&
+      Math.abs(toneClip.duration - 0.5) < 0.002 &&
+      early !== null &&
+      early > -0.02 &&
+      early < 0.05,
+    JSON.stringify({ waited: ran, early, reached, toneClip }),
+  );
+
  await mutate(`
    const p = __gm.state.project;
    p.assets = p.assets.filter((a) => !a.id.startsWith('cp68-') && a.id !== ${JSON.stringify(madeId)});
@@ -8,7 +8,7 @@
 * whose player is still loading. A copied item is pasted under the pointer or at the playhead. A
 * video dragged in wears a sprite the composition keeps to itself. An item dragged in the picture
 * follows the pointer while the button is down. A band picks items, with Alt clips, and copied clips
- * are pasted where they fit.
+ * are pasted where they fit. A sound dragged in is an audio item, heard while the playhead runs.
 *
 * Everything it adds carries a `ct69` id and goes again at the end.
 */
@@ -486,6 +486,111 @@ export async function compositionTab(t) {
    JSON.stringify({ frameWidth, along, alone }),
  );

+  // 69o (#418). A sound dragged onto a track is an audio item as long as one pass of it, picked on
+  // the composition, the inspector naming it. Played from the middle, the composition plans it as a
+  // clip that far into the sound, on the item's own bus and to the item's end; Pause ends it, and a
+  // scrub plays nothing.
+  const sounds = await evaluate(`fetch('/feuerwerk/api/index.php?r=sounds').then((r) => r.json())`);
+  const blip = sounds.find((item) => item.bytes === 4044);
+  await mutate(`
+    const p = __gm.state.project;
+    p.assets.push({ id: 'ct69-sound', type: 'sound', name: 'Blip 69', props: {}, file: ${JSON.stringify(blip?.file ?? '')} });
+    const c = p.assets.find((a) => a.id === '${CUT}');
+    c.objects = [];
+    c.items = [];
+    c.clips = [];`);
+  const seconds = await evaluate(`__gm.sound.load(${JSON.stringify(blip?.file ?? '')})`);
+  const soundDrop = await dragAsset('ct69-sound', await json(`__gm.composition.laneAt('ct69-t2', 10)`));
+  m = await model();
+  const tone = m.items[0];
+  const tonePick = await json('__gm.state.selection');
+  const toneSection = await json(`__gm.section('SOUND ITEM')`);
+  await mutate(`
+    const c = __gm.state.project.assets.find((a) => a.id === '${CUT}');
+    c.items[0].length = 20;`);
+  await evaluate('__gm.composition.scrub(15)');
+  await sleep(200);
+  const scrubbed = (await json('__gm.sound.playbacks()')).filter((p) => p.name === `composition:${CUT}`);
+  const toneAt = await json('__gm.composition.playAt()');
+  await mouse('mouseMoved', toneAt.x, toneAt.y);
+  await t.click(toneAt.x, toneAt.y);
+  let heard = [];
+  for (let i = 0; i < 40 && !heard.some((p) => p.name === `composition:${CUT}` && p.clips.length > 0); i++) {
+    await sleep(50);
+    heard = await json('__gm.sound.playbacks()');
+  }
+  const toneView = await json('__gm.composition.view()');
+  const toneSound = await json('__gm.composition.sound()');
+  const playing = heard.find((p) => p.name === `composition:${CUT}`);
+  const toneClip = playing?.clips[0];
+  await t.click(toneAt.x, toneAt.y);
+  await sleep(300);
+  const paused = (await json('__gm.sound.playbacks()')).filter((p) => p.name === `composition:${CUT}`);
+  check(
+    'a sound dragged onto a track is an audio item, heard from the playhead on its own bus until the item ends',
+    seconds > 0 &&
+      soundDrop?.what === 'item' &&
+      tone?.audio?.kind === 'sound' &&
+      tone.audio.asset === 'ct69-sound' &&
+      tone.start === 10 &&
+      tone.length === Math.max(1, Math.round(seconds * 10)) &&
+      m.objects.length === 0 &&
+      tonePick?.kind === 'composition' &&
+      JSON.stringify(tonePick.itemIds) === JSON.stringify([tone.id]) &&
+      toneSection !== null &&
+      scrubbed.length === 0 &&
+      toneClip?.sound === 'ct69-sound' &&
+      toneClip.bus === `composition:${CUT}:${tone.id}` &&
+      toneSound.playing &&
+      toneSound.playings.includes(playing.id) &&
+      toneClip.offset > 0.5 &&
+      toneClip.offset < 0.8 &&
+      Math.abs(toneClip.offset + toneClip.duration - 2) < 0.002 &&
+      paused.length === 0,
+    JSON.stringify({
+      seconds,
+      soundDrop,
+      tone,
+      tonePick,
+      scrubbed,
+      toneClip,
+      paused: paused.length,
+      toneView,
+      heard,
+      toneSection,
+      toneSound,
+    }),
+  );
+
+  // 69o-ii. A pattern and a song leave the Assets panel as well and land as audio items as long as
+  // their preview promised – the panel once let only sounds go.
+  await mutate(`
+    const p = __gm.state.project;
+    p.assets.push(
+      { id: 'ct69-pattern', type: 'pattern', name: 'Riff 69', props: { bars: 1 }, notes: [] },
+      { id: 'ct69-song', type: 'song', name: 'Song 69', props: {}, tracks: [], items: [], graph: { nodes: [], edges: [] } },
+    );
+    p.assets.find((a) => a.id === '${CUT}').items = [];`);
+  const patternDrop = await dragAsset('ct69-pattern', await json(`__gm.composition.laneAt('ct69-t1', 10)`));
+  const songDrop = await dragAsset('ct69-song', await json(`__gm.composition.laneAt('ct69-t2', 10)`));
+  m = await model();
+  const kinds = m.items.map((i) => ({
+    kind: i.audio?.kind,
+    asset: i.audio?.asset,
+    start: i.start,
+    length: i.length,
+  }));
+  const landed = (drop, kind) =>
+    drop?.what === 'item' &&
+    kinds.some(
+      (i) => i.kind === kind && i.asset === drop.assetId && i.start === 10 && i.length === drop.length,
+    );
+  check(
+    'a pattern and a song dragged onto a track are audio items as long as their preview',
+    landed(patternDrop, 'pattern') && landed(songDrop, 'song') && m.items.length === 2,
+    JSON.stringify({ patternDrop, songDrop, kinds }),
+  );
+
  await showTab('scene');
  await mutate(`
    const p = __gm.state.project;