Play sounds, patterns and songs on the tracks of a composition (#418)
feuerwerk / gate (push) Successful in 4m31s
feuerwerk / suite (push) Successful in 18m21s

A trailer needs its music and effects on the same timeline as its
pictures. Until now a composition could only place objects, so sound
had to be started by hand from a behavior and drifted against the
picture.

- Audio items: a sound, a clip of a sound, a pattern or a song dragged
  from the asset panel onto a track becomes an item as long as one
  pass of it. It may share a track with object and video items and
  overlap them.
- Editing works as for other items: move, resize, trim on the left
  (moves the offset), fit to media, delete, copy and paste, box
  selection. Sounds draw their waveform, the colour tells the kind.
- The inspector shows what it plays, start, length, offset, volume
  (-60 to +12 dB) and bus. The bus is "From asset", the master or one
  of the buses, as asked for in the issue.
- An item longer than its material plays in rounds. Sounds and
  patterns start over, a song runs on in its loop. Loop marks on the
  item follow the same rule, a song's first mark sits at its loop end.
- compositionAudio.ts turns the items into sequences, each with its
  own tempo. CompositionSound plays them, in the tab and in a game.
- In the tab the items are heard from the playhead on, straight into
  the master like every tab. A scrub is silent. Playing a song or
  pattern in its own tab stops the composition, and the other way
  round.
- In a game they are planned on the audio clock for the moment the
  composition clock reaches them, with the scheduler's LEAD made up.
  While the clock stands (waiting for a video, paused, placement off)
  they are silent and resume at the right spot.

Found along the way:

- The asset panel never let a song out of its list, so a song could
  not be dragged anywhere. Song and scene tab reject it as they reject
  a pattern.
- The asset panel scrolled to the selected asset on every rebuild of
  its list, so folding a group far below the selection jumped back up.
  Only a new selection scrolls now.
- At the end of the list, folding a group pulled every row down by the
  rows that went away. ScrollContainer has a new opt-in holdOffset,
  set only in the asset panel, that keeps the offset until the user
  scrolls up.

Limits: a time scale other than 1 or a frame longer than 0.1 s moves
the picture and not the sound. A baked song plays its notes here, not
the bake.

Tests:
- tsc, eslint, Prettier clean. Gate on melektaus-atx green,
  309 files, 4788 tests (4772 plus 16 new).
- Unit: composition-audio.test.ts 13 new, scroll-container 3 new.
  Counter-checks: without catching up on load time or without the
  silence on pause the unit tests go red.
- e2e 68h: the sound is planned 1.2 ms after the composition reaches
  0.5 s. Without the LEAD make-up it is 54 ms late and red.
- e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a
  song onto a track, 20 frames each.
- e2e 04: folding Sound at the end of the list keeps its header at
  y 234. With the old panel it jumps to 984 and the check is red.
- Not run in a browser: the loop marks of a song item, copy and paste
  of audio items, the bus choice.
- README and wiki count 2072 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-13 21:28:36 +02:00
1 parent 30b618d9a9
commit 77ac9d1aaf
30 files changed
+1571 -101

No files matched your search

+29 -4
View File
@@ -15,9 +15,11 @@ import { findAssetOf } from '../../core/model/Project';
import type { PropertyValue } from '../../core/model/Properties';
import { findVideo, VIDEO_WAIT_MS } from '../../core/model/video';
import { VIDEO_CLOCK_KEY, VIDEO_TIME_KEY, videoDrivenOf } from '../../core/properties/videos';
import { CompositionSound } from '../compositionSound';
import type { Entity } from '../Entity';
import type { World } from '../World';
import { bind } from './AnimationSystem';
import { type AudioSink, SILENT_AUDIO } from './Audio';
import type { System } from './System';
export interface CompositionFinish {
@@ -40,6 +42,8 @@ interface Run {
live: Map<string, Spawned>;
/** Seconds the clock has stood so far for a video without its picture, see `waits`. */
waited: number;
/** Its audio items (#418), heard while its clock runs. */
sound: CompositionSound;
}
/** What the view that shows a run knows of its driven videos (#404). */
@@ -57,26 +61,39 @@ export interface VideoPictures {
*
* While the view still waits for the picture of a video an item brings in, the composition's
* clock stands, so the video starts at its first frame and not where the clock has got to meanwhile.
* Its audio items are heard only while the clock runs: they stop while it stands – for a video, a
* placement switched off, a paused game – and start again where it stands.
*/
export class CompositionSystem implements System {
/** A paused frame stops the sound of every composition; nothing else moves. */
readonly runsWhilePaused = true;
finished: CompositionFinish[] = [];
private readonly runs = new Map<Entity, Run>();
constructor(
private readonly plays = true,
private readonly pictures: VideoPictures | null = null,
private readonly audio: AudioSink = SILENT_AUDIO,
) {}
update(world: World, dt: number): void {
this.finished = [];
for (const placement of this.runs.keys())
if (!world.entities.includes(placement)) this.runs.delete(placement);
for (const [placement, run] of this.runs)
if (!world.entities.includes(placement)) {
run.sound.stop();
this.runs.delete(placement);
}
if (world.frozen) {
for (const run of this.runs.values()) run.sound.stop();
return;
}
const started = world.entities.filter((e) => e.type === 'composition' && e.composition);
for (const placement of started) this.advance(world, placement, dt);
}
stop(): void {
this.finished = [];
for (const run of this.runs.values()) run.sound.stop();
this.runs.clear();
}
@@ -96,8 +113,13 @@ export class CompositionSystem implements System {
return;
}
run = this.runs.get(placement);
if (!run) this.runs.set(placement, (run = { id: player.run, live: new Map(), waited: 0 }));
if (player.shown && isHandleEnabled(placement) && !this.waits(run, dt)) player.time += dt;
if (!run)
this.runs.set(
placement,
(run = { id: player.run, live: new Map(), waited: 0, sound: new CompositionSound(this.audio) }),
);
const runs = player.shown && isHandleEnabled(placement) && !this.waits(run, dt);
if (runs) player.time += dt;
player.shown = true;
const length = compositionLength(world.project, comp);
let frame = compositionFrameAt(comp, player.time);
@@ -111,6 +133,8 @@ export class CompositionSystem implements System {
return;
}
this.show(world, comp, placement, run, frame);
if (runs) run.sound.start(world.project, comp, player.time);
else run.sound.stop();
}
/** Whether a video of a live item still has no picture; at most `VIDEO_WAIT_MS` at a stretch. */
@@ -131,6 +155,7 @@ export class CompositionSystem implements System {
private end(world: World, placement: Entity): void {
const run = this.runs.get(placement);
if (!run) return;
run.sound.stop();
for (const { entity } of run.live.values()) if (world.entities.includes(entity)) world.remove(entity);
this.runs.delete(placement);
}