Play sounds, patterns and songs on the tracks of a composition (#418)
A trailer needs its music and effects on the same timeline as its pictures. Until now a composition could only place objects, so sound had to be started by hand from a behavior and drifted against the picture. - Audio items: a sound, a clip of a sound, a pattern or a song dragged from the asset panel onto a track becomes an item as long as one pass of it. It may share a track with object and video items and overlap them. - Editing works as for other items: move, resize, trim on the left (moves the offset), fit to media, delete, copy and paste, box selection. Sounds draw their waveform, the colour tells the kind. - The inspector shows what it plays, start, length, offset, volume (-60 to +12 dB) and bus. The bus is "From asset", the master or one of the buses, as asked for in the issue. - An item longer than its material plays in rounds. Sounds and patterns start over, a song runs on in its loop. Loop marks on the item follow the same rule, a song's first mark sits at its loop end. - compositionAudio.ts turns the items into sequences, each with its own tempo. CompositionSound plays them, in the tab and in a game. - In the tab the items are heard from the playhead on, straight into the master like every tab. A scrub is silent. Playing a song or pattern in its own tab stops the composition, and the other way round. - In a game they are planned on the audio clock for the moment the composition clock reaches them, with the scheduler's LEAD made up. While the clock stands (waiting for a video, paused, placement off) they are silent and resume at the right spot. Found along the way: - The asset panel never let a song out of its list, so a song could not be dragged anywhere. Song and scene tab reject it as they reject a pattern. - The asset panel scrolled to the selected asset on every rebuild of its list, so folding a group far below the selection jumped back up. Only a new selection scrolls now. - At the end of the list, folding a group pulled every row down by the rows that went away. ScrollContainer has a new opt-in holdOffset, set only in the asset panel, that keeps the offset until the user scrolls up. Limits: a time scale other than 1 or a frame longer than 0.1 s moves the picture and not the sound. A baked song plays its notes here, not the bake. Tests: - tsc, eslint, Prettier clean. Gate on melektaus-atx green, 309 files, 4788 tests (4772 plus 16 new). - Unit: composition-audio.test.ts 13 new, scroll-container 3 new. Counter-checks: without catching up on load time or without the silence on pause the unit tests go red. - e2e 68h: the sound is planned 1.2 ms after the composition reaches 0.5 s. Without the LEAD make-up it is 54 ms late and red. - e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a song onto a track, 20 frames each. - e2e 04: folding Sound at the end of the list keeps its header at y 234. With the old panel it jumps to 984 and the check is red. - Not run in a browser: the loop marks of a song item, copy and paste of audio items, the bus choice. - README and wiki count 2072 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
30b618d9a9
commit
77ac9d1aaf
30 files changed
+1571
-101
No files matched your search
@@ -15,9 +15,11 @@ import { findAssetOf } from '../../core/model/Project';
|
||||
import type { PropertyValue } from '../../core/model/Properties';
|
||||
import { findVideo, VIDEO_WAIT_MS } from '../../core/model/video';
|
||||
import { VIDEO_CLOCK_KEY, VIDEO_TIME_KEY, videoDrivenOf } from '../../core/properties/videos';
|
||||
import { CompositionSound } from '../compositionSound';
|
||||
import type { Entity } from '../Entity';
|
||||
import type { World } from '../World';
|
||||
import { bind } from './AnimationSystem';
|
||||
import { type AudioSink, SILENT_AUDIO } from './Audio';
|
||||
import type { System } from './System';
|
||||
|
||||
export interface CompositionFinish {
|
||||
@@ -40,6 +42,8 @@ interface Run {
|
||||
live: Map<string, Spawned>;
|
||||
/** Seconds the clock has stood so far for a video without its picture, see `waits`. */
|
||||
waited: number;
|
||||
/** Its audio items (#418), heard while its clock runs. */
|
||||
sound: CompositionSound;
|
||||
}
|
||||
|
||||
/** What the view that shows a run knows of its driven videos (#404). */
|
||||
@@ -57,26 +61,39 @@ export interface VideoPictures {
|
||||
*
|
||||
* While the view still waits for the picture of a video an item brings in, the composition's
|
||||
* clock stands, so the video starts at its first frame and not where the clock has got to meanwhile.
|
||||
* Its audio items are heard only while the clock runs: they stop while it stands – for a video, a
|
||||
* placement switched off, a paused game – and start again where it stands.
|
||||
*/
|
||||
export class CompositionSystem implements System {
|
||||
/** A paused frame stops the sound of every composition; nothing else moves. */
|
||||
readonly runsWhilePaused = true;
|
||||
finished: CompositionFinish[] = [];
|
||||
private readonly runs = new Map<Entity, Run>();
|
||||
|
||||
constructor(
|
||||
private readonly plays = true,
|
||||
private readonly pictures: VideoPictures | null = null,
|
||||
private readonly audio: AudioSink = SILENT_AUDIO,
|
||||
) {}
|
||||
|
||||
update(world: World, dt: number): void {
|
||||
this.finished = [];
|
||||
for (const placement of this.runs.keys())
|
||||
if (!world.entities.includes(placement)) this.runs.delete(placement);
|
||||
for (const [placement, run] of this.runs)
|
||||
if (!world.entities.includes(placement)) {
|
||||
run.sound.stop();
|
||||
this.runs.delete(placement);
|
||||
}
|
||||
if (world.frozen) {
|
||||
for (const run of this.runs.values()) run.sound.stop();
|
||||
return;
|
||||
}
|
||||
const started = world.entities.filter((e) => e.type === 'composition' && e.composition);
|
||||
for (const placement of started) this.advance(world, placement, dt);
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.finished = [];
|
||||
for (const run of this.runs.values()) run.sound.stop();
|
||||
this.runs.clear();
|
||||
}
|
||||
|
||||
@@ -96,8 +113,13 @@ export class CompositionSystem implements System {
|
||||
return;
|
||||
}
|
||||
run = this.runs.get(placement);
|
||||
if (!run) this.runs.set(placement, (run = { id: player.run, live: new Map(), waited: 0 }));
|
||||
if (player.shown && isHandleEnabled(placement) && !this.waits(run, dt)) player.time += dt;
|
||||
if (!run)
|
||||
this.runs.set(
|
||||
placement,
|
||||
(run = { id: player.run, live: new Map(), waited: 0, sound: new CompositionSound(this.audio) }),
|
||||
);
|
||||
const runs = player.shown && isHandleEnabled(placement) && !this.waits(run, dt);
|
||||
if (runs) player.time += dt;
|
||||
player.shown = true;
|
||||
const length = compositionLength(world.project, comp);
|
||||
let frame = compositionFrameAt(comp, player.time);
|
||||
@@ -111,6 +133,8 @@ export class CompositionSystem implements System {
|
||||
return;
|
||||
}
|
||||
this.show(world, comp, placement, run, frame);
|
||||
if (runs) run.sound.start(world.project, comp, player.time);
|
||||
else run.sound.stop();
|
||||
}
|
||||
|
||||
/** Whether a video of a live item still has no picture; at most `VIDEO_WAIT_MS` at a stretch. */
|
||||
@@ -131,6 +155,7 @@ export class CompositionSystem implements System {
|
||||
private end(world: World, placement: Entity): void {
|
||||
const run = this.runs.get(placement);
|
||||
if (!run) return;
|
||||
run.sound.stop();
|
||||
for (const { entity } of run.live.values()) if (world.entities.includes(entity)) world.remove(entity);
|
||||
this.runs.delete(placement);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user