Play sounds, patterns and songs on the tracks of a composition (#418)
feuerwerk / gate (push) Successful in 4m31s
feuerwerk / suite (push) Successful in 18m21s

A trailer needs its music and effects on the same timeline as its
pictures. Until now a composition could only place objects, so sound
had to be started by hand from a behavior and drifted against the
picture.

- Audio items: a sound, a clip of a sound, a pattern or a song dragged
  from the asset panel onto a track becomes an item as long as one
  pass of it. It may share a track with object and video items and
  overlap them.
- Editing works as for other items: move, resize, trim on the left
  (moves the offset), fit to media, delete, copy and paste, box
  selection. Sounds draw their waveform, the colour tells the kind.
- The inspector shows what it plays, start, length, offset, volume
  (-60 to +12 dB) and bus. The bus is "From asset", the master or one
  of the buses, as asked for in the issue.
- An item longer than its material plays in rounds. Sounds and
  patterns start over, a song runs on in its loop. Loop marks on the
  item follow the same rule, a song's first mark sits at its loop end.
- compositionAudio.ts turns the items into sequences, each with its
  own tempo. CompositionSound plays them, in the tab and in a game.
- In the tab the items are heard from the playhead on, straight into
  the master like every tab. A scrub is silent. Playing a song or
  pattern in its own tab stops the composition, and the other way
  round.
- In a game they are planned on the audio clock for the moment the
  composition clock reaches them, with the scheduler's LEAD made up.
  While the clock stands (waiting for a video, paused, placement off)
  they are silent and resume at the right spot.

Found along the way:

- The asset panel never let a song out of its list, so a song could
  not be dragged anywhere. Song and scene tab reject it as they reject
  a pattern.
- The asset panel scrolled to the selected asset on every rebuild of
  its list, so folding a group far below the selection jumped back up.
  Only a new selection scrolls now.
- At the end of the list, folding a group pulled every row down by the
  rows that went away. ScrollContainer has a new opt-in holdOffset,
  set only in the asset panel, that keeps the offset until the user
  scrolls up.

Limits: a time scale other than 1 or a frame longer than 0.1 s moves
the picture and not the sound. A baked song plays its notes here, not
the bake.

Tests:
- tsc, eslint, Prettier clean. Gate on melektaus-atx green,
  309 files, 4788 tests (4772 plus 16 new).
- Unit: composition-audio.test.ts 13 new, scroll-container 3 new.
  Counter-checks: without catching up on load time or without the
  silence on pause the unit tests go red.
- e2e 68h: the sound is planned 1.2 ms after the composition reaches
  0.5 s. Without the LEAD make-up it is 54 ms late and red.
- e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a
  song onto a track, 20 frames each.
- e2e 04: folding Sound at the end of the list keeps its header at
  y 234. With the old panel it jumps to 984 and the check is red.
- Not run in a browser: the loop marks of a song item, copy and paste
  of audio items, the bus choice.
- README and wiki count 2072 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-13 21:28:36 +02:00
1 parent 30b618d9a9
commit 77ac9d1aaf
30 files changed
+1571 -101

No files matched your search

+63 -17
View File
@@ -5,9 +5,24 @@
* into.
*/
import { videoLoopsOf } from '../properties/videos';
import type { Asset, CompositionAsset, CompositionClip, CompositionItem } from './Asset';
import { compositionFps, DEFAULT_ITEM_SECONDS, itemEnd, itemVideo } from './composition';
import type { Project } from './Project';
import type {
Asset,
CompositionAsset,
CompositionAudioKind,
CompositionClip,
CompositionItem,
} from './Asset';
import {
audioRoundSeconds,
audioSeconds,
compositionFps,
DEFAULT_ITEM_SECONDS,
fittedLength,
itemEnd,
itemVideo,
} from './composition';
import { findAsset, type Project } from './Project';
import { findSound } from './sound';
import { findVideo, videoLengthOf } from './video';
/** A stretch of a track, in frames. */
@@ -114,7 +129,8 @@ export function movedItems(
/**
* The clips that belong to the picked items: a clip belongs to the item lying on its track at its
* first frame, of two there the one in front – the later start, at the same start the later in the
* list. A clip that begins before an item and only reaches into it belongs to the one before.
* list. A clip that begins before an item and only reaches into it belongs to the one before. An
* audio item (#418) has nothing a clip could move, and so has no clips.
*/
export function carriedClips(
items: readonly CompositionItem[],
@@ -126,7 +142,7 @@ export function carriedClips(
for (const clip of clips) {
let owner: CompositionItem | undefined;
for (const item of items)
if (item.track === clip.track && item.start <= clip.start && clip.start < itemEnd(item))
if (!item.audio && item.track === clip.track && item.start <= clip.start && clip.start < itemEnd(item))
if (!owner || item.start >= owner.start) owner = item;
if (owner && picked.has(owner.id)) carried.add(clip.id);
}
@@ -308,32 +324,44 @@ export function trimmedItem(
return { ...rest, start, length: item.length - by, offset };
}
/** Where an item's video lies in the composition, in frames: where it begins, how long a pass is, whether it goes round. */
/**
* Where an item's video lies in the composition, in frames: where it begins, how long its first pass
* is, whether it goes round, and how long every pass after the first is where that differs – a song
* goes round its loop only.
*/
export interface ItemMedia {
begins: number;
length: number;
loops: boolean;
round?: number;
}
/** The video of an item as it lies on its track; null without one, or while its length is not known. */
/**
* The video of an item as it lies on its track, or what an audio item plays (#418) – a sound and a
* pattern go round, a song to the end of its loop and then round it –; null without one, or while
* its length is not known.
*/
export function itemMedia(project: Project, comp: CompositionAsset, item: CompositionItem): ItemMedia | null {
const video = itemVideo(project, comp, item);
const seconds = video ? videoLengthOf(video) : 0;
if (!video || !(seconds > 0)) return null;
const video = item.audio ? null : itemVideo(project, comp, item);
const seconds = item.audio ? audioSeconds(project, item.audio) : video ? videoLengthOf(video) : 0;
if (!(seconds > 0)) return null;
const fps = compositionFps(comp);
return {
const media = {
begins: item.start - item.offset * fps,
length: seconds * fps,
loops: videoLoopsOf(video.asset.props),
loops: item.audio ? true : videoLoopsOf(video!.asset.props),
};
const round = item.audio?.kind === 'song' ? audioRoundSeconds(project, item.audio) * fps : 0;
return round > 0 ? { ...media, round } : media;
}
/** The frames from `from` to `to` where the video begins or a pass of it ends. */
export function mediaEdges(media: ItemMedia, from: number, to: number): number[] {
const edges = media.begins >= from && media.begins <= to ? [media.begins] : [];
const first = Math.max(1, Math.ceil((from - media.begins) / media.length));
for (let pass = first; pass === 1 || media.loops; pass++) {
const end = media.begins + pass * media.length;
const round = media.round ?? media.length;
const first = media.begins + media.length;
for (let pass = Math.max(0, Math.ceil((from - first) / round)); pass === 0 || media.loops; pass++) {
const end = first + pass * round;
if (end > to) break;
if (end >= from) edges.push(end);
}
@@ -368,11 +396,28 @@ export function snappedToMedia(frame: number, media: ItemMedia | null, reach: nu
);
}
/** What an asset is laid on a track as when it is to be heard (#418): a sound, a clip of one, a pattern or a song. */
export function audioKindOf(project: Project, assetId: string): CompositionAudioKind | null {
const asset = findAsset(project, assetId);
if (asset?.type === 'pattern' || asset?.type === 'song') return asset.type;
return findSound(project, assetId) ? 'sound' : null;
}
/**
* How long an item of `asset` is when it is dropped in: as long as the video it is (#419) or its
* sprite wears, else 3 s.
* sprite wears, or what it plays as an audio item (#418); else 3 s.
*/
export function droppedLength(project: Project, comp: CompositionAsset, asset: Asset): number {
const kind = audioKindOf(project, asset.id);
if (kind)
return fittedLength(project, comp, {
id: '',
track: '',
start: 0,
length: 1,
offset: 0,
audio: { kind, asset: asset.id, volume: 0, bus: '' },
});
const texture =
asset.type === 'sprite' ? asset.props.texture : asset.type === 'video' ? asset.id : undefined;
const video = typeof texture === 'string' ? findVideo(project, texture) : null;
@@ -381,8 +426,9 @@ export function droppedLength(project: Project, comp: CompositionAsset, asset: A
return seconds > 0 ? Math.max(1, Math.round(seconds * fps)) : Math.round(DEFAULT_ITEM_SECONDS * fps);
}
/** What an item wears that its length follows: the video or clip's id, '' for none. */
/** What an item wears that its length follows: the video or clip's id, what an audio item plays, '' for none. */
export function mediaKey(project: Project, comp: CompositionAsset, item: CompositionItem): string {
if (item.audio) return item.audio.asset;
const video = itemVideo(project, comp, item);
return video ? (video.clip?.id ?? video.asset.id) : '';
}