Play sounds, patterns and songs on the tracks of a composition (#418)
A trailer needs its music and effects on the same timeline as its pictures. Until now a composition could only place objects, so sound had to be started by hand from a behavior and drifted against the picture. - Audio items: a sound, a clip of a sound, a pattern or a song dragged from the asset panel onto a track becomes an item as long as one pass of it. It may share a track with object and video items and overlap them. - Editing works as for other items: move, resize, trim on the left (moves the offset), fit to media, delete, copy and paste, box selection. Sounds draw their waveform, the colour tells the kind. - The inspector shows what it plays, start, length, offset, volume (-60 to +12 dB) and bus. The bus is "From asset", the master or one of the buses, as asked for in the issue. - An item longer than its material plays in rounds. Sounds and patterns start over, a song runs on in its loop. Loop marks on the item follow the same rule, a song's first mark sits at its loop end. - compositionAudio.ts turns the items into sequences, each with its own tempo. CompositionSound plays them, in the tab and in a game. - In the tab the items are heard from the playhead on, straight into the master like every tab. A scrub is silent. Playing a song or pattern in its own tab stops the composition, and the other way round. - In a game they are planned on the audio clock for the moment the composition clock reaches them, with the scheduler's LEAD made up. While the clock stands (waiting for a video, paused, placement off) they are silent and resume at the right spot. Found along the way: - The asset panel never let a song out of its list, so a song could not be dragged anywhere. Song and scene tab reject it as they reject a pattern. - The asset panel scrolled to the selected asset on every rebuild of its list, so folding a group far below the selection jumped back up. Only a new selection scrolls now. - At the end of the list, folding a group pulled every row down by the rows that went away. ScrollContainer has a new opt-in holdOffset, set only in the asset panel, that keeps the offset until the user scrolls up. Limits: a time scale other than 1 or a frame longer than 0.1 s moves the picture and not the sound. A baked song plays its notes here, not the bake. Tests: - tsc, eslint, Prettier clean. Gate on melektaus-atx green, 309 files, 4788 tests (4772 plus 16 new). - Unit: composition-audio.test.ts 13 new, scroll-container 3 new. Counter-checks: without catching up on load time or without the silence on pause the unit tests go red. - e2e 68h: the sound is planned 1.2 ms after the composition reaches 0.5 s. Without the LEAD make-up it is 54 ms late and red. - e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a song onto a track, 20 frames each. - e2e 04: folding Sound at the end of the list keeps its header at y 234. With the old panel it jumps to 984 and the check is red. - Not run in a browser: the loop marks of a song item, copy and paste of audio items, the bus choice. - README and wiki count 2072 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
30b618d9a9
commit
77ac9d1aaf
30 files changed
+1571
-101
No files matched your search
@@ -5,9 +5,24 @@
|
||||
* into.
|
||||
*/
|
||||
import { videoLoopsOf } from '../properties/videos';
|
||||
import type { Asset, CompositionAsset, CompositionClip, CompositionItem } from './Asset';
|
||||
import { compositionFps, DEFAULT_ITEM_SECONDS, itemEnd, itemVideo } from './composition';
|
||||
import type { Project } from './Project';
|
||||
import type {
|
||||
Asset,
|
||||
CompositionAsset,
|
||||
CompositionAudioKind,
|
||||
CompositionClip,
|
||||
CompositionItem,
|
||||
} from './Asset';
|
||||
import {
|
||||
audioRoundSeconds,
|
||||
audioSeconds,
|
||||
compositionFps,
|
||||
DEFAULT_ITEM_SECONDS,
|
||||
fittedLength,
|
||||
itemEnd,
|
||||
itemVideo,
|
||||
} from './composition';
|
||||
import { findAsset, type Project } from './Project';
|
||||
import { findSound } from './sound';
|
||||
import { findVideo, videoLengthOf } from './video';
|
||||
|
||||
/** A stretch of a track, in frames. */
|
||||
@@ -114,7 +129,8 @@ export function movedItems(
|
||||
/**
|
||||
* The clips that belong to the picked items: a clip belongs to the item lying on its track at its
|
||||
* first frame, of two there the one in front – the later start, at the same start the later in the
|
||||
* list. A clip that begins before an item and only reaches into it belongs to the one before.
|
||||
* list. A clip that begins before an item and only reaches into it belongs to the one before. An
|
||||
* audio item (#418) has nothing a clip could move, and so has no clips.
|
||||
*/
|
||||
export function carriedClips(
|
||||
items: readonly CompositionItem[],
|
||||
@@ -126,7 +142,7 @@ export function carriedClips(
|
||||
for (const clip of clips) {
|
||||
let owner: CompositionItem | undefined;
|
||||
for (const item of items)
|
||||
if (item.track === clip.track && item.start <= clip.start && clip.start < itemEnd(item))
|
||||
if (!item.audio && item.track === clip.track && item.start <= clip.start && clip.start < itemEnd(item))
|
||||
if (!owner || item.start >= owner.start) owner = item;
|
||||
if (owner && picked.has(owner.id)) carried.add(clip.id);
|
||||
}
|
||||
@@ -308,32 +324,44 @@ export function trimmedItem(
|
||||
return { ...rest, start, length: item.length - by, offset };
|
||||
}
|
||||
|
||||
/** Where an item's video lies in the composition, in frames: where it begins, how long a pass is, whether it goes round. */
|
||||
/**
|
||||
* Where an item's video lies in the composition, in frames: where it begins, how long its first pass
|
||||
* is, whether it goes round, and how long every pass after the first is where that differs – a song
|
||||
* goes round its loop only.
|
||||
*/
|
||||
export interface ItemMedia {
|
||||
begins: number;
|
||||
length: number;
|
||||
loops: boolean;
|
||||
round?: number;
|
||||
}
|
||||
|
||||
/** The video of an item as it lies on its track; null without one, or while its length is not known. */
|
||||
/**
|
||||
* The video of an item as it lies on its track, or what an audio item plays (#418) – a sound and a
|
||||
* pattern go round, a song to the end of its loop and then round it –; null without one, or while
|
||||
* its length is not known.
|
||||
*/
|
||||
export function itemMedia(project: Project, comp: CompositionAsset, item: CompositionItem): ItemMedia | null {
|
||||
const video = itemVideo(project, comp, item);
|
||||
const seconds = video ? videoLengthOf(video) : 0;
|
||||
if (!video || !(seconds > 0)) return null;
|
||||
const video = item.audio ? null : itemVideo(project, comp, item);
|
||||
const seconds = item.audio ? audioSeconds(project, item.audio) : video ? videoLengthOf(video) : 0;
|
||||
if (!(seconds > 0)) return null;
|
||||
const fps = compositionFps(comp);
|
||||
return {
|
||||
const media = {
|
||||
begins: item.start - item.offset * fps,
|
||||
length: seconds * fps,
|
||||
loops: videoLoopsOf(video.asset.props),
|
||||
loops: item.audio ? true : videoLoopsOf(video!.asset.props),
|
||||
};
|
||||
const round = item.audio?.kind === 'song' ? audioRoundSeconds(project, item.audio) * fps : 0;
|
||||
return round > 0 ? { ...media, round } : media;
|
||||
}
|
||||
|
||||
/** The frames from `from` to `to` where the video begins or a pass of it ends. */
|
||||
export function mediaEdges(media: ItemMedia, from: number, to: number): number[] {
|
||||
const edges = media.begins >= from && media.begins <= to ? [media.begins] : [];
|
||||
const first = Math.max(1, Math.ceil((from - media.begins) / media.length));
|
||||
for (let pass = first; pass === 1 || media.loops; pass++) {
|
||||
const end = media.begins + pass * media.length;
|
||||
const round = media.round ?? media.length;
|
||||
const first = media.begins + media.length;
|
||||
for (let pass = Math.max(0, Math.ceil((from - first) / round)); pass === 0 || media.loops; pass++) {
|
||||
const end = first + pass * round;
|
||||
if (end > to) break;
|
||||
if (end >= from) edges.push(end);
|
||||
}
|
||||
@@ -368,11 +396,28 @@ export function snappedToMedia(frame: number, media: ItemMedia | null, reach: nu
|
||||
);
|
||||
}
|
||||
|
||||
/** What an asset is laid on a track as when it is to be heard (#418): a sound, a clip of one, a pattern or a song. */
|
||||
export function audioKindOf(project: Project, assetId: string): CompositionAudioKind | null {
|
||||
const asset = findAsset(project, assetId);
|
||||
if (asset?.type === 'pattern' || asset?.type === 'song') return asset.type;
|
||||
return findSound(project, assetId) ? 'sound' : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* How long an item of `asset` is when it is dropped in: as long as the video it is (#419) or its
|
||||
* sprite wears, else 3 s.
|
||||
* sprite wears, or what it plays as an audio item (#418); else 3 s.
|
||||
*/
|
||||
export function droppedLength(project: Project, comp: CompositionAsset, asset: Asset): number {
|
||||
const kind = audioKindOf(project, asset.id);
|
||||
if (kind)
|
||||
return fittedLength(project, comp, {
|
||||
id: '',
|
||||
track: '',
|
||||
start: 0,
|
||||
length: 1,
|
||||
offset: 0,
|
||||
audio: { kind, asset: asset.id, volume: 0, bus: '' },
|
||||
});
|
||||
const texture =
|
||||
asset.type === 'sprite' ? asset.props.texture : asset.type === 'video' ? asset.id : undefined;
|
||||
const video = typeof texture === 'string' ? findVideo(project, texture) : null;
|
||||
@@ -381,8 +426,9 @@ export function droppedLength(project: Project, comp: CompositionAsset, asset: A
|
||||
return seconds > 0 ? Math.max(1, Math.round(seconds * fps)) : Math.round(DEFAULT_ITEM_SECONDS * fps);
|
||||
}
|
||||
|
||||
/** What an item wears that its length follows: the video or clip's id, '' for none. */
|
||||
/** What an item wears that its length follows: the video or clip's id, what an audio item plays, '' for none. */
|
||||
export function mediaKey(project: Project, comp: CompositionAsset, item: CompositionItem): string {
|
||||
if (item.audio) return item.audio.asset;
|
||||
const video = itemVideo(project, comp, item);
|
||||
return video ? (video.clip?.id ?? video.asset.id) : '';
|
||||
}
|
||||
Reference in new issue
Block a user