From 77ac9d1aafd21272e4b43b44f7b1da4af30c5aff Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Tue, 13 Oct 2026 21:28:36 +0200 Subject: [PATCH] Play sounds, patterns and songs on the tracks of a composition (#418) A trailer needs its music and effects on the same timeline as its pictures. Until now a composition could only place objects, so sound had to be started by hand from a behavior and drifted against the picture. - Audio items: a sound, a clip of a sound, a pattern or a song dragged from the asset panel onto a track becomes an item as long as one pass of it. It may share a track with object and video items and overlap them. - Editing works as for other items: move, resize, trim on the left (moves the offset), fit to media, delete, copy and paste, box selection. Sounds draw their waveform, the colour tells the kind. - The inspector shows what it plays, start, length, offset, volume (-60 to +12 dB) and bus. The bus is "From asset", the master or one of the buses, as asked for in the issue. - An item longer than its material plays in rounds. Sounds and patterns start over, a song runs on in its loop. Loop marks on the item follow the same rule, a song's first mark sits at its loop end. - compositionAudio.ts turns the items into sequences, each with its own tempo. CompositionSound plays them, in the tab and in a game. - In the tab the items are heard from the playhead on, straight into the master like every tab. A scrub is silent. Playing a song or pattern in its own tab stops the composition, and the other way round. - In a game they are planned on the audio clock for the moment the composition clock reaches them, with the scheduler's LEAD made up. While the clock stands (waiting for a video, paused, placement off) they are silent and resume at the right spot. Found along the way: - The asset panel never let a song out of its list, so a song could not be dragged anywhere. Song and scene tab reject it as they reject a pattern. - The asset panel scrolled to the selected asset on every rebuild of its list, so folding a group far below the selection jumped back up. Only a new selection scrolls now. - At the end of the list, folding a group pulled every row down by the rows that went away. ScrollContainer has a new opt-in holdOffset, set only in the asset panel, that keeps the offset until the user scrolls up. Limits: a time scale other than 1 or a frame longer than 0.1 s moves the picture and not the sound. A baked song plays its notes here, not the bake. Tests: - tsc, eslint, Prettier clean. Gate on melektaus-atx green, 309 files, 4788 tests (4772 plus 16 new). - Unit: composition-audio.test.ts 13 new, scroll-container 3 new. Counter-checks: without catching up on load time or without the silence on pause the unit tests go red. - e2e 68h: the sound is planned 1.2 ms after the composition reaches 0.5 s. Without the LEAD make-up it is 54 ms late and red. - e2e 04 and 69 ALL PASSED, 86 checks. 69o-ii drags a pattern and a song onto a track, 20 frames each. - e2e 04: folding Sound at the end of the list keeps its header at y 234. With the old panel it jumps to 984 and the check is red. - Not run in a browser: the loop marks of a song item, copy and paste of audio items, the bus choice. - README and wiki count 2072 checks. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- src/core/commands/compositionCommands.ts | 4 +- src/core/model/Asset.ts | 18 +- src/core/model/clipboard.ts | 5 +- src/core/model/composition.ts | 89 ++++- src/core/model/compositionEdit.ts | 80 +++- src/core/music/Scheduler.ts | 4 +- src/core/music/compositionAudio.ts | 198 ++++++++++ src/core/music/playbacks.ts | 10 +- src/core/music/sequence.ts | 4 + src/editor/Editor.ts | 1 + src/editor/actions.ts | 6 +- src/editor/clipboard.ts | 31 +- src/editor/composition/CompositionEditor.ts | 44 ++- src/editor/composition/CompositionTimeline.ts | 237 ++++++++++-- src/editor/debug/documentHooks.ts | 5 + src/editor/inspector/Inspector.ts | 7 +- src/editor/inspector/compositionSection.ts | 123 +++++- src/editor/panels/AssetPanel.ts | 21 +- src/editor/state.ts | 19 +- src/runtime/compositionSound.ts | 92 +++++ src/runtime/systems/BehaviorSystem.ts | 2 +- src/runtime/systems/CompositionSystem.ts | 33 +- src/ui/widgets/ScrollContainer.ts | 18 +- src/viewport/SoundLibrary.ts | 4 + tests/core/composition-audio.test.ts | 360 ++++++++++++++++++ tests/ui/scroll-container.test.ts | 38 ++ tools/e2e/04-assets.mjs | 54 +++ tools/e2e/68-composition.mjs | 56 ++- tools/e2e/69-composition-tab.mjs | 107 +++++- 30 files changed, 1571 insertions(+), 101 deletions(-) create mode 100644 src/core/music/compositionAudio.ts create mode 100644 src/runtime/compositionSound.ts create mode 100644 tests/core/composition-audio.test.ts diff --git a/README.md b/README.md index f077ff63..b1594fb2 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ npm run build # writes dist/ and regenerates index.html; the folder is wh The browser suite and the three example games need a machine with a GPU: ```bash -node tools/seed-e2e.mjs && node tools/e2e.mjs # the editor, 2068 checks (GM_STEPS=20 for one step) +node tools/seed-e2e.mjs && node tools/e2e.mjs # the editor, 2072 checks (GM_STEPS=20 for one step) # every tool signs in first – see the Accounts page node tools/seed-jumprun.mjs && node tools/smoke-jumprun.mjs node tools/seed-tetris.mjs && node tools/smoke-tetris.mjs diff --git a/src/core/commands/compositionCommands.ts b/src/core/commands/compositionCommands.ts index 8a767676..d6d6c45f 100644 --- a/src/core/commands/compositionCommands.ts +++ b/src/core/commands/compositionCommands.ts @@ -102,9 +102,9 @@ export function pasteCompositionItems( const comp = requireComposition(project, assetId); const paste = new PasteObjectsCommand(assetId, clip, {}, { x: 0, y: 0 }); const ids = new Map(clip.roots.map((r, n) => [r.id, paste.rootIds[n]!])); - for (const c of clip.clips ?? []) ids.set(c.id, newId()); + for (const c of [...(clip.clips ?? []), ...(clip.audio ?? [])]) ids.set(c.id, newId()); const { items, clips } = pastedItemsWithClips( - clip.roots.map((r) => copied.get(r.id)!), + [...clip.roots.map((r) => copied.get(r.id)!), ...(clip.audio ?? [])], clip.clips ?? [], comp.clips, ids, diff --git a/src/core/model/Asset.ts b/src/core/model/Asset.ts index 96d1936f..0939df1f 100644 --- a/src/core/model/Asset.ts +++ b/src/core/model/Asset.ts @@ -274,7 +274,8 @@ export interface CompositionTrack { * `offset` is how many seconds into its video the item begins, which a trim of the left edge moves; * below 0 the item begins before the video and shows its first picture until then. `fitMedia` says * its length still follows that video, or the default while there is none. Items of a track may - * overlap. + * overlap. An item with `audio` spawns no object: it plays a sound, a pattern or a song (#418), and + * `offset` is how far into that it begins. */ export interface CompositionItem { id: string; @@ -283,6 +284,21 @@ export interface CompositionItem { length: number; offset: number; fitMedia?: boolean; + audio?: CompositionAudio; +} + +export type CompositionAudioKind = 'sound' | 'pattern' | 'song'; + +/** + * What an audio item plays (#418), round and round for as long as it lasts – a song round its loop. + * `asset` is a sound, a clip of one, a pattern or a song; `volume` is in decibels. `bus` is the bus + * of the mix it is heard on: `''` where the routing sends its asset, else a bus or the master. + */ +export interface CompositionAudio { + kind: CompositionAudioKind; + asset: string; + volume: number; + bus: string; } /** diff --git a/src/core/model/clipboard.ts b/src/core/model/clipboard.ts index c6a082b0..525f4267 100644 --- a/src/core/model/clipboard.ts +++ b/src/core/model/clipboard.ts @@ -43,6 +43,8 @@ export interface ObjectClip { clips?: CompositionClip[]; /** The sprites of a composition's own the objects wear (#419), back for a paste after a cut. */ sprites?: SpriteAsset[]; + /** Audio items copied with them (#418): they have no object. */ + audio?: CompositionItem[]; } export interface GraphClip { @@ -53,10 +55,11 @@ export interface GraphClip { annotations: { kind: AnnotationKind; annotation: GraphAnnotation }[]; } -/** Animation clips of a composition (#417), for a paste into the same one or another. */ +/** Animation clips or audio items of a composition (#417, #418), for a paste into the same one or another. */ export interface CompositionClipsClip { kind: 'compositionClips'; clips: CompositionClip[]; + items?: CompositionItem[]; } export type Clip = ObjectClip | GraphClip | CompositionClipsClip; diff --git a/src/core/model/composition.ts b/src/core/model/composition.ts index 36718d09..3b60091a 100644 --- a/src/core/model/composition.ts +++ b/src/core/model/composition.ts @@ -4,11 +4,15 @@ * game read the same cut; `compositionHolder` writes it out as objects for a view to draw. */ import { newId } from '../ids'; +import { patternLength, patternTempo } from '../music/pattern'; +import { ITEM_MIN_DB, songLoop, songTempo, SOUND_ITEM_MAX_DB } from '../music/song'; +import { secondsOfTicks } from '../music/time'; import { COMPOSITION_FPS_KEY, COMPOSITION_LOOP_KEY, DEFAULT_COMPOSITION_FPS, } from '../properties/compositions'; +import { playedSoundById } from '../properties/playedSound'; import { resolveSceneObjectProps } from '../properties/resolve'; import { VIDEO_CLOCK_KEY, VIDEO_TIME_KEY, videoDrivenOf } from '../properties/videos'; import { animationDuration, fpsOf, frameCount, targetNamePath } from './animation'; @@ -18,6 +22,8 @@ import type { AnimationTrack, Asset, CompositionAsset, + CompositionAudio, + CompositionAudioKind, CompositionClip, CompositionItem, CompositionTrack, @@ -62,9 +68,47 @@ export function itemEnd(item: CompositionItem): number { return item.start + item.length; } -/** The object an item spawns; undefined once it is gone, and then the item does nothing. */ +/** The object an item spawns; undefined once it is gone, and then the item does nothing – or for an audio item. */ export function itemObject(comp: CompositionAsset, item: CompositionItem): SceneObject | undefined { - return comp.objects.find((o) => o.id === item.id); + return item.audio ? undefined : comp.objects.find((o) => o.id === item.id); +} + +/** Whether an item stands on its track: an audio item always (#418), any other while it has its object. */ +export function itemStands(comp: CompositionAsset, item: CompositionItem): boolean { + return item.audio !== undefined || itemObject(comp, item) !== undefined; +} + +/** Seconds a sound file lasts once its bytes are there, as the editor's sound library knows it (#418). */ +let soundSeconds: (file: string) => number | null = () => null; + +export function setSoundMeasure(measure: (file: string) => number | null): void { + soundSeconds = measure; +} + +/** + * Seconds the first pass of what an audio item plays lasts: its sound once loaded, its pattern, its + * song to the end of its loop – it goes round there, what lies behind is never heard –; 0 while that + * is not known or the asset is gone. + */ +export function audioSeconds(project: Project, audio: CompositionAudio): number { + if (audio.kind === 'sound') { + const file = playedSoundById(project, audio.asset); + return (file && soundSeconds(file)) || 0; + } + if (audio.kind === 'pattern') { + const pattern = findAssetOf(project, audio.asset, 'pattern'); + return pattern ? secondsOfTicks(patternLength(pattern), patternTempo(pattern.props)) : 0; + } + const song = findAssetOf(project, audio.asset, 'song'); + return song ? secondsOfTicks(songLoop(song).end, songTempo(song.props)) : 0; +} + +/** Seconds every pass after the first lasts: a song's loop, else the same as the first. */ +export function audioRoundSeconds(project: Project, audio: CompositionAudio): number { + const song = audio.kind === 'song' ? findAssetOf(project, audio.asset, 'song') : undefined; + if (!song) return audioSeconds(project, audio); + const loop = songLoop(song); + return secondsOfTicks(loop.end - loop.start, songTempo(song.props)); } /** @@ -86,7 +130,7 @@ export function clipLength(project: Project, comp: CompositionAsset, clip: Compo /** The frame after the last item or clip ends: how long one run of the composition is. */ export function compositionLength(project: Project, comp: CompositionAsset): number { let end = 0; - for (const item of comp.items) if (itemObject(comp, item)) end = Math.max(end, itemEnd(item)); + for (const item of comp.items) if (itemStands(comp, item)) end = Math.max(end, itemEnd(item)); for (const clip of comp.clips) end = Math.max(end, clip.start + clipLength(project, comp, clip)); return end; } @@ -108,13 +152,17 @@ function videoWorn(props: PropertyBag, project: Project): VideoRef | null { } /** - * The length an item with `fitMedia` takes: from its start to the end of its video, or the default - * while it wears none or the video's length is not known yet. + * The length an item with `fitMedia` takes: from its start to the end of its video, or of what an + * audio item plays; the default while it has none or its length is not known yet. */ export function fittedLength(project: Project, comp: CompositionAsset, item: CompositionItem): number { const fps = compositionFps(comp); - const video = itemVideo(project, comp, item); - const seconds = video ? videoLengthOf(video) - item.offset : 0; + const video = item.audio ? null : itemVideo(project, comp, item); + const seconds = item.audio + ? audioSeconds(project, item.audio) - item.offset + : video + ? videoLengthOf(video) - item.offset + : 0; return seconds > 0 ? Math.max(1, Math.round(seconds * fps)) : Math.round(DEFAULT_ITEM_SECONDS * fps); } @@ -314,8 +362,9 @@ export function heldCompositionTracks(raw: unknown): CompositionTrack[] { } /** - * Held to what an item may be: the id of an object it spawns and no other item does, a track of - * the composition, a start in whole frames from 0, at least a frame long, a finite offset. + * Held to what an item may be: the id of an object it spawns and no other item does – an audio item + * an id of its own and what it plays –, a track of the composition, a start in whole frames from 0, + * at least a frame long, a finite offset. */ export function heldCompositionItems( raw: unknown, @@ -329,7 +378,9 @@ export function heldCompositionItems( const id = text(i.id); const start = Math.round(finite(i.start)); const length = Math.round(finite(i.length)); - if (!placed.has(id) || items.some((other) => other.id === id) || !known.has(text(i.track))) continue; + const audio = heldAudio(i.audio); + if (!id || (audio ? placed.has(id) : !placed.has(id))) continue; + if (items.some((other) => other.id === id) || !known.has(text(i.track))) continue; if (!(start >= 0) || !(length >= 1)) continue; const offset = finite(i.offset); const item: CompositionItem = { @@ -340,11 +391,29 @@ export function heldCompositionItems( offset: Number.isFinite(offset) ? offset : 0, }; if (i.fitMedia === true) item.fitMedia = true; + if (audio) item.audio = audio; items.push(item); } return items; } +const AUDIO_KINDS: readonly CompositionAudioKind[] = ['sound', 'pattern', 'song']; + +/** Held to what an audio item plays: a kind, an asset id kept even once the asset is gone, a volume. */ +function heldAudio(raw: unknown): CompositionAudio | null { + if (typeof raw !== 'object' || raw === null) return null; + const a = raw as Record; + const kind = AUDIO_KINDS.find((k) => k === a.kind); + if (!kind || !text(a.asset)) return null; + const volume = finite(a.volume); + return { + kind, + asset: text(a.asset), + volume: Number.isFinite(volume) ? Math.max(ITEM_MIN_DB, Math.min(SOUND_ITEM_MAX_DB, volume)) : 0, + bus: text(a.bus), + }; +} + /** Held like an item; an animation that is gone is kept, as an animation keeps a root that is gone. */ export function heldCompositionClips(raw: unknown, tracks: readonly CompositionTrack[]): CompositionClip[] { const known = new Set(tracks.map((t) => t.id)); diff --git a/src/core/model/compositionEdit.ts b/src/core/model/compositionEdit.ts index 8b379e07..2f00aa84 100644 --- a/src/core/model/compositionEdit.ts +++ b/src/core/model/compositionEdit.ts @@ -5,9 +5,24 @@ * into. */ import { videoLoopsOf } from '../properties/videos'; -import type { Asset, CompositionAsset, CompositionClip, CompositionItem } from './Asset'; -import { compositionFps, DEFAULT_ITEM_SECONDS, itemEnd, itemVideo } from './composition'; -import type { Project } from './Project'; +import type { + Asset, + CompositionAsset, + CompositionAudioKind, + CompositionClip, + CompositionItem, +} from './Asset'; +import { + audioRoundSeconds, + audioSeconds, + compositionFps, + DEFAULT_ITEM_SECONDS, + fittedLength, + itemEnd, + itemVideo, +} from './composition'; +import { findAsset, type Project } from './Project'; +import { findSound } from './sound'; import { findVideo, videoLengthOf } from './video'; /** A stretch of a track, in frames. */ @@ -114,7 +129,8 @@ export function movedItems( /** * The clips that belong to the picked items: a clip belongs to the item lying on its track at its * first frame, of two there the one in front – the later start, at the same start the later in the - * list. A clip that begins before an item and only reaches into it belongs to the one before. + * list. A clip that begins before an item and only reaches into it belongs to the one before. An + * audio item (#418) has nothing a clip could move, and so has no clips. */ export function carriedClips( items: readonly CompositionItem[], @@ -126,7 +142,7 @@ export function carriedClips( for (const clip of clips) { let owner: CompositionItem | undefined; for (const item of items) - if (item.track === clip.track && item.start <= clip.start && clip.start < itemEnd(item)) + if (!item.audio && item.track === clip.track && item.start <= clip.start && clip.start < itemEnd(item)) if (!owner || item.start >= owner.start) owner = item; if (owner && picked.has(owner.id)) carried.add(clip.id); } @@ -308,32 +324,44 @@ export function trimmedItem( return { ...rest, start, length: item.length - by, offset }; } -/** Where an item's video lies in the composition, in frames: where it begins, how long a pass is, whether it goes round. */ +/** + * Where an item's video lies in the composition, in frames: where it begins, how long its first pass + * is, whether it goes round, and how long every pass after the first is where that differs – a song + * goes round its loop only. + */ export interface ItemMedia { begins: number; length: number; loops: boolean; + round?: number; } -/** The video of an item as it lies on its track; null without one, or while its length is not known. */ +/** + * The video of an item as it lies on its track, or what an audio item plays (#418) – a sound and a + * pattern go round, a song to the end of its loop and then round it –; null without one, or while + * its length is not known. + */ export function itemMedia(project: Project, comp: CompositionAsset, item: CompositionItem): ItemMedia | null { - const video = itemVideo(project, comp, item); - const seconds = video ? videoLengthOf(video) : 0; - if (!video || !(seconds > 0)) return null; + const video = item.audio ? null : itemVideo(project, comp, item); + const seconds = item.audio ? audioSeconds(project, item.audio) : video ? videoLengthOf(video) : 0; + if (!(seconds > 0)) return null; const fps = compositionFps(comp); - return { + const media = { begins: item.start - item.offset * fps, length: seconds * fps, - loops: videoLoopsOf(video.asset.props), + loops: item.audio ? true : videoLoopsOf(video!.asset.props), }; + const round = item.audio?.kind === 'song' ? audioRoundSeconds(project, item.audio) * fps : 0; + return round > 0 ? { ...media, round } : media; } /** The frames from `from` to `to` where the video begins or a pass of it ends. */ export function mediaEdges(media: ItemMedia, from: number, to: number): number[] { const edges = media.begins >= from && media.begins <= to ? [media.begins] : []; - const first = Math.max(1, Math.ceil((from - media.begins) / media.length)); - for (let pass = first; pass === 1 || media.loops; pass++) { - const end = media.begins + pass * media.length; + const round = media.round ?? media.length; + const first = media.begins + media.length; + for (let pass = Math.max(0, Math.ceil((from - first) / round)); pass === 0 || media.loops; pass++) { + const end = first + pass * round; if (end > to) break; if (end >= from) edges.push(end); } @@ -368,11 +396,28 @@ export function snappedToMedia(frame: number, media: ItemMedia | null, reach: nu ); } +/** What an asset is laid on a track as when it is to be heard (#418): a sound, a clip of one, a pattern or a song. */ +export function audioKindOf(project: Project, assetId: string): CompositionAudioKind | null { + const asset = findAsset(project, assetId); + if (asset?.type === 'pattern' || asset?.type === 'song') return asset.type; + return findSound(project, assetId) ? 'sound' : null; +} + /** * How long an item of `asset` is when it is dropped in: as long as the video it is (#419) or its - * sprite wears, else 3 s. + * sprite wears, or what it plays as an audio item (#418); else 3 s. */ export function droppedLength(project: Project, comp: CompositionAsset, asset: Asset): number { + const kind = audioKindOf(project, asset.id); + if (kind) + return fittedLength(project, comp, { + id: '', + track: '', + start: 0, + length: 1, + offset: 0, + audio: { kind, asset: asset.id, volume: 0, bus: '' }, + }); const texture = asset.type === 'sprite' ? asset.props.texture : asset.type === 'video' ? asset.id : undefined; const video = typeof texture === 'string' ? findVideo(project, texture) : null; @@ -381,8 +426,9 @@ export function droppedLength(project: Project, comp: CompositionAsset, asset: A return seconds > 0 ? Math.max(1, Math.round(seconds * fps)) : Math.round(DEFAULT_ITEM_SECONDS * fps); } -/** What an item wears that its length follows: the video or clip's id, '' for none. */ +/** What an item wears that its length follows: the video or clip's id, what an audio item plays, '' for none. */ export function mediaKey(project: Project, comp: CompositionAsset, item: CompositionItem): string { + if (item.audio) return item.audio.asset; const video = itemVideo(project, comp, item); return video ? (video.clip?.id ?? video.asset.id) : ''; } diff --git a/src/core/music/Scheduler.ts b/src/core/music/Scheduler.ts index 95d393fb..9f9bcdac 100644 --- a/src/core/music/Scheduler.ts +++ b/src/core/music/Scheduler.ts @@ -196,7 +196,7 @@ export class Scheduler { note: due.note, velocity: due.velocity, instrument: due.instrument, - tempo: this.tempo, + tempo: due.tempo ?? this.tempo, ...(due.volume !== undefined && { volume: due.volume }), ...(due.bus !== undefined && { bus: due.bus }), }); @@ -212,7 +212,7 @@ export class Scheduler { fx: due.fx, volume: due.volume, bus: due.bus, - tempo: this.tempo, + tempo: due.tempo ?? this.tempo, }); if (voice !== 0) this.voices.push(voice); } diff --git a/src/core/music/compositionAudio.ts b/src/core/music/compositionAudio.ts new file mode 100644 index 00000000..8bd50699 --- /dev/null +++ b/src/core/music/compositionAudio.ts @@ -0,0 +1,198 @@ +/** + * What a composition plays of its audio items (#418), as sequences the scheduler plays on the audio + * clock. Time is counted in milliseconds – ticks at `COMPOSITION_TEMPO` –, so a frame of any `fps` + * lands within a millisecond. A pattern or a song keeps its own tempo inside its item: its notes are + * laid out here in the composition's time, each with the tempo a Delay synced to beats counts by. + * + * Every item is heard on a bus of its own at its volume, through the sound FX of its pattern or song; + * a song's tracks are buses of that item. One playing goes to one place of the mix, so the items are + * split into one sequence for each place they are sent to. + */ +import type { CompositionAsset, CompositionItem } from '../model/Asset'; +import { compositionFps, compositionLength, compositionLoops } from '../model/composition'; +import { findAssetOf, type Project } from '../model/Project'; +import { findSound } from '../model/sound'; +import { pickedFx } from '../properties/assetFx'; +import { MASTER_MIX, type MixRoute, playedRoute } from './mixer'; +import { fxVoiceOf } from './needs'; +import { patternInstrument, patternLength, patternSequence, patternTempo } from './pattern'; +import type { BusLevel } from './playbacks'; +import { + clipsIn, + type LoopRange, + notesIn, + type Sequence, + type SequenceClip, + type SequenceNote, +} from './sequence'; +import { dbToGain, songLoop, songSequence, songTempo } from './song'; +import { songBuses } from './songBuses'; +import { secondsOfTicks, ticksOfSeconds } from './time'; + +/** The tempo a composition's sequences count at: 1000 ticks to the second. */ +export const COMPOSITION_TEMPO = 125; + +/** Ticks of a composition's sequence: milliseconds. */ +export function compositionTicks(seconds: number): number { + return Math.round(ticksOfSeconds(seconds, COMPOSITION_TEMPO)); +} + +/** One sequence of a composition and where it goes. */ +export interface CompositionPlaying { + route: MixRoute; + sequence: Sequence; + buses: BusLevel[]; +} + +/** The bus an audio item is heard on, at its volume; a song's tracks go into it. */ +export function audioItemBus(compId: string, itemId: string): string { + return `composition:${compId}:${itemId}`; +} + +/** + * Every audio item of the composition on a track there is, as sequences – one for each place of the + * mix they go to, in the order the first item for it lies in the list. With `routed` off all go to + * the master, as one. + */ +export function compositionPlayings( + project: Project, + comp: CompositionAsset, + routed = true, +): CompositionPlaying[] { + const fps = compositionFps(comp); + const tracks = new Set(comp.tracks.map((t) => t.id)); + const byRoute = new Map(); + for (const item of comp.items) { + if (!item.audio || !tracks.has(item.track)) continue; + const laid = laidOut(project, comp.id, item, fps); + if (!laid) continue; + const route = routed ? playedRoute(project, item.audio.bus, laid.routedAs) : MASTER_MIX; + const key = JSON.stringify(route); + let playing = byRoute.get(key); + if (!playing) byRoute.set(key, (playing = { route, sequence: { notes: [], clips: [] }, buses: [] })); + (playing.sequence.notes as SequenceNote[]).push(...laid.notes); + (playing.sequence.clips as SequenceClip[]).push(...laid.clips); + playing.buses.push(...laid.buses); + } + const playings = [...byRoute.values()]; + for (const { sequence } of playings) { + (sequence.notes as SequenceNote[]).sort((a, b) => a.tick - b.tick || a.note - b.note); + (sequence.clips as SequenceClip[]).sort((a, b) => a.tick - b.tick); + } + return playings; +} + +/** How long one run of the composition is in ticks, and the loop it goes round where it loops. */ +export function compositionSpan( + project: Project, + comp: CompositionAsset, +): { length: number; loop: LoopRange | null } { + const length = compositionTicks(compositionLength(project, comp) / compositionFps(comp)); + return { length, loop: compositionLoops(comp) && length > 0 ? { start: 0, end: length } : null }; +} + +interface Laid { + notes: SequenceNote[]; + clips: SequenceClip[]; + buses: BusLevel[]; + /** The asset its routing is read from: the sound of a clip, else what it plays. */ + routedAs: string; +} + +/** + * An audio item in the composition's ticks: where it begins and ends, and from how far into what it + * plays. An offset below 0 starts that much later in the item. + */ +function laidOut(project: Project, compId: string, item: CompositionItem, fps: number): Laid | null { + const audio = item.audio!; + const late = Math.max(0, -item.offset); + const begin = compositionTicks(item.start / fps + late); + const end = compositionTicks((item.start + item.length) / fps); + if (end <= begin) return null; + const into = Math.max(0, item.offset); + const seconds = secondsOfTicks(end - begin, COMPOSITION_TEMPO); + const bus = audioItemBus(compId, item.id); + const own: BusLevel = { name: bus, gain: dbToGain(audio.volume), pan: 0 }; + if (audio.kind === 'sound') { + const ref = findSound(project, audio.asset); + if (!ref?.asset.file) return null; + const clip = { + tick: begin, + length: end - begin, + sound: audio.asset, + fx: pickedFx('', ref.asset.props), + offset: into, + volume: 1, + bus, + }; + return { notes: [], clips: [clip], buses: [own], routedAs: ref.asset.id }; + } + const withFx = (fxId: string, tempo: number): BusLevel => { + const fx = fxVoiceOf(project, fxId, tempo); + return fx ? { ...own, fx } : own; + }; + if (audio.kind === 'pattern') { + const pattern = findAssetOf(project, audio.asset, 'pattern'); + if (!pattern || !patternInstrument(pattern.props)) return null; + const tempo = patternTempo(pattern.props); + const length = patternLength(pattern); + const heard = played(patternSequence(pattern), { start: 0, end: length }, tempo, into, seconds); + return { + notes: heard.notes.map((n) => ({ ...n, tick: begin + n.tick, bus })), + clips: [], + buses: [withFx(pickedFx('', pattern.props), tempo)], + routedAs: pattern.id, + }; + } + const song = findAssetOf(project, audio.asset, 'song'); + if (!song) return null; + const tempo = songTempo(song.props); + const heard = played(songSequence(project, song), songLoop(song), tempo, into, seconds); + // Its tracks are the item's: two items of one song do not share them. + const prefix = `song:${song.id}`; + const renamed = (name: string) => (name.startsWith(prefix) ? `${bus}${name.slice(prefix.length)}` : name); + const tracks = songBuses(project, song).map(({ name, into: to, sends, ...level }) => ({ + ...level, + name: renamed(name), + ...(to !== undefined && { into: renamed(to) }), + ...(sends && { sends: sends.map((send) => ({ ...send, to: send.to ? renamed(send.to) : bus })) }), + })); + return { + notes: heard.notes.map((n) => ({ ...n, tick: begin + n.tick, ...(n.bus && { bus: renamed(n.bus) }) })), + clips: heard.clips.map((c) => ({ ...c, tick: begin + c.tick, bus: renamed(c.bus) })), + buses: [withFx(pickedFx('', song.props), tempo), ...tracks], + routedAs: song.id, + }; +} + +/** + * What a pattern or a song plays from `into` seconds on for `seconds`, round its loop, in ticks of + * the composition from 0: each note and clip with its own tempo, none ringing past the end. + */ +function played( + sequence: Sequence, + loop: LoopRange, + tempo: number, + into: number, + seconds: number, +): { notes: SequenceNote[]; clips: SequenceClip[] } { + const from = Math.round(ticksOfSeconds(into, tempo)); + const to = from + Math.ceil(ticksOfSeconds(seconds, tempo)); + const span = compositionTicks(seconds); + const at = (ticks: number) => compositionTicks(secondsOfTicks(ticks - from, tempo)); + const notes = notesIn(sequence, 0, loop, from, to).flatMap((n) => { + const tick = at(n.at); + const length = Math.min(compositionTicks(secondsOfTicks(n.length, tempo)), span - tick); + if (tick >= span || length <= 0) return []; + const { at: _, ...note } = n; + return [{ ...note, tick, length: Math.max(1, length), tempo }]; + }); + const clips = clipsIn(sequence, 0, loop, from, to, true).flatMap((c) => { + const tick = at(c.at); + const length = Math.min(compositionTicks(secondsOfTicks(c.length, tempo)), span - tick); + if (tick >= span || length <= 0) return []; + const { at: _, skip, ...clip } = c; + return [{ ...clip, tick, length, offset: c.offset + secondsOfTicks(skip, tempo), tempo }]; + }); + return { notes, clips }; +} diff --git a/src/core/music/playbacks.ts b/src/core/music/playbacks.ts index d1202146..5a38de99 100644 --- a/src/core/music/playbacks.ts +++ b/src/core/music/playbacks.ts @@ -4,6 +4,7 @@ import type { BakePlayback } from './bake'; import type { MixSend } from './mixer'; import { LEAD, type ScheduledClip, type ScheduledNote, Scheduler, type SchedulerHost } from './Scheduler'; import type { LoopRange, Sequence } from './sequence'; +import { ticksOfSeconds } from './time'; /** What a sequence is made of: loaded before the first of its notes and clips is planned. */ export interface SequenceNeeds { @@ -56,6 +57,11 @@ export interface SequencePlay { strikeClip?(clip: ScheduledClip): number; /** Its bake (#340), played in place of its notes and clips where the host can. */ bake?: BakePlay; + /** + * Where it starts, in ticks, as heard `LEAD` from now: a composition's sound (#418). Where it first + * waits for its files, it starts as much further in, to keep to the clock it was asked for by. + */ + from?: number; } /** A bake as it is played: its file, the level it is heard at, and which stretches of it. */ @@ -158,9 +164,11 @@ export class Playbacks { baked: runner instanceof BakedRun, }; this.list.push(playback); + const asked = this.host.now(); void this.host.prepare(play.needs, id).then(() => { if (!this.list.includes(playback) || playback.fading !== null) return; - runner.start(0); + const from = play.from; + runner.start(from === undefined ? 0 : from + ticksOfSeconds(this.host.now() - asked, play.tempo)); // From the first note heard, which is later where the buses are (#346). const at = runner.startedAt ?? this.host.now() + LEAD; this.host.level(id, play.volume ?? 1, at, Math.max(0, play.fadeIn ?? 0)); diff --git a/src/core/music/sequence.ts b/src/core/music/sequence.ts index 7c117c65..3354df32 100644 --- a/src/core/music/sequence.ts +++ b/src/core/music/sequence.ts @@ -19,6 +19,8 @@ export interface SequenceNote { volume?: number; /** The track of a song it is heard on (#334); straight out where left out. */ bus?: string; + /** What a Delay synced to beats counts by, where it is not the sequence's: a pattern in a composition (#418). */ + tempo?: number; } /** @@ -34,6 +36,8 @@ export interface SequenceClip { offset: number; volume: number; bus: string; + /** As a note's. */ + tempo?: number; } export interface Sequence { diff --git a/src/editor/Editor.ts b/src/editor/Editor.ts index f1eb15ed..412788a2 100644 --- a/src/editor/Editor.ts +++ b/src/editor/Editor.ts @@ -1420,6 +1420,7 @@ export class Editor implements PlayHost { this.console.refresh(); if (this.shows('animation')) this.animationEditor.tick(step); if (this.shows('composition')) this.compositionEditor.tick(dt); + else this.compositionEditor.hidden(); if (this.shows('tile')) this.tileEditor.tick(step); // The preview is the editor's own, so it runs on while the game is paused. if (this.shows('curve')) this.curveEditor.tick(dt); diff --git a/src/editor/actions.ts b/src/editor/actions.ts index 4f779802..e55fd9f6 100644 --- a/src/editor/actions.ts +++ b/src/editor/actions.ts @@ -238,11 +238,11 @@ export function removeSelection(state: EditorState): boolean { else removeTrack(state, s.assetId, s.trackId); return true; case 'composition': - // The clips when some are picked, else the track and what lies on it. + // The items or clips when some are picked, else the track and what lies on it. if ( state.execute( - s.clipIds.length > 0 - ? removeFromComposition(state.project, s.assetId, [], s.clipIds) + s.clipIds.length > 0 || (s.itemIds ?? []).length > 0 + ? removeFromComposition(state.project, s.assetId, s.itemIds ?? [], s.clipIds) : removeCompositionTrack(state.project, s.assetId, s.trackId), ) ) diff --git a/src/editor/clipboard.ts b/src/editor/clipboard.ts index 2afc50b5..bfa31ece 100644 --- a/src/editor/clipboard.ts +++ b/src/editor/clipboard.ts @@ -60,12 +60,31 @@ export function copyObjectIds(state: EditorState, holderId: string, ids: readonl return put(state, clip, roots, counted(roots.length, clip.items ? 'item' : 'object')); } -/** Copies animation clips of a composition; false when none of them is there. */ -export function copyCompositionClips(state: EditorState, assetId: string, ids: readonly string[]): boolean { +/** + * Copies animation clips or items of a composition: with an item that is an object among them as + * objects, the audio items (#418) going along; false when none of them is there. + */ +export function copyCompositionClips( + state: EditorState, + assetId: string, + ids: readonly string[], + itemIds: readonly string[] = [], +): boolean { const asset = findAssetOf(state.project, assetId, 'composition'); - const clips = asset?.clips.filter((c) => ids.includes(c.id)).map((c) => ({ ...c })) ?? []; - if (clips.length === 0) return false; - return put(state, { kind: 'compositionClips', clips }, [], counted(clips.length, 'clip')); + if (!asset) return false; + const picked = asset.items.filter((i) => itemIds.includes(i.id)); + const audio = picked.filter((i) => i.audio).map((i) => structuredClone(i)); + const objects = picked.filter((i) => !i.audio).map((i) => i.id); + if (objects.length > 0) { + const clip = copyObjects(state.project, asset, objects); + if (!clip) return false; + const roots = clip.roots.map((r) => r.id); + return put(state, { ...clip, audio }, roots, counted(roots.length + audio.length, 'item')); + } + const clips = asset.clips.filter((c) => ids.includes(c.id)).map((c) => ({ ...c })); + if (clips.length === 0 && audio.length === 0) return false; + const noun = audio.length > 0 ? counted(audio.length, 'item') : counted(clips.length, 'clip'); + return put(state, { kind: 'compositionClips', clips, items: audio }, [], noun); } /** @@ -75,7 +94,7 @@ export function copyCompositionClips(state: EditorState, assetId: string, ids: r export function copySelection(state: EditorState): boolean { const s = state.selection; if (s?.kind === 'object') return copyObjectIds(state, s.holderId, s.objectIds); - if (s?.kind === 'composition') return copyCompositionClips(state, s.assetId, s.clipIds); + if (s?.kind === 'composition') return copyCompositionClips(state, s.assetId, s.clipIds, s.itemIds); if (s?.kind === 'node' || s?.kind === 'annotation') { const graph = findGraphAsset(state.project, s.assetId)?.graph; const nodeIds = s.kind === 'node' ? s.nodeIds : []; diff --git a/src/editor/composition/CompositionEditor.ts b/src/editor/composition/CompositionEditor.ts index 72ff98f3..0d76d19a 100644 --- a/src/editor/composition/CompositionEditor.ts +++ b/src/editor/composition/CompositionEditor.ts @@ -1,8 +1,11 @@ import type { Asset, CompositionAsset } from '../../core/model/Asset'; import { compositionFps, compositionLength } from '../../core/model/composition'; import { VIDEO_WAIT_MS } from '../../core/model/video'; +import { CompositionSound } from '../../runtime/compositionSound'; import { Button, formatNumber, VStack, type Widget } from '../../ui'; +import { soundLibrary } from '../../viewport/SoundLibrary'; import { AssetTabEditor } from '../AssetTabEditor'; +import { claimTransport, releaseTransport } from '../music/transport'; import type { EditorState } from '../state'; import { CompositionCanvas } from './CompositionCanvas'; import { CompositionTimeline } from './CompositionTimeline'; @@ -21,6 +24,12 @@ export class CompositionEditor extends AssetTabEditor this.canvas.pause(); + constructor(state: EditorState) { super(state, { infoWidth: INFO_WIDTH }); this.addTrackButton = new Button('+ Track', { onClick: () => this.canvas.addTrack() }); @@ -31,7 +40,15 @@ export class CompositionEditor extends AssetTabEditor this.syncPlayButton(); + timeline.onPlaying = () => { + this.syncPlayButton(); + if (timeline.isPlaying) claimTransport(this.halt); + else { + releaseTransport(this.halt); + this.sound.stop(); + } + }; + state.on('project', () => (this.edited = true)); this.mount(timeline, [this.addTrackButton, this.playButton]); } @@ -88,10 +105,33 @@ export class CompositionEditor extends AssetTabEditor = { + sound: 0x2f6a78, + pattern: 0x5b4a8f, + song: 0x7a3f6a, +}; const DEFAULT_VIEW: CompositionView = { panX: 0, pxPerTick: 2, panY: 0, trackHeight: TRACK_HEIGHT }; /** Room at the top of an item for its name; marks of its video keep below it. */ const LABEL_STRIP = 15; @@ -182,6 +202,8 @@ export class CompositionTimeline extends Widget { private hover: { id: string; part: Part } | null = null; /** Where the pointer was last, in local coordinates: a paste lands there while it is over the lanes. */ private pointer: Vec2 | null = null; + /** Sound files whose waveform is on its way. */ + private readonly loading = new Set(); private needsFit = true; /** Where the playhead stands, in frames – fractional while playing. */ playhead = 0; @@ -338,10 +360,38 @@ export class CompositionTimeline extends Widget { const picked = this.pickedTrack(asset); const row = spot?.row ?? (picked >= 0 ? picked : undefined); const at = spot?.frame ?? Math.floor(this.playhead + 1e-6); + if (clip.kind === 'compositionClips' && (clip.items ?? []).length > 0) + return this.pasteAudio(asset, clip.items!, clip.clips, at, row); if (clip.kind === 'compositionClips') return this.pasteClips(asset, clip.clips, at, row); const pasted = pasteCompositionItems(this.state.project, asset.id, clip, at, row); if (!pasted || !this.state.execute(pasted.command)) return false; - this.state.select(objectSelection(asset.id, pasted.itemIds)); + this.pickItems(pasted.itemIds); + return true; + } + + /** Copied audio items (#418), and clips copied with them, put in as items are. */ + private pasteAudio( + asset: CompositionAsset, + copied: readonly CompositionItem[], + copiedClips: readonly CompositionClip[], + at: number, + row: number | undefined, + ): boolean { + const ids = new Map([...copied, ...copiedClips].map((c) => [c.id, newId()])); + const { items, clips } = pastedItemsWithClips( + copied, + copiedClips, + asset.clips, + ids, + at, + asset.tracks.map((t) => t.id), + this.lengthOf(asset), + row, + ); + const label = items.length === 1 ? 'Paste item' : 'Paste items'; + const parts = { items: [...asset.items, ...items], clips: [...asset.clips, ...clips] }; + if (!this.state.execute(new SetCompositionCommand(asset.id, parts, label))) return false; + this.pickItems(items.map((i) => i.id)); return true; } @@ -463,18 +513,39 @@ export class CompositionTimeline extends Widget { return (clip) => Math.max(1, clipLength(this.state.project, asset, clip)); } - /** The items of the composition there is an object for: one whose object went is not on the tracks. */ + /** The items of the composition on its tracks: one whose object went is not. */ private liveItems(asset: CompositionAsset): CompositionItem[] { - return asset.items.filter((i) => itemObject(asset, i)); + return asset.items.filter((i) => itemStands(asset, i)); } private pickedItems(asset: CompositionAsset): string[] { const s = this.state.selection; + if (s?.kind === 'composition' && s.assetId === asset.id) return s.itemIds ?? []; return s?.kind === 'object' && s.holderId === asset.id ? s.objectIds.filter((id) => asset.items.some((i) => i.id === id)) : []; } + /** + * Picks items: as objects while they all are – the hierarchy and the inspector show them –, on the + * composition once an audio item is among them (#418); the composition itself for none. + */ + private pickItems(ids: readonly string[]): void { + const asset = this.asset(); + if (!asset) return; + const items = ids.flatMap((id) => asset.items.find((i) => i.id === id) ?? []); + if (items.length === 0) this.state.select({ kind: 'asset', assetId: asset.id }); + else if (!items.some((i) => i.audio)) this.state.select(objectSelection(asset.id, [...ids])); + else + this.state.select({ + kind: 'composition', + assetId: asset.id, + trackId: items[0]!.track, + clipIds: [], + itemIds: items.map((i) => i.id), + }); + } + private pickedClips(asset: CompositionAsset): string[] { const s = this.state.selection; return s?.kind === 'composition' && s.assetId === asset.id ? s.clipIds : []; @@ -551,11 +622,10 @@ export class CompositionTimeline extends Widget { const item = asset.items.find((i) => i.id === hit.id)!; const picked = this.pickedItems(asset); if (toggle) { - this.state.toggleObject(asset.id, item.id); + this.pickItems(picked.includes(item.id) ? picked.filter((id) => id !== item.id) : [...picked, item.id]); return; } - if (!picked.includes(item.id) || hit.part !== 'body') - this.state.select(objectSelection(asset.id, [item.id])); + if (!picked.includes(item.id) || hit.part !== 'body') this.pickItems([item.id]); const media = itemMedia(this.state.project, asset, item); if (hit.part === 'end') this.gesture = { kind: 'resize', anchor: item, item, media }; else if (hit.part === 'start') this.gesture = { kind: 'trim', anchor: item, item, media }; @@ -653,7 +723,9 @@ export class CompositionTimeline extends Widget { gesture.anchor, this.snapped(frame, gesture.media), compositionFps(asset), - gesture.media !== null || itemVideo(this.state.project, asset, gesture.anchor) !== null, + gesture.media !== null || + gesture.anchor.audio !== undefined || + itemVideo(this.state.project, asset, gesture.anchor) !== null, ); break; } @@ -743,7 +815,7 @@ export class CompositionTimeline extends Widget { break; case 'move': { if (!gesture.moved) { - if (gesture.what === 'item') this.state.select(objectSelection(asset.id, [gesture.anchor.id])); + if (gesture.what === 'item') this.pickItems([gesture.anchor.id]); break; } const parts = this.movedParts(asset, gesture, gesture.frames, gesture.tracks); @@ -788,9 +860,7 @@ export class CompositionTimeline extends Widget { const caught = band.active ? itemsInBand(this.view, asset.tracks, this.liveItems(asset), band.start, band.current) : []; - const ids = [...new Set([...(band.add ? this.pickedItems(asset) : []), ...caught])]; - if (ids.length > 0) this.state.select(objectSelection(asset.id, ids)); - else this.state.select({ kind: 'asset', assetId: asset.id }); + this.pickItems([...new Set([...(band.add ? this.pickedItems(asset) : []), ...caught])]); } private commit( @@ -804,17 +874,21 @@ export class CompositionTimeline extends Widget { /** * Follows an asset dragged in from the panel: a placeable one, a picture or a video (#419) lands as - * an item under the pointer, as long as its media or 3 s, over whatever lies there; an animation as - * a clip one pass long, where it fits whole. + * an item under the pointer, as long as its media or 3 s, over whatever lies there; a sound, a clip + * of one, a pattern or a song as an audio item as long as one pass of it (#418); an animation as a + * clip one pass long, where it fits whole. */ private followDrop(asset: CompositionAsset, assetId: string, local: Vec2, g: Vec2): void { - const dragged = findAsset(this.state.project, assetId); + const project = this.state.project; + const dragged = findAsset(project, assetId); + const audio = audioKindOf(project, assetId); const row = trackAtY(this.view, asset.tracks.length, g.y); const inside = local.x >= HEAD_WIDTH && local.x <= this.w && local.y >= RULER_HEIGHT && local.y <= this.h; const picture = dragged ? wornPicture(dragged) : null; const usable = - dragged && (dragged.type === 'animation' || isPlaceableType(dragged.type) || picture !== null); - if (!inside || row === null || !dragged || !usable) { + audio !== null || + (dragged && (dragged.type === 'animation' || isPlaceableType(dragged.type) || picture !== null)); + if (!inside || row === null || !usable) { if (this.drop) { this.drop = null; this.redraw(); @@ -823,7 +897,19 @@ export class CompositionTimeline extends Widget { } const track = asset.tracks[row]!.id; const at = Math.max(0, Math.round(xToTick(this.view, g.x))); - if (dragged.type === 'animation') { + if (audio) { + const file = audio === 'sound' ? playedSoundById(project, assetId) : ''; + if (file) this.ensureLoaded(file); + const item = { id: '', track, start: at, length: 1, offset: 0, audio: audioOf(audio, assetId) }; + this.drop = { + what: 'item', + assetId, + track, + start: at, + length: fittedLength(project, asset, item), + rejection: '', + }; + } else if (dragged!.type === 'animation') { const length = clipLength(this.state.project, asset, { id: '', track, @@ -852,8 +938,8 @@ export class CompositionTimeline extends Widget { assetId, track, start: at, - length: droppedLength(this.state.project, asset, dragged), - rejection: picture === null ? placementRejection(this.state.project, asset, dragged) : '', + length: droppedLength(this.state.project, asset, dragged!), + rejection: picture === null ? placementRejection(this.state.project, asset, dragged!) : '', }; this.redraw(); } @@ -894,6 +980,13 @@ export class CompositionTimeline extends Widget { return; } const item = { track: drop.track, start: drop.start, length: drop.length, offset: 0, fitMedia: true }; + const audio = audioKindOf(this.state.project, drop.assetId); + if (audio) { + const added = { ...item, id: newId(), audio: audioOf(audio, drop.assetId) }; + const command = new SetCompositionCommand(asset.id, { items: [...asset.items, added] }, 'Add item'); + if (this.state.execute(command)) this.pickItems([added.id]); + return; + } const dragged = findAsset(this.state.project, drop.assetId); const picture = dragged ? wornPicture(dragged) : null; const { command, objectId } = @@ -927,15 +1020,30 @@ export class CompositionTimeline extends Widget { const row = local.y >= RULER_HEIGHT ? trackAtY(this.view, asset.tracks.length, g.y) : null; const hit = local.x >= HEAD_WIDTH && local.y >= RULER_HEIGHT ? this.hitAt(asset, g) : null; if (hit?.kind === 'item') { - if (!this.pickedItems(asset).includes(hit.id)) this.state.select(objectSelection(asset.id, [hit.id])); + if (!this.pickedItems(asset).includes(hit.id)) this.pickItems([hit.id]); const ids = this.pickedItems(asset); + const audio = asset.items.find((i) => i.id === hit.id)?.audio; + const played = + audio && + (audio.kind === 'sound' + ? findSound(this.state.project, audio.asset)?.asset + : findAsset(this.state.project, audio.asset)); new Menu([ + ...(audio + ? [ + { + label: `Open ${audio.kind}`, + disabled: !played, + onSelect: () => played && this.state.openAsset(played.id), + }, + ] + : []), { label: 'Fit to media', disabled: ids.length !== 1, onSelect: () => fitToMedia({ state: this.state }, asset.id, hit.id), }, - makeProjectAssetEntry({ state: this.state }, asset.id, hit.id), + ...(audio ? [] : [makeProjectAssetEntry({ state: this.state }, asset.id, hit.id)]), { separator: true }, { label: ids.length > 1 ? `Remove ${ids.length} items` : 'Remove item', @@ -1087,7 +1195,7 @@ export class CompositionTimeline extends Widget { const hint = !asset ? '' : asset.items.length === 0 && asset.clips.length === 0 - ? 'Drag a sprite, text, camera or container onto a track, an animation to play on it.' + ? 'Drag a sprite, text, camera or container onto a track, an animation to play on it, a sound, pattern or song to hear.' : ''; this.hint.text = hint; this.hint.visible = hint !== ''; @@ -1098,7 +1206,12 @@ export class CompositionTimeline extends Widget { const c = Theme.colors; const s = this.state.selection; const lit = - s?.kind === 'composition' && s.assetId === asset.id && s.clipIds.length === 0 ? s.trackId : ''; + s?.kind === 'composition' && + s.assetId === asset.id && + s.clipIds.length === 0 && + (s.itemIds ?? []).length === 0 + ? s.trackId + : ''; const end = HEAD_WIDTH + tickToX(this.view, compositionLength(this.state.project, asset)); const zero = HEAD_WIDTH + tickToX(this.view, 0); asset.tracks.forEach((track, row) => { @@ -1137,7 +1250,13 @@ export class CompositionTimeline extends Widget { if (box.x > this.w || box.x + box.w < HEAD_WIDTH) continue; const lit = picked.has(item.id); const media = itemMedia(this.state.project, asset, item); - const color = itemVideo(this.state.project, asset, item) ? VIDEO_COLOR : ITEM_COLOR; + const color = item.audio + ? this.audioName(item) === null + ? REJECTED_COLOR + : AUDIO_COLORS[item.audio.kind] + : itemVideo(this.state.project, asset, item) + ? VIDEO_COLOR + : ITEM_COLOR; this.lanes .roundRect(box.x, box.y, box.w, box.h, 3) .fill({ color, alpha: lit ? 1 : 0.85 }) @@ -1147,10 +1266,15 @@ export class CompositionTimeline extends Widget { alpha: lit ? 1 : 0.45, alignment: 1, }); + if (item.audio?.kind === 'sound') this.drawWave(item, box); if (media) this.drawMediaMarks(item, media, box); const object = itemObject(asset, item); const placed = object && findAsset(this.state.project, object.assetId); - const name = object ? objectName(object, placed) : ''; + const name = item.audio + ? (this.audioName(item) ?? '(missing)') + : object + ? objectName(object, placed) + : ''; const seen = HEAD_WIDTH + tickToX(this.view, uncoveredEnd(shown, item)) - box.x; if (seen > 18) labels.push({ text: name, x: Math.max(box.x, HEAD_WIDTH) + 4, y: box.y + 2, w: seen - 8 }); @@ -1213,6 +1337,62 @@ export class CompositionTimeline extends Widget { } } + /** What an audio item plays, by name – "Sound / Clip" for a clip –; null once it is gone. */ + private audioName(item: CompositionItem): string | null { + const audio = item.audio!; + const project = this.state.project; + const there = + audio.kind === 'sound' + ? findSound(project, audio.asset) !== null + : findAssetOf(project, audio.asset, audio.kind) !== undefined; + return there ? assetRefLabel(project, audio.asset) : null; + } + + /** + * The waveform of a sound item, round and round as it plays, from where it starts in it, below its + * name – the Song tab's. + */ + private drawWave(item: CompositionItem, box: Box): void { + const asset = this.asset(); + const played = asset && playedSoundById(this.state.project, item.audio!.asset); + if (!asset || !played) return; + const peaks = soundLibrary.peaks(played); + const seconds = soundLibrary.duration(played); + if (!peaks || !seconds) { + this.ensureLoaded(played); + return; + } + const fps = compositionFps(asset); + const width = tickToX(this.view, seconds * fps) - tickToX(this.view, 0); + if (!(width > 0)) return; + const mid = box.y + LABEL_STRIP + (box.h - LABEL_STRIP) / 2; + const half = (box.h - LABEL_STRIP - 4) / 2; + const from = Math.max(box.x, HEAD_WIDTH); + const to = Math.min(box.x + box.w, this.w); + const begins = HEAD_WIDTH + tickToX(this.view, item.start - item.offset * fps); + for (let round = Math.floor((from - begins) / width); ; round++) { + const start = Math.round(begins + round * width); + if (start >= to) break; + for (let dx = Math.max(0, Math.ceil((from - start) / 2) * 2); dx < width; dx += 2) { + const x = start + dx; + if (x >= to) break; + // Before a sound that begins inside the item, it is silent. + if (x < begins) continue; + const reach = Math.max(0.5, peakOver(peaks, dx / width, (dx + 2) / width) * half); + this.lanes.rect(x, mid - reach, 1.5, reach * 2).fill({ color: 0xffffff, alpha: 0.6 }); + } + } + } + + private ensureLoaded(file: string): void { + if (this.loading.has(file) || soundLibrary.peaks(file) !== null) return; + this.loading.add(file); + void soundLibrary.loadPicture(file).then(() => { + this.loading.delete(file); + this.redraw(); + }); + } + /** The end of an item the pointer would drag, lit – over an item that covers it, too. */ private drawGrip(box: Box, part: 'start' | 'end'): void { const reach = Math.min(EDGE_REACH, box.w / 3); @@ -1307,6 +1487,11 @@ export class CompositionTimeline extends Widget { } } +/** A new audio item's, as its asset says: unity, on the bus its routing sends it to. */ +function audioOf(kind: CompositionAudioKind, asset: string): NonNullable { + return { kind, asset, volume: 0, bus: '' }; +} + /** Puts the texts where they are wanted, reusing the ones there are and hiding the rest. */ function syncTexts(pool: Text[], layer: Container, wanted: readonly Placed[], style: TextStyle): void { wanted.forEach((place, i) => { diff --git a/src/editor/debug/documentHooks.ts b/src/editor/debug/documentHooks.ts index d76e6b04..c51ccbb2 100644 --- a/src/editor/debug/documentHooks.ts +++ b/src/editor/debug/documentHooks.ts @@ -96,6 +96,11 @@ export function documentHooks({ state, editor }: HookContext) { objectScreen: (objectId: string) => editor.compositionEditor.preview.objectScreen(objectId), /** Where the picture draws a sprite right now; null while it is left out, a video without its picture. */ drawn: (objectId: string) => editor.compositionEditor.preview.drawnRect(objectId), + /** Whether its audio items are heard (#418), and the numbers of their playings. */ + sound: () => ({ + playing: editor.compositionEditor.sound.playing, + playings: [...editor.compositionEditor.sound.playings], + }), }, tile: { current: () => editor.tileEditor.currentAssetId, diff --git a/src/editor/inspector/Inspector.ts b/src/editor/inspector/Inspector.ts index b68dc2ee..a6512a0c 100644 --- a/src/editor/inspector/Inspector.ts +++ b/src/editor/inspector/Inspector.ts @@ -146,7 +146,8 @@ export class Inspector extends Panel { else if (s.kind === 'track') buildTrackSection(this.host, s.assetId, s.trackId, s.keys); else if (s.kind === 'edge') buildEdgeSection(this.host, s.assetId, s.edgeId, s.edgeIds); else if (s.kind === 'song') buildSongSection(this.host, s.assetId, s.trackId, s.itemIds); - else if (s.kind === 'composition') buildCompositionSection(this.host, s.assetId, s.trackId, s.clipIds); + else if (s.kind === 'composition') + buildCompositionSection(this.host, s.assetId, s.trackId, s.clipIds, s.itemIds); else if (s.objectIds.length > 1) buildObjectsSection(this.host, s.holderId, s.objectIds); else buildObjectSection(this.host, s.holderId, s.objectId); this.scroll.setContent(this.host.body); @@ -167,7 +168,7 @@ export class Inspector extends Panel { if (s.kind === 'edge') return edgeSignature(this.state.project, s.assetId, s.edgeId, s.edgeIds); if (s.kind === 'song') return songSignature(this.state.project, s.assetId, s.trackId, s.itemIds); if (s.kind === 'composition') - return compositionSignature(this.state.project, s.assetId, s.trackId, s.clipIds); + return compositionSignature(this.state.project, s.assetId, s.trackId, s.clipIds, s.itemIds); return objectSignature(this.state, s.holderId, s.objectIds, s.objectId); } @@ -208,7 +209,7 @@ export class Inspector extends Panel { : s.kind === 'song' ? refreshSongSection(this.host, s.assetId, s.trackId, s.itemIds) : s.kind === 'composition' - ? refreshCompositionSection(this.host, s.assetId, s.trackId, s.clipIds) + ? refreshCompositionSection(this.host, s.assetId, s.trackId, s.clipIds, s.itemIds) : s.objectIds.length > 1 || refreshObjectSection(this.host, s.holderId, s.objectId); if (!ok) this.rebuild(); diff --git a/src/editor/inspector/compositionSection.ts b/src/editor/inspector/compositionSection.ts index b34896b4..1d2c43bd 100644 --- a/src/editor/inspector/compositionSection.ts +++ b/src/editor/inspector/compositionSection.ts @@ -1,6 +1,12 @@ import { MakeProjectAssetCommand } from '../../core/commands/assetCommands'; import { SetCompositionCommand } from '../../core/commands/compositionCommands'; -import type { CompositionAsset, CompositionItem } from '../../core/model/Asset'; +import type { + Asset, + CompositionAsset, + CompositionAudio, + CompositionAudioKind, + CompositionItem, +} from '../../core/model/Asset'; import { clipLength, compositionFps, @@ -10,6 +16,8 @@ import { } from '../../core/model/composition'; import { heldMove, movedWithClips, resizedItem } from '../../core/model/compositionEdit'; import { findAsset, findAssetOf, type Project } from '../../core/model/Project'; +import { findSound } from '../../core/model/sound'; +import { ITEM_MIN_DB, SOUND_ITEM_MAX_DB } from '../../core/music/song'; import type { PropertyDef } from '../../core/properties/PropertySchema'; import { Button, formatNumber, type MenuEntry } from '../../ui'; import { createFieldRow } from './fieldFactory'; @@ -21,6 +29,36 @@ const LENGTH_ROW = 'composition:length'; const OFFSET_ROW = 'composition:offset'; const CLIP_START_ROW = 'composition:clipStart'; const ANIMATION_ROW = 'composition:animation'; +const PLAYS_ROW = 'composition:plays'; +const VOLUME_ROW = 'composition:volume'; +const BUS_ROW = 'composition:bus'; + +const AUDIO_TITLES: Record = { + sound: 'SOUND ITEM', + pattern: 'PATTERN ITEM', + song: 'SONG ITEM', +}; + +const VOLUME_DEF: PropertyDef = { + key: VOLUME_ROW, + label: 'Volume', + kind: 'number', + default: 0, + min: ITEM_MIN_DB, + max: SOUND_ITEM_MAX_DB, + step: 0.5, + decimals: 1, +}; + +const BUS_DEF: PropertyDef = { + key: BUS_ROW, + label: 'Bus', + kind: 'asset', + assetTypes: ['bus'], + none: 'From asset', + master: true, + default: '', +}; const seconds = (key: string, label: string, negative = false): PropertyDef => ({ key, @@ -32,15 +70,21 @@ const seconds = (key: string, label: string, negative = false): PropertyDef => ( decimals: 3, }); -/** Structure-defining facts of a track or clip selection: which it is, and whether it is still there. */ +/** Structure-defining facts of a track, clip or item selection: which it is, and whether it is still there. */ export function compositionSignature( project: Project, assetId: string, trackId: string, clipIds: string[], + itemIds: string[] = [], ): string { const comp = findAssetOf(project, assetId, 'composition'); if (!comp?.tracks.some((t) => t.id === trackId)) return 'missing'; + if (itemIds.length === 1) { + const audio = comp.items.find((i) => i.id === itemIds[0])?.audio; + return `item:${itemIds[0]}:${audio ? playedAsset(project, audio)?.id : ''}${timingSignature(project, assetId, itemIds[0]!)}`; + } + if (itemIds.length > 0) return `items:${itemIds.join(',')}`; return clipIds.length > 0 ? `clips:${clipIds.join(',')}` : 'track'; } @@ -50,10 +94,17 @@ export function buildCompositionSection( assetId: string, trackId: string, clipIds: string[], + itemIds: string[] = [], ): void { const comp = findAssetOf(host.state.project, assetId, 'composition'); const track = comp?.tracks.find((t) => t.id === trackId); if (!comp || !track) return; + if (itemIds.length > 0) { + const item = itemIds.length === 1 ? comp.items.find((i) => i.id === itemIds[0]) : undefined; + if (item?.audio) buildAudioItem(host, assetId, item, item.audio); + else addTitle(host, `${itemIds.length} ITEMS`); + return; + } if (clipIds.length === 0) { addTitle(host, 'TRACK'); addNameField(host, track.name, { @@ -99,10 +150,20 @@ export function refreshCompositionSection( assetId: string, trackId: string, clipIds: string[], + itemIds: string[] = [], ): boolean { const comp = findAssetOf(host.state.project, assetId, 'composition'); const track = comp?.tracks.find((t) => t.id === trackId); if (!comp || !track) return false; + if (itemIds.length > 0) { + const audio = itemIds.length === 1 ? comp.items.find((i) => i.id === itemIds[0])?.audio : undefined; + if (audio) { + refreshTimingRows(host, assetId, itemIds[0]!); + host.fields.get(VOLUME_ROW)?.setValue(audio.volume); + host.fields.get(BUS_ROW)?.setValue(audio.bus); + } + return true; + } if (clipIds.length === 0) host.nameField?.setValue(track.name); const clip = clipIds.length === 1 ? comp.clips.find((c) => c.id === clipIds[0]) : undefined; if (clip) host.fields.get(CLIP_START_ROW)?.setValue(clip.start / compositionFps(comp)); @@ -129,7 +190,7 @@ export function addTimingRows(host: InspectorHost, holderId: string, objectId: s if (!found) return; const { comp, item } = found; const fps = compositionFps(comp); - const video = itemVideo(host.state.project, comp, item) !== null; + const video = item.audio !== undefined || itemVideo(host.state.project, comp, item) !== null; addTitle(host, 'TIMING'); const write = (label: string, edit: (current: CompositionItem) => CompositionItem) => { const current = compositionItemOf(host.state.project, holderId, objectId); @@ -190,9 +251,11 @@ export function addTimingRows(host: InspectorHost, holderId: string, objectId: s addNote( host, item.fitMedia - ? video - ? 'Its length follows its video.' - : 'Its length follows a video it is given, 3 s while it has none.' + ? item.audio + ? 'Its length follows what it plays.' + : video + ? 'Its length follows its video.' + : 'Its length follows a video it is given, 3 s while it has none.' : 'Its length is its own.', 'small', ); @@ -214,7 +277,7 @@ export function refreshTimingRows(host: InspectorHost, holderId: string, objectI export function timingSignature(project: Project, holderId: string, objectId: string): string { const found = compositionItemOf(project, holderId, objectId); if (!found) return ''; - const video = itemVideo(project, found.comp, found.item) !== null; + const video = found.item.audio !== undefined || itemVideo(project, found.comp, found.item) !== null; const fitted = found.item.length === fittedLength(project, found.comp, found.item); return `|timing:${video}:${found.item.fitMedia === true}:${fitted}`; } @@ -248,6 +311,52 @@ export function makeProjectAssetEntry( }; } +/** + * An audio item (#418): what it plays, when, how loud and on which bus. It plays round and round for + * as long as it lasts; its offset is how far into what it plays it begins. + */ +function buildAudioItem( + host: InspectorHost, + assetId: string, + item: CompositionItem, + audio: CompositionAudio, +): void { + addTitle(host, AUDIO_TITLES[audio.kind]); + const played = playedAsset(host.state.project, audio); + if (played) assetLink(host, PLAYS_ROW, 'Plays', played); + else addNote(host, `Its ${audio.kind} is gone.`, 'warning'); + addTimingRows(host, assetId, item.id); + const edit = (label: string, change: Partial) => { + const comp = findAssetOf(host.state.project, assetId, 'composition'); + if (!comp) return; + const items = comp.items.map((i) => + i.id === item.id && i.audio ? { ...i, audio: { ...i.audio, ...change } } : i, + ); + if (SetCompositionCommand.changes(host.state.project, assetId, { items })) + host.state.execute(new SetCompositionCommand(assetId, { items }, label)); + }; + const volume = createFieldRow(host, VOLUME_DEF, audio.volume, { + onInput: () => undefined, + onChange: (value) => edit('Set item volume', { volume: Number(value) || 0 }), + }); + host.fields.set(VOLUME_ROW, volume); + host.body.add(volume.row); + const bus = createFieldRow(host, BUS_DEF, audio.bus, { + onInput: () => undefined, + onChange: (value) => edit('Set item bus', { bus: typeof value === 'string' ? value : '' }), + }); + host.fields.set(BUS_ROW, bus); + host.body.add(bus.row); + addNote(host, 'Volume in dB. It plays round and round for as long as it lasts.', 'small'); +} + +/** The asset an audio item plays, for a clip of a sound the sound; undefined once it is gone. */ +function playedAsset(project: Project, audio: CompositionAudio): Asset | undefined { + return audio.kind === 'sound' + ? findSound(project, audio.asset)?.asset + : findAssetOf(project, audio.asset, audio.kind); +} + function withOffset(item: CompositionItem, offset: number): CompositionItem { const { fitMedia: _, ...rest } = item; return { ...rest, offset: Number.isFinite(offset) ? offset : 0 }; diff --git a/src/editor/panels/AssetPanel.ts b/src/editor/panels/AssetPanel.ts index c55d6ce8..9115dd79 100644 --- a/src/editor/panels/AssetPanel.ts +++ b/src/editor/panels/AssetPanel.ts @@ -78,6 +78,11 @@ export class AssetPanel extends Panel { private readonly foldedCategories = new Set(); /** Ids of the rows as shown, headings included. */ private rows: string[] = []; + /** + * The selected asset the list last scrolled to. Only a new selection scrolls: a rebuild – folding + * a group, any edit – leaves the list where the user put it. + */ + private scrolledTo: string | null = null; private readonly stopListening: () => void; constructor( @@ -112,6 +117,8 @@ export class AssetPanel extends Panel { }; // Room to scroll the last group up off the panel's bottom edge, a few rows' worth. this.scroll.tailSpace = 6 * this.list.rowHeight; + // Folding a group leaves the rows above it, and the heading under the pointer, where they are. + this.scroll.holdOffset = true; scrollTween(this.scroll); this.scroll.setContent(this.list); this.content.add(this.scroll, 1); @@ -121,6 +128,7 @@ export class AssetPanel extends Panel { state.on('load', () => { this.unfolded.clear(); this.foldedCategories.clear(); + this.scrolledTo = null; this.refresh(); }); state.on('selection', () => this.syncSelection()); @@ -274,7 +282,8 @@ export class AssetPanel extends Panel { return; } this.list.select(id, true); - if (id) this.scrollTo(id); + if (id && id !== this.scrolledTo) this.scrollTo(id); + this.scrolledTo = id; } private scrollTo(id: string): void { @@ -445,12 +454,16 @@ export class AssetPanel extends Panel { /** * Hands the asset to the view under the pointer: a scene places it, a song lays out a pattern or a - * sound (#334), a composition an animation as a clip (#417) and a picture or video as an item - * (#419). A scene has nowhere to go. + * sound (#334), a composition an animation as a clip (#417), a picture or video as an item (#419) + * and a sound, pattern or song as an audio item (#418). A scene has nowhere to go. */ private startPlacementDrag(assetId: string): void { const asset = findAsset(this.state.project, assetId); - const laid = asset?.type === 'pattern' || asset?.type === 'sound' || asset?.type === 'animation'; + const laid = + asset?.type === 'pattern' || + asset?.type === 'song' || + asset?.type === 'sound' || + asset?.type === 'animation'; if (!asset || !(isPlaceableType(asset.type) || laid || wornPicture(asset) !== null)) return; startDrag({ kind: 'asset', id: assetId }, asset.name); } diff --git a/src/editor/state.ts b/src/editor/state.ts index e89f352f..8c3214cb 100644 --- a/src/editor/state.ts +++ b/src/editor/state.ts @@ -83,9 +83,10 @@ export type Selection = | { kind: 'song'; assetId: string; trackId: string; itemIds: string[] } /** * A track of a composition (#417) and the animation clips picked on it or on others: `clipIds` is - * empty when the track itself was clicked. Its items are objects, picked as an object selection. + * empty when the track itself was clicked. Items that are objects only are picked as an object + * selection; `itemIds` holds picked items once an audio item is among them (#418). */ - | { kind: 'composition'; assetId: string; trackId: string; clipIds: string[] } + | { kind: 'composition'; assetId: string; trackId: string; clipIds: string[]; itemIds?: string[] } | null; /** @@ -857,11 +858,14 @@ export class EditorState { const asset = findAssetOf(this.project, s.assetId, 'composition'); if (!asset) this.select(null); else { - // The same as a song's: a clip that died leaves, a track that died hands over to the composition. + // The same as a song's: a clip or item that died leaves, a track that died hands over to the composition. const clips = s.clipIds.filter((id) => asset.clips.some((clip) => clip.id === id)); + const picked = s.itemIds ?? []; + const items = picked.filter((id) => asset.items.some((item) => item.id === id)); const trackAlive = asset.tracks.some((track) => track.id === s.trackId); if (!trackAlive) this.select({ kind: 'asset', assetId: s.assetId }); - else if (clips.length !== s.clipIds.length) this.select({ ...s, clipIds: clips }); + else if (clips.length !== s.clipIds.length || items.length !== picked.length) + this.select({ ...s, clipIds: clips, itemIds: items }); } } } @@ -1089,7 +1093,12 @@ export function sameSelection(a: Selection, b: Selection): boolean { if (a.kind === 'song' && b.kind === 'song') return a.assetId === b.assetId && a.trackId === b.trackId && sameIds(a.itemIds, b.itemIds); if (a.kind === 'composition' && b.kind === 'composition') - return a.assetId === b.assetId && a.trackId === b.trackId && sameIds(a.clipIds, b.clipIds); + return ( + a.assetId === b.assetId && + a.trackId === b.trackId && + sameIds(a.clipIds, b.clipIds) && + sameIds(a.itemIds ?? [], b.itemIds ?? []) + ); return false; } diff --git a/src/runtime/compositionSound.ts b/src/runtime/compositionSound.ts new file mode 100644 index 00000000..5690148b --- /dev/null +++ b/src/runtime/compositionSound.ts @@ -0,0 +1,92 @@ +import type { CompositionAsset } from '../core/model/Asset'; +import type { Project } from '../core/model/Project'; +import { + COMPOSITION_TEMPO, + compositionPlayings, + compositionSpan, + compositionTicks, +} from '../core/music/compositionAudio'; +import { sequenceNeeds } from '../core/music/needs'; +import { LEAD } from '../core/music/Scheduler'; +import type { AudioSink } from './systems/Audio'; +import { playClip, strikeNote } from './systems/behaviorContext'; + +/** How long the sound of a composition takes to fall silent when it stops. */ +const HUSH = 0.02; + +/** + * The sound of one composition as it plays (#418): its audio items as playings of the sink, started + * from a time of the composition and stopped as a whole. Whoever plays the composition starts it + * while its clock runs and stops it while the clock stands, so the two keep together. + */ +export class CompositionSound { + private ids: number[] = []; + private started = false; + /** What it plays, to tell an edit that changes it from one that does not. */ + private heard = ''; + + constructor( + private readonly audio: AudioSink, + private readonly owner = '', + ) {} + + get playing(): boolean { + return this.started; + } + + /** The numbers of its playings (test hook). */ + get playings(): readonly number[] { + return this.ids; + } + + /** + * Plays the composition from `seconds` into it, as heard `LEAD` from now – unless it plays already. + * `routed` off sends it all to the master, as the tabs preview a piece. + */ + start(project: Project, comp: CompositionAsset, seconds: number, routed = true): void { + if (this.started) return; + this.started = true; + const playings = compositionPlayings(project, comp, routed); + this.heard = keyOf(playings); + const { length, loop } = compositionSpan(project, comp); + const sources = { project, audio: this.audio }; + for (const { route, sequence, buses } of playings) { + const id = this.audio.playSequence( + `composition:${comp.id}`, + { + sequence, + tempo: COMPOSITION_TEMPO, + length, + loop, + needs: sequenceNeeds(project, sequence), + buses, + ...(typeof route === 'string' ? { bus: route } : { sends: route }), + from: compositionTicks(seconds + LEAD), + strike: (note) => strikeNote(sources, note, this.owner), + strikeClip: (clip) => playClip(sources, clip), + }, + this.owner, + ); + if (id !== 0) this.ids.push(id); + } + } + + /** Whether the composition plays something else now than when it was started. */ + changed(project: Project, comp: CompositionAsset, routed = true): boolean { + return this.started && keyOf(compositionPlayings(project, comp, routed)) !== this.heard; + } + + stop(): void { + for (const id of this.ids) this.audio.stopPlayback(id, HUSH); + this.ids = []; + this.started = false; + this.heard = ''; + } +} + +/** The playings as text, a sound FX by its plan's presence: a plan is no plain data. */ +function keyOf(playings: ReturnType): string { + return JSON.stringify(playings, (key, value: unknown) => + key === 'fx' && typeof value === 'object' ? 1 : value, + ); +} diff --git a/src/runtime/systems/BehaviorSystem.ts b/src/runtime/systems/BehaviorSystem.ts index 6bcd9fcb..2b5a0ad3 100644 --- a/src/runtime/systems/BehaviorSystem.ts +++ b/src/runtime/systems/BehaviorSystem.ts @@ -148,7 +148,7 @@ export function addGameSystems( }: GameSystemOptions = {}, ): BehaviorSystem { const collisions = physics ? new PhysicsSystem(physics) : new CollisionSystem(); - const compositions = new CompositionSystem(playsCompositions, pictures); + const compositions = new CompositionSystem(playsCompositions, pictures, audio); const animations = new AnimationSystem(); const sources = { collisions, animations, compositions, audio, clock, saves, recorder, breakpoint }; const behaviors = new BehaviorSystem(project, input, sources, shouldRun); diff --git a/src/runtime/systems/CompositionSystem.ts b/src/runtime/systems/CompositionSystem.ts index 41f53d7f..e66b9551 100644 --- a/src/runtime/systems/CompositionSystem.ts +++ b/src/runtime/systems/CompositionSystem.ts @@ -15,9 +15,11 @@ import { findAssetOf } from '../../core/model/Project'; import type { PropertyValue } from '../../core/model/Properties'; import { findVideo, VIDEO_WAIT_MS } from '../../core/model/video'; import { VIDEO_CLOCK_KEY, VIDEO_TIME_KEY, videoDrivenOf } from '../../core/properties/videos'; +import { CompositionSound } from '../compositionSound'; import type { Entity } from '../Entity'; import type { World } from '../World'; import { bind } from './AnimationSystem'; +import { type AudioSink, SILENT_AUDIO } from './Audio'; import type { System } from './System'; export interface CompositionFinish { @@ -40,6 +42,8 @@ interface Run { live: Map; /** Seconds the clock has stood so far for a video without its picture, see `waits`. */ waited: number; + /** Its audio items (#418), heard while its clock runs. */ + sound: CompositionSound; } /** What the view that shows a run knows of its driven videos (#404). */ @@ -57,26 +61,39 @@ export interface VideoPictures { * * While the view still waits for the picture of a video an item brings in, the composition's * clock stands, so the video starts at its first frame and not where the clock has got to meanwhile. + * Its audio items are heard only while the clock runs: they stop while it stands – for a video, a + * placement switched off, a paused game – and start again where it stands. */ export class CompositionSystem implements System { + /** A paused frame stops the sound of every composition; nothing else moves. */ + readonly runsWhilePaused = true; finished: CompositionFinish[] = []; private readonly runs = new Map(); constructor( private readonly plays = true, private readonly pictures: VideoPictures | null = null, + private readonly audio: AudioSink = SILENT_AUDIO, ) {} update(world: World, dt: number): void { this.finished = []; - for (const placement of this.runs.keys()) - if (!world.entities.includes(placement)) this.runs.delete(placement); + for (const [placement, run] of this.runs) + if (!world.entities.includes(placement)) { + run.sound.stop(); + this.runs.delete(placement); + } + if (world.frozen) { + for (const run of this.runs.values()) run.sound.stop(); + return; + } const started = world.entities.filter((e) => e.type === 'composition' && e.composition); for (const placement of started) this.advance(world, placement, dt); } stop(): void { this.finished = []; + for (const run of this.runs.values()) run.sound.stop(); this.runs.clear(); } @@ -96,8 +113,13 @@ export class CompositionSystem implements System { return; } run = this.runs.get(placement); - if (!run) this.runs.set(placement, (run = { id: player.run, live: new Map(), waited: 0 })); - if (player.shown && isHandleEnabled(placement) && !this.waits(run, dt)) player.time += dt; + if (!run) + this.runs.set( + placement, + (run = { id: player.run, live: new Map(), waited: 0, sound: new CompositionSound(this.audio) }), + ); + const runs = player.shown && isHandleEnabled(placement) && !this.waits(run, dt); + if (runs) player.time += dt; player.shown = true; const length = compositionLength(world.project, comp); let frame = compositionFrameAt(comp, player.time); @@ -111,6 +133,8 @@ export class CompositionSystem implements System { return; } this.show(world, comp, placement, run, frame); + if (runs) run.sound.start(world.project, comp, player.time); + else run.sound.stop(); } /** Whether a video of a live item still has no picture; at most `VIDEO_WAIT_MS` at a stretch. */ @@ -131,6 +155,7 @@ export class CompositionSystem implements System { private end(world: World, placement: Entity): void { const run = this.runs.get(placement); if (!run) return; + run.sound.stop(); for (const { entity } of run.live.values()) if (world.entities.includes(entity)) world.remove(entity); this.runs.delete(placement); } diff --git a/src/ui/widgets/ScrollContainer.ts b/src/ui/widgets/ScrollContainer.ts index 4775bdb0..5b89d1a5 100644 --- a/src/ui/widgets/ScrollContainer.ts +++ b/src/ui/widgets/ScrollContainer.ts @@ -24,6 +24,14 @@ export class ScrollContainer extends Widget { * is made this much taller, so a list lays its empty area (clicks, context menu) over it as well. */ tailSpace = 0; + /** + * Content that shrinks keeps its offset: the rows above stay where they were, and the room the + * gone ones leave stays below until the view is scrolled up off it. Without it the offset is + * clamped to the shorter content, and every row slides down by what went away. Opt-in. + */ + holdOffset = false; + /** The height the content asks for, tail included – without the room held for gone rows. */ + private needed = 0; /** * Carries out a wheel step, by default at once. A panel can hand it to a transition instead, * which makes the step and then shows the way to where it led. @@ -84,7 +92,9 @@ export class ScrollContainer extends Widget { const clamped = clamp(y, 0, this.maxScroll); if (clamped === this.scrollY) return; this.scrollY = clamped; - this.applyScroll(); + // Room held for gone rows is let go as the view leaves it. + if (this.holdOffset && this.contentHeight > Math.max(this.needed, this.h)) this.relayoutContent(); + else this.applyScroll(); } /** Scrolls just far enough for the content band [top, top + height) to be inside the view. */ @@ -96,11 +106,13 @@ export class ScrollContainer extends Widget { /** Re-measure the content after it changed (rows added/removed). */ relayoutContent(): void { if (!this.content) return; + const held = this.holdOffset ? this.scrollY + this.h : 0; // Wrapped text grows taller when the scrollbar takes its width, so measure again at that width. let needed = this.content.heightFor(this.w) + this.tailSpace; - const width = Math.max(0, this.w - (needed > this.h ? Theme.scrollbarWidth : 0)); + const width = Math.max(0, this.w - (Math.max(needed, held) > this.h ? Theme.scrollbarWidth : 0)); if (width !== this.w) needed = this.content.heightFor(width) + this.tailSpace; - const height = Math.max(needed, this.h); + this.needed = needed; + const height = Math.max(needed, held, this.h); // Content shorter than the viewport is pinned to the viewport height either way, so `resize` // would see no change and skip the layout -- leaving rows added since stacked at the origin. const unchanged = this.content.w === width && this.content.h === height; diff --git a/src/viewport/SoundLibrary.ts b/src/viewport/SoundLibrary.ts index 33307e17..a12309e2 100644 --- a/src/viewport/SoundLibrary.ts +++ b/src/viewport/SoundLibrary.ts @@ -63,6 +63,7 @@ import { import { type Levels, type Spectrum, SpectrumTaps } from './spectrum'; import { decodeSpan, StreamFeed, StreamFile } from './streaming'; import { wake } from './wake'; +import { setSoundMeasure } from '../core/model/composition'; /** A voice is no louder than a sound item of a song turned all the way up. */ const MAX_VOLUME = dbToGain(SOUND_ITEM_MAX_DB); @@ -2964,3 +2965,6 @@ function readMeter(meter: BusMeter): [number, number] { } export const soundLibrary = new SoundLibrary(); + +/** A composition fits an audio item to the length of its sound once the bytes are here (#418). */ +setSoundMeasure((file) => soundLibrary.duration(file)); diff --git a/tests/core/composition-audio.test.ts b/tests/core/composition-audio.test.ts new file mode 100644 index 00000000..7cd0ca0b --- /dev/null +++ b/tests/core/composition-audio.test.ts @@ -0,0 +1,360 @@ +/** + * Sounds, patterns and songs as items of a composition (#418): kept without an object, laid out as + * sequences in milliseconds on buses of their own, and heard in a game while the composition's + * clock runs. + */ +import { describe, expect, it } from 'vitest'; +import { + CreateAssetCommand, + CreateSoundCommand, + SetAssetPropertyCommand, +} from '../../src/core/commands/assetCommands'; +import { SetCompositionCommand } from '../../src/core/commands/compositionCommands'; +import { SetLoopCommand, SetPatternNotesCommand } from '../../src/core/commands/patternCommands'; +import { PlaceAssetCommand } from '../../src/core/commands/sceneObjectCommands'; +import { SetSongCommand } from '../../src/core/commands/songCommands'; +import type { + CompositionAsset, + CompositionItem, + InstrumentAsset, + SongAsset, +} from '../../src/core/model/Asset'; +import { compositionLength, fittedLength, setSoundMeasure } from '../../src/core/model/composition'; +import { carriedClips, itemMedia, mediaMarks } from '../../src/core/model/compositionEdit'; +import { findAssetOf } from '../../src/core/model/Project'; +import { audioItemBus, compositionPlayings, compositionSpan } from '../../src/core/music/compositionAudio'; +import { MASTER_MIX, mixBusName } from '../../src/core/music/mixer'; +import { Playbacks, type SequencePlay } from '../../src/core/music/playbacks'; +import { LEAD } from '../../src/core/music/Scheduler'; +import { newItem } from '../../src/core/music/song'; +import { MASTER_BUS } from '../../src/core/properties/buses'; +import { SILENT_AUDIO } from '../../src/runtime/systems/Audio'; +import { addGameSystems } from '../../src/runtime/systems/BehaviorSystem'; +import { Input } from '../../src/runtime/systems/Input'; +import { World } from '../../src/runtime/World'; +import { graphEditor, makeBehavior, makeProject } from '../helpers'; + +const FILE = `${'e5'.repeat(32)}.wav`; +const Q = 480; + +/** + * A composition "Cut" at 10 fps placed in a scene, a sound "Blip", a one-bar pattern "Riff" at + * 120 BPM (two seconds) with notes on beats 1 and 3, and a song "Theme" playing it from its start. + */ +function makeCut() { + const f = makeProject(); + const scene = new CreateAssetCommand('scene', 'Level'); + const comp = new CreateAssetCommand('composition', 'Cut', { fps: 10 }); + const sound = new CreateSoundCommand('Blip', FILE); + const ins = new CreateAssetCommand('instrument', 'Lead'); + const pattern = new CreateAssetCommand('pattern', 'Riff'); + const song = new CreateAssetCommand('song', 'Theme'); + const bus = new CreateAssetCommand('bus', 'Music'); + for (const c of [scene, comp, sound, ins, pattern, song, bus]) f.stack.execute(c); + (ins.asset as InstrumentAsset).graph.nodes.find((n) => n.type === 'InstrumentSample')!.params.sound = + sound.asset.id; + f.stack.execute(new SetAssetPropertyCommand(pattern.asset.id, 'instrument', ins.asset.id)); + f.stack.execute(new SetAssetPropertyCommand(pattern.asset.id, 'bars', 1)); + f.stack.execute( + new SetPatternNotesCommand(pattern.asset.id, [ + { id: 'n1', tick: 0, length: Q, note: 60, velocity: 1 }, + { id: 'n2', tick: 2 * Q, length: Q, note: 64, velocity: 1 }, + ]), + ); + const theme = () => findAssetOf(f.project, song.asset.id, 'song') as SongAsset; + const track = theme().tracks[0]!; + f.stack.execute( + new SetSongCommand(song.asset.id, { + items: [newItem({ track: track.id, kind: 'pattern', asset: pattern.asset.id, tick: 0, length: 4 * Q })], + }), + ); + const place = new PlaceAssetCommand(scene.asset.id, comp.asset.id, { x: 0, y: 0 }); + f.stack.execute(place); + const cut = () => findAssetOf(f.project, comp.asset.id, 'composition') as CompositionAsset; + const lay = (...items: CompositionItem[]) => + f.stack.execute(new SetCompositionCommand(cut().id, { items })); + const audioItem = ( + id: string, + kind: 'sound' | 'pattern' | 'song', + start: number, + length: number, + more: Partial & { volume?: number; bus?: string } = {}, + ): CompositionItem => ({ + id, + track: cut().tracks[0]!.id, + start, + length, + offset: more.offset ?? 0, + audio: { + kind, + asset: { sound: sound.asset.id, pattern: pattern.asset.id, song: song.asset.id }[kind], + volume: more.volume ?? 0, + bus: more.bus ?? '', + }, + }); + return { + ...f, + sceneId: scene.asset.id, + placement: place.objectId, + soundId: sound.asset.id, + patternId: pattern.asset.id, + songId: song.asset.id, + busId: bus.asset.id, + cut, + lay, + audioItem, + }; +} + +describe('audio items of a composition', () => { + it('are kept without an object, and count to its length', () => { + const f = makeCut(); + f.lay(f.audioItem('a', 'sound', 5, 20, { volume: 40 }), f.audioItem('b', 'pattern', 0, 10)); + expect(f.cut().items.map((i) => i.id)).toEqual(['a', 'b']); + // Held to the most a sound item may be turned up by. + expect(f.cut().items[0]!.audio).toEqual({ kind: 'sound', asset: f.soundId, volume: 12, bus: '' }); + expect(f.cut().objects).toEqual([]); + expect(compositionLength(f.project, f.cut())).toBe(25); + }); + + it('drops one that plays nothing or an unknown kind', () => { + const f = makeCut(); + const bad = { + ...f.audioItem('a', 'sound', 0, 10), + audio: { kind: 'video', asset: 'x', volume: 0, bus: '' }, + }; + const empty = { + ...f.audioItem('b', 'sound', 0, 10), + audio: { kind: 'sound', asset: '', volume: 0, bus: '' }, + }; + f.lay(bad as CompositionItem, empty as CompositionItem); + expect(f.cut().items).toEqual([]); + }); + + it('take the length of a pattern or song, of a sound once it is measured', () => { + const f = makeCut(); + f.lay(f.audioItem('p', 'pattern', 0, 1, { offset: 0.5 }), f.audioItem('s', 'sound', 0, 1)); + const [pattern, sound] = f.cut().items; + expect(fittedLength(f.project, f.cut(), pattern!)).toBe(15); + expect(fittedLength(f.project, f.cut(), sound!)).toBe(30); + setSoundMeasure((file) => (file === FILE ? 1.25 : null)); + try { + expect(fittedLength(f.project, f.cut(), sound!)).toBe(13); + expect(itemMedia(f.project, f.cut(), sound!)).toEqual({ begins: 0, length: 12.5, loops: true }); + } finally { + setSoundMeasure(() => null); + } + }); + + it('mark a song to the end of its loop, then every time round it', () => { + const f = makeCut(); + f.stack.execute(new SetLoopCommand('song', f.songId, { start: Q, end: 3 * Q })); + f.lay(f.audioItem('s', 'song', 0, 40)); + const song = f.cut().items[0]!; + expect(fittedLength(f.project, f.cut(), song)).toBe(15); + const media = itemMedia(f.project, f.cut(), song)!; + expect(media).toEqual({ begins: 0, length: 15, loops: true, round: 10 }); + expect(mediaMarks(song, media).seams).toEqual([15, 25, 35]); + }); + + it('own no clips: a clip on one belongs to the object item it starts on', () => { + const f = makeCut(); + const sound = f.audioItem('a', 'sound', 0, 20); + const object: CompositionItem = { id: 'o', track: sound.track, start: 0, length: 20, offset: 0 }; + const clip = { id: 'c', track: sound.track, animation: 'x', start: 5 }; + expect([...carriedClips([object, sound], [clip], new Set(['a']))]).toEqual([]); + expect([...carriedClips([object, sound], [clip], new Set(['o']))]).toEqual(['c']); + }); +}); + +describe('the sequences of a composition', () => { + it('lay a sound in milliseconds on a bus of its own at its volume, late by an offset below 0', () => { + const f = makeCut(); + f.lay( + f.audioItem('a', 'sound', 5, 20, { volume: -6, offset: 0.25 }), + f.audioItem('b', 'sound', 30, 10, { offset: -0.3 }), + ); + const [playing] = compositionPlayings(f.project, f.cut()); + expect(playing!.route).toBe(MASTER_MIX); + expect(playing!.sequence.clips).toEqual([ + { + tick: 500, + length: 2000, + sound: f.soundId, + fx: '', + offset: 0.25, + volume: 1, + bus: audioItemBus(f.cut().id, 'a'), + }, + { + tick: 3300, + length: 700, + sound: f.soundId, + fx: '', + offset: 0, + volume: 1, + bus: audioItemBus(f.cut().id, 'b'), + }, + ]); + expect(playing!.buses.map((b) => [b.name, Number(b.gain.toFixed(3))])).toEqual([ + [audioItemBus(f.cut().id, 'a'), 0.501], + [audioItemBus(f.cut().id, 'b'), 1], + ]); + }); + + it('play a pattern round and round at its own tempo, cut where the item ends', () => { + const f = makeCut(); + // 3.5 s from half a second in: beat 3 (1 s), then round two (1.5 s, 2.5 s), then the next (3.5 s) is cut. + f.lay(f.audioItem('p', 'pattern', 10, 35, { offset: 0.5 })); + const notes = compositionPlayings(f.project, f.cut())[0]!.sequence.notes; + expect(notes.map((n) => [n.tick, n.length, n.note, n.tempo])).toEqual([ + [1500, 500, 64, 120], + [2500, 500, 60, 120], + [3500, 500, 64, 120], + ]); + }); + + it('give a song item its own copy of the song’s buses, sending into the item’s', () => { + const f = makeCut(); + f.lay(f.audioItem('s', 'song', 0, 20), f.audioItem('t', 'song', 0, 20)); + const [playing] = compositionPlayings(f.project, f.cut()); + const bus = audioItemBus(f.cut().id, 's'); + const track = + f.cut().tracks.length && (findAssetOf(f.project, f.songId, 'song') as SongAsset).tracks[0]!.id; + const names = playing!.buses.map((b) => b.name); + expect(names).toContain(`${bus}:${track}`); + expect(names).toContain(`${audioItemBus(f.cut().id, 't')}:${track}`); + expect(playing!.buses.find((b) => b.name === `${bus}:${track}`)!.sends).toEqual([ + expect.objectContaining({ to: bus }), + ]); + const own = playing!.sequence.notes.filter((n) => n.bus?.startsWith(`${bus}:`)); + expect(own.map((n) => [n.tick, n.note])).toEqual([ + [0, 60], + [1000, 64], + ]); + }); + + it('go one to each place of the mix their items are sent to, or all to the master unrouted', () => { + const f = makeCut(); + f.lay( + f.audioItem('a', 'sound', 0, 10), + f.audioItem('b', 'sound', 0, 10, { bus: f.busId }), + f.audioItem('c', 'pattern', 0, 10, { bus: MASTER_BUS }), + ); + const routes = compositionPlayings(f.project, f.cut()).map((p) => [p.route, p.buses.length]); + expect(routes).toEqual([ + [MASTER_MIX, 2], + [mixBusName(f.busId), 1], + ]); + expect(compositionPlayings(f.project, f.cut(), false).map((p) => p.route)).toEqual([MASTER_MIX]); + }); + + it('run as long as the composition, round it where it loops', () => { + const f = makeCut(); + f.lay(f.audioItem('a', 'sound', 5, 20)); + expect(compositionSpan(f.project, f.cut())).toEqual({ length: 2500, loop: null }); + f.stack.execute(new SetAssetPropertyCommand(f.cut().id, 'loop', true)); + expect(compositionSpan(f.project, f.cut())).toEqual({ length: 2500, loop: { start: 0, end: 2500 } }); + }); +}); + +describe('a playing started from a tick (#418)', () => { + it('starts as much further in as it waited for its files', async () => { + let time = 0; + let loaded = () => undefined as void; + const started: number[] = []; + const playbacks = new Playbacks({ + now: () => time, + latency: () => 0, + playing: () => false, + stop: () => undefined, + wake: () => () => undefined, + prepare: () => new Promise((resolve) => (loaded = resolve)), + level: () => undefined, + after: () => undefined, + }); + const play: SequencePlay = { + sequence: { notes: [{ tick: 0, length: 1, note: 60, velocity: 1, instrument: 'x' }] }, + tempo: 125, + length: 10000, + loop: null, + needs: { layers: [], files: [] }, + from: 1000, + strike: (note) => (started.push(note.at), 1), + }; + playbacks.start(1, 'composition:x', play, ''); + time = 0.25; + loaded(); + await Promise.resolve(); + const runner = playbacks.current()[0]!.runner as unknown as { position(): { ticks: number } }; + expect(runner.position().ticks).toBe(1250); + }); +}); + +/** A sink that writes down what it is asked to play and stop. */ +function recordingAudio() { + const sink = { + ...SILENT_AUDIO, + started: [] as { id: number; name: string; play: SequencePlay }[], + stopped: [] as number[], + playSequence: (name: string, play: SequencePlay) => { + const id = 100 + sink.started.length; + sink.started.push({ id, name, play }); + return id; + }, + stopPlayback: (id: number) => { + sink.stopped.push(id); + }, + }; + return sink; +} + +describe('a composition played in a game', () => { + function run(f: ReturnType) { + const behavior = makeBehavior(f, 'Director'); + f.stack.execute(new PlaceAssetCommand(f.sceneId, behavior.id, { x: 0, y: 0 }, f.placement)); + const edit = graphEditor(f, behavior); + const start = edit.addNode('OnStart'); + const owner = edit.addNode('Owner'); + const play = edit.addNode('PlayComposition'); + edit.connect(start, 'exec', play, 'exec'); + edit.connect(owner, 'object', play, 'object'); + const audio = recordingAudio(); + const world = World.instantiate(f.project, f.sceneId); + addGameSystems(world, f.project, new Input(), { audio }); + return { audio, world, frame: (times = 1) => Array.from({ length: times }, () => world.tick(0.1)) }; + } + + it('is heard from where its clock stands once it runs, and stops with its end', () => { + const f = makeCut(); + f.lay(f.audioItem('a', 'sound', 0, 5)); + const game = run(f); + game.frame(); + // The first tick shows frame 0; the clock and the sound run from the next. + expect(game.audio.started).toEqual([]); + game.frame(); + expect(game.audio.started).toHaveLength(1); + expect(game.audio.started[0]!.name).toBe(`composition:${f.cut().id}`); + expect(game.audio.started[0]!.play.from).toBe(Math.round((0.1 + LEAD) * 1000)); + game.frame(2); + expect(game.audio.started).toHaveLength(1); + game.frame(3); + expect(game.audio.stopped).toEqual([100]); + }); + + it('stands silent while the game is paused, and goes on from where its clock stood', () => { + const f = makeCut(); + f.lay(f.audioItem('a', 'sound', 0, 50)); + const game = run(f); + game.frame(3); + expect(game.audio.started).toHaveLength(1); + game.world.paused = true; + game.frame(2); + expect(game.audio.stopped).toEqual([100]); + expect(game.audio.started).toHaveLength(1); + game.world.paused = false; + game.frame(); + expect(game.audio.started).toHaveLength(2); + expect(game.audio.started[1]!.play.from).toBe(Math.round((0.3 + LEAD) * 1000)); + }); +}); diff --git a/tests/ui/scroll-container.test.ts b/tests/ui/scroll-container.test.ts index 31566920..56ee39cb 100644 --- a/tests/ui/scroll-container.test.ts +++ b/tests/ui/scroll-container.test.ts @@ -67,6 +67,44 @@ describe('a scroll container with tail space', () => { }); }); +describe('a scroll container holding its offset', () => { + const scrolledToEnd = (holdOffset: boolean) => { + const scroll = new ScrollContainer(); + scroll.holdOffset = holdOffset; + scroll.resize(200, 100); + const rows = new VStack(); + for (let i = 0; i < 10; i++) rows.add(new Spacer(20)); + scroll.setContent(rows); + scroll.scrollTo(100); + // Three rows go, as a folded group's would. + rows.removeChildren(7); + scroll.relayoutContent(); + return { scroll, rows }; + }; + + it('keeps the offset when the content shrinks, the room left below', () => { + const { scroll, rows } = scrolledToEnd(true); + expect(scroll.offset).toBe(100); + expect(rows.h).toBe(200); + }); + + it('lets the room go as the view is scrolled up off it', () => { + const { scroll, rows } = scrolledToEnd(true); + scroll.scrollTo(70); + expect(rows.h).toBe(170); + expect(scroll.maxScroll).toBe(70); + scroll.scrollTo(20); + expect(rows.h).toBe(140); + expect(scroll.maxScroll).toBe(40); + }); + + it('without it clamps the offset, sliding every row down', () => { + const { scroll, rows } = scrolledToEnd(false); + expect(scroll.offset).toBe(40); + expect(rows.h).toBe(140); + }); +}); + describe('a scroll container and its scrollbar', () => { const filled = (viewportH: number, rowCount: number) => { const scroll = new ScrollContainer(); diff --git a/tools/e2e/04-assets.mjs b/tools/e2e/04-assets.mjs index 5a6a34e9..0c393703 100644 --- a/tools/e2e/04-assets.mjs +++ b/tools/e2e/04-assets.mjs @@ -57,6 +57,7 @@ export async function assets(t) { await addMenuOnRightClick(t); await emptyGroupsOnSwitch(t); + await foldKeepsPlace(t); // 4b. Drag the Crate asset from the panel into the scene view. Driven step by step rather than // with `drag()`, because the preview only exists while the button is still down. @@ -681,3 +682,56 @@ async function emptyGroupsOnSwitch(t) { JSON.stringify({ before: shape(before), off: shape(off) }), ); } + +/** + * Folding a group leaves the list where it stood: it does not jump back to the selected asset, and + * at the end of the list, where the room below lets it scroll, the rows do not slide down by the + * ones that went away. The heading clicked stays under the pointer, folded and unfolded. + */ +async function foldKeepsPlace(t) { + const { check, sleep, json, mutate, mouse, click, select, showTab } = t; + const before = await json('__gm.state.selection'); + await mutate(` + const p = __gm.state.project; + for (let i = 1; i <= 30; i++) + p.assets.push({ id: 'fold04-' + i, type: 'sprite', name: 'Fold ' + String(i).padStart(2, '0'), + props: { width: 10, height: 10, color: 0xffffff, texture: '' } }); + p.assets.push({ id: 'fold04-sound', type: 'sound', name: 'Fold Sound', props: {}, file: '' });`); + // The Crate is selected far above; the Sounds group, unfolded, sits at the end of the list. + await select({ kind: 'asset', assetId: 'crate' }); + await json(`__gm.assetRow('fold04-sound')`); + const rect = await json('__gm.assetsRect()'); + await mouse('mouseWheel', rect.x + rect.w / 2, rect.y + rect.h / 2, 'none', { deltaX: 0, deltaY: 5000 }); + await sleep(500); + const at = async () => ({ + heading: (await json(`__gm.assetRowShown('group:sound')`))?.y, + crate: (await json(`__gm.assetRowShown('crate')`))?.y, + }); + const start = await at(); + const heading = await json(`__gm.assetRowShown('group:sound')`); + await click(heading.x, heading.y); + await sleep(200); + const folded = await at(); + const groupFolded = !(await json('__gm.assetGroups()')).find((g) => g.type === 'sound')?.expanded; + await sleep(450); + await click(heading.x, heading.y); + await sleep(200); + const unfolded = await at(); + check( + 'folding a group at the end of the list leaves its heading under the pointer, the selection out of view', + start.heading > rect.y && + start.heading < rect.y + rect.h && + start.crate < rect.y && + groupFolded && + folded.heading === start.heading && + unfolded.heading === start.heading && + folded.crate === start.crate && + unfolded.crate === start.crate, + JSON.stringify({ rect, start, folded, groupFolded, unfolded }), + ); + await mutate(` + const p = __gm.state.project; + p.assets = p.assets.filter((a) => !a.id.startsWith('fold04-'));`); + await select(before ?? null); + await showTab('scene'); +} diff --git a/tools/e2e/68-composition.mjs b/tools/e2e/68-composition.mjs index ec75fdea..ee42facd 100644 --- a/tools/e2e/68-composition.mjs +++ b/tools/e2e/68-composition.mjs @@ -10,7 +10,8 @@ import { MAKE_OBJECT } from './lib/scaffold.mjs'; * item is left. Nothing of the run stays in the scene. An item's video is waited for: the clock * stands until the Game tab has its first picture, for a clip that starts later the picture there, * and then the video plays at once. A recording that runs meanwhile takes no frame. When the item - * ends, its video stops and goes quiet with it. + * ends, its video stops and goes quiet with it. A sound item is planned once the clock runs, timed + * to the cut. * * Everything it adds carries a `cp68` id and goes again at the end. */ @@ -368,6 +369,59 @@ export async function composition(t) { }), ); + // 68h (#418). A sound item half a second in, beside the video item from 68d: the cut waits for the + // video, and its sound with it – planned once the clock runs, for the second of the audio + // clock the cut reaches the item at, and to its end. + const sounds = await evaluate(`fetch('/feuerwerk/api/index.php?r=sounds').then((r) => r.json())`); + const blip = sounds.find((item) => item.bytes === 4044); + await mutate(` + const p = __gm.state.project; + p.assets.push({ id: 'cp68-blip', type: 'sound', name: 'Blip 68', props: {}, file: ${JSON.stringify(blip?.file ?? '')} }); + p.assets.find((a) => a.id === 'cp68-screen').props.texture = 'cp68-video'; + const c = p.assets.find((a) => a.id === 'cp68-cut'); + c.items = [ + { id: 'cp68-v', track: 'cp68-front', start: 0, length: 120, offset: 0 }, + { id: 'cp68-tone', track: 'cp68-back', start: 30, length: 30, offset: 0, + audio: { kind: 'sound', asset: 'cp68-blip', volume: 0, bus: '' } }, + ];`); + await evaluate(`{ + window.cp68Sound = new Promise((done) => { + const rows = []; + const until = performance.now() + 20000; + const step = () => { + if (__gm.state.mode === 'play') { + const time = ${clock}; + const tone = __gm.sound.playbacks().find((p) => p.name === 'composition:cp68-cut'); + rows.push({ time, clock: __gm.sound.clock(), planned: tone?.clips.length ?? 0, clip: tone?.clips[0] ?? null }); + if (time !== null && time >= 0.6) return done(JSON.stringify(rows)); + } + if (performance.now() > until) return done(JSON.stringify(rows)); + requestAnimationFrame(step); + }; + step(); + }); + null; + }`); + await play(); + const toneRows = JSON.parse(await evaluate('window.cp68Sound')); + await stop(); + const ran = toneRows.findIndex((row) => row.time > 0); + const reached = toneRows.find((row) => row.time >= 0.5); + const toneClip = toneRows.at(-1)?.clip; + const early = reached && toneClip ? reached.clock - toneClip.at : null; + check( + 'a sound item is planned only once the cut’s clock runs, for when the cut reaches it, to its end', + ran > 0 && + toneRows.slice(0, ran).every((row) => row.planned === 0) && + toneClip?.sound === 'cp68-blip' && + toneClip.offset === 0 && + Math.abs(toneClip.duration - 0.5) < 0.002 && + early !== null && + early > -0.02 && + early < 0.05, + JSON.stringify({ waited: ran, early, reached, toneClip }), + ); + await mutate(` const p = __gm.state.project; p.assets = p.assets.filter((a) => !a.id.startsWith('cp68-') && a.id !== ${JSON.stringify(madeId)}); diff --git a/tools/e2e/69-composition-tab.mjs b/tools/e2e/69-composition-tab.mjs index 081caa11..2d42ced2 100644 --- a/tools/e2e/69-composition-tab.mjs +++ b/tools/e2e/69-composition-tab.mjs @@ -8,7 +8,7 @@ * whose player is still loading. A copied item is pasted under the pointer or at the playhead. A * video dragged in wears a sprite the composition keeps to itself. An item dragged in the picture * follows the pointer while the button is down. A band picks items, with Alt clips, and copied clips - * are pasted where they fit. + * are pasted where they fit. A sound dragged in is an audio item, heard while the playhead runs. * * Everything it adds carries a `ct69` id and goes again at the end. */ @@ -486,6 +486,111 @@ export async function compositionTab(t) { JSON.stringify({ frameWidth, along, alone }), ); + // 69o (#418). A sound dragged onto a track is an audio item as long as one pass of it, picked on + // the composition, the inspector naming it. Played from the middle, the composition plans it as a + // clip that far into the sound, on the item's own bus and to the item's end; Pause ends it, and a + // scrub plays nothing. + const sounds = await evaluate(`fetch('/feuerwerk/api/index.php?r=sounds').then((r) => r.json())`); + const blip = sounds.find((item) => item.bytes === 4044); + await mutate(` + const p = __gm.state.project; + p.assets.push({ id: 'ct69-sound', type: 'sound', name: 'Blip 69', props: {}, file: ${JSON.stringify(blip?.file ?? '')} }); + const c = p.assets.find((a) => a.id === '${CUT}'); + c.objects = []; + c.items = []; + c.clips = [];`); + const seconds = await evaluate(`__gm.sound.load(${JSON.stringify(blip?.file ?? '')})`); + const soundDrop = await dragAsset('ct69-sound', await json(`__gm.composition.laneAt('ct69-t2', 10)`)); + m = await model(); + const tone = m.items[0]; + const tonePick = await json('__gm.state.selection'); + const toneSection = await json(`__gm.section('SOUND ITEM')`); + await mutate(` + const c = __gm.state.project.assets.find((a) => a.id === '${CUT}'); + c.items[0].length = 20;`); + await evaluate('__gm.composition.scrub(15)'); + await sleep(200); + const scrubbed = (await json('__gm.sound.playbacks()')).filter((p) => p.name === `composition:${CUT}`); + const toneAt = await json('__gm.composition.playAt()'); + await mouse('mouseMoved', toneAt.x, toneAt.y); + await t.click(toneAt.x, toneAt.y); + let heard = []; + for (let i = 0; i < 40 && !heard.some((p) => p.name === `composition:${CUT}` && p.clips.length > 0); i++) { + await sleep(50); + heard = await json('__gm.sound.playbacks()'); + } + const toneView = await json('__gm.composition.view()'); + const toneSound = await json('__gm.composition.sound()'); + const playing = heard.find((p) => p.name === `composition:${CUT}`); + const toneClip = playing?.clips[0]; + await t.click(toneAt.x, toneAt.y); + await sleep(300); + const paused = (await json('__gm.sound.playbacks()')).filter((p) => p.name === `composition:${CUT}`); + check( + 'a sound dragged onto a track is an audio item, heard from the playhead on its own bus until the item ends', + seconds > 0 && + soundDrop?.what === 'item' && + tone?.audio?.kind === 'sound' && + tone.audio.asset === 'ct69-sound' && + tone.start === 10 && + tone.length === Math.max(1, Math.round(seconds * 10)) && + m.objects.length === 0 && + tonePick?.kind === 'composition' && + JSON.stringify(tonePick.itemIds) === JSON.stringify([tone.id]) && + toneSection !== null && + scrubbed.length === 0 && + toneClip?.sound === 'ct69-sound' && + toneClip.bus === `composition:${CUT}:${tone.id}` && + toneSound.playing && + toneSound.playings.includes(playing.id) && + toneClip.offset > 0.5 && + toneClip.offset < 0.8 && + Math.abs(toneClip.offset + toneClip.duration - 2) < 0.002 && + paused.length === 0, + JSON.stringify({ + seconds, + soundDrop, + tone, + tonePick, + scrubbed, + toneClip, + paused: paused.length, + toneView, + heard, + toneSection, + toneSound, + }), + ); + + // 69o-ii. A pattern and a song leave the Assets panel as well and land as audio items as long as + // their preview promised – the panel once let only sounds go. + await mutate(` + const p = __gm.state.project; + p.assets.push( + { id: 'ct69-pattern', type: 'pattern', name: 'Riff 69', props: { bars: 1 }, notes: [] }, + { id: 'ct69-song', type: 'song', name: 'Song 69', props: {}, tracks: [], items: [], graph: { nodes: [], edges: [] } }, + ); + p.assets.find((a) => a.id === '${CUT}').items = [];`); + const patternDrop = await dragAsset('ct69-pattern', await json(`__gm.composition.laneAt('ct69-t1', 10)`)); + const songDrop = await dragAsset('ct69-song', await json(`__gm.composition.laneAt('ct69-t2', 10)`)); + m = await model(); + const kinds = m.items.map((i) => ({ + kind: i.audio?.kind, + asset: i.audio?.asset, + start: i.start, + length: i.length, + })); + const landed = (drop, kind) => + drop?.what === 'item' && + kinds.some( + (i) => i.kind === kind && i.asset === drop.assetId && i.start === 10 && i.length === drop.length, + ); + check( + 'a pattern and a song dragged onto a track are audio items as long as their preview', + landed(patternDrop, 'pattern') && landed(songDrop, 'song') && m.items.length === 2, + JSON.stringify({ patternDrop, songDrop, kinds }), + ); + await showTab('scene'); await mutate(` const p = __gm.state.project;