From 4203ca83ea099b8aaeb6082a9055d1f457dfd556 Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Tue, 13 Oct 2026 21:28:38 +0200 Subject: [PATCH] Compositions: sounds, patterns and songs as items on a track (#418) - Compositions has a new section on audio items: how they are dragged in, what the inspector offers, how rounds and loop marks work, and how they sound in the tab and in a game. - Concepts says that folding in the asset panel keeps the scroll position. - Development counts 2072 editor checks. Co-Authored-By: Claude Opus 5.5 --- Compositions.md | 65 ++++++++++++++++++++++++++++++++++++++++++------- Concepts.md | 3 +++ Development.md | 2 +- 3 files changed, 60 insertions(+), 10 deletions(-) diff --git a/Compositions.md b/Compositions.md index 8cfd54b..e22b4a9 100644 --- a/Compositions.md +++ b/Compositions.md @@ -1,7 +1,8 @@ A **composition** lays objects out in time (#417): a cut of video clips, a cutscene, a title that comes in and goes again. It has **tracks**, and on each track **items** and **animation clips**. An -item is one object that lives from its start to its end; a clip plays an animation on the items of -its track while it runs. Placed in a scene, a composition plays when a behavior says so. +item is one object that lives from its start to its end, or a sound, a pattern or a song heard while +it lasts (#418); a clip plays an animation on the items of its track while it runs. Placed in a +scene, a composition plays when a behavior says so. ## The asset @@ -19,8 +20,30 @@ of objects beside scenes and containers (`ObjectHolder` in `src/core/model/Proje What is stored (`CompositionAsset` in `src/core/model/Asset.ts`): `tracks` (id, name), `items` (id = the object's id, track, start and length in frames, offset in seconds into its video, -`fitMedia`) and `clips` (id, track, animation, start in frames). An item whose object is gone stays -in the list, so an undo finds it again, and plays as nothing. +`fitMedia`, and for an audio item `audio`) and `clips` (id, track, animation, start in frames). An +item whose object is gone stays in the list, so an undo finds it again, and plays as nothing. + +### Audio items + +A **sound**, a **clip of a sound**, a **pattern** or a **song** dragged onto a track is an audio item +(#418). It has no object; `audio` says what it plays (`kind`, `asset`), its **volume** in dB +(-60 to +12) and its **bus**. It shares a track with object items and may overlap them. + +- **It plays for as long as it lasts**, and stops at its end. Longer than what it plays, it goes + round and round: a sound and a pattern from their start, a song round the loop of its ruler, as + `Play song` with Loop does. The **offset** is how far into it the item begins, in seconds; a trim + of the left edge moves it. Below 0 the item is silent until then. +- **A pattern or a song keeps its own tempo** inside the item. A Delay synced to beats counts by it. +- **The bus.** *From asset* (the default) sends it where the [Master routing](Sounds#master-routing) + sends its sound, pattern or song; *Master* or a bus sends it there instead. Each item is heard on + a bus of its own at its volume, through the sound FX of its pattern or song; a song's tracks are + buses of that item, so two items of one song do not share them. +- **Animation clips do not belong to an audio item**: there is nothing on it they could move. +- **What is gone plays nothing.** The item stays, drawn red, named *(missing)*. + +The sequences are made in `src/core/music/compositionAudio.ts`: everything in milliseconds of the +composition, one sequence for each place of the mix the items go to. `CompositionSound` +(`src/runtime/compositionSound.ts`) plays them through the scheduler, in the tab and in a game. ### Sprites of its own @@ -79,6 +102,10 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play** same way, wearing a sprite the composition keeps for that picture (see *Sprites of its own* below). A video's item is as long as the video. A clip of a video is chosen afterwards, on the Texture row of the item's inspector. +- **Dragging a sound, a clip of one, a pattern or a song onto a track** (#418) makes an audio item + there, as long as one pass of it, picked; a song's first pass ends where its loop does. A sound + has to be loaded to know its length; the drag loads it, and until it is there the item is 3 s + long. - **Dragging an animation onto a track** makes a clip at that frame, if it fits between the clips there. - **Dragging an item** moves it in time and from track to track, past and over other items; it @@ -97,13 +124,19 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play** item cuts into the video on that side: there would be more of it. A line with a triangle, like a loop in the Song tab, marks where the video starts or ends inside the item; with Loop on the video, one line per round. An overlap is darkened, and a line marks the end of the item below. + An audio item shows the same marks for what it plays, a line at every round of it; for a song + the first at the end of its loop, then one per time round the loop. A sound item draws its + waveform; sounds are blue-green, patterns violet, songs plum. - **Click** picks an item or a clip, `Ctrl`-click adds or removes one. A click on a track head picks the track, a click in the ruler moves the playhead there, and dragging in it scrubs. - **A band** dragged from an empty spot picks the items it touches. With `Alt` held it picks the clips instead and may start anywhere, on an item too. With `Ctrl` it adds to what is picked. A - pick holds items or clips, never both. + pick holds items or clips, never both. Items that are objects only are picked as objects, so the + hierarchy and the inspector show them; with an audio item among them the pick is the + composition's, and the inspector shows the audio item, or the count for several. - **The right-click menu** on an item: *Fit to media*, *Make project asset* (only for an item - wearing a sprite of the composition's own), *Remove item*. On a clip: *Open animation*, + wearing a sprite of the composition's own), *Remove item*. On an audio item *Open sound* + (*pattern*, *song*) instead of *Make project asset*. On a clip: *Open animation*, *Remove clip*. On a track head or an empty stretch of a track, as in the Song tab: *Add track above*, *Add track below*, *Move up*, *Move down*, *Delete track*. - **Copy and paste.** `Ctrl+C` on picked items keeps their objects and their timing, `Ctrl+V` in @@ -113,7 +146,8 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play** their distances and their tracks to each other. The same copy pastes into a scene as plain objects; objects copied from a scene do not paste into a composition, they have no timing. Copied items take their clips along; where one of those would lie over a clip, items and clips - move on together to the first frame where they fit. **Picked clips** are copied and pasted the same way, into this composition or another. As clips + move on together to the first frame where they fit. Audio items are copied with them, and alone + the same way. **Picked clips** are copied and pasted the same way, into this composition or another. As clips of a track never overlap, a paste that would lie over a clip moves on, all its clips together, to the first frame where they fit. - `Delete` removes the picked items with their objects and their clips, the picked clips or the @@ -121,6 +155,10 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play** - **Play** plays from the playhead at the composition's frame rate, round with its Loop; from the end it starts at 0. The wheel zooms the time, the middle button or `Shift`-drag pans. Both glide like the other tabs (see [Concepts](Concepts)). +- **Audio items are heard while it plays**, from where the playhead stands, straight into the + master as in every tab. Their bus is heard in a game. A scrub is silent. While the playhead + waits for a video, the sound waits too. An edit while it plays is heard at once. Playing a song + or a pattern in its tab stops the composition, and the other way round. **The picture** is a scene view whose objects are the items alive at the playhead. A click picks an item or its child, dragging moves the real object. It takes nothing from the asset panel and @@ -130,7 +168,9 @@ the links of their video in the [Master routing](Sounds#master-routing). **The hierarchy** lists the items alive at the playhead; re-parenting is off there, because items stay at the top. **The inspector** of an item shows its object's rows and a block **TIMING**: -*Start*, *Length* and, for a video item, *Offset*, all in seconds, and **Fit to media**. A track +*Start*, *Length* and, for a video item, *Offset*, all in seconds, and **Fit to media**. An audio +item shows **SOUND ITEM** (*PATTERN*, *SONG*): what it plays, the same timing rows with *Offset*, +**Volume** in dB and **Bus** (*From asset*, *Master* or a bus). A track shows its name; a clip its animation and its start, with the length of one pass. **Fit to media.** A new item's length follows its video: when an override gives its sprite another @@ -167,6 +207,13 @@ it picks the placement, its items cannot be picked one by one there. still seeks to it — the composition's clock stands, for at most 3 s at a time. The rest of the game runs on. The Composition tab holds its playhead the same way. A recording takes no frame meanwhile (see [Videos](Videos#recording-the-game-view)). +- **Its audio items are heard while its clock runs** (#418), on the bus each one names. They are + planned on the audio clock like the notes of a song, for the moment the clock reaches them, so + sound and picture keep together. While the clock stands (waiting for a video, the placement + switched off, the game paused), they are silent, and start again where it stands. `Stop + composition` and its end stop them. A time scale other than 1 moves the picture and not the + sound, and so does a frame longer than 0.1 s, which the game clock counts as 0.1 s: after one, + the sound is ahead of the picture until it starts again after the clock stood. - **The editor run plays none** (*Run in editor*): it writes into the saved project, and the items would stay there. `Play composition` says so in the log there. @@ -176,6 +223,6 @@ with the same row binding the `AnimationSystem` uses. ## Not yet -- Sounds and songs as items on a track (#418). +- A baked song plays its notes inside a composition, not its bake. - Dragging a clip of a video straight out of the Video tab onto a track. - A composition inside a composition, and *Sort with parent* for a placement. diff --git a/Concepts.md b/Concepts.md index 81ae762..aa46aa0 100644 --- a/Concepts.md +++ b/Concepts.md @@ -321,6 +321,9 @@ below); quick steps add up and go on at the pace the list had, and dragging the a row ends the glide where it is. Below the last row the asset panel always keeps room for six more, so its end can be scrolled up away from the bottom edge; clicks and the context menu there work as on any empty part of the list. +Folding a group leaves the list where it stands: the heading clicked stays under the pointer, and +rows that fold away near the end leave their room below until the list is scrolled up. Only a new +selection scrolls the list to its row. **Panel sizes are remembered.** Every divider stays where it was dragged, after a reload too: the ones left and right of the centre column, the one above the console, the one in the left column, diff --git a/Development.md b/Development.md index d6d4fae..9362026 100644 --- a/Development.md +++ b/Development.md @@ -21,7 +21,7 @@ php tests/php/releases.test.php # the release stor php tests/php/files.test.php # uploaded files in a folder per account (#344), the same way php tests/php/videos.test.php # the video store, its queue and the worker (#402), with the # real ffmpeg: the CI image and feuerwerk-video have it -node tools/seed-e2e.mjs && node tools/e2e.mjs # 2068 editor checks, screenshots in /tmp/gm-e2e-*.png +node tools/seed-e2e.mjs && node tools/e2e.mjs # 2072 editor checks, screenshots in /tmp/gm-e2e-*.png GM_STEPS=20 node tools/e2e.mjs # or 1-12, or 6,20 — a subset while working on one step ./e2e.sh start [GM_STEPS] # the same run detached, see below ./e2e.sh status # live progress, or the last run's result