Compositions: sounds, patterns and songs as items on a track (#418)

- Compositions has a new section on audio items: how they are dragged
  in, what the inspector offers, how rounds and loop marks work, and
  how they sound in the tab and in a game.
- Concepts says that folding in the asset panel keeps the scroll
  position.
- Development counts 2072 editor checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adminandClaude Opus 5.5 committed 2026-10-13 21:28:38 +02:00
1 parent 3311619147
commit 4203ca83ea
3 files changed
+60 -10

No files matched your search

+56 -9
@@ -1,7 +1,8 @@
A **composition** lays objects out in time (#417): a cut of video clips, a cutscene, a title that
comes in and goes again. It has **tracks**, and on each track **items** and **animation clips**. An
item is one object that lives from its start to its end; a clip plays an animation on the items of
its track while it runs. Placed in a scene, a composition plays when a behavior says so.
item is one object that lives from its start to its end, or a sound, a pattern or a song heard while
it lasts (#418); a clip plays an animation on the items of its track while it runs. Placed in a
scene, a composition plays when a behavior says so.
## The asset
@@ -19,8 +20,30 @@ of objects beside scenes and containers (`ObjectHolder` in `src/core/model/Proje
What is stored (`CompositionAsset` in `src/core/model/Asset.ts`): `tracks` (id, name), `items` (id
= the object's id, track, start and length in frames, offset in seconds into its video,
`fitMedia`) and `clips` (id, track, animation, start in frames). An item whose object is gone stays
in the list, so an undo finds it again, and plays as nothing.
`fitMedia`, and for an audio item `audio`) and `clips` (id, track, animation, start in frames). An
item whose object is gone stays in the list, so an undo finds it again, and plays as nothing.
### Audio items
A **sound**, a **clip of a sound**, a **pattern** or a **song** dragged onto a track is an audio item
(#418). It has no object; `audio` says what it plays (`kind`, `asset`), its **volume** in dB
(-60 to +12) and its **bus**. It shares a track with object items and may overlap them.
- **It plays for as long as it lasts**, and stops at its end. Longer than what it plays, it goes
round and round: a sound and a pattern from their start, a song round the loop of its ruler, as
`Play song` with Loop does. The **offset** is how far into it the item begins, in seconds; a trim
of the left edge moves it. Below 0 the item is silent until then.
- **A pattern or a song keeps its own tempo** inside the item. A Delay synced to beats counts by it.
- **The bus.** *From asset* (the default) sends it where the [Master routing](Sounds#master-routing)
sends its sound, pattern or song; *Master* or a bus sends it there instead. Each item is heard on
a bus of its own at its volume, through the sound FX of its pattern or song; a song's tracks are
buses of that item, so two items of one song do not share them.
- **Animation clips do not belong to an audio item**: there is nothing on it they could move.
- **What is gone plays nothing.** The item stays, drawn red, named *(missing)*.
The sequences are made in `src/core/music/compositionAudio.ts`: everything in milliseconds of the
composition, one sequence for each place of the mix the items go to. `CompositionSound`
(`src/runtime/compositionSound.ts`) plays them through the scheduler, in the tab and in a game.
### Sprites of its own
@@ -79,6 +102,10 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play**
same way, wearing a sprite the composition keeps for that picture (see *Sprites of its own*
below). A video's item is as long as the video. A clip of a video is chosen afterwards, on the
Texture row of the item's inspector.
- **Dragging a sound, a clip of one, a pattern or a song onto a track** (#418) makes an audio item
there, as long as one pass of it, picked; a song's first pass ends where its loop does. A sound
has to be loaded to know its length; the drag loads it, and until it is there the item is 3 s
long.
- **Dragging an animation onto a track** makes a clip at that frame, if it fits between the clips
there.
- **Dragging an item** moves it in time and from track to track, past and over other items; it
@@ -97,13 +124,19 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play**
item cuts into the video on that side: there would be more of it. A line with a triangle, like a
loop in the Song tab, marks where the video starts or ends inside the item; with Loop on the
video, one line per round. An overlap is darkened, and a line marks the end of the item below.
An audio item shows the same marks for what it plays, a line at every round of it; for a song
the first at the end of its loop, then one per time round the loop. A sound item draws its
waveform; sounds are blue-green, patterns violet, songs plum.
- **Click** picks an item or a clip, `Ctrl`-click adds or removes one. A click on a track head picks
the track, a click in the ruler moves the playhead there, and dragging in it scrubs.
- **A band** dragged from an empty spot picks the items it touches. With `Alt` held it picks the
clips instead and may start anywhere, on an item too. With `Ctrl` it adds to what is picked. A
pick holds items or clips, never both.
pick holds items or clips, never both. Items that are objects only are picked as objects, so the
hierarchy and the inspector show them; with an audio item among them the pick is the
composition's, and the inspector shows the audio item, or the count for several.
- **The right-click menu** on an item: *Fit to media*, *Make project asset* (only for an item
wearing a sprite of the composition's own), *Remove item*. On a clip: *Open animation*,
wearing a sprite of the composition's own), *Remove item*. On an audio item *Open sound*
(*pattern*, *song*) instead of *Make project asset*. On a clip: *Open animation*,
*Remove clip*. On a track head or an empty stretch of a track, as in the Song tab: *Add track
above*, *Add track below*, *Move up*, *Move down*, *Delete track*.
- **Copy and paste.** `Ctrl+C` on picked items keeps their objects and their timing, `Ctrl+V` in
@@ -113,7 +146,8 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play**
their distances and their tracks to each other. The same copy pastes into a scene as plain
objects; objects copied from a scene do not paste into a composition, they have no timing.
Copied items take their clips along; where one of those would lie over a clip, items and clips
move on together to the first frame where they fit. **Picked clips** are copied and pasted the same way, into this composition or another. As clips
move on together to the first frame where they fit. Audio items are copied with them, and alone
the same way. **Picked clips** are copied and pasted the same way, into this composition or another. As clips
of a track never overlap, a paste that would lie over a clip moves on, all its clips together,
to the first frame where they fit.
- `Delete` removes the picked items with their objects and their clips, the picked clips or the
@@ -121,6 +155,10 @@ the clips as narrower bars above them. The header bar has **+ Track**, **Play**
- **Play** plays from the playhead at the composition's frame rate, round with its Loop; from the
end it starts at 0. The wheel zooms the time, the middle button or `Shift`-drag pans. Both glide
like the other tabs (see [Concepts](Concepts)).
- **Audio items are heard while it plays**, from where the playhead stands, straight into the
master as in every tab. Their bus is heard in a game. A scrub is silent. While the playhead
waits for a video, the sound waits too. An edit while it plays is heard at once. Playing a song
or a pattern in its tab stops the composition, and the other way round.
**The picture** is a scene view whose objects are the items alive at the playhead. A click picks
an item or its child, dragging moves the real object. It takes nothing from the asset panel and
@@ -130,7 +168,9 @@ the links of their video in the [Master routing](Sounds#master-routing).
**The hierarchy** lists the items alive at the playhead; re-parenting is off there, because items
stay at the top. **The inspector** of an item shows its object's rows and a block **TIMING**:
*Start*, *Length* and, for a video item, *Offset*, all in seconds, and **Fit to media**. A track
*Start*, *Length* and, for a video item, *Offset*, all in seconds, and **Fit to media**. An audio
item shows **SOUND ITEM** (*PATTERN*, *SONG*): what it plays, the same timing rows with *Offset*,
**Volume** in dB and **Bus** (*From asset*, *Master* or a bus). A track
shows its name; a clip its animation and its start, with the length of one pass.
**Fit to media.** A new item's length follows its video: when an override gives its sprite another
@@ -167,6 +207,13 @@ it picks the placement, its items cannot be picked one by one there.
still seeks to it — the composition's clock stands, for at most 3 s at a time.
The rest of the game runs on. The Composition tab holds its playhead the same way. A recording
takes no frame meanwhile (see [Videos](Videos#recording-the-game-view)).
- **Its audio items are heard while its clock runs** (#418), on the bus each one names. They are
planned on the audio clock like the notes of a song, for the moment the clock reaches them, so
sound and picture keep together. While the clock stands (waiting for a video, the placement
switched off, the game paused), they are silent, and start again where it stands. `Stop
composition` and its end stop them. A time scale other than 1 moves the picture and not the
sound, and so does a frame longer than 0.1 s, which the game clock counts as 0.1 s: after one,
the sound is ahead of the picture until it starts again after the clock stood.
- **The editor run plays none** (*Run in editor*): it writes into the saved project, and the items
would stay there. `Play composition` says so in the log there.
@@ -176,6 +223,6 @@ with the same row binding the `AnimationSystem` uses.
## Not yet
- Sounds and songs as items on a track (#418).
- A baked song plays its notes inside a composition, not its bake.
- Dragging a clip of a video straight out of the Video tab onto a track.
- A composition inside a composition, and *Sort with parent* for a placement.
+3
@@ -321,6 +321,9 @@ below); quick steps add up and go on at the pace the list had, and dragging the
a row ends the glide where it is.
Below the last row the asset panel always keeps room for six more, so its end can be scrolled up
away from the bottom edge; clicks and the context menu there work as on any empty part of the list.
Folding a group leaves the list where it stands: the heading clicked stays under the pointer, and
rows that fold away near the end leave their room below until the list is scrolled up. Only a new
selection scrolls the list to its row.
**Panel sizes are remembered.** Every divider stays where it was dragged, after a reload too: the
ones left and right of the centre column, the one above the console, the one in the left column,
+1 -1
@@ -21,7 +21,7 @@ php tests/php/releases.test.php # the release stor
php tests/php/files.test.php # uploaded files in a folder per account (#344), the same way
php tests/php/videos.test.php # the video store, its queue and the worker (#402), with the
# real ffmpeg: the CI image and feuerwerk-video have it
node tools/seed-e2e.mjs && node tools/e2e.mjs # 2068 editor checks, screenshots in /tmp/gm-e2e-*.png
node tools/seed-e2e.mjs && node tools/e2e.mjs # 2072 editor checks, screenshots in /tmp/gm-e2e-*.png
GM_STEPS=20 node tools/e2e.mjs # or 1-12, or 6,20 — a subset while working on one step
./e2e.sh start [GM_STEPS] # the same run detached, see below
./e2e.sh status # live progress, or the last run's result