MCP: what is heard, under audio in get_runtime (#436)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adminandClaude Opus 5.5 committed 2026-10-18 22:15:25 +02:00
1 parent 3b3179c9a0
commit 29891b37d3
1 file changed
+34 -3
+34 -3
@@ -68,7 +68,7 @@ the badge says *Agent offline* meanwhile.
| `select` | Selects an asset, objects of a scene or container, or nodes of a graph |
| `play` | Play, pause, resume, stop. With `until`, the game runs on to that game time and holds there. Says so when no camera is on, since the game then shows nothing |
| `play_input` | Keys and the mouse in the running game, step by step, see below |
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, the last console lines |
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, what is heard (`audio`), the last console lines |
| `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once |
There is also a resource, `feuerwerk://view`, with the same answer as `get_view`. An agent that
@@ -247,6 +247,34 @@ it, then `get_runtime` to see what happened.
- `get_view` warns when the scene Play starts has no camera switched on, and while playing when the
game has none.
## Hearing the game
The agent has no ears, so `get_runtime` tells it under `audio` what you would hear. Whether music
really sounds, it reads there, not only in the `silent` of `get_asset`, which is worked out from the
data and misses a bus at 0, a send that leads nowhere, a sound FX that filters everything out or a
sound that did not load.
- **`clock`:** the state of the audio clock and the second it is at. While a pause holds it, the
state is the one it had as the pause took hold. `suspended` there means the browser held it while
the game ran: nothing was heard and notes stay `planned`, and `audio` says so in a `hint`. Chrome
starts it only once the page has had a click or a key, so after a reload the user clicks into the
editor once before the agent plays.
- **`levels`:** the master, each bus of the mix, each pattern, song or composition playing and the
tracks of a song, each with `peak` and `rms` in dB, or `silent` below -90 dB. A level is taken
after the volume and the pan of its bus, over the last `window` seconds (about 0.7, so the gap
between two notes is not taken for silence).
- **Held while paused.** A pause holds the audio clock, and with it the levels: `play` with `until`
and a read after it give the last 0.7 s before that moment.
- **`voices`:** what sounds now, a sound by its name (a clip as `Sound / Clip`), a note as its
instrument and note name (`E4`), the bake of a pattern or song as such, each with its bus and the
object it belongs to (`by`), `released` once its note is let go and `ringing` once only its echoes
are left. At most 40 are listed and the rest counted. `planned` counts the notes already set to
start a moment later.
- **`playing`:** the patterns, songs and compositions playing, with `ended`, `fading` or `baked`.
- **`waiting` and `failed`:** voices whose bytes have not come yet, and files that would not load.
- **Measured only while the agent is switched on.** Each bus then has a second tap beside the meter
of the Song tab, an analyser for each side; switched off, they are let go.
## How it works
```
@@ -341,7 +369,9 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
- `tests/editor/agent-play.test.ts`: a held key walks the object as far as the game seconds say, a
tapped one fires a key-down once, a kept one stays down across calls, a click on an object lands
and one beside it does not, a call that gives up when the game stands still and lets go of its key,
calls turned down before anything runs, `get_runtime`, and the camera hint. Pausing at a game
calls turned down before anything runs, `get_runtime` with what is heard (levels by bus name in
dB, the inner buses of a track left out, voices by sound or note, planned, waiting and failed),
and the camera hint. Pausing at a game
time: `play` with `until` landing on it exactly from a start, a resume and a pause, what it turns
down, giving up when the time stands still, a pause that came first, and a last `pause` step after
a wait and after a tap.
@@ -355,5 +385,6 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
the agent coming up in its tab, pixel art it paints read back as rows and its cel saved to the server, notes it writes read back by
name and a pattern and a sprite it lays on a composition opening in its tab, a sound you uploaded listed, made an asset nothing uses yet, and a preset filling an instrument, the object moving when played, the agent holding the arrow and clicking the
object itself and reading with `get_runtime` how far it went, `play` with `until` and a last
`pause` step holding the game at the time asked for, one Ctrl+Z taking it back, a delete waiting for
`pause` step holding the game at the time asked for, a sound looping on the master and on a bus
at 0 heard and silent, held as they are while paused, one Ctrl+Z taking it back, a delete waiting for
the user's Cancel; and the badge switching it off.