MCP: what is heard, under audio in get_runtime (#436)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent
3b3179c9a0
commit
29891b37d3
1 file changed
+34
-3
+34
-3
@@ -68,7 +68,7 @@ the badge says *Agent offline* meanwhile.
|
||||
| `select` | Selects an asset, objects of a scene or container, or nodes of a graph |
|
||||
| `play` | Play, pause, resume, stop. With `until`, the game runs on to that game time and holds there. Says so when no camera is on, since the game then shows nothing |
|
||||
| `play_input` | Keys and the mouse in the running game, step by step, see below |
|
||||
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, the last console lines |
|
||||
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, what is heard (`audio`), the last console lines |
|
||||
| `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once |
|
||||
|
||||
There is also a resource, `feuerwerk://view`, with the same answer as `get_view`. An agent that
|
||||
@@ -247,6 +247,34 @@ it, then `get_runtime` to see what happened.
|
||||
- `get_view` warns when the scene Play starts has no camera switched on, and while playing when the
|
||||
game has none.
|
||||
|
||||
## Hearing the game
|
||||
|
||||
The agent has no ears, so `get_runtime` tells it under `audio` what you would hear. Whether music
|
||||
really sounds, it reads there, not only in the `silent` of `get_asset`, which is worked out from the
|
||||
data and misses a bus at 0, a send that leads nowhere, a sound FX that filters everything out or a
|
||||
sound that did not load.
|
||||
|
||||
- **`clock`:** the state of the audio clock and the second it is at. While a pause holds it, the
|
||||
state is the one it had as the pause took hold. `suspended` there means the browser held it while
|
||||
the game ran: nothing was heard and notes stay `planned`, and `audio` says so in a `hint`. Chrome
|
||||
starts it only once the page has had a click or a key, so after a reload the user clicks into the
|
||||
editor once before the agent plays.
|
||||
- **`levels`:** the master, each bus of the mix, each pattern, song or composition playing and the
|
||||
tracks of a song, each with `peak` and `rms` in dB, or `silent` below -90 dB. A level is taken
|
||||
after the volume and the pan of its bus, over the last `window` seconds (about 0.7, so the gap
|
||||
between two notes is not taken for silence).
|
||||
- **Held while paused.** A pause holds the audio clock, and with it the levels: `play` with `until`
|
||||
and a read after it give the last 0.7 s before that moment.
|
||||
- **`voices`:** what sounds now, a sound by its name (a clip as `Sound / Clip`), a note as its
|
||||
instrument and note name (`E4`), the bake of a pattern or song as such, each with its bus and the
|
||||
object it belongs to (`by`), `released` once its note is let go and `ringing` once only its echoes
|
||||
are left. At most 40 are listed and the rest counted. `planned` counts the notes already set to
|
||||
start a moment later.
|
||||
- **`playing`:** the patterns, songs and compositions playing, with `ended`, `fading` or `baked`.
|
||||
- **`waiting` and `failed`:** voices whose bytes have not come yet, and files that would not load.
|
||||
- **Measured only while the agent is switched on.** Each bus then has a second tap beside the meter
|
||||
of the Song tab, an analyser for each side; switched off, they are let go.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
@@ -341,7 +369,9 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
|
||||
- `tests/editor/agent-play.test.ts`: a held key walks the object as far as the game seconds say, a
|
||||
tapped one fires a key-down once, a kept one stays down across calls, a click on an object lands
|
||||
and one beside it does not, a call that gives up when the game stands still and lets go of its key,
|
||||
calls turned down before anything runs, `get_runtime`, and the camera hint. Pausing at a game
|
||||
calls turned down before anything runs, `get_runtime` with what is heard (levels by bus name in
|
||||
dB, the inner buses of a track left out, voices by sound or note, planned, waiting and failed),
|
||||
and the camera hint. Pausing at a game
|
||||
time: `play` with `until` landing on it exactly from a start, a resume and a pause, what it turns
|
||||
down, giving up when the time stands still, a pause that came first, and a last `pause` step after
|
||||
a wait and after a tap.
|
||||
@@ -355,5 +385,6 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
|
||||
the agent coming up in its tab, pixel art it paints read back as rows and its cel saved to the server, notes it writes read back by
|
||||
name and a pattern and a sprite it lays on a composition opening in its tab, a sound you uploaded listed, made an asset nothing uses yet, and a preset filling an instrument, the object moving when played, the agent holding the arrow and clicking the
|
||||
object itself and reading with `get_runtime` how far it went, `play` with `until` and a last
|
||||
`pause` step holding the game at the time asked for, one Ctrl+Z taking it back, a delete waiting for
|
||||
`pause` step holding the game at the time asked for, a sound looping on the master and on a bus
|
||||
at 0 heard and silent, held as they are while paused, one Ctrl+Z taking it back, a delete waiting for
|
||||
the user's Cancel; and the badge switching it off.
|
||||
Reference in new issue
Block a user