diff --git a/MCP.md b/MCP.md index 78b7a22..ce0d7e7 100644 --- a/MCP.md +++ b/MCP.md @@ -68,7 +68,7 @@ the badge says *Agent offline* meanwhile. | `select` | Selects an asset, objects of a scene or container, or nodes of a graph | | `play` | Play, pause, resume, stop. With `until`, the game runs on to that game time and holds there. Says so when no camera is on, since the game then shows nothing | | `play_input` | Keys and the mouse in the running game, step by step, see below | -| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, the last console lines | +| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, what is heard (`audio`), the last console lines | | `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once | There is also a resource, `feuerwerk://view`, with the same answer as `get_view`. An agent that @@ -247,6 +247,34 @@ it, then `get_runtime` to see what happened. - `get_view` warns when the scene Play starts has no camera switched on, and while playing when the game has none. +## Hearing the game + +The agent has no ears, so `get_runtime` tells it under `audio` what you would hear. Whether music +really sounds, it reads there, not only in the `silent` of `get_asset`, which is worked out from the +data and misses a bus at 0, a send that leads nowhere, a sound FX that filters everything out or a +sound that did not load. + +- **`clock`:** the state of the audio clock and the second it is at. While a pause holds it, the + state is the one it had as the pause took hold. `suspended` there means the browser held it while + the game ran: nothing was heard and notes stay `planned`, and `audio` says so in a `hint`. Chrome + starts it only once the page has had a click or a key, so after a reload the user clicks into the + editor once before the agent plays. +- **`levels`:** the master, each bus of the mix, each pattern, song or composition playing and the + tracks of a song, each with `peak` and `rms` in dB, or `silent` below -90 dB. A level is taken + after the volume and the pan of its bus, over the last `window` seconds (about 0.7, so the gap + between two notes is not taken for silence). +- **Held while paused.** A pause holds the audio clock, and with it the levels: `play` with `until` + and a read after it give the last 0.7 s before that moment. +- **`voices`:** what sounds now, a sound by its name (a clip as `Sound / Clip`), a note as its + instrument and note name (`E4`), the bake of a pattern or song as such, each with its bus and the + object it belongs to (`by`), `released` once its note is let go and `ringing` once only its echoes + are left. At most 40 are listed and the rest counted. `planned` counts the notes already set to + start a moment later. +- **`playing`:** the patterns, songs and compositions playing, with `ended`, `fading` or `baked`. +- **`waiting` and `failed`:** voices whose bytes have not come yet, and files that would not load. +- **Measured only while the agent is switched on.** Each bus then has a second tap beside the meter + of the Song tab, an analyser for each side; switched off, they are let go. + ## How it works ``` @@ -341,7 +369,9 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen - `tests/editor/agent-play.test.ts`: a held key walks the object as far as the game seconds say, a tapped one fires a key-down once, a kept one stays down across calls, a click on an object lands and one beside it does not, a call that gives up when the game stands still and lets go of its key, - calls turned down before anything runs, `get_runtime`, and the camera hint. Pausing at a game + calls turned down before anything runs, `get_runtime` with what is heard (levels by bus name in + dB, the inner buses of a track left out, voices by sound or note, planned, waiting and failed), + and the camera hint. Pausing at a game time: `play` with `until` landing on it exactly from a start, a resume and a pause, what it turns down, giving up when the time stands still, a pause that came first, and a last `pause` step after a wait and after a tap. @@ -355,5 +385,6 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen the agent coming up in its tab, pixel art it paints read back as rows and its cel saved to the server, notes it writes read back by name and a pattern and a sprite it lays on a composition opening in its tab, a sound you uploaded listed, made an asset nothing uses yet, and a preset filling an instrument, the object moving when played, the agent holding the arrow and clicking the object itself and reading with `get_runtime` how far it went, `play` with `until` and a last - `pause` step holding the game at the time asked for, one Ctrl+Z taking it back, a delete waiting for + `pause` step holding the game at the time asked for, a sound looping on the master and on a bus + at 0 heard and silent, held as they are while paused, one Ctrl+Z taking it back, a delete waiting for the user's Cancel; and the badge switching it off.