MCP: hold the game at a game time (#435)

admin committed 2026-10-18 21:27:00 +02:00
1 parent 286e3daa8d
commit 3b3179c9a0
1 file changed
+21 -4
+21 -4
@@ -66,7 +66,7 @@ the badge says *Agent offline* meanwhile.
| `run_commands` | Changes the project, see below |
| `open` | Brings up an asset's tab, or the scene or game tab |
| `select` | Selects an asset, objects of a scene or container, or nodes of a graph |
| `play` | Play, pause, resume, stop. Says so when no camera is on, since the game then shows nothing |
| `play` | Play, pause, resume, stop. With `until`, the game runs on to that game time and holds there. Says so when no camera is on, since the game then shows nothing |
| `play_input` | Keys and the mouse in the running game, step by step, see below |
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, the last console lines |
| `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once |
@@ -220,10 +220,23 @@ it, then `get_runtime` to see what happened.
`"at": {"x": .., "y": ..}` a point in world units; `down`, `up` and `move` do just that.
`{"wheel": -1}` turns the wheel, `{"wait": 1}` lets the game run on. Keys go by their
`KeyboardEvent.key` name (`a`, `ArrowUp`, `Enter`), and `Space` is the space bar.
- **A moment of the game is held, not caught.** Between two calls the game runs on, often for
seconds, so a call that only reads comes too late for a short moment. `play` with `until` (game
seconds, as `get_runtime` gives them, at most 30 ahead) lets the game run to that time and pauses
it there: `{"action": "play", "until": 0.5}` starts it, `resume` and `pause` go on from where it
is. The frame that would go past the time is cut short, so the game stands at 0.5 s exactly, not a
frame later. A last `play_input` step `{"pause": true}` does the same after the step before: after
a wait at its time exactly, after a key or a click at once. `get_runtime` and `screenshot` then
show that moment.
- **Until is called off** when the user pauses first, or a `Break` node does (the answer says it
paused before the time), and when a `Load scene` or `Restart game` starts the time over. A
recording not in real time keeps its own frames, so there the pause comes with the first frame at
or past the time.
- **Time is the game's.** Every step takes at least one frame, so a tap is seen as a press and then
a release. Hold and wait count game seconds, at most 30 per call. A call over 5 seconds tells the
relay it takes longer.
- **What the answer tells:** the game time it took, the keys still held, where the mouse is,
- **What the answer tells:** the game time it took and the one it is at, whether it is paused,
the keys still held, where the mouse is,
whether a click on an object landed on it, and the new console lines.
- **A tapped or held key is let go again**, even when a step fails. Only `keep` leaves one down.
- **It needs the game playing, not paused, and the editor tab in front.** A tab in the background
@@ -328,7 +341,10 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
- `tests/editor/agent-play.test.ts`: a held key walks the object as far as the game seconds say, a
tapped one fires a key-down once, a kept one stays down across calls, a click on an object lands
and one beside it does not, a call that gives up when the game stands still and lets go of its key,
calls turned down before anything runs, `get_runtime`, and the camera hint.
calls turned down before anything runs, `get_runtime`, and the camera hint. Pausing at a game
time: `play` with `until` landing on it exactly from a start, a resume and a pause, what it turns
down, giving up when the time stands still, a pause that came first, and a last `pause` step after
a wait and after a tap.
- `tests/editor/graph-framing.test.ts`: how far the camera goes after nodes: not at all when they
are in sight, just far enough otherwise, zooming out only for what does not fit.
- e2e step 71 (`tools/e2e/71-mcp.mjs`): a token made in the dialog, the agent switched on in the
@@ -338,5 +354,6 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
added far away, a Variables node with a port and a group framing every drawn node, an animation keyed by
the agent coming up in its tab, pixel art it paints read back as rows and its cel saved to the server, notes it writes read back by
name and a pattern and a sprite it lays on a composition opening in its tab, a sound you uploaded listed, made an asset nothing uses yet, and a preset filling an instrument, the object moving when played, the agent holding the arrow and clicking the
object itself and reading with `get_runtime` how far it went, one Ctrl+Z taking it back, a delete waiting for
object itself and reading with `get_runtime` how far it went, `play` with `until` and a last
`pause` step holding the game at the time asked for, one Ctrl+Z taking it back, a delete waiting for
the user's Cancel; and the badge switching it off.