MCP: let the agent play the game and read its state (#429)

- Tool table with play_input and get_runtime.
- New section Playing the game: steps, targets, game time, the hidden
  tab, the user's own mouse over the Game tab.
- The camera hint in get_view and play.
- Tests for the play stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adminandClaude Opus 5.5 committed 2026-10-18 15:01:46 +02:00
1 parent 3e7956c442
commit c854b1ed07
1 file changed
+36 -4
+36 -4
@@ -65,7 +65,9 @@ the badge says *Agent offline* meanwhile.
| `run_commands` | Changes the project, see below |
| `open` | Brings up an asset's tab, or the scene or game tab |
| `select` | Selects an asset, objects of a scene or container, or nodes of a graph |
| `play` | Play, pause, resume, stop |
| `play` | Play, pause, resume, stop. Says so when no camera is on, since the game then shows nothing |
| `play_input` | Keys and the mouse in the running game, step by step, see below |
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, the last console lines |
| `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once |
There is also a resource, `feuerwerk://view`, with the same answer as `get_view`. An agent that
@@ -122,6 +124,31 @@ A behavior acts on the object it is placed under. The smallest walking object is
sprite, a behavior placed with the sprite as `parent`, and in the behavior an `OnKeyHeld` whose
`exec` and `dt` feed a `Move`, with an `Owner` on its `object`.
## Playing the game
The agent can test what it built without you pressing a key: `play`, then `play_input` to play
it, then `get_runtime` to see what happened.
- **`play_input` takes steps, and each does one thing.** `{"key": "ArrowLeft", "hold": 0.5}` holds a
key, `{"key": "Space"}` taps it for one frame, `"keep": "down"` leaves a key held after the call
and `"keep": "up"` lets it go. `{"mouse": "click", "at": {"object": "<id>"}}` clicks an object,
`"at": {"x": .., "y": ..}` a point in world units; `down`, `up` and `move` do just that.
`{"wheel": -1}` turns the wheel, `{"wait": 1}` lets the game run on. Keys go by their
`KeyboardEvent.key` name (`a`, `ArrowUp`, `Enter`), and `Space` is the space bar.
- **Time is the game's.** Every step takes at least one frame, so a tap is seen as a press and then
a release. Hold and wait count game seconds, at most 30 per call. A call over 5 seconds tells the
relay it takes longer.
- **What the answer tells:** the game time it took, the keys still held, where the mouse is,
whether a click on an object landed on it, and the new console lines.
- **A tapped or held key is let go again**, even when a step fails. Only `keep` leaves one down.
- **It needs the game playing, not paused, and the editor tab in front.** A tab in the background
runs no frames, so the call says so right away. If the game time stands still for 25 seconds
(paused by a behavior, the tab sent to the background), the call stops and names the step.
- **Your own mouse over the game view** still moves the game's mouse. While the agent clicks, leave
it outside the Game tab.
- `get_view` warns when the scene Play starts has no camera switched on, and while playing when the
game has none.
## How it works
```
@@ -141,8 +168,8 @@ Claude Code --- MCP over HTTPS ---> /mcp ---> relay (tools/mcp/) <--- WebSocket
`src/editor/agent/tools.json`, and passes a call to the tab of the token's account and the answer
back. A call to an account without a tab on gets "no editor connected" and never reaches another
account's tab.
- A call has 30 seconds. One that asks you something (a delete) tells the relay to wait, and then
has two minutes.
- A call has 30 seconds. One that asks you something (a delete) or plays the game for more than 5
seconds tells the relay to wait, and then has two minutes.
- `tools.json` holds each tool's argument schema; the editor checks every call against it
(`argCheck.ts`). The commands and their schemas are in `commandCatalog.ts`, the all-or-nothing
undo step in `editPlan.ts`, the reading tools in `readTools.ts`.
@@ -166,11 +193,16 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
the arrow keys" (no overlaps, no link over a node, none backwards, at most 8 crossings with one
Owner for four moves and 4 with one each), chains running straight, nodes that stay, groups kept
clear, loops, and what `lint_graph` counts.
- `tests/editor/agent-play.test.ts`: a held key walks the object as far as the game seconds say, a
tapped one fires a key-down once, a kept one stays down across calls, a click on an object lands
and one beside it does not, a call that gives up when the game stands still and lets go of its key,
calls turned down before anything runs, `get_runtime`, and the camera hint.
- `tests/editor/graph-framing.test.ts`: how far the camera goes after nodes: not at all when they
are in sight, just far enough otherwise, zooming out only for what does not fit.
- e2e step 71 (`tools/e2e/71-mcp.mjs`): a token made in the dialog, the agent switched on in the
account menu, an MCP client reading the view, hearing a tab switch and taking a screenshot; the
agent building a walking object whose graph comes up with its nodes selected and arranged,
Arrange in the header untangling it as one undo step, the open tab's camera going after a node
added far away, the object moving when played, one Ctrl+Z taking it back, a delete waiting for
added far away, the object moving when played, the agent holding the arrow and clicking the
object itself and reading with `get_runtime` how far it went, one Ctrl+Z taking it back, a delete waiting for
the user's Cancel; and the badge switching it off.