MCP: let the agent play the game and read its state (#429)
- Tool table with play_input and get_runtime. - New section Playing the game: steps, targets, game time, the hidden tab, the user's own mouse over the Game tab. - The camera hint in get_view and play. - Tests for the play stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent
3e7956c442
commit
c854b1ed07
1 file changed
+36
-4
+36
-4
@@ -65,7 +65,9 @@ the badge says *Agent offline* meanwhile.
|
||||
| `run_commands` | Changes the project, see below |
|
||||
| `open` | Brings up an asset's tab, or the scene or game tab |
|
||||
| `select` | Selects an asset, objects of a scene or container, or nodes of a graph |
|
||||
| `play` | Play, pause, resume, stop |
|
||||
| `play` | Play, pause, resume, stop. Says so when no camera is on, since the game then shows nothing |
|
||||
| `play_input` | Keys and the mouse in the running game, step by step, see below |
|
||||
| `get_runtime` | The running game: time, pause, camera, keys held, mouse, every object with its position (behaviors with their variables), the Variables assets, the last console lines |
|
||||
| `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once |
|
||||
|
||||
There is also a resource, `feuerwerk://view`, with the same answer as `get_view`. An agent that
|
||||
@@ -122,6 +124,31 @@ A behavior acts on the object it is placed under. The smallest walking object is
|
||||
sprite, a behavior placed with the sprite as `parent`, and in the behavior an `OnKeyHeld` whose
|
||||
`exec` and `dt` feed a `Move`, with an `Owner` on its `object`.
|
||||
|
||||
## Playing the game
|
||||
|
||||
The agent can test what it built without you pressing a key: `play`, then `play_input` to play
|
||||
it, then `get_runtime` to see what happened.
|
||||
|
||||
- **`play_input` takes steps, and each does one thing.** `{"key": "ArrowLeft", "hold": 0.5}` holds a
|
||||
key, `{"key": "Space"}` taps it for one frame, `"keep": "down"` leaves a key held after the call
|
||||
and `"keep": "up"` lets it go. `{"mouse": "click", "at": {"object": "<id>"}}` clicks an object,
|
||||
`"at": {"x": .., "y": ..}` a point in world units; `down`, `up` and `move` do just that.
|
||||
`{"wheel": -1}` turns the wheel, `{"wait": 1}` lets the game run on. Keys go by their
|
||||
`KeyboardEvent.key` name (`a`, `ArrowUp`, `Enter`), and `Space` is the space bar.
|
||||
- **Time is the game's.** Every step takes at least one frame, so a tap is seen as a press and then
|
||||
a release. Hold and wait count game seconds, at most 30 per call. A call over 5 seconds tells the
|
||||
relay it takes longer.
|
||||
- **What the answer tells:** the game time it took, the keys still held, where the mouse is,
|
||||
whether a click on an object landed on it, and the new console lines.
|
||||
- **A tapped or held key is let go again**, even when a step fails. Only `keep` leaves one down.
|
||||
- **It needs the game playing, not paused, and the editor tab in front.** A tab in the background
|
||||
runs no frames, so the call says so right away. If the game time stands still for 25 seconds
|
||||
(paused by a behavior, the tab sent to the background), the call stops and names the step.
|
||||
- **Your own mouse over the game view** still moves the game's mouse. While the agent clicks, leave
|
||||
it outside the Game tab.
|
||||
- `get_view` warns when the scene Play starts has no camera switched on, and while playing when the
|
||||
game has none.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
@@ -141,8 +168,8 @@ Claude Code --- MCP over HTTPS ---> /mcp ---> relay (tools/mcp/) <--- WebSocket
|
||||
`src/editor/agent/tools.json`, and passes a call to the tab of the token's account and the answer
|
||||
back. A call to an account without a tab on gets "no editor connected" and never reaches another
|
||||
account's tab.
|
||||
- A call has 30 seconds. One that asks you something (a delete) tells the relay to wait, and then
|
||||
has two minutes.
|
||||
- A call has 30 seconds. One that asks you something (a delete) or plays the game for more than 5
|
||||
seconds tells the relay to wait, and then has two minutes.
|
||||
- `tools.json` holds each tool's argument schema; the editor checks every call against it
|
||||
(`argCheck.ts`). The commands and their schemas are in `commandCatalog.ts`, the all-or-nothing
|
||||
undo step in `editPlan.ts`, the reading tools in `readTools.ts`.
|
||||
@@ -166,11 +193,16 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen
|
||||
the arrow keys" (no overlaps, no link over a node, none backwards, at most 8 crossings with one
|
||||
Owner for four moves and 4 with one each), chains running straight, nodes that stay, groups kept
|
||||
clear, loops, and what `lint_graph` counts.
|
||||
- `tests/editor/agent-play.test.ts`: a held key walks the object as far as the game seconds say, a
|
||||
tapped one fires a key-down once, a kept one stays down across calls, a click on an object lands
|
||||
and one beside it does not, a call that gives up when the game stands still and lets go of its key,
|
||||
calls turned down before anything runs, `get_runtime`, and the camera hint.
|
||||
- `tests/editor/graph-framing.test.ts`: how far the camera goes after nodes: not at all when they
|
||||
are in sight, just far enough otherwise, zooming out only for what does not fit.
|
||||
- e2e step 71 (`tools/e2e/71-mcp.mjs`): a token made in the dialog, the agent switched on in the
|
||||
account menu, an MCP client reading the view, hearing a tab switch and taking a screenshot; the
|
||||
agent building a walking object whose graph comes up with its nodes selected and arranged,
|
||||
Arrange in the header untangling it as one undo step, the open tab's camera going after a node
|
||||
added far away, the object moving when played, one Ctrl+Z taking it back, a delete waiting for
|
||||
added far away, the object moving when played, the agent holding the arrow and clicking the
|
||||
object itself and reading with `get_runtime` how far it went, one Ctrl+Z taking it back, a delete waiting for
|
||||
the user's Cancel; and the badge switching it off.
|
||||
Reference in new issue
Block a user