From b8f8921f17d54d84e6b55710399c36f39a5dc928 Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Sun, 18 Oct 2026 11:09:00 +0200 Subject: [PATCH] MCP: let an AI agent read and edit the open project (#429) Stage 2 of #429 gives the agent tools to read the project and change it, so the MCP page now explains them. - MCP: new section "Changing the project" with run_commands, the "as"/"$name" references, one undo step per call, the rollback and the delete question. The tool list names the read tools, catalog, open, select, play and undo/redo. - Architecture: the new modules in src/editor/agent/. Co-Authored-By: Claude Opus 5.5 --- Architecture.md | 6 ++++- MCP.md | 59 +++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 60 insertions(+), 5 deletions(-) diff --git a/Architecture.md b/Architecture.md index bea961e..0b90f49 100644 --- a/Architecture.md +++ b/Architecture.md @@ -337,7 +337,11 @@ src/editor/ App → Editor (layout + ticker), PlaySession (the run), auth/LoginScreen + auth/PasswordDialog + auth/loginRules + auth/FormStatus (the sign-in screen and the change-password dialog in the canvas, what either may complain about, and the warning line and busy state both show, #177/#182), session (what a guest or an account may do, `sessionAllows`, #428), guestStores (a guest's stores: they list what the server offers and refuse every write before a request is made, #428), autosave, agent/ (#429: AgentLink, the tab's WebSocket to the MCP relay with its retries; agentView, what `get_view` answers and the one line the agent is told when it changes, as plain functions; - agentTools, the tools of tools.json, which the relay reads as well; screenshot; AgentTokensDialog), + agentTools, the tools of tools.json, which the relay reads as well, checked by argCheck; + commandCatalog, the commands an agent runs, each with its argument schema; editPlan, AgentEdit, + the steps of one call as one undo step, built one after another, all or nothing; readTools, + list_assets, get_asset, get_graph and catalog as plain functions of the project; editorToolHost, + the editor behind the tools and the question before a delete; screenshot; AgentTokensDialog), polygonEdit (points of an outline: what is under the pointer, whether a point is inside, put / move / take away — scene view and Texture tab; and the same for a ring of a vector art shape, where a point may carry Bezier handles: `ringGripAt`, splitting a curved segment diff --git a/MCP.md b/MCP.md index d0001e3..af6a465 100644 --- a/MCP.md +++ b/MCP.md @@ -3,7 +3,8 @@ tab**, not on the stored project: you see everything it does as it happens, and with Ctrl+Z like your own work. It speaks MCP (the Model Context Protocol) to a relay on the server, and the relay passes its tool calls to your tab. -So far the agent sees what you see; changing the project comes next (see the issue). +The agent sees what you see, reads the project and changes it through the same commands your +clicks run. Laying out the graphs it builds comes next (see the issue). ## Setting it up @@ -51,8 +52,17 @@ the badge says *Agent offline* meanwhile. | Tool | What it does | |---|---| -| `get_view` | What you are looking at: the project, the tab and the document it shows, the selection with names (objects, nodes by type, frames, keys ...), the camera of the view, whether the game is playing or paused, tabs in windows of their own, and the menu or dialog on top | +| `get_view` | What you are looking at: the project, the tab and the document it shows, the selection with names (objects, nodes by type, frames, keys ...), the camera of the view, whether the game is playing or paused, tabs in windows of their own, and the menu or dialog on top (a menu by its first entries) | | `screenshot` | The whole editor as drawn, as a PNG, at most `maxWidth` pixels wide (1280 unless asked otherwise) | +| `list_assets` | The assets with ids, names and types, by type or a part of the name | +| `get_asset` | One asset as stored, with the rows of its properties and the named objects of a scene or container. A graph is only counted, pixels and map cells too | +| `get_graph` | A graph: every node with its settings and its ports as they are now, the edges, the groups and notes | +| `catalog` | What there is to make: the commands with their arguments, the nodes a graph can hold (all of them, or the best matches of a search in full), the asset types and their properties, the variable kinds | +| `run_commands` | Changes the project, see below | +| `open` | Brings up an asset's tab, or the scene or game tab | +| `select` | Selects an asset, objects of a scene or container, or nodes of a graph | +| `play` | Play, pause, resume, stop | +| `undo`, `redo` | Like Ctrl+Z and Ctrl+Y, up to 20 steps at once | There is also a resource, `feuerwerk://view`, with the same answer as `get_view`. An agent that subscribes to it hears when you switch the tab, the document or the selection, start or stop @@ -60,6 +70,38 @@ playing, or open a dialog; moving the camera is no news. Not every client follow so the relay also tells the agent with its next tool answer that the view has changed since it last looked, in one line. +## Changing the project + +`run_commands` takes a list of steps, each a command of the catalog with its arguments, and runs +them **as one undo step**: in the Edit history it is called `Agent: ` and the label the agent gave +it. One Ctrl+Z takes the whole call back. + +- **All or nothing.** The arguments of every step are checked against the command's schema; if a + step fails, the steps before it are taken back and the answer names the step and why + (`Step 8 (connect): Cannot connect number to exec`). Nothing of a failed call stays. +- **Steps build on each other.** A step with `"as": "walk"` gives later steps its id as `"$walk"` + (and `"$walk.closing"` for the second node a Find adds). Each step is built after the one before + has run, so one call can make a behavior and fill its graph. +- **Nodes come from the palette.** `add_node` takes the id of an entry of the catalog for that + graph: `OnKeyHeld`, `Move`, `RunBehavior:` for another behavior. Without `x` and `y` a node + goes right of everything the graph holds; the real layout comes with the next stage. +- **Deleting asks you.** A call that deletes an asset that was there before opens a dialog + *The agent asks: Delete "..."?* with the same text as your own Delete. The call waits for your + answer, up to 110 seconds; Cancel, Escape or no answer leave everything as it was. +- **While the game runs** a change goes into the running copy, as yours would, and the answer says + it is gone when the game stops. + +The 23 commands so far: assets (`create_asset`, `rename_asset`, `duplicate_asset`, +`delete_asset`, `set_asset_property`, `reset_asset_property`, `set_start_scene`), variables +(`add_variable`, `remove_variable`, `rename_variable`), placed objects (`place_asset`, +`remove_objects`, `move_object`, `rename_object`, `set_override`, `clear_override`, `set_parent`) +and graphs (`add_node`, `remove_node`, `move_node`, `set_node_param`, `connect`, `disconnect`). +Textures, fonts, sounds and videos need a file and cannot be made by an agent. + +A behavior acts on the object it is placed under. The smallest walking object is therefore a +sprite, a behavior placed with the sprite as `parent`, and in the behavior an `OnKeyHeld` whose +`exec` and `dt` feed a `Move`, with an `Owner` on its `object`. + ## How it works ``` @@ -79,6 +121,11 @@ Claude Code --- MCP over HTTPS ---> /mcp ---> relay (tools/mcp/) <--- WebSocket `src/editor/agent/tools.json`, and passes a call to the tab of the token's account and the answer back. A call to an account without a tab on gets "no editor connected" and never reaches another account's tab. +- A call has 30 seconds. One that asks you something (a delete) tells the relay to wait, and then + has two minutes. +- `tools.json` holds each tool's argument schema; the editor checks every call against it + (`argCheck.ts`). The commands and their schemas are in `commandCatalog.ts`, the all-or-nothing + undo step in `editPlan.ts`, the reading tools in `readTools.ts`. What is behind it on the server, and the routes it uses: [API](API), [Development](Development). @@ -90,6 +137,10 @@ What is behind it on the server, and the routes it uses: [API](API), [Developmen - `tests/php/agents.test.php`: tokens and tickets on disk. `tests/php/rights.test.php` asks the new routes as everybody, with and without the relay's key. - `tests/editor/agent-view.test.ts`: what `get_view` answers, its one line, the tools' arguments. +- `tests/editor/agent-edit.test.ts`: a walking object built in one call and taken back whole by one + undo, a failing step that leaves nothing, argument checks, deletes that ask, the reading tools + and the catalog. - e2e step 71 (`tools/e2e/71-mcp.mjs`): a token made in the dialog, the agent switched on in the - account menu, an MCP client reading the view, hearing a tab switch and taking a screenshot, - and the badge switching it off. + account menu, an MCP client reading the view, hearing a tab switch and taking a screenshot; the + agent building a walking object that moves when played, one Ctrl+Z taking it back, a delete + waiting for the user's Cancel; and the badge switching it off.