Let the agent play the game and read its state through MCP (#429)
An agent could build a game object but not see whether it works. It had to ask the user to press the keys. Now it can play the game in the editor itself, send keys and mouse input and read what happened. - play_input sends steps into the running game: hold or tap a key, keep it down or let it go, click, press, release or move the mouse, turn the wheel, wait. Holding and waiting count game seconds. - A mouse target is a point in world units or an object by id or name. A click on an object reports whether it landed on it. - The answer names the game time used, the keys still held, the mouse and the new console lines. - One tool instead of press_keys and pointer from the plan. A step list can then mix both, like holding Shift during a click. - get_runtime reads the running game: objects with id, type and position, the own variables of their behaviors, the Variables assets, the camera, keys, mouse and the console. Named objects come with all their properties. - A hidden editor tab gets no frames. play_input says so at once. If game time stands still for 25 s it stops, names the step and lets go of every key it held. - get_view and play say when the scene has no active camera. The first agent test missed that and only saw it on a screenshot. - The relay instructions tell the agent to test what it built. Found in the live test on a real project: - A click on an object by its name reported hit false. The hit test looked the target up by id only. - get_runtime listed the built-in switches of a behavior (enabled, runInEditor) as its variables. Tests: - agent-play (new), agent-edit, agent-view, relay green. - 5 counter-checks red: hold counts one frame, key never released, click ignores the object, hidden tab not refused, no variable filter. - e2e 71: 21/21. The agent holds ArrowLeft 0.5 s (x -120 to -270) and clicks the object (hit true). - Gate: CHECK EXIT 0, 319 files, 4944 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f296f427ef
commit
1943539a7a
12 files changed
+843
-4
No files matched your search
@@ -43,6 +43,9 @@ const INSTRUCTIONS = [
|
||||
'Get or Owner node next to its user, not over a long link; check with lint_graph.',
|
||||
'The editor follows your edits: after run_commands the user sees the graph, scene or asset it worked',
|
||||
'on, with what it touched selected. Use open and select to show anything else.',
|
||||
'Test what you built yourself: play, then play_input sends keys and mouse clicks into the game,',
|
||||
'and get_runtime reads where the objects are, their variables and the console. A game needs a',
|
||||
'camera in its scene to show anything; get_view says when it has none.',
|
||||
'When a tool answers that no editor is connected, ask the user to open the editor and switch on',
|
||||
'"Agent" in its account menu.',
|
||||
].join(' ');
|
||||
|
||||
Reference in new issue
Block a user