Let the agent hold the game at a given game time (#435)

The agent could not look at a short moment of a running game. Its
calls took longer than the moment lasted, so an intro window from
0.5 to 2 s slipped past twice in the 5d test. Now the editor itself
stops the clock at the time the agent names.

- play takes until: play, resume or pause let the game run on to
  that game time and hold it there. The call answers once the game
  is paused, so get_runtime and screenshot show that moment.
- The frame that would step past the time is cut short
  (frameWithin in src/core/time.ts, used in PlaySession). The game
  stands at 0.5 s, not one frame later.
- play_input takes a last step {"pause": true}. After a wait it
  holds the game at the time of the wait, after a key or click at
  once. Its answer names the game time and whether it is paused.
- A pause that comes first (toolbar or a Break node), a Load scene
  or a Restart game call the until off, and the answer says so.
- When the clock stands still, the call gives up after 25 s with
  the time it is at. A tab in the background is refused at once.
- A recording that does not run in real time is not cut. There the
  game stops in the first frame that reaches the time.
- get_runtime and play_input give times to the millisecond, not to
  the hundredth, so 0.505 no longer reads as 0.51.
- The relay's instructions name both ways.

Tests:
- Unit agent-play (6 new) and time (2 new): 19/19, related editor
  and relay tests 1158/1158.
- 6 counter-checks, all compile, all red. One stayed green at
  first: 0.3 s is exactly 18 frames. The test now takes 0.31 s.
- e2e 71 (atx): ALL PASSED, 27 checks. New 71h3 holds at 0.505 s
  and 0.8 s and after a wait of 0.31 s. Without the cut in
  PlaySession it went red at 0.5166 s.
- Gate (atx): CHECK EXIT 0, 327 files, 5033 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/editor/Editor.ts

@@ -1668,9 +1668,10 @@ export class Editor implements PlayHost {
     // and so does a run that waits for its videos at the start (#404).
     // A single step out of the pause moves by one fixed step, however long the frame took.
     // A recording not in real time (#423) moves it by one frame of the recording, or holds it.
+    // A pause the agent asked for at a time (#435) cuts the frame that would go past it.
     const stepping = this.play.beginFrame();
     const recorded = stepping ? null : this.play.recordingStep();
-    const frame = stepping ? FIXED_STEP : (recorded ?? dt);
+    const frame = stepping ? FIXED_STEP : (recorded ?? this.play.frameBeforePause(dt, gameClock.scale));
     const held = this.state.paused && !stepping;
     const step = gameClock.advance(frame, held || this.play.isGamePaused || this.play.isWaiting);
     this.play.tick(step, timeStep(frame, gameClock.scale, held));

src/editor/PlaySession.ts

@@ -12,7 +12,7 @@ import { type BehaviorSystem, behaviorError, type Trace } from '../runtime/syste
 import type { AnimationPlayer } from '../core/model/animation';
 import type { PropertyBag } from '../core/model/Properties';
 import { VIDEO_WAIT_MS } from '../core/model/video';
-import { gameClock } from '../core/time';
+import { frameWithin, gameClock } from '../core/time';
 import { loadPhysics, readyPhysics } from '../runtime/physics/engine';
 import { Input } from '../runtime/systems/Input';
 import type { World } from '../runtime/World';
@@ -95,6 +95,8 @@ export class PlaySession {
   private stepping = false;
   /** The object whose `Break` node paused the run, until it is resumed or stepped on. */
   private breakAt: string | null = null;
+  /** The time the agent wants the run held at (#435), on the world it named; a `Load scene` calls it off. */
+  private pausing: { world: World; time: number } | null = null;
 
   constructor(
     private readonly state: EditorState,
@@ -162,10 +164,24 @@ export class PlaySession {
     }
     this.state.setPaused(!this.state.paused);
     this.breakAt = null;
+    this.pausing = null;
     this.applyPause(true);
     ui().focus.blur();
   }
 
+  /** Holds the run once its time reaches `time` (#435), as if the pause button had been pressed; null calls it off. */
+  pauseAt(time: number | null): void {
+    this.pausing = time === null || !this.world ? null : { world: this.world, time };
+  }
+
+  /** Called before the clock moves: the frame cut short so the run does not go past the time `pauseAt` named. */
+  frameBeforePause(dt: number, scale: number): number {
+    const pausing = this.pausing;
+    if (!pausing || pausing.world !== this.world || this.state.paused || this.isGamePaused || this.isWaiting)
+      return dt;
+    return frameWithin(dt, scale, pausing.time - pausing.world.time);
+  }
+
   /** Lets a paused run on by exactly one frame, at the next tick; does nothing unless paused. */
   step(): void {
     if (this.isPaused) this.stepAsked = true;
@@ -356,6 +372,7 @@ export class PlaySession {
     this.videoWait = null;
     this.stepAsked = false;
     this.breakAt = null;
+    this.pausing = null;
     const world = this.world;
     this.run = null;
     this.input.detach();
@@ -420,6 +437,21 @@ export class PlaySession {
     // A game that pauses itself is looked at like one paused from the toolbar, and so is a step.
     this.follow((world.paused && !wasPaused) || this.stepping ? this.panels.flush() : update);
     this.pauseAtBreak();
+    this.pauseWhenDue();
+  }
+
+  /** The run reached the time `pauseAt` named: it holds there. A pause that came first calls it off. */
+  private pauseWhenDue(): void {
+    const pausing = this.pausing;
+    if (!pausing) return;
+    if (pausing.world !== this.world || this.state.paused) {
+      this.pausing = null;
+      return;
+    }
+    if (pausing.world.time < pausing.time - 1e-9) return;
+    this.pausing = null;
+    this.state.setPaused(true);
+    this.applyPause(true);
   }
 
   /**

src/editor/agent/agentTools.ts

@@ -11,12 +11,16 @@ import type { StepTarget } from './commandCatalog';
 import { cameraOfWorld } from '../../viewport/activeCamera';
 import { pixelCels } from '../../viewport/PixelCels';
 import {
+  exact,
+  IN_BACKGROUND,
   type InputStep,
   inputProblem,
   inputRefusal,
   LONG_DEADLINE_MS,
   LONG_INPUT_SECONDS,
+  MAX_INPUT_SECONDS,
   type PlayDriver,
+  runUntil,
   SHORT_DEADLINE_MS,
   sendInput,
   totalSeconds,
@@ -164,6 +168,7 @@ async function run(name: string, args: Record<string, unknown>, host: ToolHost):
     case 'select':
       return outcome(host.select(args as unknown as AgentSelection), 'Selected.');
     case 'play': {
+      if (args.until !== undefined) return playUntil(args.action as PlayAction, args.until as number, host);
       const refusal = host.play(args.action as PlayAction);
       if (refusal !== null) return failure(refusal);
       const facts = args.action === 'play' ? host.runtime?.() : null;
@@ -215,6 +220,41 @@ async function playInput(steps: InputStep[], host: ToolHost): Promise<ToolResult
   return answer(await sendInput(steps, host.game, long ? LONG_DEADLINE_MS : SHORT_DEADLINE_MS));
 }
 
+/** `play`, `resume` or `pause` with `until` (#435): the game runs on to that time and holds there. */
+async function playUntil(action: PlayAction, until: number, host: ToolHost): Promise<ToolResult> {
+  const game = host.game;
+  if (!game) return failure('This editor plays no game');
+  if (action === 'stop') return failure('until goes with play, resume or pause');
+  if (game.hidden()) return failure(IN_BACKGROUND);
+  const run = action === 'play' ? null : game.run();
+  if (action !== 'play') {
+    if (!run) return failure('The game is not running: start it with play first');
+    if (action === 'pause' && game.paused())
+      return failure('The game is paused already: resume it with until');
+    if (action === 'resume' && !game.paused())
+      return failure('The game is running already: pause it with until');
+  }
+  const now = run?.world.time ?? 0;
+  if (until <= now + 1e-6) return failure(`The game is at ${exact(now)} s already: until has to lie ahead`);
+  const ahead = until - now;
+  if (ahead > MAX_INPUT_SECONDS)
+    return failure(`until is ${exact(ahead)} s ahead; at most ${MAX_INPUT_SECONDS} s go in one call`);
+  if (action !== 'pause') {
+    const refusal = host.play(action);
+    if (refusal !== null) return failure(refusal);
+  }
+  const long = ahead > LONG_INPUT_SECONDS;
+  if (long) host.takesLong?.();
+  const time = await runUntil(game, until, long ? LONG_DEADLINE_MS : SHORT_DEADLINE_MS);
+  const facts = action === 'play' ? host.runtime?.() : null;
+  const blind = facts && !cameraOfWorld(facts.world) ? ` ${NO_CAMERA_HINT}` : '';
+  const text =
+    time < until - 1e-6
+      ? `Paused at ${time} s of game time, before ${until} s: the user or a Break node paused it first.`
+      : `Paused at ${time} s of game time.${blind}`;
+  return done(text, { time, paused: true });
+}
+
 async function runCommands(args: Record<string, unknown>, host: ToolHost): Promise<ToolResult> {
   const steps = args.steps as EditStep[];
   const deletes = deletedAssets(steps);

src/editor/agent/editorToolHost.ts

@@ -25,6 +25,7 @@ export interface EditorParts {
     play(paused: boolean): void;
     stop(): void;
     togglePause(): void;
+    pauseAt(time: number | null): void;
     agentRun(): { world: World; input: Input; shared: SharedState } | null;
   };
   /** The user's "Follow agent" switch. */
@@ -82,6 +83,10 @@ export function editorToolHost(parts: EditorParts, wait: () => void): ToolHost {
       hidden: () => document.visibilityState === 'hidden',
       frame: () => nextFrame(parts.app),
       now: () => performance.now(),
+      pause: () => {
+        if (parts.play.isPlaying && !state.paused) parts.play.togglePause();
+      },
+      pauseAt: (time) => parts.play.pauseAt(time),
     },
     runtime: () => {
       const run = parts.play.agentRun();

src/editor/agent/playInput.ts

@@ -14,6 +14,8 @@ export interface InputStep {
   at?: { x?: number; y?: number; object?: string };
   wheel?: number;
   wait?: number;
+  /** Holds the game after the step before (#435); it comes last. */
+  pause?: boolean;
 }
 
 /** What `sendInput` needs of the editor. */
@@ -28,6 +30,10 @@ export interface PlayDriver {
   frame(): Promise<void>;
   /** Wall-clock milliseconds. */
   now(): number;
+  /** Pauses the run, as the toolbar does. */
+  pause(): void;
+  /** Pauses the run once its time reaches `time`, with the frame before cut short to land on it; null calls it off. */
+  pauseAt(time: number | null): void;
 }
 
 export interface InputOutcome {
@@ -38,6 +44,9 @@ export interface InputOutcome {
   held: string[];
   /** Where the mouse is, in world units, and whether it is over the game's picture. */
   mouse: { x: number; y: number; inside: boolean; down: boolean };
+  /** The game's time when the steps were done, and whether a `pause` step holds it there. */
+  time: number;
+  paused: boolean;
   /** What the console got meanwhile. */
   log: string[];
   /** Per click on an object: whether it landed on it. */
@@ -63,8 +72,12 @@ export function keyOf(name: string): string {
 export function inputProblem(steps: readonly InputStep[]): string | null {
   for (const [index, step] of steps.entries()) {
     const at = `Step ${index + 1}`;
-    const kinds = (['key', 'mouse', 'wheel', 'wait'] as const).filter((kind) => step[kind] !== undefined);
-    if (kinds.length !== 1) return `${at} has to do one thing: key, mouse, wheel or wait`;
+    const kinds = (['key', 'mouse', 'wheel', 'wait', 'pause'] as const).filter(
+      (kind) => step[kind] !== undefined,
+    );
+    if (kinds.length !== 1) return `${at} has to do one thing: key, mouse, wheel, wait or pause`;
+    if (step.pause !== undefined && (step.pause !== true || index !== steps.length - 1))
+      return `${at}: {"pause": true} ends the steps, it comes last`;
     if (step.key === undefined && (step.hold !== undefined || step.keep !== undefined))
       return `${at}: hold and keep go with a key`;
     if (step.hold !== undefined && step.keep !== undefined) return `${at}: a key is held or kept, not both`;
@@ -89,11 +102,13 @@ export function totalSeconds(steps: readonly InputStep[]): number {
 export function inputRefusal(driver: PlayDriver): string | null {
   if (!driver.run()) return 'The game is not running: start it with play first';
   if (driver.paused()) return 'The game is paused: resume it first';
-  if (driver.hidden())
-    return 'The editor tab is in the background, where the browser runs no frames: ask the user to bring it to the front';
+  if (driver.hidden()) return IN_BACKGROUND;
   return null;
 }
 
+export const IN_BACKGROUND =
+  'The editor tab is in the background, where the browser runs no frames: ask the user to bring it to the front';
+
 /**
  * Plays the steps into the running game, each one taking at least one frame, so an edge a step
  * makes is seen before the next one. A tapped or held key and a clicked button are let go again
@@ -109,28 +124,70 @@ export async function sendInput(
   const logStart = first.world.log.length;
   const startWorld = first.world;
   const hits: InputOutcome['hits'] = [];
-  for (const [index, step] of steps.entries()) {
-    try {
-      await playStep(step, clock, (object, hit) => hits.push({ step: index + 1, object, hit }));
-    } catch (error) {
-      const why = error instanceof Error ? error.message : String(error);
-      throw new Error(`Step ${index + 1}: ${why} (after ${round(clock.seconds)} s of game time)`, {
-        cause: error,
-      });
+  const pauses = steps.at(-1)?.pause === true;
+  try {
+    for (const [index, step] of steps.entries()) {
+      try {
+        if (step.pause) {
+          if (!driver.paused()) driver.pause();
+          continue;
+        }
+        // A wait the pause comes after ends on its time exactly, not a frame beyond.
+        if (pauses && index === steps.length - 2 && (step.wait ?? 0) > 0)
+          driver.pauseAt(clock.run().world.time + step.wait!);
+        await playStep(step, clock, (object, hit) => hits.push({ step: index + 1, object, hit }));
+      } catch (error) {
+        const why = error instanceof Error ? error.message : String(error);
+        throw new Error(`Step ${index + 1}: ${why} (after ${round(clock.seconds)} s of game time)`, {
+          cause: error,
+        });
+      }
     }
+  } finally {
+    if (pauses) driver.pauseAt(null);
   }
   const { world, input } = clock.run();
   const { x, y, inside, down } = world.pointer;
   return {
-    seconds: round(clock.seconds),
+    seconds: exact(clock.seconds),
     frames: clock.frames,
     held: [...input.heldKeys],
     mouse: { x: round(x), y: round(y), inside, down },
+    time: exact(world.time),
+    paused: driver.paused(),
     log: world === startWorld ? world.log.slice(logStart) : [...world.log],
     hits,
   };
 }
 
+/**
+ * Lets the run go on until its time reaches `until` and holds it there (#435): the time it holds at,
+ * earlier when the user or a `Break` node paused it first.
+ */
+export async function runUntil(driver: PlayDriver, until: number, deadlineMs: number): Promise<number> {
+  const deadline = driver.now() + deadlineMs;
+  const world = driver.run()?.world;
+  driver.pauseAt(until);
+  try {
+    for (;;) {
+      const run = driver.run();
+      if (!run) throw new Error('The game stopped before it got there');
+      if (run.world !== world)
+        throw new Error(
+          'A Load scene or Restart game started the time over before it got there; the game runs on',
+        );
+      if (driver.paused()) return exact(run.world.time);
+      if (driver.now() > deadline)
+        throw new Error(
+          `The game time stood still at ${exact(run.world.time)} s: paused by a behavior, or the editor tab went to the background`,
+        );
+      await driver.frame();
+    }
+  } finally {
+    driver.pauseAt(null);
+  }
+}
+
 async function playStep(
   step: InputStep,
   clock: GameClock,
@@ -255,3 +312,8 @@ class GameClock {
 function round(value: number): number {
   return Math.round(value * 100) / 100;
 }
+
+/** A game time to the millisecond: a pause lands on it, and a hundredth would hide a frame. */
+export function exact(value: number): number {
+  return Math.round(value * 1000) / 1000;
+}

src/editor/agent/runtimeView.ts

@@ -8,6 +8,7 @@ import type { Entity } from '../../runtime/Entity';
 import type { World } from '../../runtime/World';
 import { cameraOfItems, cameraOfWorld } from '../../viewport/activeCamera';
 import { collectHolderItems } from '../../viewport/items';
+import { exact } from './playInput';
 
 /** The running game as `get_runtime` reads it (#429). */
 export interface RuntimeFacts {
@@ -55,7 +56,7 @@ export function runtimeView(
   const lines = options.console ?? CONSOLE_LINES;
   return {
     scene: { id: world.sceneId, name: findAsset(project, world.sceneId)?.name ?? '' },
-    time: round(world.time),
+    time: exact(world.time),
     paused: facts.paused ? 'toolbar' : world.paused ? 'by the game' : false,
     camera: camera
       ? { id: camera.id, x: round(camera.x), y: round(camera.y), ...cameraFrame(camera.props) }

src/editor/agent/tools.json

@@ -187,10 +187,17 @@
   {
     "name": "play",
     "title": "Play the game",
-    "description": "Plays the game in the editor, pauses, resumes or stops it. Edits made while it runs change the running copy only and are gone when it stops.",
+    "description": "Plays the game in the editor, pauses, resumes or stops it. With until, play, resume or pause let the game run on to that game time, as get_runtime gives it, and hold it there, exactly: the call answers once it is paused, and get_runtime and screenshot then show that moment. Edits made while it runs change the running copy only and are gone when it stops.",
     "inputSchema": {
       "type": "object",
-      "properties": { "action": { "type": "string", "enum": ["play", "pause", "resume", "stop"] } },
+      "properties": {
+        "action": { "type": "string", "enum": ["play", "pause", "resume", "stop"] },
+        "until": {
+          "type": "number",
+          "minimum": 0,
+          "description": "Game seconds to pause at, at most 30 ahead. A Load scene or Restart game starts the time over and calls it off."
+        }
+      },
       "required": ["action"],
       "additionalProperties": false
     },
@@ -199,7 +206,7 @@
   {
     "name": "play_input",
     "title": "Keys and mouse in the running game",
-    "description": "Plays keys and the mouse into the running game like a player, step by step; each step takes at least one frame, and hold and wait count game seconds. A step does one thing: {\"key\": \"ArrowLeft\", \"hold\": 0.5} holds a key; {\"key\": \"Space\"} taps it for one frame; {\"key\": \"Shift\", \"keep\": \"down\"} presses it and leaves it held after the call, \"keep\": \"up\" lets it go; {\"mouse\": \"click\", \"at\": {\"object\": \"<object id>\"}} or {\"mouse\": \"click\", \"at\": {\"x\": 10, \"y\": -20}} clicks the left button there (world units, as get_runtime gives them); \"down\", \"up\" and \"move\" do just that; {\"wheel\": -1} turns the wheel; {\"wait\": 1} lets the game run on. Keys go by their KeyboardEvent.key name: \"a\", \"ArrowUp\", \"Enter\", \"Space\". Answers the game time it took, the keys still held, where the mouse is, whether a click on an object landed on it, and the new console lines; read positions and variables with get_runtime. Needs the game playing, not paused, and the editor tab in front.",
+    "description": "Plays keys and the mouse into the running game like a player, step by step; each step takes at least one frame, and hold and wait count game seconds. A step does one thing: {\"key\": \"ArrowLeft\", \"hold\": 0.5} holds a key; {\"key\": \"Space\"} taps it for one frame; {\"key\": \"Shift\", \"keep\": \"down\"} presses it and leaves it held after the call, \"keep\": \"up\" lets it go; {\"mouse\": \"click\", \"at\": {\"object\": \"<object id>\"}} or {\"mouse\": \"click\", \"at\": {\"x\": 10, \"y\": -20}} clicks the left button there (world units, as get_runtime gives them); \"down\", \"up\" and \"move\" do just that; {\"wheel\": -1} turns the wheel; {\"wait\": 1} lets the game run on; {\"pause\": true}, the last step, holds the game right after the step before, a wait to its time exactly, so get_runtime and screenshot show that moment. Keys go by their KeyboardEvent.key name: \"a\", \"ArrowUp\", \"Enter\", \"Space\". Answers the game time it took and the game time it is at, whether it is paused, the keys still held, where the mouse is, whether a click on an object landed on it, and the new console lines; read positions and variables with get_runtime. Needs the game playing, not paused, and the editor tab in front.",
     "inputSchema": {
       "type": "object",
       "properties": {
@@ -228,7 +235,8 @@
                 "additionalProperties": false
               },
               "wheel": { "type": "integer", "minimum": -10, "maximum": 10 },
-              "wait": { "type": "number", "minimum": 0, "maximum": 10, "description": "Game seconds." }
+              "wait": { "type": "number", "minimum": 0, "maximum": 10, "description": "Game seconds." },
+              "pause": { "type": "boolean", "description": "true: holds the game here; the last step." }
             },
             "additionalProperties": false
           }