Let the agent read what is heard in the running game (#436)

The agent has no ears. It could start music and sounds, but not tell
whether anything reached the speakers: a song without an instrument, a
bus at volume 0 or a browser that holds the audio clock all looked the
same from outside.

- get_runtime has a section audio. Under levels it gives the peak and
  RMS in dB of the master, each bus of the mix, each pattern, song or
  composition playing and the tracks of a song. Below -90 dB it says
  silent.
- The levels come from a tap of their own after the level and pan of
  each bus, an AnalyserNode per side as for the meters of the Song tab.
  It looks back about 0.7 s, so the gap between two notes is not
  silence. A pause holds the values, which fits play with until.
- Under voices it names what sounds now: a sound by its asset, a note
  of an instrument with its note name, each with its bus and object.
  Next to them planned, playing, waiting (bytes not here yet) and
  failed.
- Under clock it gives the state of the audio clock, with a hint when
  it stands still while the game runs. The state is told as it was
  before a pause, else the pause would hide that case.
- The buses are tapped only while the agent is switched on.
- The note now travels with a voice from the Start of a pattern or song
  to the SoundLibrary, so a voice can be named by it.
- tools.json and the relay's instructions tell the agent to check under
  audio that music and sounds are really heard.

Found on the user's project: started right after F5, before a click in
the page, the browser kept the audio clock suspended for the whole run.
Nothing was heard and the notes stayed planned. That is what clock now
shows.

Tests:
- Unit: sound-library (taps, levels, voices, clock), agent-play
  (audioView). Pattern and song tests expect the note in their timing.
- 9 counter-checks, 8 red. Not dropping a tap with its bus stays green,
  as the bus is cut off and gone anyway.
- e2e 71 (melektaus-atx): 28/28. New 71h4: a sound on the master and
  one on a bus at volume 0, both voices named, the master loud, the bus
  silent, the values held in the pause.
- Gate (melektaus-atx): CHECK EXIT 0, 327 files, 5037 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/editor/Editor.ts

@@ -984,6 +984,7 @@ export class Editor implements PlayHost {
               play: this.play,
               follows: followsAgent,
               stores,
+              heard: () => soundLibrary.heard(),
             },
             wait,
           ),
@@ -993,7 +994,11 @@ export class Editor implements PlayHost {
         this.state.setNotice('Agent switched off: another tab of your account has it now');
       },
     });
-    link.onChange = () => this.syncAgentBadge();
+    link.onChange = () => {
+      this.syncAgentBadge();
+      // The buses are tapped for what the agent hears only while it is switched on (#436).
+      soundLibrary.watch(link.on);
+    };
     return link;
   }
 

src/editor/agent/audioView.ts

@@ -0,0 +1,171 @@
+import { noteName } from '../../core/instrument/notes';
+import { findAsset, type Project } from '../../core/model/Project';
+import { clipsOf } from '../../core/model/sound';
+import { MASTER_MIX } from '../../core/music/mixer';
+import { playedSoundOf } from '../../core/properties/playedSound';
+import type { World } from '../../runtime/World';
+import type { HeardSound, HeardVoice } from '../../viewport/SoundLibrary';
+
+/** Below this a bus counts as silent: far under anything a speaker gives out. */
+export const SILENT_DB = -90;
+/** The most voices an answer lists; the rest are counted. */
+export const MAX_VOICES = 40;
+
+export const STALLED_HINT =
+  'The browser holds the audio clock while the game runs, so nothing is heard and notes stay planned. ' +
+  'The browser starts it only once the user has clicked or pressed a key in the editor page, and not ' +
+  'without a sound output: ask the user.';
+
+const INSTRUMENT = 'instrument:';
+const PLAYING_BUS = /^play:(\d+)(?::(.*))?$/;
+/** The bus of a track of a song played on its own: `play:<n>:song:<song>:<track>`. */
+const SONG_TRACK = /^song:([^:]+):([^:#]+)$/;
+
+/**
+ * What `get_runtime` says of the sound (#436): the level of the master and of every bus that is
+ * heard by itself, the voices sounding now with what they play and who asked, the patterns, songs
+ * and compositions playing, and what is not heard for lack of its bytes. Names stand in for ids.
+ */
+export function audioView(heard: HeardSound, project: Project, world: World): object {
+  const names = new SoundNames(project, heard, world);
+  const sounding = heard.voices.filter((voice) => voice.startsIn === 0);
+  const planned = heard.voices.length - sounding.length;
+  // A pause holds the clock; its state is the one it had while the game ran.
+  const { state, time } = heard.clock;
+  return {
+    clock: { state, time: Math.round(time * 1000) / 1000 },
+    ...(state === 'suspended' && { hint: STALLED_HINT }),
+    window: Math.round(heard.window * 100) / 100,
+    levels: heard.buses
+      .filter((bus) => shownBus(bus.name))
+      .sort((a, b) => Number(b.name === MASTER_MIX) - Number(a.name === MASTER_MIX))
+      .map(({ name, peak, rms }) => levelRow(names.bus(name), peak, rms)),
+    voices: sounding.slice(0, MAX_VOICES).map((voice) => ({
+      ...names.voice(voice),
+      ...(voice.released && { released: true }),
+      ...(voice.ringing && { ringing: true }),
+    })),
+    ...(sounding.length > MAX_VOICES && { moreVoices: sounding.length - MAX_VOICES }),
+    ...(planned > 0 && { planned }),
+    playing: heard.playings.map((playing) => ({
+      ...names.playing(playing.name),
+      ...(names.by(playing.owner) && { by: names.by(playing.owner) }),
+      ...(!playing.playing && { ended: true }),
+      ...(playing.fading !== null && { fading: true }),
+      ...(playing.baked && { baked: true }),
+    })),
+    ...(heard.waiting.length > 0 && { waiting: heard.waiting.map((voice) => names.voice(voice)) }),
+    ...(heard.failed.length > 0 && { failed: heard.failed.map((file) => names.file(file)) }),
+  };
+}
+
+/**
+ * The buses worth a level of their own: the master, those of the mix, the master of each playing
+ * and the tracks of a song. What lies inside a track – its items, its own bus for them – is heard on
+ * the track.
+ */
+function shownBus(name: string): boolean {
+  if (name === MASTER_MIX || name.startsWith('mix:')) return true;
+  const playing = PLAYING_BUS.exec(name);
+  return playing !== null && (playing[2] === undefined || SONG_TRACK.test(playing[2]));
+}
+
+function levelRow(bus: string, peak: readonly number[], rms: readonly number[]): object {
+  const loudest = decibels(Math.max(...peak));
+  if (loudest < SILENT_DB) return { bus, silent: true };
+  const power = rms.reduce((sum, side) => sum + side * side, 0) / rms.length;
+  return { bus, peak: round(loudest), rms: round(decibels(Math.sqrt(power))) };
+}
+
+function decibels(gain: number): number {
+  return gain > 0 ? 20 * Math.log10(gain) : -Infinity;
+}
+
+function round(db: number): number {
+  return Number.isFinite(db) ? Math.round(db * 10) / 10 : SILENT_DB;
+}
+
+/** Names for what the sink knows by file, by bus and by the id of an entity. */
+class SoundNames {
+  private readonly files = new Map<string, Record<string, string>>();
+  private readonly playings: ReadonlyMap<number, string>;
+
+  constructor(
+    private readonly project: Project,
+    heard: HeardSound,
+    private readonly world: World,
+  ) {
+    for (const asset of project.assets) {
+      if (asset.type === 'sound') {
+        this.files.set(asset.file, { sound: asset.name });
+        for (const clip of clipsOf(asset))
+          this.files.set(playedSoundOf({ asset, clip }), { sound: `${asset.name} / ${clip.name}` });
+      }
+      if ((asset.type === 'pattern' || asset.type === 'song') && asset.bake)
+        this.files.set(asset.bake.file, { bake: `${asset.type} ${asset.name}` });
+    }
+    this.playings = new Map(heard.playings.map(({ id, name }) => [id, this.label(name)]));
+  }
+
+  voice(voice: HeardVoice): object {
+    const by = this.by(voice.owner);
+    return {
+      ...this.file(voice.file, voice.note),
+      bus: this.bus(voice.bus),
+      ...(by && { by }),
+    };
+  }
+
+  /** What a file plays: a sound or a clip of one, a note of an instrument, the bake of a pattern or song. */
+  file(file: string, note?: number): Record<string, string> {
+    if (file.startsWith(INSTRUMENT)) {
+      const id = file.slice(INSTRUMENT.length);
+      return {
+        instrument: findAsset(this.project, id)?.name ?? id,
+        ...(note !== undefined && { note: noteName(note) }),
+      };
+    }
+    return this.files.get(file) ?? { file };
+  }
+
+  bus(name: string): string {
+    if (name === '' || name === MASTER_MIX) return 'master';
+    if (name.startsWith('mix:')) {
+      const id = name.slice('mix:'.length);
+      return findAsset(this.project, id)?.name ?? id;
+    }
+    const playing = PLAYING_BUS.exec(name);
+    if (!playing) return name;
+    const of = this.playings.get(Number(playing[1])) ?? `playing ${playing[1]}`;
+    const track = SONG_TRACK.exec(playing[2] ?? '');
+    if (!track) return of;
+    const song = findAsset(this.project, track[1]!);
+    const named = song?.type === 'song' ? song.tracks.find((t) => t.id === track[2]) : undefined;
+    return `${of}, track ${named?.name ?? track[2]}`;
+  }
+
+  playing(name: string): Record<string, string> {
+    const [kind, id] = splitName(name);
+    return kind ? { [kind]: findAsset(this.project, id)?.name ?? id } : { name };
+  }
+
+  /** The object a voice's behavior sits on, or the behavior itself; '' for no owner. */
+  by(owner: string): string {
+    if (owner === '') return '';
+    const entity = this.world.entities.find((e) => e.id === owner);
+    if (!entity) return owner;
+    return entity.parent?.name ?? entity.name;
+  }
+
+  private label(name: string): string {
+    const [kind, id] = splitName(name);
+    return kind ? `${kind} ${findAsset(this.project, id)?.name ?? id}` : name;
+  }
+}
+
+/** `pattern:<id>`, `song:<id>` or `composition:<id>` as kind and id; no kind for anything else. */
+function splitName(name: string): [string, string] {
+  const at = name.indexOf(':');
+  const kind = at > 0 ? name.slice(0, at) : '';
+  return ['pattern', 'song', 'composition'].includes(kind) ? [kind, name.slice(at + 1)] : ['', name];
+}

src/editor/agent/editorToolHost.ts

@@ -13,6 +13,7 @@ import type { StepTarget } from './commandCatalog';
 import { screenshot } from './screenshot';
 import type { FileAssetType } from './storedFiles';
 import type { FileStore } from '../../core/store/FileStore';
+import type { HeardSound } from '../../viewport/SoundLibrary';
 
 /** What the agent's tools reach of the editor beyond its state. */
 export interface EditorParts {
@@ -32,6 +33,8 @@ export interface EditorParts {
   follows(): boolean;
   /** Where the files of each kind are kept (#431). */
   stores: Record<FileAssetType, FileStore>;
+  /** What is heard now (#436). */
+  heard(): HeardSound;
 }
 
 /** How long the user has to answer an agent's question before it counts as a no. */
@@ -92,7 +95,14 @@ export function editorToolHost(parts: EditorParts, wait: () => void): ToolHost {
       const run = parts.play.agentRun();
       if (!run) return null;
       const { world, input, shared } = run;
-      return { world, shared, project: state.project, heldKeys: input.heldKeys, paused: state.paused };
+      return {
+        world,
+        shared,
+        project: state.project,
+        heldKeys: input.heldKeys,
+        paused: state.paused,
+        audio: parts.heard(),
+      };
     },
     takesLong: wait,
     files: (type) => parts.stores[type].list(),

src/editor/agent/runtimeView.ts

@@ -8,6 +8,8 @@ import type { Entity } from '../../runtime/Entity';
 import type { World } from '../../runtime/World';
 import { cameraOfItems, cameraOfWorld } from '../../viewport/activeCamera';
 import { collectHolderItems } from '../../viewport/items';
+import type { HeardSound } from '../../viewport/SoundLibrary';
+import { audioView } from './audioView';
 import { exact } from './playInput';
 
 /** The running game as `get_runtime` reads it (#429). */
@@ -19,6 +21,8 @@ export interface RuntimeFacts {
   heldKeys: ReadonlySet<string>;
   /** Paused from the toolbar; the game's own pause is the world's. */
   paused: boolean;
+  /** What is heard (#436); left out where the run plays to nobody. */
+  audio?: HeardSound;
 }
 
 /** How many console lines an answer carries unless the agent asks for more. */
@@ -66,6 +70,7 @@ export function runtimeView(
     mouse: { ...world.pointer, x: round(world.pointer.x), y: round(world.pointer.y) },
     objects: picked.map((entity) => entityRow(entity, project, named.length > 0)),
     variables: sharedRows(facts.shared, project),
+    ...(facts.audio && { audio: audioView(facts.audio, project, world) }),
     console: lines > 0 ? world.log.slice(-lines) : [],
     consoleLines: world.log.length,
   };

src/editor/agent/tools.json

@@ -250,7 +250,7 @@
   {
     "name": "get_runtime",
     "title": "Read the running game",
-    "description": "The game as it runs now: game time, whether it is paused, the camera, the keys held and the mouse, every object with its id, type and position in world units (behaviors with their variables), the values of the Variables assets, and the last console lines. Name objects (ids or names) to get all their properties.",
+    "description": "The game as it runs now: game time, whether it is paused, the camera, the keys held and the mouse, every object with its id, type and position in world units (behaviors with their variables), the values of the Variables assets, the sound (audio), and the last console lines. Under audio: the audio clock (with a hint when it stands still while the game runs: then nothing is heard); the level of the master, of each bus of the mix, of each pattern, song or composition playing and of the tracks of a song, as peak and RMS in dB over the last window seconds (silent below -90 dB; held while the game is paused); the voices sounding now (a sound, or an instrument with its note), each with its bus and the object it belongs to (by); planned, the notes already set to start; the patterns, songs and compositions playing; waiting, the voices whose bytes have not arrived; and failed, the files that would not load. Name objects (ids or names) to get all their properties.",
     "inputSchema": {
       "type": "object",
       "properties": {

src/runtime/systems/Audio.ts

@@ -119,6 +119,8 @@ export interface NoteTiming {
   tempo?: number;
   /** The bus it is heard on: a track of a song (#334) or one of the mix (#325); the master where left out. */
   bus?: string;
+  /** The note it is, for whoever asks what is heard (#436). */
+  note?: number;
 }
 
 /** When and how a clip of a song plays (#334); see `playClip`. */

src/runtime/systems/behaviorContext.ts

@@ -349,14 +349,18 @@ export function playNote(
 ): number {
   const asset = findAssetOf(project, instrumentId, 'instrument');
   if (!asset) return 0;
-  const built = instrumentBuildFor(asset, heldNote(struck.note), instrumentSourcesOf(project)).layers;
+  const note = heldNote(struck.note);
+  const built = instrumentBuildFor(asset, note, instrumentSourcesOf(project)).layers;
   if (built.length === 0) return 0;
   const layers = fx ? built.map((layer) => throughScopedFx(fx.plan, layer, FX_SCOPE)) : built;
   const velocity = Number.isFinite(struck.velocity) ? Math.max(0, Math.min(1, struck.velocity)) : 1;
   const level = Math.max(0, Math.min(1, struck.volume * velocity));
   const duration = Number.isFinite(struck.duration) ? Math.max(0, struck.duration) : 0;
   const values = { ...params, [VELOCITY_PARAM]: velocity, ...scoped(fx?.params ?? {}) };
-  return audio.playNote(instrumentVoiceName(instrumentId), layers, values, level, owner, duration, timing);
+  return audio.playNote(instrumentVoiceName(instrumentId), layers, values, level, owner, duration, {
+    ...timing,
+    note,
+  });
 }
 
 /** Where the parameters of a note's own sound FX go by (#346): no name of a parameter has a colon. */

src/viewport/SoundLibrary.ts

@@ -79,6 +79,11 @@ const MAX_SEND_DELAY = 1;
 
 /** Samples a level meter looks back over (#359): a frame of 40 ms at 48 kHz misses no peak. */
 const METER_SPAN = 2048;
+/**
+ * Samples what the agent hears looks back over (#436): the most an analyser holds, about 0.7 s, so
+ * the gap between two notes does not read as silence.
+ */
+const WATCH_SPAN = 32768;
 /** The loudest the limiter at the output lets a sample be (#359), and how fast it lets go again. */
 const GUARD_DB = -1;
 const GUARD_RELEASE = 0.1;
@@ -157,6 +162,8 @@ interface Voice {
   owner: string;
   /** The bus it is heard on, `''` straight out. */
   bus: string;
+  /** The note it is, for a note of an instrument (#436). */
+  note?: number;
   /** When it started on the audio clock, what its Envelopes count from. */
   at: number;
   sources: Source[];
@@ -252,6 +259,8 @@ interface Start {
   bus?: string;
   /** The copy `file` streams from (#349); its bytes are not loaded then, its head is. */
   stream?: StreamFile;
+  /** The note it is, for a note of an instrument (#436). */
+  note?: number;
 }
 
 /**
@@ -260,6 +269,10 @@ interface Start {
  */
 interface Pending {
   id: number;
+  /** What it plays and who asked, as its voice will have them (#436). */
+  file: string;
+  owner: string;
+  note?: number;
   bus: string;
   fx: Layered;
   timed: { params: FxParams; time: number; ease: EaseShare }[];
@@ -298,6 +311,13 @@ export class SoundLibrary implements AudioSink {
    */
   private guard: { input: GainNode; limiter: AudioWorkletNode | null; latency: number } | null = null;
   private guardMeter: BusMeter | null = null;
+  /** Whether every bus is tapped for what the agent hears (#436). */
+  private watching = false;
+  /**
+   * The state of the audio clock as a pause took hold (#436): one the browser had suspended already
+   * stays so after the pause, and nothing was heard before it either.
+   */
+  private heldFrom = 'none';
 
   /**
    * `guarded` false leaves the limiter out: for checks that follow every node of a voice. With
@@ -509,7 +529,20 @@ export class SoundLibrary implements AudioSink {
     const fx = { layers, params, tempo: timing.tempo ?? DEFAULT_TEMPO };
     const at = timing.at ?? 0;
     const bus = timing.bus ?? '';
-    return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration, at, bus });
+    const note = timing.note;
+    return this.begin({
+      id: 0,
+      file: name,
+      files,
+      volume,
+      loop: false,
+      owner,
+      fx,
+      duration,
+      at,
+      bus,
+      ...(note !== undefined && { note }),
+    });
   }
 
   /**
@@ -716,23 +749,23 @@ export class SoundLibrary implements AudioSink {
     return [...this.buses.values()].filter((bus) => bus.meter).length;
   }
 
-  private tapMeter(from: AudioNode): BusMeter {
+  private tapMeter(from: AudioNode, span = METER_SPAN): BusMeter {
     const context = this.audio();
     const split = context.createChannelSplitter(2);
     const sides = [0, 1].map((channel) => {
       const side = context.createAnalyser();
-      side.fftSize = METER_SPAN;
+      side.fftSize = span;
       split.connect(side, channel);
       return side;
     });
     from.connect(split);
-    return { split, sides, samples: new Float32Array(METER_SPAN) };
+    return { split, sides, samples: new Float32Array(span) };
   }
 
-  private dropMeter(bus: Bus): void {
+  private dropMeter(bus: Bus, which: 'meter' | 'watch' = 'meter'): void {
-    const meter = bus.meter;
+    const meter = bus[which];
     if (!meter) return;
-    bus.meter = null;
+    bus[which] = null;
     try {
       bus.pan.disconnect(meter.split);
     } catch {
@@ -741,10 +774,63 @@ export class SoundLibrary implements AudioSink {
     meter.split.disconnect();
   }
 
-  /** Cuts the pan of a bus from where it went; its meter stays on. */
+  /** Cuts the pan of a bus from where it went; its meters stay on. */
   private cutPan(bus: Bus): void {
     bus.pan.disconnect();
     if (bus.meter) bus.pan.connect(bus.meter.split);
+    if (bus.watch) bus.pan.connect(bus.watch.split);
+  }
+
+  /**
+   * Taps every bus for what the agent hears (#436), those made from now on as well, or lets the taps
+   * go: on only while the agent is switched on, so the editor pays nothing for it otherwise.
+   */
+  watch(on: boolean): void {
+    if (this.watching === on) return;
+    this.watching = on;
+    for (const bus of this.buses.values()) {
+      if (!on) this.dropMeter(bus, 'watch');
+      else bus.watch ??= this.tapMeter(bus.pan, WATCH_SPAN);
+    }
+  }
+
+  /**
+   * What is heard now (#436): every voice and playing, the voices still waiting on their bytes, the
+   * files that would not load, and the level of each bus tapped by `watch` over the last
+   * `WATCH_SPAN` samples, after its level and pan – held where a pause holds the clock.
+   */
+  heard(): HeardSound {
+    const now = this.context?.currentTime ?? 0;
+    const voiceOf = (voice: Voice | Pending): HeardVoice => ({
+      id: voice.id,
+      file: voice.file,
+      owner: voice.owner,
+      bus: voice.bus,
+      ...(voice.note !== undefined && { note: voice.note }),
+    });
+    return {
+      window: this.context ? WATCH_SPAN / this.context.sampleRate : 0,
+      clock: { state: this.held ? this.heldFrom : this.clockState(), time: now, held: this.held },
+      voices: this.voices.map((voice) => ({
+        ...voiceOf(voice),
+        startsIn: Math.max(0, voice.at - now),
+        released: voice.released,
+        ringing: voice.ringing,
+      })),
+      waiting: this.pending.filter((waiting) => !waiting.stopped).map(voiceOf),
+      failed: [...this.failed],
+      playings: this.playbacks().map(({ id, name, owner, playing, fading, baked }) => ({
+        id,
+        name,
+        owner,
+        playing,
+        fading,
+        baked,
+      })),
+      buses: [...this.buses].flatMap(([name, bus]) =>
+        bus.watch ? [{ name, ...readLevels(bus.watch) }] : [],
+      ),
+    };
   }
 
   /** Where the voices of a bus go in; the master for `''` (#325). */
@@ -776,8 +862,10 @@ export class SoundLibrary implements AudioSink {
       chain: null,
       leaving: 0,
       meter: null,
+      watch: null,
     };
     made.input.connect(made.gain).connect(made.pan).connect(this.outOf(target));
+    if (this.watching) made.watch = this.tapMeter(made.pan, WATCH_SPAN);
     this.buses.set(name, made);
     return made;
   }
@@ -1134,6 +1222,7 @@ export class SoundLibrary implements AudioSink {
 
   private dropBus(name: string, bus: Bus): void {
     this.dropMeter(bus);
+    this.dropMeter(bus, 'watch');
     this.dropSends(bus);
     this.unchain(bus);
     bus.input.disconnect();
@@ -1565,6 +1654,9 @@ export class SoundLibrary implements AudioSink {
     const askedAt = this.cancelsFor(start.file, start.owner);
     const waiting: Pending = {
       id,
+      file: start.file,
+      owner: start.owner,
+      ...(start.note !== undefined && { note: start.note }),
       bus: start.bus ?? '',
       fx: { ...start.fx, params: { ...start.fx.params } },
       timed: [],
@@ -1738,6 +1830,7 @@ export class SoundLibrary implements AudioSink {
    * wakes the clock – a pattern or song reading it every few milliseconds would undo the pause.
    */
   setSuspended(suspended: boolean): void {
+    if (suspended && !this.held) this.heldFrom = this.clockState();
     this.held = suspended;
     const context = this.live();
     if (!context) return;
@@ -2212,6 +2305,7 @@ export class SoundLibrary implements AudioSink {
     clip: asked,
     bus = '',
     stream,
+    note,
   }: Start): void {
     const context = this.audio();
     // A context starts suspended where the page has not been interacted with; without this the
@@ -2279,6 +2373,7 @@ export class SoundLibrary implements AudioSink {
       file,
       owner,
       bus,
+      ...(note !== undefined && { note }),
       at: when,
       sources,
       ending: new Set(),
@@ -2946,6 +3041,8 @@ interface Bus {
   leaving: number;
   /** Its level meter (#359), while one is asked after. */
   meter: BusMeter | null;
+  /** What the agent hears of it (#436), while the agent is switched on. */
+  watch: BusMeter | null;
 }
 
 /** A tap after the pan of a bus: an analyser for each side, so a peak panned hard is not halved. */
@@ -3073,6 +3170,64 @@ function readMeter(meter: BusMeter): [number, number] {
   return [left!, right!];
 }
 
+/**
+ * The loudest sample and the RMS of each side over the stretch a meter looks back over (#436), 1 at
+ * full scale.
+ */
+function readLevels(meter: BusMeter): { peak: [number, number]; rms: [number, number] } {
+  const [left, right] = meter.sides.map((side) => {
+    side.getFloatTimeDomainData(meter.samples);
+    let peak = 0;
+    let squares = 0;
+    for (const sample of meter.samples) {
+      peak = Math.max(peak, Math.abs(sample));
+      squares += sample * sample;
+    }
+    return { peak, rms: Math.sqrt(squares / meter.samples.length) };
+  });
+  return { peak: [left!.peak, right!.peak], rms: [left!.rms, right!.rms] };
+}
+
+/** One voice as `heard` gives it (#436): a sound by its file, a note by `instrument:<id>` and its note. */
+export interface HeardVoice {
+  id: number;
+  file: string;
+  /** The behavior it belongs to, `''` for none. */
+  owner: string;
+  /** Its bus, `''` the master. */
+  bus: string;
+  note?: number;
+}
+
+/** What `SoundLibrary.heard` gives (#436). */
+export interface HeardSound {
+  /** Seconds the levels look back over; 0 before there is any sound. */
+  window: number;
+  /**
+   * The audio clock: `running`, `suspended` or `none` before there is any sound – while a pause holds
+   * it, as it was when the pause took hold –, the second it is at, and whether a pause holds it.
+   * Suspended, nothing is heard.
+   */
+  clock: { state: string; time: number; held: boolean };
+  /** Voices started, a planned one `startsIn` seconds from now. */
+  voices: (HeardVoice & { startsIn: number; released: boolean; ringing: boolean })[];
+  /** Voices waiting on their bytes. */
+  waiting: HeardVoice[];
+  /** Files that would not load. */
+  failed: string[];
+  /** Patterns, songs and compositions playing, by `pattern:<id>`, `song:<id>`, `composition:<id>`. */
+  playings: {
+    id: number;
+    name: string;
+    owner: string;
+    playing: boolean;
+    fading: number | null;
+    baked: boolean;
+  }[];
+  /** The tapped buses by their name in the sink: `mix`, `mix:<id>`, `play:<n>`, `play:<n>:<track>`. */
+  buses: { name: string; peak: [number, number]; rms: [number, number] }[];
+}
+
 export const soundLibrary = new SoundLibrary();
 
 /** A composition fits an audio item to the length of its sound once the bytes are here (#418). */

tests/core/pattern-play.test.ts

@@ -236,7 +236,15 @@ describe('Play pattern', () => {
     expect([asked?.owner, asked?.play.loop]).toEqual(['', null]);
     asked!.play.strike({ at: 3, duration: 0.25, note: 60, velocity: 0.5, instrument: f.lead.id, tempo: 90 });
     expect(audio.notes).toEqual([
-      { name: `instrument:${f.lead.id}`, volume: 0.25, owner: '', duration: 0.25, at: 3, tempo: 90 },
+      {
+        name: `instrument:${f.lead.id}`,
+        volume: 0.25,
+        owner: '',
+        duration: 0.25,
+        at: 3,
+        tempo: 90,
+        note: 60,
+      },
     ]);
   });
 

tests/core/song-play.test.ts

@@ -341,7 +341,7 @@ describe('Play song', () => {
       tempo: 90,
     });
     expect(audio.notes).toEqual([
-      { name: `instrument:${f.lead.id}`, volume: 1, owner: '', at: 3, tempo: 90, bus },
+      { name: `instrument:${f.lead.id}`, volume: 1, owner: '', at: 3, tempo: 90, bus, note: 60 },
     ]);
     expect(audio.clips).toEqual([
       { file: FILE, clip: { at: 4, offset: 0.5, duration: 2, volume: 0.5, bus } },

tests/editor/agent-play.test.ts

@@ -7,6 +7,8 @@ import type { AgentView } from '../../src/editor/agent/agentView';
 import { frameWithin } from '../../src/core/time';
 import { type PlayDriver, pointOf, sendInput } from '../../src/editor/agent/playInput';
 import { cameraHint, NO_CAMERA_HINT } from '../../src/editor/agent/runtimeView';
+import type { HeardSound } from '../../src/viewport/SoundLibrary';
+import { STALLED_HINT } from '../../src/editor/agent/audioView';
 import {
   graphEditor,
   makeBehavior,
@@ -352,6 +354,96 @@ describe('get_runtime', () => {
     );
     expect(text(await runTool('get_runtime', {}, hostFor(f, started, driver, false)))).toMatch(/not running/);
   });
+
+  it('says what is heard: levels in dB by bus name, voices and playings by asset, the rest left out (#436)', async () => {
+    const { f, started, driver } = game();
+    const made = (type: 'sound' | 'instrument' | 'song' | 'bus', name: string) => {
+      const command = new CreateAssetCommand(type, name);
+      f.stack.execute(command);
+      return command.asset;
+    };
+    const sound = made('sound', 'Jump');
+    if (sound.type === 'sound') sound.file = 'jump.wav';
+    const instrument = made('instrument', 'Lead');
+    const song = made('song', 'Theme');
+    const track = song.type === 'song' ? song.tracks[0]! : null;
+    const bus = made('bus', 'Music');
+    const owner = started.world.entities.find((entity) => entity.type === 'behavior')!.id;
+    const trackBus = `play:7:song:${song.id}:${track!.id}`;
+    const quiet = { released: false, ringing: false, startsIn: 0 };
+    const level = (peak: number, rms: number): Pick<HeardSound['buses'][number], 'peak' | 'rms'> => ({
+      peak: [peak, peak / 2],
+      rms: [rms, rms],
+    });
+    const heard: HeardSound = {
+      window: 32768 / 48000,
+      clock: { state: 'running', time: 12.34567, held: false },
+      voices: [
+        { id: 1, file: 'jump.wav', owner, bus: '', ...quiet },
+        {
+          id: 2,
+          file: `instrument:${instrument.id}`,
+          owner: '',
+          bus: trackBus,
+          note: 64,
+          ...quiet,
+          released: true,
+        },
+        {
+          id: 3,
+          file: `instrument:${instrument.id}`,
+          owner: '',
+          bus: trackBus,
+          note: 65,
+          ...quiet,
+          startsIn: 0.2,
+        },
+      ],
+      waiting: [{ id: 4, file: 'boom.wav', owner: '', bus: `mix:${bus.id}` }],
+      failed: ['boom.wav'],
+      playings: [{ id: 7, name: `song:${song.id}`, owner, playing: true, fading: null, baked: false }],
+      buses: [
+        { name: trackBus, ...level(0.5, 0.1) },
+        { name: `${trackBus}:#items`, ...level(0.5, 0.1) },
+        { name: 'mix', ...level(1, 0.01) },
+        { name: `mix:${bus.id}`, ...level(0, 0) },
+        { name: 'play:7', ...level(0.5, 0.1) },
+      ],
+    };
+    const host = hostFor(f, started, driver);
+    const facts = host.runtime!()!;
+    host.runtime = () => ({ ...facts, audio: heard });
+
+    const view = (await runTool('get_runtime', {}, host)).structuredContent as { audio: object };
+    expect(view.audio).toEqual({
+      clock: { state: 'running', time: 12.346 },
+      window: 0.68,
+      levels: [
+        { bus: 'master', peak: 0, rms: -40 },
+        { bus: `song Theme, track ${track!.name}`, peak: -6, rms: -20 },
+        { bus: 'Music', silent: true },
+        { bus: 'song Theme', peak: -6, rms: -20 },
+      ],
+      voices: [
+        { sound: 'Jump', bus: 'master', by: 'Box' },
+        { instrument: 'Lead', note: 'E4', bus: `song Theme, track ${track!.name}`, released: true },
+      ],
+      planned: 1,
+      playing: [{ song: 'Theme', by: 'Box' }],
+      waiting: [{ file: 'boom.wav', bus: 'Music' }],
+      failed: [{ file: 'boom.wav' }],
+    });
+
+    // A clock the browser holds while the game runs means nothing is heard, paused since or not.
+    const audioWith = async (clock: HeardSound['clock']) => {
+      host.runtime = () => ({ ...facts, audio: { ...heard, clock } });
+      return ((await runTool('get_runtime', {}, host)).structuredContent as { audio: { hint?: string } })
+        .audio;
+    };
+    expect(await audioWith({ state: 'running', time: 3, held: true })).not.toHaveProperty('hint');
+    expect((await audioWith({ state: 'suspended', time: 0, held: false })).hint).toBe(STALLED_HINT);
+    expect((await audioWith({ state: 'suspended', time: 0, held: true })).hint).toBe(STALLED_HINT);
+  });
 });
 
 describe('the camera', () => {

tests/viewport/sound-library.test.ts

@@ -2342,6 +2342,80 @@ describe('a sound FX on a bus (#346)', () => {
     expect(disconnects).toEqual(['panner', 'splitter']);
   });
 
+  it('taps every bus for the agent while it watches, a later one too, with its peak and RMS (#436)', async () => {
+    const library = new SoundLibrary(false);
+    await library.setBuses([{ name: 'song:s:t', gain: 1, pan: 0 }]);
+    expect(library.heard().buses).toEqual([]);
+    library.watch(true);
+    expect(analysers.map((side) => side.fftSize)).toEqual([32768, 32768, 32768, 32768]);
+    await library.setBuses([
+      { name: 'song:s:t', gain: 1, pan: 0 },
+      { name: 'song:s:u', gain: 1, pan: 0 },
+    ]);
+    expect(library.heard().buses.map((bus) => bus.name)).toEqual(['mix', 'song:s:t', 'song:s:u']);
+    // The fake holds one sample in the middle of the stretch and silence round it.
+    analysers[2]!.sample = -0.5;
+    analysers[3]!.sample = 0.25;
+    const track = library.heard().buses[1]!;
+    expect(track).toMatchObject({ name: 'song:s:t', peak: [0.5, 0.25] });
+    expect(track.rms[0]).toBeCloseTo(0.5 / Math.sqrt(32768), 9);
+    expect(track.rms[1]).toBeCloseTo(0.25 / Math.sqrt(32768), 9);
+    expect(library.heard().window).toBeCloseTo(32768 / 48000);
+    // The Song tab's meter is another tap: letting it go leaves the agent's.
+    library.busPeaks(['song:s:t']);
+    library.unmeter();
+    expect(library.heard().buses).toHaveLength(3);
+    // Moved elsewhere, a bus keeps its tap.
+    wires.length = 0;
+    await library.setBuses([{ name: 'song:s:t', gain: 1, pan: 0, into: 'song:s:u' }]);
+    expect(wires.filter((wire) => wire === 'panner > splitter')).toHaveLength(1);
+    disconnects.length = 0;
+    library.watch(false);
+    expect(library.heard().buses).toEqual([]);
+    expect(disconnects.filter((node) => node === 'splitter')).toHaveLength(3);
+  });
+
+  it('says what plays: each voice with its owner, bus and note, those planned, waiting or failed (#436)', async () => {
+    const library = new SoundLibrary(false);
+    await play(library, FILE, { owner: 'o1' });
+    const gate = {
+      pitch: [],
+      stages: [],
+      source: { file: FILE, volume: 1, loop: true, loopStart: 0, loopEnd: 0, gate: true },
+    } as FxPlan;
+    library.playNote('instrument:i', [gate], {}, 1, '', 0, { at: 5, bus: 'song:s:t', note: 64 });
+    library.play(OTHER, 1, false, 'o2');
+    vi.stubGlobal('fetch', () => Promise.resolve({ ok: false, status: 404 }));
+    vi.spyOn(console, 'error').mockImplementation(() => {});
+    await library.load('gone.wav');
+    const heard = library.heard();
+    expect(heard.voices).toEqual([
+      { id: 1, file: FILE, owner: 'o1', bus: '', startsIn: 0, released: false, ringing: false },
+      {
+        id: 2,
+        file: 'instrument:i',
+        owner: '',
+        bus: 'song:s:t',
+        note: 64,
+        startsIn: 5,
+        released: false,
+        ringing: false,
+      },
+    ]);
+    expect(heard.waiting).toEqual([{ id: 3, file: OTHER, owner: 'o2', bus: '' }]);
+    expect(heard.failed).toEqual(['gone.wav']);
+    library.release(2);
+    expect(library.heard().voices[1]).toMatchObject({ released: true });
+    // Paused, the clock is told as it ran before: held by the pause, or by the browser already.
+    expect(heard.clock).toEqual({ state: 'running', time: 0, held: false });
+    library.setSuspended(true);
+    contextState = 'suspended';
+    expect(library.heard().clock).toEqual({ state: 'running', time: 0, held: true });
+    library.setSuspended(false);
+    library.setSuspended(true);
+    expect(library.heard().clock).toEqual({ state: 'suspended', time: 0, held: true });
+  });
+
   it('makes the chain of a track in the Song tab once, and again only for another sound FX', async () => {
     const library = new SoundLibrary(false);
     const track = { name: 'song:s:t', gain: 1, pan: 0, fx: voice(crush) };

tools/e2e/71-mcp.mjs

@@ -637,6 +637,69 @@ export async function mcp(t) {
       }),
     );
 
+    // 71h4. What the agent hears (#436): the sound of 71g9 loops twice from the start of the game, once
+    // on the master and once on a bus at volume 0. Held at a game time, get_runtime names both voices,
+    // the master is heard, the quiet bus is silent, and the levels stay as they are while it is paused.
+    const noisy = await callTool('run_commands', {
+      label: 'noise',
+      steps: [
+        { command: 'add_file_asset', args: { type: 'sound', file: wav.file }, as: 'tone' },
+        { command: 'create_asset', args: { type: 'bus', name: `${TOKEN_NAME} quiet` }, as: 'quiet' },
+        { command: 'set_asset_property', args: { asset: '$quiet', key: 'volume', value: 0 } },
+        { command: 'create_asset', args: { type: 'behavior', name: `${TOKEN_NAME} noise` }, as: 'noise' },
+        { command: 'place_asset', args: { in: sceneId, asset: '$noise', x: 0, y: 0, parent: playerId } },
+        { command: 'add_node', args: { asset: '$noise', node: 'OnStart' }, as: 'start' },
+        {
+          command: 'add_node',
+          args: { asset: '$noise', node: 'PlaySound', params: { sound: '$tone', loop: 'true' } },
+          as: 'loud',
+        },
+        {
+          command: 'add_node',
+          args: {
+            asset: '$noise',
+            node: 'PlaySound',
+            params: { sound: '$tone', loop: 'true', bus: '$quiet' },
+          },
+          as: 'hushed',
+        },
+        ...['$loud', '$hushed'].map((node) => ({
+          command: 'connect',
+          args: { asset: '$noise', from: { node: '$start', port: 'exec' }, to: { node, port: 'exec' } },
+        })),
+      ],
+    });
+    await settle();
+    const quietName = `${TOKEN_NAME} quiet`;
+    const heldAt = await callTool('play', { action: 'play', until: 0.6 });
+    const heardFirst = (await callTool('get_runtime', { console: 0 }))?.structuredContent?.audio;
+    await sleep(400);
+    const heardLater = (await callTool('get_runtime', { console: 0 }))?.structuredContent?.audio;
+    await callTool('play', { action: 'stop' });
+    await waitUntil('__gm.state.mode === "edit"', { label: 'the noisy game stopped' });
+    await callTool('undo');
+    await settle();
+    const levelOf = (audio, bus) => audio?.levels?.find((row) => row.bus === bus);
+    const toneVoices = (heardFirst?.voices ?? []).filter((voice) => voice.sound !== undefined);
+    check(
+      'get_runtime says what is heard: both voices by sound and bus, the master loud, a bus at 0 silent, held while paused',
+      noisy?.isError === false &&
+        heldAt?.structuredContent?.paused === true &&
+        toneVoices.length === 2 &&
+        toneVoices.some((voice) => voice.bus === 'master') &&
+        toneVoices.some((voice) => voice.bus === quietName) &&
+        heardFirst.window > 0.6 &&
+        levelOf(heardFirst, 'master')?.peak > -12 &&
+        levelOf(heardFirst, quietName)?.silent === true &&
+        JSON.stringify(heardLater?.levels) === JSON.stringify(heardFirst?.levels),
+      JSON.stringify({
+        text: noisy?.content?.[0]?.text?.slice(0, 300),
+        heldAt: heldAt?.structuredContent ?? heldAt?.content,
+        heardFirst,
+        laterLevels: heardLater?.levels,
+      }),
+    );
+
     // 71i. One Ctrl+Z of the user takes the agent's whole step back.
     await press('z', { ctrl: true });
     await settle();

tools/mcp/relay.mjs

@@ -56,7 +56,9 @@ const INSTRUCTIONS = [
   'The editor follows your edits: after run_commands the user sees the graph, scene or asset it worked',
   'on, with what it touched selected. Use open and select to show anything else.',
   'Test what you built yourself: play, then play_input sends keys and mouse clicks into the game,',
-  'and get_runtime reads where the objects are, their variables and the console. To look at one',
+  'and get_runtime reads where the objects are, their variables and the console, and under audio',
+  'what you would hear: the voices sounding and the level of each bus, silent where nothing reaches',
+  'it. Check there that music and sounds are really heard. To look at one',
   'moment, play with until holds the game at that game time, as does a last play_input step',
   '{"pause": true}; a call that only reads comes too late for it, the game runs on meanwhile. A game needs a',
   'camera in its scene to show anything; get_view says when it has none.',