Let the agent read what is heard in the running game (#436)

The agent has no ears. It could start music and sounds, but not tell
whether anything reached the speakers: a song without an instrument, a
bus at volume 0 or a browser that holds the audio clock all looked the
same from outside.

- get_runtime has a section audio. Under levels it gives the peak and
  RMS in dB of the master, each bus of the mix, each pattern, song or
  composition playing and the tracks of a song. Below -90 dB it says
  silent.
- The levels come from a tap of their own after the level and pan of
  each bus, an AnalyserNode per side as for the meters of the Song tab.
  It looks back about 0.7 s, so the gap between two notes is not
  silence. A pause holds the values, which fits play with until.
- Under voices it names what sounds now: a sound by its asset, a note
  of an instrument with its note name, each with its bus and object.
  Next to them planned, playing, waiting (bytes not here yet) and
  failed.
- Under clock it gives the state of the audio clock, with a hint when
  it stands still while the game runs. The state is told as it was
  before a pause, else the pause would hide that case.
- The buses are tapped only while the agent is switched on.
- The note now travels with a voice from the Start of a pattern or song
  to the SoundLibrary, so a voice can be named by it.
- tools.json and the relay's instructions tell the agent to check under
  audio that music and sounds are really heard.

Found on the user's project: started right after F5, before a click in
the page, the browser kept the audio clock suspended for the whole run.
Nothing was heard and the notes stayed planned. That is what clock now
shows.

Tests:
- Unit: sound-library (taps, levels, voices, clock), agent-play
  (audioView). Pattern and song tests expect the note in their timing.
- 9 counter-checks, 8 red. Not dropping a tap with its bus stays green,
  as the bus is cut off and gone anyway.
- e2e 71 (melektaus-atx): 28/28. New 71h4: a sound on the master and
  one on a bus at volume 0, both voices named, the master loud, the bus
  silent, the values held in the pause.
- Gate (melektaus-atx): CHECK EXIT 0, 327 files, 5037 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-18 22:14:07 +02:00
1 parent dd2096e0df
commit b5e071ca68
14 files changed
+607 -16

No files matched your search

+3 -1
View File
@@ -56,7 +56,9 @@ const INSTRUCTIONS = [
'The editor follows your edits: after run_commands the user sees the graph, scene or asset it worked',
'on, with what it touched selected. Use open and select to show anything else.',
'Test what you built yourself: play, then play_input sends keys and mouse clicks into the game,',
'and get_runtime reads where the objects are, their variables and the console. To look at one',
'and get_runtime reads where the objects are, their variables and the console, and under audio',
'what you would hear: the voices sounding and the level of each bus, silent where nothing reaches',
'it. Check there that music and sounds are really heard. To look at one',
'moment, play with until holds the game at that game time, as does a last play_input step',
'{"pause": true}; a call that only reads comes too late for it, the game runs on meanwhile. A game needs a',
'camera in its scene to show anything; get_view says when it has none.',