From 452b31df54b6186813326d32e9597c4f7c881f0c Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Tue, 15 Sep 2026 09:20:53 +0200 Subject: [PATCH] A game that stays silent is only half a game, so the pages say what a sound is (#45) --- API.md | 19 +++++++++--- Architecture.md | 29 ++++++++++++++---- Behaviors.md | 7 +++-- Concepts.md | 5 +-- Development.md | 17 ++++++++--- Home.md | 4 ++- Sounds.md | 81 +++++++++++++++++++++++++++++++++++++++++++++++++ _Sidebar.md | 1 + 8 files changed, 144 insertions(+), 19 deletions(-) create mode 100644 Sounds.md diff --git a/API.md b/API.md index 8c5385b..3d8b395 100644 --- a/API.md +++ b/API.md @@ -13,6 +13,9 @@ All routes go through `api/index.php?r=…` (the site's nginx does not pass `PAT | GET | `fonts` | `[{file,name,bytes,updatedAt}]` — what the font picker offers | | POST | `fonts&name=Serif.ttf` | **raw font bytes** → `{file,name,bytes}` (201) | | DELETE | `fonts/{file}` | 204 | +| GET | `sounds` | `[{file,name,bytes,updatedAt}]` — what the sound picker offers | +| POST | `sounds&name=jump.wav` | **raw sound bytes** → `{file,name,bytes}` (201) | +| DELETE | `sounds/{file}` | 204 | Every project document carries `formatVersion` (`PROJECT_FORMAT_VERSION` in `core/model/Project.ts`, currently 2). There are no load-time migrations: `serialize.ts` reads what it recognises and drops @@ -20,10 +23,18 @@ the rest. Uploads send the bytes as the body rather than `multipart/form-data`: no boundary parsing in PHP, and PHP's `upload_max_filesize` (2 MB) never applies — only `post_max_size`. The API has its -own caps behind nginx's: 8 MB per request, 4 MB per image or font (413). The stored -files are served statically from `data/textures/` and `data/fonts/`, so reading one never runs -PHP. An image is downscaled and re-encoded in the browser when it would not fit; a font cannot -be, so an oversized typeface is refused rather than shrunk. +own caps behind nginx's: 20 MB per request, 4 MB per image or font and 16 MB per sound (413) — +a single music track is worth more than four typefaces. **nginx's own limit is the one that +actually bites** (1 MB today, see `MAX_UPLOAD_BYTES` in `textureImport.ts`): raising the API's +cap without raising that one changes nothing. The stored files are served statically from +`data/textures/`, `data/fonts/` and `data/sounds/`, so reading one never runs PHP. An image is +downscaled and re-encoded in the browser when it would not fit; a font or a sound cannot be, so +an oversized one is refused rather than shrunk. + +The format of a font or a sound is read from the file's **own signature**, never from the +`Content-Type` of the request: browsers name audio inconsistently (`audio/mp4`, `audio/x-m4a`, +nothing at all for the same `.m4a`). MP3 is the one with no single signature — either an `ID3` +tag or a frame header of eleven set bits, which is what `SoundStore::mp3Extension` demands. **`name` is the original file name**, kept beside the stored file as `{file}.json` and returned with it. The stored file is always `{sha256}.{ext}`, so the name is a caption and never a path: diff --git a/Architecture.md b/Architecture.md index eaadcc4..58f4b9a 100644 --- a/Architecture.md +++ b/Architecture.md @@ -55,17 +55,18 @@ src/core/graph/ graph model, NodeRegistry/NodeContext/values, interpreter src/core/particles/ nodes (the catalogue), compile (a graph → one function per output), emission (which particles are alive at a second — a pure rule, no state) src/core/store/ serialize (untrusted JSON → Project), http (one API call), ProjectStore/HttpProjectStore, - TextureStore/HttpTextureStore, FontStore/HttpFontStore + TextureStore/HttpTextureStore, FontStore/HttpFontStore, SoundStore/HttpSoundStore src/core/time.ts the global clock: one scaled step a frame, shared by the world, the particles, the shaders and the animation preview -src/runtime/ World, Entity, System interface, Input tracker, CollisionSystem, AnimationSystem, BehaviorSystem +src/runtime/ World, Entity, System interface, Input tracker, CollisionSystem, AnimationSystem, BehaviorSystem, + Audio (the AudioSink a run plays into — the counterpart of Input on the way out) src/ui/ widget toolkit (generic, knows nothing about assets) src/viewport/ view (transforms), items (what to draw), symbols (how to draw it), - TextureLibrary, FontLibrary, ShaderLibrary, ParticleLibrary, + TextureLibrary, FontLibrary, SoundLibrary, ShaderLibrary, ParticleLibrary, particles (one ParticleContainer per source), activeCamera — no editor concepts, so tests reach it freely src/editor/ App → Editor (layout + ticker), PlaySession (the run), ProjectSession (load/create), EditorRun (run in editor), state, actions, shortcuts, debugHook, autosave, - fileImport (drop/paste) → textureImport / fontImport, + fileImport (drop/paste) → textureImport / fontImport / soundImport, framing (fit/marquee/snap maths), AssetTabEditor (the shared shell of the asset tabs), TabStrip + tabRow (the tab row and what it does when the tabs outgrow it) src/editor/viewport/ WorldViewport base + SceneViewport/GameViewport — editing tools, hence not in src/viewport/ @@ -85,8 +86,8 @@ src/editor/particles/ ParticlesEditor (graph canvas left of a preview panel, sp ParticlePreview (the panel: the asset's own particles, wearing its own rows) src/editor/list/ ListEditor (a row per item), slot (which list is being edited, and how it is written back) src/editor/console/ log (ring buffer, argument formatting, the console.* capture), ConsolePanel (the tail under the tabs) -api/ PHP backend: index.php (routing), lib/{Store,TextureStore,FontStore,OriginalName,Response}.php - data/projects/ persisted projects, data/textures/ + data/fonts/ content-addressed uploads +api/ PHP backend: index.php (routing), lib/{Store,TextureStore,FontStore,SoundStore,OriginalName,Response}.php + data/projects/ persisted projects, data/textures/ + data/fonts/ + data/sounds/ content-addressed uploads tests/ Vitest unit tests tools/ seed-*, e2e (steps in tools/e2e/, rig in tools/e2e/lib/), smoke-*, lib/, assets/ (the walk demo's sheet) ``` @@ -144,6 +145,22 @@ step from there**; a second clock beside it is the thing to avoid, not to add. N the other way round: **whoever keeps an absolute second of a clock has to survive that clock going back to zero**, which it does at both ends of a run. A particle field keeps its origin that way, so one in the future means the clock was pulled out from under it and it starts over. +**The time scale is deliberately not part of one of them: pitch.** A game in slow motion keeps +its sounds at normal speed — a tape being slowed is a sound effect of its own, not the meaning of +"half speed" — so the audio device is left alone and only what the world *does* is scaled. +A pause is the exception, because a voice playing over a still image is nobody's idea of paused: +the audio context is suspended and resumes mid-note. + +**Everything the graph does to the outside world goes through the context, audio included.** +`NodeContext.playSound` takes an asset id, not a file: resolving it — and folding the asset's own +volume and loop setting together with the node's (`loop` is `boolean | undefined`, and `undefined` +is what leaves the answer to the asset) — is the `BehaviorSystem`'s job, since it is the one +holding the project. Below that line, `AudioSink` (`runtime/systems/Audio.ts`) knows only +files and voices, and `viewport/SoundLibrary` implements it. That keeps `core/` free of the +browser the way it is free of Pixi, lets a test run a world that plays into a recorder instead of +a device, and is the reason a run started before anything wired a sink in plays silently rather +than crashing. + **A predicate named after what a thing looks like will be asked what a thing *is*.** `isSymbolType` reads as "draws an icon", but four callers use it for "is editor decoration": the running world, the Game tab, the draw order and the upright placement all key off it. Giving a particle source an icon diff --git a/Behaviors.md b/Behaviors.md index d8cfe98..c9776af 100644 --- a/Behaviors.md +++ b/Behaviors.md @@ -99,7 +99,7 @@ shown on the node and editable in the inspector. - **Node set** (`src/core/graph/nodes/`, one file per node): events `On start`, `On enable`, `On update`, `On disable`, `On destroy` (see *Lifecycle*), `On key down/up/held`, `On timer`, `On collision enter/stay/exit`, `On click` (the mouse pressed on the owner — anywhere when it has none), `On animation finished`, `On state entered`, `On state exited` (one output per state, see *State machines*), `Start` · flow `Branch`, `Sequence`, `Repeat`, `Run behavior`, `End`, and the three that move or follow a state machine, `State branch`, `Set state`, `State transition` (see *State machines*) · data `Self`, `Owner`, `Parent`, `Object by name`, `Name of`, `Is asset` (is this object an instance of asset X — how you tell what you collided with), - `Variables`, `Properties` (every property row of the target's type, the transform rows `scale`, `rotation`, `mirrorX`, `mirrorY` included — how a graph turns a sprite), `Get position` (local and world), `Is enabled`, literals `Number`/`Text`/`Boolean`/`Texture`, `Add/Subtract/Multiply/Divide`, + `Variables`, `Properties` (every property row of the target's type, the transform rows `scale`, `rotation`, `mirrorX`, `mirrorY` included — how a graph turns a sprite), `Get position` (local and world), `Is enabled`, literals `Number`/`Text`/`Boolean`/`Texture`/`State machine`/`Sound`, `Add/Subtract/Multiply/Divide`, `Min/Max/Abs/Sqrt/Clamp`, `Floor/Ceil/Round`, `Modulo`, `Random`, `Time`, `To text`, `To number`, `Join text`, `Substring`, `Find text`, `Equals`, lists `List` (a constant), `Fill` (one value repeated — how an empty board is made), `Count`, @@ -109,7 +109,10 @@ shown on the node and editable in the inspector. normalized direction, so diagonals are not faster), `Compare`, `And/Or/Not`, `Key is down`, `Mouse position`, `Mouse button`, `Is hovered`, `Is touching`, `Current state` (name and id), `Is in state` (one boolean per state) · actions `Move` (dx, dy, scale, dt — connect an event's dt for per-second movement; in the object's own axes, which a turned parent turns too), `Set position`, `Spawn` - (asset setting + x/y/z, optional parent → outputs the new object), `Destroy`, `Emit` (starts a particle source over, see *Particles*), `Log`, `Stop game`, `Restart game`. A node type is one file exporting a `NodeDef` + (asset setting + x/y/z, optional parent → outputs the new object), `Destroy`, `Emit` (starts a particle source over, see *Particles*), + `Play sound` / `Stop sound` (see [Sounds](Sounds): the asset says how loud and whether it loops, the node's + `volume` multiplies that and its `loop` either leaves it alone or overrules it in either + direction; every play is a voice of its own, so two in a row overlap), `Log`, `Stop game`, `Restart game`. A node type is one file exporting a `NodeDef` (ports, settings, `run`), listed in `nodes/index.ts`; palette, ports, validation and inspector fields are generated from it. A port may declare a `default` for its inline value (e.g. `Move.scale = 1`), shown on the node and in the inspector. diff --git a/Concepts.md b/Concepts.md index bd65c8d..e7cd9a6 100644 --- a/Concepts.md +++ b/Concepts.md @@ -1,8 +1,9 @@ | Term | Meaning | |---|---| -| **Asset** | Named thing with a type (`scene`, `camera`, `sprite`, `text`, `behavior`, `collider`, `container`, `texture`, `animation`, `statemachine`, `shader`, `font`, `particles`, `variables`) and typed properties. Assets have **no position**. | -| **Texture** | An uploaded image. Scenes, textures, animations, state machines, shaders, fonts and Variables assets are the seven types that cannot be placed: a texture has no position, it is *worn* by a sprite through the ordinary property `texture`. See [Textures](Textures). | +| **Asset** | Named thing with a type (`scene`, `camera`, `sprite`, `text`, `behavior`, `collider`, `container`, `texture`, `animation`, `statemachine`, `shader`, `font`, `sound`, `particles`, `variables`) and typed properties. Assets have **no position**. | +| **Texture** | An uploaded image. Scenes, textures, animations, state machines, shaders, fonts, sounds and Variables assets are the eight types that cannot be placed: a texture has no position, it is *worn* by a sprite through the ordinary property `texture`. See [Textures](Textures). | | **Font** | An uploaded typeface, worn by a text through the ordinary property `font`; no font means the editor's own. Uploaded and picked exactly like a texture, and stored the same way. See [Fonts](Fonts). | +| **Sound** | An uploaded audio file. Uploaded, picked and stored exactly like a typeface — but nothing *wears* it: it is neither on screen nor in the scene, and a graph plays it with `Play sound`. Its own rows say how loud it is and whether it loops; the node scales the one and can overrule the other either way. See [Sounds](Sounds). | | **State machine** | A named set of **states**, edited as nodes in its own tab. A behavior puts it into a state with `Set state`, moves it along a named transition with `State transition` and branches on it with `State branch`. Its **Scope** says where the current state lives: **Local** gives every behavior instance one of its own, **Global** keeps a single one for the whole run. See [State machines](State-machines). | | **Shader** | A node graph that becomes the GLSL a sprite is drawn with, edited in its own tab. A sprite wears it through `shader`; the numbers the graph declares are rows below that dropdown. See [Shaders](Shaders). | | **Particles** | A node graph that says what *one* particle is as a function of its age and of a random number drawn once for it — no forces, no simulation. Unlike the other graph assets it **is** placeable: the object made from it is a source, whose rows say how many go out and for how long they live. Because a particle is a formula, the same second always draws the same picture and the tab's slider can jump to any of them exactly. See [Particles](Particles). | diff --git a/Development.md b/Development.md index f2caa60..b4a4491 100644 --- a/Development.md +++ b/Development.md @@ -12,7 +12,7 @@ npm run check # typecheck + lint + format:check + tests, the gate CI runs # code is lost (tail's is reported), so a red run reads as green npm run build # writes dist/ and regenerates index.html ./build.sh # the same under `set -o pipefail`, for callers that pipe the output -node tools/seed-e2e.mjs && node tools/e2e.mjs # 885 editor checks, screenshots in /tmp/gm-e2e-*.png +node tools/seed-e2e.mjs && node tools/e2e.mjs # 901 editor checks, screenshots in /tmp/gm-e2e-*.png GM_STEPS=20 node tools/e2e.mjs # or 1-12, or 6,20 — a subset while working on one step ./e2e.sh start [GM_STEPS] # the same run detached, see below ./e2e.sh status # live progress, or the last run's result @@ -64,7 +64,7 @@ The e2e run is a list of steps in `tools/e2e/` (hierarchy, inspector, play, asse graph, parenting, colliders, containers, text/enabled, menus, textures, frames, overflow, aspect, container tab, run-in-editor, pause, nested behaviors, key repeat, list tab, struct variable, variable nodes, property nodes, animations, state machines, console, shaders, transforms, fonts, -particles, shared scope, render on change) that build on each other's +particles, shared scope, render on change, sounds) that build on each other's scene state and share the CDP driver in `tools/lib/harness.mjs` — `press`, `undo`, `play`, `object(id)`, `world()`, `mutate(js)` and friends, so a check reads like the interaction it exercises. @@ -78,7 +78,7 @@ The same holds for the **`+` menu**: it has one entry per asset type and scrolls last of them sit under the mask, where `locate` still finds an entry but a click on it lands on nothing. A step picks a type with `pickAssetType(t, 'Font')` from the same file, which scrolls it in first. The fourteenth type (Variables) pushed `Font` under that edge and took the font step down, -because only the particles step had carried the workaround of its own. +because only the particles step had carried the workaround of its own. Sound is the fifteenth. **What no colour check can see: whether anything was drawn at all.** `__gm.pixelAt` re-renders the stage on demand, so it answers correctly on a screen that has been frozen for a minute — which is @@ -91,6 +91,15 @@ tab whose picture stands still — the scene viewport may be wearing a Time shad and animation previews are moving pictures by definition. Screenshots ask for a frame first (`__gm.wake()` in `harness.mjs`) rather than trusting what the compositor still holds. +**Nothing is ever heard, and that is fine.** The browser runs with `--mute-audio`, plus +`--autoplay-policy=no-user-gesture-required` so a voice still *starts* — otherwise the sound +checks would be measuring a browser that refuses to play rather than an editor that never asked +it to. `__gm.sound.stats()` (voices started, stopped, in flight, and the last one with its volume +and loop flag) is the witness, for the reason `renderStats` is one. The seed's two sounds are +**generated WAVs** (`tools/lib/audio.mjs`), not files: unlike the DejaVu typefaces there is no +audio file the host and the CI image are guaranteed to share, and the bytes are a pure function of +the arguments, so storage stays content-addressed and a check can name a file by its exact size. + **A check that demands an exact number must be handed the number, not something it can be derived from.** The particle preview used to hold second 0.5 by computing the clock reading that *would* mean 0.5 since the field started; the round trip through floating point misses by the last bit about one @@ -183,4 +192,4 @@ widths to find click points. The **render device is not in this repo** — `/dev and a 1 GB `/dev/shm` come from `container.options` in the runner's own `config.yaml`, which applies to every job on that runner. Without them chromium falls back to llvmpipe, which gets WebGL 1.0 only. Measured in the job container: `ANGLE (Intel, Vulkan 1.3.230, HD Graphics 630)`, -WebGL 2.0, and the same 885 checks as here. +WebGL 2.0, and the same 901 checks as here. diff --git a/Home.md b/Home.md index 2c12d73..359ff39 100644 --- a/Home.md +++ b/Home.md @@ -15,6 +15,7 @@ Live: · Widget playground: