Play sounds as notes of an instrument asset #331
game-maker / suite (push) Failing after 2m40s
game-maker / gate (push) Successful in 3m14s

A sound could only be played as it was, one at a time. An instrument
is a node graph that makes a note a sound: Notes nodes bring the note
in, Samples play sound assets for it, the effects of a sound FX shape
it and the Output sums the layers. Its tab plays it from a keyboard,
a behavior with Play note. It is what patterns and songs (#330) will
play.

- New asset type instrument, under the heading Music, not placeable.
  A new one starts as Notes into a Sample into the Output.
- Notes are counted as MIDI does: 60 is C4, names like C#4, sharps
  only, the lowest C-1. Notes has From and To, outputs any, note,
  frequency and velocity, and with Single notes a trigger per note,
  each with an optional name for a drum kit.
- New port type trigger, only into the trigger of a source. Inputs
  can take any number of edges now (PortDef.many): a trigger, and
  the Output, which plays everything into it side by side.
- Sample plays a sound asset. Follow note shifts it by note - root
  through the pitch worklet of #317, so it keeps its length. Gate
  fades it out at the end of the note, One-shot plays through. Loop
  runs between Loop start and Loop end, also in the worklet.
- Parameters declares the instrument's own parameters with a
  default. Sound FX sends the sound through a sound FX asset, its
  parameters as inputs, the sound set in where its Input was.
- compileInstrument walks the graph once per note. A trigger is on
  only for the notes it stands for, a silent source silences what it
  alone feeds, a Mix with a silent side is the other side. velocity
  and the parameters stay open for the voice. Builds are cached per
  note until the graph or a sound or FX it names changes.
- The player takes a note as one voice of several layers, each way
  with its own file and volume. release fades the gated ways out,
  one-shots play on. playNote can release after a duration.
- Play note plays one note with velocity, volume, duration and
  detached, and hands out voice for Set sound FX parameters and
  Stop sound. Velocity is always loudness. Instrument parameters
  picks values for the instrument's parameters like Sound FX
  parameters does for an FX, renames follow.
- The tab has the graph, a header field per parameter, Stop, and
  128 keys below. Keys without sound are grey. A key plays while it
  is held. Focused, the computer keyboard plays as in a tracker by
  KeyboardEvent.code, the arrows move the octave.

Findings:
- A label of the keys set fill on the shared small text style and
  darkened every small label of the editor. Caught in a screenshot
  before the commit, the keys have a style of their own now.
- Suite task 523 on cf6799a failed two checks of e2e 34 that timed
  audio by the page's clock: a 2.95 s echo tail was gone after 1 s,
  and Quad in was further along than 0.7 s allow. Both time on the
  audio clock now through a new hook __gm.sound.clock().

Tests:
- instrument.test.ts (new, 22): notes, asset, palette, round trip,
  compile per note, drum kit, zones, Mix, velocity and parameters,
  Sound FX node with rename, problems, cache, Play note, the keys
- sound-library +7 (layers, release, duration, early release, loop
  region in buffer and worklet), pitch-player +2 (loop region),
  node-catalog +4, graph adjusted. 582 tests in the touched files.
- e2e 4 and 34 on melektaus-atx: ALL PASSED, 34 with 44 checks (+7)
- Gate on melektaus-atx: CHECK EXIT 0, 243 test files, 3750 tests
- Counter-checks: any always on, release ending one-shots, Sound FX
  inputs ignored, loop region ignored. 3, 2, 1 and 1 tests failed.
- Browser probe on stein2: keys, held key, Single notes names and
  the header looked at in screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-06 10:07:40 +02:00
1 parent cf6799a784
commit 8a4e4a6762
54 files changed
+2561 -143

No files matched your search

+185 -52
View File
@@ -48,22 +48,28 @@ interface Source {
stop(): void;
ended(then: () => void): void;
fade: GainNode | null;
/** Ends with the note (#331): a release fades it out, where everything else plays on. */
gate: boolean;
}
/**
* One sound in flight: its sources, its own gain, the file it came from and who is making it. A
* voice has a source per way through its sound FX (#316), started together and as long as each
* other. Its knobs are in the order `levelsOf` lists the numbers of its chain.
* other. A note of an instrument (#331) is one voice too, its layers side by side, each way with a
* file of its own. Its knobs are in the order `levelsOf` lists the numbers of its chains.
*/
interface Voice {
/** What `play` returned for it, the handle a `tune` reaches it by. */
id: number;
/** The file it plays, or for a note what `playNote` was given to name its instrument by. */
file: string;
/** The behavior this voice belongs to; `''` for one nothing owns (see `AudioSink.play`). */
owner: string;
sources: Source[];
/** Calls the end of one source in, once however it comes: by itself or by a release. */
finish: ((source: Source) => void) | null;
gain: GainNode;
fx: FxVoice;
fx: Layered;
knobs: Knob[];
/** Whether its FX can ring on after the sources end (#324), and whether that is all it does now. */
rings: boolean;
@@ -72,11 +78,26 @@ interface Voice {
quiet: (() => void) | null;
}
/** What one voice is wired from: the file, and what has been made for it so far. */
/**
* The plans of a voice and the values of their parameters: one plan for a sound played through its
* FX, one per layer for a note (#331).
*/
interface Layered {
layers: readonly FxPlan[];
params: FxParams;
}
/** One way of a chain: what a source plays. */
type Way = Exclude<FxChain, { mix: number }>;
/**
* What one voice is wired from: its buffers by file, the file and loop a way without a source of
* its own plays (the sound a `play` was given), and what has been made for it so far.
*/
interface Wiring {
context: AudioContext;
buffers: ReadonlyMap<string, AudioBuffer>;
file: string;
buffer: AudioBuffer;
loop: boolean;
rings: boolean;
sources: Source[];
@@ -85,16 +106,32 @@ interface Wiring {
stages: number;
}
/** Everything a voice is started with, the same whether it waits on its bytes first or not. */
interface Start {
id: number;
file: string;
/** The files its ways read, every one of them loaded before it starts. */
files: readonly string[];
volume: number;
loop: boolean;
owner: string;
fx: Layered;
/** Seconds until a note is released (#331); 0 holds it until it is released or stopped. */
duration: number;
}
/**
* A play still waiting on its bytes: what a quick `tune` in the meantime writes into, and the timed
* ones, which have to start with the voice rather than run out before it is heard.
*/
interface Pending {
id: number;
fx: FxVoice;
fx: Layered;
timed: { params: FxParams; time: number; ease: EaseShare }[];
/** Stopped by its number before it started. */
stopped: boolean;
/** A note released before it started: it starts and is released at once, so a one-shot is heard. */
released: boolean;
}
/**
@@ -159,29 +196,82 @@ export class SoundLibrary implements AudioSink {
*/
play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE): number {
if (file === '' || this.failed.has(file)) return 0;
const layered = { layers: [fx.plan], params: fx.params };
return this.begin({ id: 0, file, files: [file], volume, loop, owner, fx: layered, duration: 0 });
}
/**
* One note of an instrument (#331): its layers played side by side as one voice, each way with its
* own file. `name` stands for the instrument where a file would stand – what `levels` finds it by.
* A `duration` above 0 releases it that many seconds after it starts.
*/
playNote(
name: string,
layers: readonly FxPlan[],
params: FxParams,
volume: number,
owner = '',
duration = 0,
): number {
const files = [...new Set(layers.flatMap(filesOf))].filter(
(file) => file !== '' && !this.failed.has(file),
);
if (files.length === 0) return 0;
const fx = { layers, params };
return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration });
}
/**
* The end of a note (#331): the ways that end with it fade out, the others play on, and the voice
* goes once they have all ended – or rings on first where it has echoes.
*/
release(id: number): void {
const waiting = this.pending.find((other) => other.id === id);
if (waiting) waiting.released = true;
const voice = this.voices.find((other) => other.id === id);
if (!voice) return;
const context = this.audio();
const gated = voice.sources.filter((source) => source.gate);
for (const source of gated) if (source.fade) fadeOut(source.fade.gain, context.currentTime);
if (gated.length === 0) return;
this.after(context.currentTime + FADE, () => {
for (const source of gated) {
source.stop();
voice.finish?.(source);
}
});
}
private begin(start: Start): number {
const id = this.nextId++;
const buffer = this.buffers.get(file);
const pitching = pitches(fx.plan) ? this.pitchWorklet() : null;
if (buffer && (!pitching || this.canPitch !== null)) {
this.start(id, file, buffer, volume, loop, owner, fx);
const asked: Start = { ...start, id };
const loaded = start.files.every((file) => this.buffers.has(file));
const pitching = start.fx.layers.some(pitches) ? this.pitchWorklet() : null;
if (loaded && (!pitching || this.canPitch !== null)) {
this.start(asked);
return id;
}
const askedAt = this.cancelsFor(file, owner);
const askedAt = this.cancelsFor(start.file, start.owner);
const waiting: Pending = {
id,
fx: { plan: fx.plan, params: { ...fx.params } },
fx: { layers: start.fx.layers, params: { ...start.fx.params } },
timed: [],
stopped: false,
released: false,
};
this.pending.push(waiting);
void Promise.all([this.load(file), pitching]).then(([loaded]) => {
this.pending = this.pending.filter((other) => other !== waiting);
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
if (!loaded || waiting.stopped || this.cancelsFor(file, owner) !== askedAt) return;
this.start(id, file, loaded, volume, loop, owner, waiting.fx);
for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease);
});
void Promise.all([Promise.all(start.files.map((file) => this.load(file))), pitching]).then(
([buffers]) => {
this.pending = this.pending.filter((other) => other !== waiting);
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
if (buffers.every((buffer) => !buffer) || waiting.stopped) return;
if (this.cancelsFor(start.file, start.owner) !== askedAt) return;
this.start({ ...asked, fx: waiting.fx });
for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease);
if (waiting.released) this.release(id);
},
);
return id;
}
@@ -192,12 +282,12 @@ export class SoundLibrary implements AudioSink {
// A quick change is where the voice starts; a timed one – and anything after it, so the order
// holds – runs from the moment it is heard.
if (time > 0 || waiting.timed.length > 0) waiting.timed.push({ params, time, ease });
else waiting.fx = { plan: waiting.fx.plan, params: { ...waiting.fx.params, ...params } };
else waiting.fx = { layers: waiting.fx.layers, params: { ...waiting.fx.params, ...params } };
return true;
}
const voice = this.voices.find((other) => other.id === id);
if (!voice) return false;
voice.fx = { plan: voice.fx.plan, params: { ...voice.fx.params, ...params } };
voice.fx = { layers: voice.fx.layers, params: { ...voice.fx.params, ...params } };
this.glide(voice, time, ease);
// A longer feedback makes for a longer tail: it is counted again from where the change ends.
if (voice.ringing) this.ringFor(voice, Math.max(0, time) + this.tailOf(voice.fx));
@@ -289,6 +379,14 @@ export class SoundLibrary implements AudioSink {
.map((voice) => voice.knobs.map((knob) => knob.param?.value ?? knob.target));
}
/**
* Seconds on the audio clock, 0 before there is one (test hook): what a check times a voice by,
* since that clock and the page's need not run alike.
*/
clock(): number {
return this.context?.currentTime ?? 0;
}
/** Where the pitch worklet is served from, for a check that runs it in an offline context (test hook). */
pitchWorkletUrl(): string {
return pitchWorklet;
@@ -347,7 +445,7 @@ export class SoundLibrary implements AudioSink {
this.stopped += going.filter((voice) => !(letRing && voice.ringing)).length;
for (const voice of going) {
if (letRing && voice.rings) {
if (!voice.ringing) this.release(voice);
if (!voice.ringing) this.ringOut(voice);
continue;
}
this.cut(voice);
@@ -369,7 +467,7 @@ export class SoundLibrary implements AudioSink {
}
/** Fades the sources of a voice with echoes out and lets the echoes ring on (#324). */
private release(voice: Voice): void {
private ringOut(voice: Voice): void {
voice.ringing = true;
const context = this.audio();
for (const source of voice.sources) if (source.fade) fadeOut(source.fade.gain, context.currentTime);
@@ -410,8 +508,12 @@ export class SoundLibrary implements AudioSink {
this.voices = this.voices.filter((other) => other !== voice);
}
private tailOf(fx: FxVoice): number {
return fxRings(fx.plan) ? fxTailOf(fxChainOf(fx.plan, fx.params)) : 0;
/** The longest tail of its layers: they are heard side by side. */
private tailOf(fx: Layered): number {
return fx.layers.reduce(
(longest, plan) => (fxRings(plan) ? Math.max(longest, fxTailOf(fxChainOf(plan, fx.params))) : longest),
0,
);
}
/**
@@ -436,26 +538,18 @@ export class SoundLibrary implements AudioSink {
};
}
private start(
id: number,
file: string,
buffer: AudioBuffer,
volume: number,
loop: boolean,
owner: string,
fx: FxVoice,
): void {
private start({ id, file, files, volume, loop, owner, fx, duration }: Start): void {
const context = this.audio();
// A context starts suspended where the page has not been interacted with; without this the
// voice would run silently and end as if it had played.
if (context.state === 'suspended') void context.resume();
const gain = context.createGain();
gain.gain.value = Math.max(0, Math.min(1, volume));
const rings = fxRings(fx.plan);
const rings = fx.layers.some(fxRings);
const wiring: Wiring = {
context,
buffers: this.buffers,
file,
buffer,
loop,
rings,
sources: [],
@@ -463,21 +557,24 @@ export class SoundLibrary implements AudioSink {
knobs: [],
stages: 0,
};
this.wire(wiring, fxChainOf(fx.plan, fx.params)).connect(gain).connect(context.destination);
for (const plan of fx.layers) this.wire(wiring, fxChainOf(plan, fx.params)).connect(gain);
gain.connect(context.destination);
const { sources } = wiring;
this.last = {
file,
volume: gain.gain.value,
loop,
layers: fx.layers.length,
pitches: wiring.pitches,
stages: wiring.stages,
duration: buffer.duration,
duration: Math.max(0, ...files.map((name) => this.buffers.get(name)?.duration ?? 0)),
};
const voice: Voice = {
id,
file,
owner,
sources,
finish: null,
gain,
fx,
knobs: wiring.knobs,
@@ -485,14 +582,15 @@ export class SoundLibrary implements AudioSink {
ringing: false,
quiet: null,
};
let playing = sources.length;
for (const source of sources)
source.ended(() => {
if (--playing === 0) this.ended(voice);
});
const going = new Set(sources);
voice.finish = (source) => {
if (going.delete(source) && going.size === 0) this.ended(voice);
};
for (const source of sources) source.ended(() => voice.finish?.(source));
this.voices.push(voice);
this.started += 1;
for (const source of sources) source.start();
if (duration > 0) this.after(context.currentTime + duration, () => this.release(id));
}
/**
@@ -513,7 +611,7 @@ export class SoundLibrary implements AudioSink {
const shareB = gainNode(context, chain.mix);
wayB.connect(shareB).connect(tail);
knobs.push({ param: shareB.gain, target: chain.mix, even: 'value' });
} else tail = this.source(wiring, chain.pitch, chain.shifted);
} else tail = this.source(wiring, chain);
for (const stage of chain.stages) {
const made = stageNodes(context, stage);
tail.connect(made.input);
@@ -527,21 +625,29 @@ export class SoundLibrary implements AudioSink {
/**
* A way's sound: the pitch worklet where the way has a Pitch, which keeps the length whatever the
* pitch does (#317), and otherwise the buffer as it is. Without the worklet a pitched way plays at
* its own pitch rather than not at all.
* its own pitch rather than not at all. A way with a source of its own (#331) plays that file, at
* its volume, and can fade out by itself when its note ends.
*/
private source(wiring: Wiring, pitch: number, shifted: boolean): AudioNode {
const sound = this.sound(wiring, pitch, shifted);
if (!wiring.rings) return sound;
const fade = gainNode(wiring.context, 1);
private source(wiring: Wiring, chain: Way): AudioNode {
const sound = this.sound(wiring, chain);
const own = chain.source;
if (!wiring.rings && !own) return sound;
const fade = gainNode(wiring.context, own?.volume ?? 1);
wiring.sources.at(-1)!.fade = fade;
return sound.connect(fade);
}
private sound(wiring: Wiring, pitch: number, shifted: boolean): AudioNode {
const { context, file, buffer, loop } = wiring;
private sound(wiring: Wiring, { pitch, shifted, source: own }: Way): AudioNode {
const { context } = wiring;
const file = own?.file ?? wiring.file;
const loop = own ? own.loop : wiring.loop;
const gate = own?.gate ?? false;
// A file that could not be loaded leaves its way silent rather than the whole note.
const buffer = wiring.buffers.get(file) ?? context.createBuffer(1, 1, context.sampleRate);
wiring.pitches.push(pitch);
if (shifted && this.canPitch) {
const options: PitchOptions = { file, step: buffer.sampleRate / context.sampleRate, loop };
if (own && loop) Object.assign(options, loopIn(buffer, own.loopStart, own.loopEnd));
if (!this.sent.has(file)) {
options.channels = Array.from({ length: buffer.numberOfChannels }, (_, c) =>
buffer.getChannelData(c),
@@ -566,6 +672,7 @@ export class SoundLibrary implements AudioSink {
};
},
fade: null,
gate,
});
wiring.knobs.push({ param: node.parameters.get('pitch')!, target: pitch, even: 'value' });
return node;
@@ -573,6 +680,10 @@ export class SoundLibrary implements AudioSink {
const source = context.createBufferSource();
source.buffer = buffer;
source.loop = loop;
if (own && loop) {
source.loopStart = own.loopStart;
source.loopEnd = own.loopEnd;
}
wiring.sources.push({
start: () => source.start(),
stop: () => source.stop(),
@@ -580,6 +691,7 @@ export class SoundLibrary implements AudioSink {
source.onended = then;
},
fade: null,
gate,
});
if (shifted) wiring.knobs.push({ param: null, target: pitch, even: 'value' });
return source;
@@ -590,7 +702,8 @@ export class SoundLibrary implements AudioSink {
* clock along `ease`, or else in a quick glide.
*/
private glide(voice: Voice, time: number, ease: EaseShare): void {
const levels = levelsOf(fxChainOf(voice.fx.plan, voice.fx.params));
const levels: number[] = [];
for (const plan of voice.fx.layers) levelsOf(fxChainOf(plan, voice.fx.params), levels);
const now = this.audio().currentTime;
voice.knobs.forEach((knob, i) => {
const level = levels[i];
@@ -634,6 +747,8 @@ interface LastVoice {
file: string;
volume: number;
loop: boolean;
/** How many layers of a note it played side by side (#331); 1 for a sound. */
layers: number;
/** Semitones of each way (#316), and how many gain and filter nodes it passed. */
pitches: number[];
stages: number;
@@ -646,6 +761,24 @@ function pitches(plan: FxPlan): boolean {
return 'mix' in plan ? pitches(plan.a) || pitches(plan.b) : plan.pitch.length > 0;
}
/** The files the ways of a plan read with a source of their own (#331). */
function filesOf(plan: FxPlan): string[] {
if ('mix' in plan) return [...filesOf(plan.a), ...filesOf(plan.b)];
return plan.source ? [plan.source.file] : [];
}
/**
* Where a loop runs in the samples of a sound, as the worklet takes it: between `start` and `end`
* seconds, 0 at the end being the end of the sound. One that does not fit is the whole sound, as
* Web Audio has it for a buffer source.
*/
function loopIn(buffer: AudioBuffer, start: number, end: number): { loopStart: number; loopEnd: number } {
const last = buffer.length;
const from = Math.round(Math.max(0, start) * buffer.sampleRate);
const to = end > 0 ? Math.min(last, Math.round(end * buffer.sampleRate)) : last;
return from < to ? { loopStart: from, loopEnd: to } : { loopStart: 0, loopEnd: last };
}
function gainNode(context: BaseAudioContext, level: number): GainNode {
const gain = context.createGain();
gain.gain.value = level;