Change the sound FX parameters of a sound while it plays #315
game-maker / gate (push) Successful in 3m14s
game-maker / suite (push) Successful in 15m39s

Until now a value handed to a sound FX counted from the start of the
voice only. An engine whose pitch follows the speed, or a loop that
gets quieter, could not be built. Every play of Play sound now gives
out the number of its voice, and a new node lets the knobs of that one
voice glide to new values while it plays.

- Play sound has a new output voice: the number of this one play, or 0
  where nothing was played. Kept in a Number variable it reaches the
  voice from another event. Two plays of one sound are two voices.
- New action node Set sound FX parameters (exec, voice, fx params).
  It reaches exactly the voice it is given, a detached one too. A
  voice that has ended, or voice 0, is not reached.
- AudioSink.play returns the voice number and takes the plan and the
  values (FxVoice) instead of a finished chain. AudioSink.tune takes a
  voice number and new values. A name the voice's FX does not have is
  left out, one not given keeps its value.
- SoundLibrary hands out the number at once, before the bytes are
  there, and never twice. A voice still waiting for its bytes starts
  with the values handed in meanwhile.
- Every knob of a voice moves with setTargetAtTime (30 ms time
  constant). A knob whose value did not change is not sent again, so a
  tune every frame costs next to nothing.
- A pitch on a playing voice glides by playbackRate. A new WSOLA copy
  in the middle of a sound is not worked out, so the voice gets shorter
  or longer while it is moved. On a loop that is not heard.
- fxChainOf now gives a chain whose shape depends on the plan alone.
  A stage fed by a parameter stays at 0 dB, and a Mix fed by a
  parameter keeps both ways. A typed-in 0 or Mix at 0/1 is still left
  out by the compiler.
- Every way of a voice has a source of its own, also at the same
  pitch, so each can glide on its own. The pitched copy is shared.
- The tab keeps the numbers of what it plays, so moving a parameter
  field is heard at once.
- Test hook __gm.sound.levels(file) reads the knobs of every voice of
  a file, oldest first, and stats() counts the voices tuned.

Tests:
- soundfx.test.ts: 29 tests (+2 for the node in a running world: the
  voice of the first of two plays of one sound, and a play that played
  nothing)
- sound-library.test.ts: 15 tests (+3: glide, reach by number,
  pending voice)
- node-catalog.test.ts: 370 tests (+2, Play sound checks its voice)
- sounds + keep-on-scene-load: 31 tests, unchanged
- Counter-test: a tune that reaches every voice of the same file makes
  the reach test fail.
- e2e 34 (melektaus-atx): ALL PASSED - 30 checks (+1). In Chrome two
  looping voices of one sound start at depth 7. The one whose number
  sits in a variable glides to 0.7938 (2^(-4/12) = 0.7937), the other
  stays at 1. Both gains stay at 0.5012.
- The gate ran before voices got a number (CHECK EXIT 0, 241 files,
  3658 tests), not after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-05 20:53:26 +02:00
1 parent 072537b2e8
commit c7e9769f90
18 files changed
+557 -100

No files matched your search

+7 -1
View File
@@ -240,6 +240,7 @@ export interface NodeContext {
* the one case where the rule is in the way: the death sound of an object that destroys itself. * the one case where the rule is in the way: the death sound of an object that destroys itself.
* `fx` names a sound FX asset the voice is played through (#311); `''` plays it as it is. * `fx` names a sound FX asset the voice is played through (#311); `''` plays it as it is.
* `fxParams` are values for its parameters; one not given is the default the FX declares. * `fxParams` are values for its parameters; one not given is the default the FX declares.
* Returns the number of the voice for `tuneSound`, or 0 where nothing is played.
*/ */
playSound( playSound(
assetId: string, assetId: string,
@@ -248,9 +249,14 @@ export interface NodeContext {
detached: boolean, detached: boolean,
fx: string, fx: string,
fxParams: FxParams, fxParams: FxParams,
): void; ): number;
/** Silences every voice of that sound; `''` silences everything this run started. */ /** Silences every voice of that sound; `''` silences everything this run started. */
stopSound(assetId: string): void; stopSound(assetId: string): void;
/**
* Hands new values to the sound FX parameters of one voice, by the number `playSound` gave it
* (#315), so a running sound follows the game. A voice that has ended is not reached.
*/
tuneSound(voice: number, fxParams: FxParams): void;
/** /**
* Asks for another scene, by asset id. The swap happens once the frame is over – a graph must be * Asks for another scene, by asset id. The swap happens once the frame is over – a graph must be
* able to finish the flow it is in, and half a frame in the old world and half in the new one is * able to finish the flow it is in, and half a frame in the old world and half in the new one is
+10 -7
View File
@@ -59,7 +59,9 @@ export const PlaySound: NodeDef = {
'the node only colors it. Each play is a voice of its own: firing this twice in a row is heard ' + 'the node only colors it. Each play is a voice of its own: firing this twice in a row is heard ' +
'twice, overlapping. The sound belongs to this object and stops when the object is destroyed, ' + 'twice, overlapping. The sound belongs to this object and stops when the object is destroyed, ' +
'unless "detached" is on. "fx" sends it through a sound FX asset; left empty, it plays as it is. ' + 'unless "detached" is on. "fx" sends it through a sound FX asset; left empty, it plays as it is. ' +
'"fx params" takes values for its parameters from a Sound FX parameters node.', '"fx params" takes values for its parameters from a Sound FX parameters node. "voice" is the ' +
'number of this one play, for Set sound FX parameters to change it while it plays; 0 where ' +
'nothing was played.',
inputs: [ inputs: [
exec(), exec(),
soundPort(), soundPort(),
@@ -69,25 +71,26 @@ export const PlaySound: NodeDef = {
soundFxPort(), soundFxPort(),
{ key: 'fxParams', label: 'fx params', type: 'fxparams' }, { key: 'fxParams', label: 'fx params', type: 'fxparams' },
], ],
outputs: [exec()], outputs: [exec(), num('voice')],
params: [], params: [],
run: (ctx, inputs) => { run: (ctx, inputs) => {
const sound = asString(inputs.sound); const sound = asString(inputs.sound);
if (sound) const voice = sound
ctx.playSound( ? ctx.playSound(
sound, sound,
asNumber(inputs.volume), asNumber(inputs.volume),
loopChoice(inputs.loop), loopChoice(inputs.loop),
asBoolean(inputs.detached), asBoolean(inputs.detached),
asString(inputs.fx), asString(inputs.fx),
fxParamsOf(inputs.fxParams), fxParamsOf(inputs.fxParams),
); )
return { next: 'exec' }; : 0;
return { next: 'exec', outputs: { voice } };
}, },
}; };
/** The numbers of a parameter bundle; anything else hands in none. */ /** The numbers of a parameter bundle; anything else hands in none. */
function fxParamsOf(value: Value | undefined): FxParams { export function fxParamsOf(value: Value | undefined): FxParams {
if (!isStruct(value)) return {}; if (!isStruct(value)) return {};
const params: Record<string, number> = {}; const params: Record<string, number> = {};
for (const [key, item] of Object.entries(value)) if (typeof item === 'number') params[key] = item; for (const [key, item] of Object.entries(value)) if (typeof item === 'number') params[key] = item;
@@ -0,0 +1,32 @@
import type { NodeDef } from '../../NodeRegistry';
import { asNumber } from '../../values';
import { exec, num } from '../shared';
import { fxParamsOf } from './PlaySound';
/**
* Changes the sound FX parameters of a sound that is already playing (#315). It reaches one voice,
* by the number its `Play sound` handed out: every play is a sound of its own, and two objects with
* the same engine sound – or one object playing it twice – each keep theirs.
*/
export const SetSoundFxParameters: NodeDef = {
type: 'SetSoundFxParameters',
label: 'Set sound FX parameters',
category: 'action',
description:
'Changes the sound FX parameters of a sound while it plays: the voice a Play sound handed out ' +
'glides to the values from a Sound FX parameters node. Keep the voice in a variable to reach it ' +
'from another event. A parameter not given keeps its value, a voice that has ended is not ' +
'reached. A pitch changes by speed here, so the sound gets shorter or longer while it is moved.',
inputs: [
exec(),
{ ...num('voice'), inline: false },
{ key: 'fxParams', label: 'fx params', type: 'fxparams' },
],
outputs: [exec()],
params: [],
run: (ctx, inputs) => {
const voice = asNumber(inputs.voice);
if (voice > 0) ctx.tuneSound(voice, fxParamsOf(inputs.fxParams));
return { next: 'exec' };
},
};
@@ -6,14 +6,14 @@ import { asNumber } from '../../values';
/** /**
* Values for the parameters of one sound FX (#311), picked from those its Input declares the way a * Values for the parameters of one sound FX (#311), picked from those its Input declares the way a
* Variables node picks variables. Each picked one is a number input starting at the FX's default; * Variables node picks variables. Each picked one is a number input starting at the FX's default;
* the bundle goes into the `fx params` input of a `Play sound`. * the bundle goes into the `fx params` input of a `Play sound` or a `Set sound FX parameters`.
*/ */
export const SoundFxParameters: NodeDef = { export const SoundFxParameters: NodeDef = {
type: SOUND_FX_PARAMETERS, type: SOUND_FX_PARAMETERS,
label: 'Sound FX parameters', label: 'Sound FX parameters',
category: 'data', category: 'data',
description: description:
'Values for the parameters of a sound FX, for the "fx params" input of a Play sound. Pick the parameters to set; one not picked keeps the default its Input gives it.', 'Values for the parameters of a sound FX, for the "fx params" input of a Play sound or a Set sound FX parameters. Pick the parameters to set; one not picked keeps the default its Input gives it, or on a playing sound the value it has.',
inputs: [], inputs: [],
outputs: [{ key: 'params', label: 'params', type: 'fxparams' }], outputs: [{ key: 'params', label: 'params', type: 'fxparams' }],
params: [SOUND_FX_SETTING], params: [SOUND_FX_SETTING],
+2
View File
@@ -23,6 +23,7 @@ import { Raycast } from './action/Raycast';
import { RaycastAll } from './action/RaycastAll'; import { RaycastAll } from './action/RaycastAll';
import { RestartGame } from './action/RestartGame'; import { RestartGame } from './action/RestartGame';
import { SetPosition } from './action/SetPosition'; import { SetPosition } from './action/SetPosition';
import { SetSoundFxParameters } from './action/SetSoundFxParameters';
import { SetZ } from './action/SetZ'; import { SetZ } from './action/SetZ';
import { Spawn } from './action/Spawn'; import { Spawn } from './action/Spawn';
import { StopGame } from './action/StopGame'; import { StopGame } from './action/StopGame';
@@ -298,6 +299,7 @@ export const BUILT_IN_NODES: readonly NodeDef[] = [
StopAnimation, StopAnimation,
PlaySound, PlaySound,
StopSound, StopSound,
SetSoundFxParameters,
SoundFxParameters, SoundFxParameters,
LoadScene, LoadScene,
Log, Log,
+19 -10
View File
@@ -50,6 +50,14 @@ export const EMPTY_PLAN: FxPlan = { pitch: [], stages: [] };
/** Values for the parameters, by name; a name the plan does not have is left out. */ /** Values for the parameters, by name; a name the plan does not have is left out. */
export type FxParams = Readonly<Record<string, number>>; export type FxParams = Readonly<Record<string, number>>;
/** How a voice goes through its sound FX: the plan and the values it starts with, kept so they can change while it plays (#315). */
export interface FxVoice {
plan: FxPlan;
params: FxParams;
}
export const DRY_VOICE: FxVoice = { plan: EMPTY_PLAN, params: {} };
/** /**
* A node of a sound FX graph (#311). It never runs: `emit` says what it does to the sounds arriving * A node of a sound FX graph (#311). It never runs: `emit` says what it does to the sounds arriving
* at its audio inputs, by input key, given the numbers at its other inputs. Input and Output have no * at its audio inputs, by input key, given the numbers at its other inputs. Input and Output have no
@@ -239,23 +247,24 @@ function amountOf(amount: FxAmount, params: FxParams): number {
return Math.max(amount.min ?? -Infinity, Math.min(amount.max ?? Infinity, given)); return Math.max(amount.min ?? -Infinity, Math.min(amount.max ?? Infinity, given));
} }
/** A stage fed by a parameter stays at 0 as well: a value handed in later has to find it there. */
function stagesOf(plan: FxPlan, params: FxParams): FxStage[] { function stagesOf(plan: FxPlan, params: FxParams): FxStage[] {
const stages: FxStage[] = []; const stages: FxStage[] = [];
for (const stage of plan.stages) { for (const stage of plan.stages) {
if (stage.kind === 'gain') { if (stage.kind === 'gain') {
const db = amountOf(stage.db, params); const db = amountOf(stage.db, params);
if (db !== 0) stages.push({ kind: 'gain', gain: 10 ** (db / 20) }); if (db !== 0 || stage.db.param !== undefined) stages.push({ kind: 'gain', gain: 10 ** (db / 20) });
continue; continue;
} }
const gain = amountOf(stage.gain, params); const gain = amountOf(stage.gain, params);
if (gain !== 0) stages.push({ ...stage, gain }); if (gain !== 0 || stage.gain.param !== undefined) stages.push({ ...stage, gain });
} }
return stages; return stages;
} }
/** /**
* The chain one voice is sent along: the plan with every parameter filled in. A Mix that ends at 0 * The chain one voice is sent along: the plan with every parameter filled in. Its shape depends on
* or 1 is left out with the sound it silences, so that one is not even started. * the plan alone, never on the values, so the values of a playing voice can change in place (#315).
*/ */
export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain { export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain {
const stages = stagesOf(plan, params); const stages = stagesOf(plan, params);
@@ -263,10 +272,10 @@ export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain {
const pitch = plan.pitch.reduce((sum, amount) => sum + amountOf(amount, params), 0); const pitch = plan.pitch.reduce((sum, amount) => sum + amountOf(amount, params), 0);
return { pitch, stages }; return { pitch, stages };
} }
const mix = amountOf(plan.mix, params); return {
if (mix <= 0 || mix >= 1) { mix: amountOf(plan.mix, params),
const heard = fxChainOf(mix <= 0 ? plan.a : plan.b, params); a: fxChainOf(plan.a, params),
return { ...heard, stages: [...heard.stages, ...stages] }; b: fxChainOf(plan.b, params),
} stages,
return { mix, a: fxChainOf(plan.a, params), b: fxChainOf(plan.b, params), stages }; };
} }
+2
View File
@@ -41,6 +41,8 @@ export function installDebugHook(app: Application, state: EditorState, editor: E
sound: { sound: {
stats: () => soundLibrary.stats(), stats: () => soundLibrary.stats(),
duration: (file: string) => soundLibrary.duration(file), duration: (file: string) => soundLibrary.duration(file),
/** Where the knobs of each voice of that file are now, oldest first, a pitch as its speed (#315). */
levels: (file: string) => soundLibrary.levels(file),
/** Decodes ahead of a check, so a play can be asserted without racing the network. */ /** Decodes ahead of a check, so a play can be asserted without racing the network. */
load: (file: string) => soundLibrary.load(file).then((buffer) => buffer?.duration ?? null), load: (file: string) => soundLibrary.load(file).then((buffer) => buffer?.duration ?? null),
stopAll: () => soundLibrary.stopAll(), stopAll: () => soundLibrary.stopAll(),
+17 -5
View File
@@ -3,7 +3,6 @@ import { findAssetOf } from '../../core/model/Project';
import { resolveAssetProps } from '../../core/properties/resolve'; import { resolveAssetProps } from '../../core/properties/resolve';
import { volumeOf } from '../../core/properties/sounds'; import { volumeOf } from '../../core/properties/sounds';
import { soundFxBuildFor, soundFxParams } from '../../core/soundfx/compile'; import { soundFxBuildFor, soundFxParams } from '../../core/soundfx/compile';
import { fxChainOf } from '../../core/soundfx/nodes';
import { Button, Dropdown, HStack, Label, Menu, NumberField, Stack, Theme } from '../../ui'; import { Button, Dropdown, HStack, Label, Menu, NumberField, Stack, Theme } from '../../ui';
import { soundLibrary } from '../../viewport/SoundLibrary'; import { soundLibrary } from '../../viewport/SoundLibrary';
import { AssetTabEditor } from '../AssetTabEditor'; import { AssetTabEditor } from '../AssetTabEditor';
@@ -40,6 +39,8 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
private defaults: Record<string, number> = {}; private defaults: Record<string, number> = {};
private paramsKey = ''; private paramsKey = '';
private valuesFor = ''; private valuesFor = '';
/** The voices the tab started that may still be playing – what a moved field reaches. */
private voices: number[] = [];
constructor(state: EditorState) { constructor(state: EditorState) {
super(state, { infoWidth: INFO_WIDTH }); super(state, { infoWidth: INFO_WIDTH });
@@ -89,8 +90,8 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
/** Sets one parameter the header plays with, as typing into its field does. */ /** Sets one parameter the header plays with, as typing into its field does. */
setPreviewValue(key: string, value: number): void { setPreviewValue(key: string, value: number): void {
if (!(key in this.values)) return; if (!(key in this.values)) return;
this.values[key] = value;
this.paramFields.get(key)?.setNumber(value); this.paramFields.get(key)?.setNumber(value);
this.setValue(key, value);
} }
/** Plays the picked sound through the graph; a graph that does not compile is not played at all. */ /** Plays the picked sound through the graph; a graph that does not compile is not played at all. */
@@ -100,11 +101,22 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
const plan = asset && soundFxBuildFor(asset).compiled; const plan = asset && soundFxBuildFor(asset).compiled;
if (!sound?.file || !plan) return; if (!sound?.file || !plan) return;
const volume = volumeOf(resolveAssetProps(this.state.project, sound)); const volume = volumeOf(resolveAssetProps(this.state.project, sound));
soundLibrary.play(sound.file, volume, false, PREVIEW_OWNER, fxChainOf(plan, this.values)); const voice = soundLibrary.play(sound.file, volume, false, PREVIEW_OWNER, {
plan,
params: { ...this.values },
});
if (voice) this.voices.push(voice);
}
/** A moved field is heard at once on what the tab is playing (#315); an ended voice is let go. */
private setValue(key: string, value: number): void {
this.values[key] = value;
this.voices = this.voices.filter((voice) => soundLibrary.tune(voice, { [key]: value }));
} }
stop(): void { stop(): void {
soundLibrary.stopOwner(PREVIEW_OWNER); soundLibrary.stopOwner(PREVIEW_OWNER);
this.voices = [];
} }
protected override infoText(asset: SoundFxAsset): string { protected override infoText(asset: SoundFxAsset): string {
@@ -152,8 +164,8 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
const field = new NumberField(values[port.key], { const field = new NumberField(values[port.key], {
width: PARAM_WIDTH, width: PARAM_WIDTH,
step: 0.1, step: 0.1,
onInput: (value) => (this.values[port.key] = value), onInput: (value) => this.setValue(port.key, value),
onChange: (value) => (this.values[port.key] = value), onChange: (value) => this.setValue(port.key, value),
}); });
this.paramFields.set(port.key, field); this.paramFields.set(port.key, field);
this.paramBar.add(field); this.paramBar.add(field);
+13 -5
View File
@@ -1,4 +1,4 @@
import type { FxChain } from '../../core/soundfx/nodes'; import type { FxParams, FxVoice } from '../../core/soundfx/nodes';
/** /**
* What the runtime needs of the outside world to make a noise – the counterpart of `Input` on the * What the runtime needs of the outside world to make a noise – the counterpart of `Input` on the
@@ -13,10 +13,17 @@ export interface AudioSink {
* Starts one voice. `volume` is final (0–1) and every call overlaps whatever is already playing. * Starts one voice. `volume` is final (0–1) and every call overlaps whatever is already playing.
* `owner` is the behavior the sound belongs to: when that behavior is destroyed the voice goes * `owner` is the behavior the sound belongs to: when that behavior is destroyed the voice goes
* with it, because a sound is something a thing in the world is doing, not a thing of its own. * with it, because a sound is something a thing in the world is doing, not a thing of its own.
* `''` means the voice has no owner and outlives whatever started it. `fx` is the line of effects * `''` means the voice has no owner and outlives whatever started it. `fx` is the sound FX the
* the voice passes on its way out (#311); `DRY` passes none. * voice passes on its way out (#311), with the values of its parameters; `DRY_VOICE` passes none.
* Returns the number of the voice, never used twice, or 0 where nothing is going to play.
*/ */
play(file: string, volume: number, loop: boolean, owner: string, fx?: FxChain): void; play(file: string, volume: number, loop: boolean, owner: string, fx?: FxVoice): number;
/**
* Hands new values to the sound FX parameters of one voice, by the number `play` gave it (#315).
* A name its FX does not have is left out, one not given keeps its value. A voice that has ended
* is not there any more and nothing happens – a detached one is reached like any other.
*/
tune(voice: number, params: FxParams): void;
/** Silences every voice of that file; `''` silences all of them. */ /** Silences every voice of that file; `''` silences all of them. */
stop(file: string): void; stop(file: string): void;
/** Silences every voice started by that owner – what destroying a behavior does. */ /** Silences every voice started by that owner – what destroying a behavior does. */
@@ -24,7 +31,8 @@ export interface AudioSink {
} }
export const SILENT_AUDIO: AudioSink = { export const SILENT_AUDIO: AudioSink = {
play: () => {}, play: () => 0,
tune: () => {},
stop: () => {}, stop: () => {},
stopOwner: () => {}, stopOwner: () => {},
}; };
+12 -5
View File
@@ -16,7 +16,7 @@ import { hasRenderLayer } from '../../core/properties/PropertySchema';
import { resolveAssetProps } from '../../core/properties/resolve'; import { resolveAssetProps } from '../../core/properties/resolve';
import { loopsOf, volumeOf } from '../../core/properties/sounds'; import { loopsOf, volumeOf } from '../../core/properties/sounds';
import { soundFxBuildFor } from '../../core/soundfx/compile'; import { soundFxBuildFor } from '../../core/soundfx/compile';
import { DRY, fxChainOf, type FxParams } from '../../core/soundfx/nodes'; import { DRY_VOICE, type FxParams } from '../../core/soundfx/nodes';
import type { Entity } from '../Entity'; import type { Entity } from '../Entity';
import { PhysicsSystem } from '../physics/PhysicsSystem'; import { PhysicsSystem } from '../physics/PhysicsSystem';
import type { World } from '../World'; import type { World } from '../World';
@@ -172,6 +172,7 @@ export function behaviorContext(
const file = assetId === '' ? '' : soundFileOf(project, assetId); const file = assetId === '' ? '' : soundFileOf(project, assetId);
if (assetId === '' || file) run.audio.stop(file); if (assetId === '' || file) run.audio.stop(file);
}, },
tuneSound: (voice, fxParams) => run.audio.tune(voice, fxParams),
spawn: (assetId, position, parent) => world.spawn(assetId, position, resolve(parent) ?? null), spawn: (assetId, position, parent) => world.spawn(assetId, position, resolve(parent) ?? null),
addJoint: (assetId, owner, connected, at) => { addJoint: (assetId, owner, connected, at) => {
const on = resolve(owner); const on = resolve(owner);
@@ -218,7 +219,7 @@ export function behaviorContext(
* node's, and its loop flag unless the node overrules it either way. The asset is what a sound * node's, and its loop flag unless the node overrules it either way. The asset is what a sound
* *is* – a node that says nothing therefore plays it as the project describes it. An FX asset that * *is* – a node that says nothing therefore plays it as the project describes it. An FX asset that
* does not compile plays the sound dry: a broken effect should not silence the game. Its parameters * does not compile plays the sound dry: a broken effect should not silence the game. Its parameters
* take the values handed in, the others their defaults. * take the values handed in, the others their defaults. Returns the number of the voice, 0 for none.
*/ */
function playSound( function playSound(
{ project, audio }: ContextSources, { project, audio }: ContextSources,
@@ -228,14 +229,20 @@ function playSound(
owner: string, owner: string,
fxId: string, fxId: string,
fxParams: FxParams, fxParams: FxParams,
): void { ): number {
const asset = findAssetOf(project, assetId, 'sound'); const asset = findAssetOf(project, assetId, 'sound');
if (!asset?.file) return; if (!asset?.file) return 0;
const props = resolveAssetProps(project, asset); const props = resolveAssetProps(project, asset);
const level = Math.max(0, Math.min(1, volumeOf(props) * volume)); const level = Math.max(0, Math.min(1, volumeOf(props) * volume));
const fx = fxId ? findAssetOf(project, fxId, 'soundfx') : undefined; const fx = fxId ? findAssetOf(project, fxId, 'soundfx') : undefined;
const plan = fx && soundFxBuildFor(fx).compiled; const plan = fx && soundFxBuildFor(fx).compiled;
audio.play(asset.file, level, loop ?? loopsOf(props), owner, plan ? fxChainOf(plan, fxParams) : DRY); return audio.play(
asset.file,
level,
loop ?? loopsOf(props),
owner,
plan ? { plan, params: fxParams } : DRY_VOICE,
);
} }
/** Where a ray of that length ends; where it has no direction, where it starts. */ /** Where a ray of that length ends; where it has no direction, where it starts. */
+143 -31
View File
@@ -1,34 +1,62 @@
import { DRY, type FxChain, type FxStage } from '../core/soundfx/nodes'; import {
DRY_VOICE,
fxChainOf,
type FxChain,
type FxParams,
type FxStage,
type FxVoice,
} from '../core/soundfx/nodes';
import { pitchShift } from '../core/soundfx/pitchShift'; import { pitchShift } from '../core/soundfx/pitchShift';
import type { AudioSink } from '../runtime/systems/Audio'; import type { AudioSink } from '../runtime/systems/Audio';
const DEFAULT_BASE = '/game-maker/data/sounds'; const DEFAULT_BASE = '/game-maker/data/sounds';
/** How many pitched copies are kept at once. */ /** How many pitched copies are kept at once. */
const SHIFTED_KEPT = 32; const SHIFTED_KEPT = 32;
/** Seconds a changed parameter takes to get most of the way there: quick, but no click (#315). */
const GLIDE = 0.03;
/**
* One number of a voice that can still change while it plays (#315): a level, a share of a Mix, the
* gain of a band – or the pitch of a source, which glides by speed from the pitch its copy was made at.
*/
type Knob =
| { kind: 'level'; param: AudioParam; target: number }
| { kind: 'pitch'; source: AudioBufferSourceNode; base: number; target: number };
/** /**
* One sound in flight: its sources, its own gain, the file it came from and who is making it. A * One sound in flight: its sources, its own gain, the file it came from and who is making it. A
* voice has a source per pitch its sound FX plays the file at (#316), started together and as long * voice has a source per way through its sound FX (#316), started together and as long as each
* as each other. * other. Its knobs are in the order `levelsOf` lists the numbers of its chain.
*/ */
interface Voice { interface Voice {
/** What `play` returned for it, the handle a `tune` reaches it by. */
id: number;
file: string; file: string;
/** The behavior this voice belongs to; `''` for one nothing owns (see `AudioSink.play`). */ /** The behavior this voice belongs to; `''` for one nothing owns (see `AudioSink.play`). */
owner: string; owner: string;
sources: AudioBufferSourceNode[]; sources: AudioBufferSourceNode[];
gain: GainNode; gain: GainNode;
fx: FxVoice;
knobs: Knob[];
} }
/** What one voice is wired from: the file, and its sources by pitch so far. */ /** What one voice is wired from: the file, and what has been made for it so far. */
interface Wiring { interface Wiring {
context: AudioContext; context: AudioContext;
file: string; file: string;
buffer: AudioBuffer; buffer: AudioBuffer;
loop: boolean; loop: boolean;
sources: Map<number, AudioBufferSourceNode>; sources: { source: AudioBufferSourceNode; pitch: number }[];
knobs: Knob[];
stages: number; stages: number;
} }
/** A play still waiting on its bytes: what a `tune` in the meantime writes into. */
interface Pending {
id: number;
fx: FxVoice;
}
/** /**
* Plays uploaded sounds, keyed by file name – the texture library's twin on the audio side. Files * Plays uploaded sounds, keyed by file name – the texture library's twin on the audio side. Files
* are content addressed, so a decoded buffer can never go stale and nothing is evicted. * are content addressed, so a decoded buffer can never go stale and nothing is evicted.
@@ -52,8 +80,12 @@ export class SoundLibrary implements AudioSink {
/** Never retried, so one broken reference cannot spam the network. */ /** Never retried, so one broken reference cannot spam the network. */
private readonly failed = new Set<string>(); private readonly failed = new Set<string>();
private voices: Voice[] = []; private voices: Voice[] = [];
private pending: Pending[] = [];
/** The number the next play is given; 0 stays free for "nothing plays". */
private nextId = 1;
private started = 0; private started = 0;
private stopped = 0; private stopped = 0;
private tuned = 0;
/** The last voice actually started, with the values it was started at – what a check reads back. */ /** The last voice actually started, with the values it was started at – what a check reads back. */
private last: LastVoice | null = null; private last: LastVoice | null = null;
/** /**
@@ -73,20 +105,44 @@ export class SoundLibrary implements AudioSink {
this.base = base.replace(/\/+$/, ''); this.base = base.replace(/\/+$/, '');
} }
play(file: string, volume: number, loop: boolean, owner = '', fx: FxChain = DRY): void { /**
if (file === '' || this.failed.has(file)) return; * The number is handed out at once, before the bytes are there: a graph can tune a voice in the
* same frame it asked for it, and the voice then starts with those values.
*/
play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE): number {
if (file === '' || this.failed.has(file)) return 0;
const id = this.nextId++;
const buffer = this.buffers.get(file); const buffer = this.buffers.get(file);
if (buffer) { if (buffer) {
this.start(file, buffer, volume, loop, owner, fx); this.start(id, file, buffer, volume, loop, owner, fx);
return; return id;
} }
const askedAt = this.cancelsFor(file, owner); const askedAt = this.cancelsFor(file, owner);
const waiting: Pending = { id, fx: { plan: fx.plan, params: { ...fx.params } } };
this.pending.push(waiting);
void this.load(file).then((loaded) => { void this.load(file).then((loaded) => {
this.pending = this.pending.filter((other) => other !== waiting);
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the // A `Stop sound` – or the destruction of the behavior that asked – that happened while the
// bytes were in flight has to win, or a sound the run already ended would start afterwards. // bytes were in flight has to win, or a sound the run already ended would start afterwards.
if (loaded && this.cancelsFor(file, owner) === askedAt) if (loaded && this.cancelsFor(file, owner) === askedAt)
this.start(file, loaded, volume, loop, owner, fx); this.start(id, file, loaded, volume, loop, owner, waiting.fx);
}); });
return id;
}
/** Says whether the voice was still there to be reached – playing, or waiting on its bytes. */
tune(id: number, params: FxParams): boolean {
const waiting = this.pending.find((other) => other.id === id);
if (waiting) {
waiting.fx = { plan: waiting.fx.plan, params: { ...waiting.fx.params, ...params } };
return true;
}
const voice = this.voices.find((other) => other.id === id);
if (!voice) return false;
voice.fx = { plan: voice.fx.plan, params: { ...voice.fx.params, ...params } };
this.glide(voice);
this.tuned += 1;
return true;
} }
stop(file: string): void { stop(file: string): void {
@@ -123,6 +179,7 @@ export class SoundLibrary implements AudioSink {
playing: number; playing: number;
started: number; started: number;
stopped: number; stopped: number;
tuned: number;
decoded: number; decoded: number;
failed: number; failed: number;
last: LastVoice | null; last: LastVoice | null;
@@ -131,12 +188,25 @@ export class SoundLibrary implements AudioSink {
playing: this.voices.length, playing: this.voices.length,
started: this.started, started: this.started,
stopped: this.stopped, stopped: this.stopped,
tuned: this.tuned,
decoded: this.buffers.size, decoded: this.buffers.size,
failed: this.failed.size, failed: this.failed.size,
last: this.last, last: this.last,
}; };
} }
/**
* Where the knobs of each voice of that file are right now, oldest first, a pitch as the speed it
* plays at – what a check reads to see a change arrive at one voice and not another (test hook).
*/
levels(file: string): number[][] {
return this.voices
.filter((voice) => voice.file === file)
.map((voice) =>
voice.knobs.map((knob) => (knob.kind === 'pitch' ? knob.source.playbackRate : knob.param).value),
);
}
/** How long a decoded sound runs, in seconds; null while it has not been loaded (test hook). */ /** How long a decoded sound runs, in seconds; null while it has not been loaded (test hook). */
duration(file: string): number | null { duration(file: string): number | null {
return this.buffers.get(file)?.duration ?? null; return this.buffers.get(file)?.duration ?? null;
@@ -177,12 +247,13 @@ export class SoundLibrary implements AudioSink {
} }
private start( private start(
id: number,
file: string, file: string,
buffer: AudioBuffer, buffer: AudioBuffer,
volume: number, volume: number,
loop: boolean, loop: boolean,
owner: string, owner: string,
fx: FxChain, fx: FxVoice,
): void { ): void {
const context = this.audio(); const context = this.audio();
// A context starts suspended where the page has not been interacted with; without this the // A context starts suspended where the page has not been interacted with; without this the
@@ -190,18 +261,18 @@ export class SoundLibrary implements AudioSink {
if (context.state === 'suspended') void context.resume(); if (context.state === 'suspended') void context.resume();
const gain = context.createGain(); const gain = context.createGain();
gain.gain.value = Math.max(0, Math.min(1, volume)); gain.gain.value = Math.max(0, Math.min(1, volume));
const wiring: Wiring = { context, file, buffer, loop, sources: new Map(), stages: 0 }; const wiring: Wiring = { context, file, buffer, loop, sources: [], knobs: [], stages: 0 };
this.wire(wiring, fx).connect(gain).connect(context.destination); this.wire(wiring, fxChainOf(fx.plan, fx.params)).connect(gain).connect(context.destination);
const sources = [...wiring.sources.values()]; const sources = wiring.sources.map((made) => made.source);
this.last = { this.last = {
file, file,
volume: gain.gain.value, volume: gain.gain.value,
loop, loop,
pitches: [...wiring.sources.keys()], pitches: wiring.sources.map((made) => made.pitch),
stages: wiring.stages, stages: wiring.stages,
duration: sources[0]!.buffer!.duration, duration: sources[0]!.buffer!.duration,
}; };
const voice: Voice = { file, owner, sources, gain }; const voice: Voice = { id, file, owner, sources, gain, fx, knobs: wiring.knobs };
let playing = sources.length; let playing = sources.length;
for (const source of sources) for (const source of sources)
source.onended = () => { source.onended = () => {
@@ -213,35 +284,60 @@ export class SoundLibrary implements AudioSink {
} }
/** /**
* The nodes a chain becomes, up to the one that carries its sound on. Two ways at the same pitch * The nodes a chain becomes, up to the one that carries its sound on. Every way has a source of
* share a source; a Mix weighs each of its sounds by its share before they are summed. * its own, so each can glide to its own pitch; a Mix weighs each of its sounds by its share before
* they are summed.
*/ */
private wire(wiring: Wiring, chain: FxChain): AudioNode { private wire(wiring: Wiring, chain: FxChain): AudioNode {
const { context } = wiring; const { context, knobs } = wiring;
let tail: AudioNode; let tail: AudioNode;
if ('mix' in chain) { if ('mix' in chain) {
tail = context.createGain(); tail = context.createGain();
this.wire(wiring, chain.a) const wayA = this.wire(wiring, chain.a);
.connect(gainNode(context, 1 - chain.mix)) const shareA = gainNode(context, 1 - chain.mix);
.connect(tail); wayA.connect(shareA).connect(tail);
this.wire(wiring, chain.b).connect(gainNode(context, chain.mix)).connect(tail); knobs.push({ kind: 'level', param: shareA.gain, target: 1 - chain.mix });
const wayB = this.wire(wiring, chain.b);
const shareB = gainNode(context, chain.mix);
wayB.connect(shareB).connect(tail);
knobs.push({ kind: 'level', param: shareB.gain, target: chain.mix });
} else tail = this.source(wiring, chain.pitch); } else tail = this.source(wiring, chain.pitch);
for (const stage of chain.stages) tail = tail.connect(fxNode(context, stage)); for (const stage of chain.stages) {
const [node, param] = fxNode(context, stage);
tail = tail.connect(node);
knobs.push({ kind: 'level', param, target: stage.gain });
}
wiring.stages += chain.stages.length; wiring.stages += chain.stages.length;
return tail; return tail;
} }
private source(wiring: Wiring, pitch: number): AudioBufferSourceNode { private source(wiring: Wiring, pitch: number): AudioBufferSourceNode {
const held = wiring.sources.get(pitch);
if (held) return held;
const { context, file, buffer, loop } = wiring; const { context, file, buffer, loop } = wiring;
const source = context.createBufferSource(); const source = context.createBufferSource();
source.buffer = pitch === 0 ? buffer : this.pitched(context, file, buffer, pitch, loop); source.buffer = pitch === 0 ? buffer : this.pitched(context, file, buffer, pitch, loop);
source.loop = loop; source.loop = loop;
wiring.sources.set(pitch, source); wiring.sources.push({ source, pitch });
wiring.knobs.push({ kind: 'pitch', source, base: pitch, target: pitch });
return source; return source;
} }
/**
* Sends every knob of a voice towards what its values say now. A pitch cannot get a new copy in
* the middle of a sound, so it glides by speed: the voice gets that much shorter or longer.
*/
private glide(voice: Voice): void {
const levels = levelsOf(fxChainOf(voice.fx.plan, voice.fx.params));
const now = this.audio().currentTime;
voice.knobs.forEach((knob, i) => {
const level = levels[i];
if (level === undefined || level === knob.target) return;
knob.target = level;
if (knob.kind === 'pitch')
knob.source.playbackRate.setTargetAtTime(2 ** ((level - knob.base) / 12), now, GLIDE);
else knob.param.setTargetAtTime(level, now, GLIDE);
});
}
/** /**
* The sound at another pitch and its own length (#312), worked out once per file, pitch and loop * The sound at another pitch and its own length (#312), worked out once per file, pitch and loop
* and kept for the next voice. The oldest goes first, so a pitch drawn at random every play cannot * and kept for the next voice. The oldest goes first, so a pitch drawn at random every play cannot
@@ -286,7 +382,7 @@ interface LastVoice {
file: string; file: string;
volume: number; volume: number;
loop: boolean; loop: boolean;
/** Semitones of each source, one per pitch (#316), and how many gain and filter nodes it passed. */ /** Semitones of each source, one per way (#316), and how many gain and filter nodes it passed. */
pitches: number[]; pitches: number[];
stages: number; stages: number;
/** Seconds of what was played: a pitched voice is as long as the sound itself (#312). */ /** Seconds of what was played: a pitched voice is as long as the sound itself (#312). */
@@ -299,14 +395,30 @@ function gainNode(context: AudioContext, level: number): GainNode {
return gain; return gain;
} }
function fxNode(context: AudioContext, stage: FxStage): AudioNode { /** The node a stage becomes, with the number of it a parameter can move. */
if (stage.kind === 'gain') return gainNode(context, stage.gain); function fxNode(context: AudioContext, stage: FxStage): [AudioNode, AudioParam] {
if (stage.kind === 'gain') {
const gain = gainNode(context, stage.gain);
return [gain, gain.gain];
}
const filter = context.createBiquadFilter(); const filter = context.createBiquadFilter();
filter.type = stage.filter; filter.type = stage.filter;
filter.frequency.value = stage.frequency; filter.frequency.value = stage.frequency;
filter.gain.value = stage.gain; filter.gain.value = stage.gain;
filter.Q.value = stage.q; filter.Q.value = stage.q;
return filter; return [filter, filter.gain];
}
/** The numbers of a chain in the order `wire` makes their knobs: a stage's `gain` is what its node holds. */
function levelsOf(chain: FxChain, into: number[] = []): number[] {
if ('mix' in chain) {
levelsOf(chain.a, into);
into.push(1 - chain.mix);
levelsOf(chain.b, into);
into.push(chain.mix);
} else into.push(chain.pitch);
for (const stage of chain.stages) into.push(stage.gain);
return into;
} }
export const soundLibrary = new SoundLibrary(); export const soundLibrary = new SoundLibrary();
+2 -1
View File
@@ -54,7 +54,8 @@ function keep(f: Fixture, sceneId: string, objectId: string): void {
function start(f: Fixture, audio?: AudioSink) { function start(f: Fixture, audio?: AudioSink) {
const stoppedOwners: string[] = []; const stoppedOwners: string[] = [];
const sink: AudioSink = audio ?? { const sink: AudioSink = audio ?? {
play: () => {}, play: () => 0,
tune: () => {},
stop: () => {}, stop: () => {},
stopOwner: (owner: string) => stoppedOwners.push(owner), stopOwner: (owner: string) => stoppedOwners.push(owner),
}; };
+17 -1
View File
@@ -1940,12 +1940,14 @@ const CASES: Record<string, Case[]> = {
name: 'no sound is a no-op that still continues the flow', name: 'no sound is a no-op that still continues the flow',
inputs: { sound: '', volume: 1, loop: '' }, inputs: { sound: '', volume: 1, loop: '' },
next: 'exec', next: 'exec',
outputs: { voice: 0 },
after: (ctx) => expect(ctx.played).toEqual([]), after: (ctx) => expect(ctx.played).toEqual([]),
}, },
{ {
name: 'hands the id on with the volume and loop it was given', name: 'hands the id on with the volume and loop it was given, and gives out the voice (#315)',
inputs: { sound: 'snd-1', volume: 0.5, loop: 'true' }, inputs: { sound: 'snd-1', volume: 0.5, loop: 'true' },
next: 'exec', next: 'exec',
outputs: { voice: 1 },
// What the two mean – a factor on the asset's volume, an answer that overrules its loop // What the two mean – a factor on the asset's volume, an answer that overrules its loop
// flag – is the BehaviorSystem's to resolve, since only it holds the project (see // flag – is the BehaviorSystem's to resolve, since only it holds the project (see
// `sounds.test.ts`). // `sounds.test.ts`).
@@ -2025,6 +2027,20 @@ const CASES: Record<string, Case[]> = {
after: (ctx) => expect(ctx.silenced).toEqual(['']), after: (ctx) => expect(ctx.silenced).toEqual(['']),
}, },
], ],
SetSoundFxParameters: [
{
name: 'hands the numbers of the bundle to the voice it is given',
inputs: { voice: 4, fxParams: { depth: 3, label: 'x' } },
next: 'exec',
after: (ctx) => expect(ctx.tuned).toEqual([{ voice: 4, fxParams: { depth: 3 } }]),
},
{
name: 'voice 0 – nothing was played – reaches nothing, and the flow carries on',
inputs: { voice: 0, fxParams: { depth: 3 } },
next: 'exec',
after: (ctx) => expect(ctx.tuned).toEqual([]),
},
],
Log: [ Log: [
{ {
name: 'logs its message and continues', name: 'logs its message and continues',
+81 -16
View File
@@ -12,7 +12,14 @@ import type { BehaviorAsset, SoundFxAsset } from '../../src/core/model/Asset';
import { isPlaceableType } from '../../src/core/model/AssetTypes'; import { isPlaceableType } from '../../src/core/model/AssetTypes';
import { assetRefChoices } from '../../src/core/properties/assetRef'; import { assetRefChoices } from '../../src/core/properties/assetRef';
import { compileSoundFx, soundFxBuildFor } from '../../src/core/soundfx/compile'; import { compileSoundFx, soundFxBuildFor } from '../../src/core/soundfx/compile';
import { DRY, EMPTY_PLAN, type FxChain, fxChainOf, type FxPlan } from '../../src/core/soundfx/nodes'; import {
DRY,
EMPTY_PLAN,
type FxChain,
fxChainOf,
type FxPlan,
type FxVoice,
} from '../../src/core/soundfx/nodes';
import { deserializeProject } from '../../src/core/store/serialize'; import { deserializeProject } from '../../src/core/store/serialize';
import type { AudioSink } from '../../src/runtime/systems/Audio'; import type { AudioSink } from '../../src/runtime/systems/Audio';
import { paletteEntries } from '../../src/editor/graph/paletteEntries'; import { paletteEntries } from '../../src/editor/graph/paletteEntries';
@@ -214,15 +221,15 @@ describe('a Mix in a sound FX graph (#316)', () => {
expect(b).toMatchObject({ pitch: 0, stages: [{ kind: 'gain' }, { kind: 'filter' }] }); expect(b).toMatchObject({ pitch: 0, stages: [{ kind: 'gain' }, { kind: 'filter' }] });
}); });
it('takes its share from a parameter, held between 0 and 1', () => { it('takes its share from a parameter, held between 0 and 1, and keeps both ways for it (#315)', () => {
const f = forked(); const f = forked();
f.input.ports = [{ key: 'blend', kind: 'number', default: 0.25 }]; f.input.ports = [{ key: 'blend', kind: 'number', default: 0.25 }];
f.edit.connect(f.input, 'blend', f.mix, 'mix'); f.edit.connect(f.input, 'blend', f.mix, 'mix');
const plan = compileSoundFx(f.fx.graph).compiled!; const plan = compileSoundFx(f.fx.graph).compiled!;
expect(fxChainOf(plan)).toMatchObject({ mix: 0.25 }); expect(fxChainOf(plan)).toMatchObject({ mix: 0.25 });
expect(fxChainOf(plan, { blend: 0.8 })).toMatchObject({ mix: 0.8 }); expect(fxChainOf(plan, { blend: 0.8 })).toMatchObject({ mix: 0.8 });
expect(fxChainOf(plan, { blend: -3 })).toMatchObject({ pitch: 12 }); expect(fxChainOf(plan, { blend: -3 })).toMatchObject({ mix: 0, a: { pitch: 12 }, b: { pitch: 0 } });
expect(fxChainOf(plan, { blend: 3 })).toMatchObject({ pitch: 0 }); expect(fxChainOf(plan, { blend: 3 })).toMatchObject({ mix: 1, a: { pitch: 12 }, b: { pitch: 0 } });
}); });
it('hands a Pitch behind it to both of its sounds', () => { it('hands a Pitch behind it to both of its sounds', () => {
@@ -249,12 +256,20 @@ describe('a Mix in a sound FX graph (#316)', () => {
}); });
}); });
/** Records what it was asked to play, with the line of effects. */ /**
function fakeAudio(): AudioSink & { played: { file: string; fx: FxChain | undefined }[] } { * Records what it was asked to play, with the line of effects its values make, and every tune. A
* voice is numbered by how many have been played, so the first is 1.
*/
function fakeAudio(): AudioSink & {
played: { file: string; owner: string; fx: FxChain | undefined }[];
tuned: { voice: number; params: Readonly<Record<string, number>> }[];
} {
const sink = { const sink = {
played: [] as { file: string; fx: FxChain | undefined }[], played: [] as { file: string; owner: string; fx: FxChain | undefined }[],
play: (file: string, _volume: number, _loop: boolean, _owner: string, fx?: FxChain) => tuned: [] as { voice: number; params: Readonly<Record<string, number>> }[],
sink.played.push({ file, fx }), play: (file: string, _volume: number, _loop: boolean, owner: string, fx?: FxVoice) =>
sink.played.push({ file, owner, fx: fx && fxChainOf(fx.plan, fx.params) }),
tune: (voice: number, params: Readonly<Record<string, number>>) => sink.tuned.push({ voice, params }),
stop: () => {}, stop: () => {},
stopOwner: () => {}, stopOwner: () => {},
}; };
@@ -286,7 +301,7 @@ describe('Play sound with fx', () => {
const f = playingScene((s) => s.fx.id); const f = playingScene((s) => s.fx.id);
const audio = fakeAudio(); const audio = fakeAudio();
startWorld(f, { behaviors: true, audio }).frame(); startWorld(f, { behaviors: true, audio }).frame();
expect(audio.played).toEqual([{ file: FILE, fx: { pitch: -12, stages: [] } }]); expect(audio.played).toMatchObject([{ file: FILE, fx: { pitch: -12, stages: [] } }]);
}); });
it('plays it dry without an fx, with one that is gone, and with one that does not compile', () => { it('plays it dry without an fx, with one that is gone, and with one that does not compile', () => {
@@ -305,7 +320,8 @@ describe('Play sound with fx', () => {
expect(soundFxBuildFor(broken.fx).compiled).toBeNull(); expect(soundFxBuildFor(broken.fx).compiled).toBeNull();
const brokenAudio = fakeAudio(); const brokenAudio = fakeAudio();
startWorld(broken, { behaviors: true, audio: brokenAudio }).frame(); startWorld(broken, { behaviors: true, audio: brokenAudio }).frame();
for (const audio of [none, gone, brokenAudio]) expect(audio.played).toEqual([{ file: FILE, fx: DRY }]); for (const audio of [none, gone, brokenAudio])
expect(audio.played).toMatchObject([{ file: FILE, fx: DRY }]);
}); });
}); });
@@ -329,11 +345,11 @@ describe('the parameters of a sound FX', () => {
expect(pitchOf(fxChainOf(plan, { other: 3 }))).toBe(-12); expect(pitchOf(fxChainOf(plan, { other: 3 }))).toBe(-12);
}); });
it('keep a stage that is 0 by default, so a value handed in can open it', () => { it('keep a stage that is 0 by default, so a value handed in can open it, even on a playing voice', () => {
const f = withParam('SoundFxGain', 'gain', [{ key: 'loud', kind: 'number' }]); const f = withParam('SoundFxGain', 'gain', [{ key: 'loud', kind: 'number' }]);
const plan = soundFxBuildFor(f.fx).compiled!; const plan = soundFxBuildFor(f.fx).compiled!;
expect(plan.stages).toHaveLength(1); expect(plan.stages).toHaveLength(1);
expect(fxChainOf(plan)).toEqual(DRY); expect(fxChainOf(plan)).toEqual({ pitch: 0, stages: [{ kind: 'gain', gain: 1 }] });
expect(fxChainOf(plan, { loud: 6 }).stages[0]).toMatchObject({ kind: 'gain' }); expect(fxChainOf(plan, { loud: 6 }).stages[0]).toMatchObject({ kind: 'gain' });
expect((fxChainOf(plan, { loud: 6 }).stages[0] as { gain: number }).gain).toBeCloseTo(10 ** (6 / 20)); expect((fxChainOf(plan, { loud: 6 }).stages[0] as { gain: number }).gain).toBeCloseTo(10 ** (6 / 20));
}); });
@@ -437,7 +453,9 @@ describe('Sound FX parameters and Play sound', () => {
const f = scene({ pick: ['depth'], values: { depth: 7 } }); const f = scene({ pick: ['depth'], values: { depth: 7 } });
const audio = fakeAudio(); const audio = fakeAudio();
startWorld(f, { behaviors: true, audio }).frame(); startWorld(f, { behaviors: true, audio }).frame();
expect(audio.played).toEqual([{ file: FILE, fx: { pitch: 7, stages: [] } }]); expect(audio.played).toMatchObject([
{ file: FILE, fx: { pitch: 7, stages: [{ kind: 'gain', gain: 1 }] } },
]);
}); });
it('starts a picked parameter at the default of the FX', () => { it('starts a picked parameter at the default of the FX', () => {
@@ -449,7 +467,7 @@ describe('Sound FX parameters and Play sound', () => {
]); ]);
const audio = fakeAudio(); const audio = fakeAudio();
startWorld(f, { behaviors: true, audio }).frame(); startWorld(f, { behaviors: true, audio }).frame();
expect(audio.played[0]!.fx).toEqual({ pitch: -12, stages: [] }); expect(audio.played[0]!.fx).toEqual({ pitch: -12, stages: [{ kind: 'gain', gain: 1 }] });
}); });
it('says so at the node when no FX is picked or it is gone', () => { it('says so at the node when no FX is picked or it is gone', () => {
@@ -475,9 +493,56 @@ describe('Sound FX parameters and Play sound', () => {
expect(f.params.params.drop).toBe(7); expect(f.params.params.drop).toBe(7);
const audio = fakeAudio(); const audio = fakeAudio();
startWorld(f, { behaviors: true, audio }).frame(); startWorld(f, { behaviors: true, audio }).frame();
expect(audio.played[0]!.fx).toEqual({ pitch: 7, stages: [] }); expect(audio.played[0]!.fx).toEqual({ pitch: 7, stages: [{ kind: 'gain', gain: 1 }] });
f.stack.undo(); f.stack.undo();
expect(f.params.ports).toEqual([{ key: 'depth', kind: 'number' }]); expect(f.params.ports).toEqual([{ key: 'depth', kind: 'number' }]);
expect(f.params.params.depth).toBe(7); expect(f.params.params.depth).toBe(7);
}); });
}); });
describe('Set sound FX parameters (#315)', () => {
/**
* Pressing x plays Jump through an FX twice, the first `sound` – then hands `depth` to the voice
* the first play gave out.
*/
function tuning(sound?: string) {
const f = makeSpriteScene('Box');
const { fx } = makeFx(f);
const created = new CreateSoundCommand('Jump', FILE);
f.stack.execute(created);
const behavior = makeBehavior(f, 'Logic');
place(f, behavior.id, { x: 0, y: 0 }, place(f, f.spriteId, { x: 0, y: 0 }));
const edit = graphEditor(f, behavior);
const first = edit.addNode('PlaySound', { sound: sound ?? created.asset.id, fx: fx.id });
const second = edit.addNode('PlaySound', { sound: created.asset.id, fx: fx.id });
const set = edit.addNode('SetSoundFxParameters');
const params = edit.addNode('SoundFxParameters', { soundFxAsset: fx.id, depth: 3 });
params.ports = [{ key: 'depth', kind: 'number' }];
edit.wire([
[edit.addNode('OnKeyDown', { key: 'x' }), 'exec', first, 'exec'],
[first, 'exec', second, 'exec'],
[second, 'exec', set, 'exec'],
[first, 'voice', set, 'voice'],
[params, 'params', set, 'fxParams'],
]);
return f;
}
it('hands the values to the one voice its Play sound gave out, not to another of the same sound', () => {
const audio = fakeAudio();
const { input, frame } = startWorld(tuning(), { behaviors: true, audio });
input.press('x');
frame();
expect(audio.played).toHaveLength(2);
expect(audio.tuned).toEqual([{ voice: 1, params: { depth: 3 } }]);
});
it('reaches nothing where its Play sound played nothing', () => {
const audio = fakeAudio();
const { input, frame } = startWorld(tuning('no-such-sound'), { behaviors: true, audio });
input.press('x');
frame();
expect(audio.played).toHaveLength(1);
expect(audio.tuned).toEqual([]);
});
});
+1
View File
@@ -42,6 +42,7 @@ function fakeAudio(): AudioSink & {
stoppedOwners: [] as string[], stoppedOwners: [] as string[],
play: (file: string, volume: number, loop: boolean, owner: string) => play: (file: string, volume: number, loop: boolean, owner: string) =>
sink.played.push({ file, volume, loop, owner }), sink.played.push({ file, volume, loop, owner }),
tune: () => {},
stop: (file: string) => sink.stopped.push(file), stop: (file: string) => sink.stopped.push(file),
stopOwner: (owner: string) => sink.stoppedOwners.push(owner), stopOwner: (owner: string) => sink.stoppedOwners.push(owner),
}; };
+4
View File
@@ -214,6 +214,7 @@ export type FakeContext = NodeContext & {
fxParams: Readonly<Record<string, number>>; fxParams: Readonly<Record<string, number>>;
}[]; }[];
silenced: string[]; silenced: string[];
tuned: { voice: number; fxParams: Readonly<Record<string, number>> }[];
/** The scenes the context was asked to load, in order. */ /** The scenes the context was asked to load, in order. */
loaded: string[]; loaded: string[];
/** The time scale and the pause the context holds, as the game nodes left them. */ /** The time scale and the pause the context holds, as the game nodes left them. */
@@ -265,9 +266,12 @@ export function fakeContext(overrides: Partial<NodeContext> = {}): FakeContext {
played: [], played: [],
silenced: [], silenced: [],
loaded: [], loaded: [],
// The number of a voice is how many have been played: unique, and never 0.
playSound: (assetId, volume, loop, detached, fx, fxParams) => playSound: (assetId, volume, loop, detached, fx, fxParams) =>
ctx.played.push({ assetId, volume, loop, detached, fx, fxParams }), ctx.played.push({ assetId, volume, loop, detached, fx, fxParams }),
stopSound: (assetId) => ctx.silenced.push(assetId), stopSound: (assetId) => ctx.silenced.push(assetId),
tuned: [],
tuneSound: (voice, fxParams) => ctx.tuned.push({ voice, fxParams }),
loadScene: (sceneId) => ctx.loaded.push(sceneId), loadScene: (sceneId) => ctx.loaded.push(sceneId),
spawn: () => null, spawn: () => null,
addJoint: () => null, addJoint: () => null,
+126 -15
View File
@@ -6,16 +6,28 @@
* The device is stubbed – nothing makes a noise in a unit test, and Web Audio does not exist here. * The device is stubbed – nothing makes a noise in a unit test, and Web Audio does not exist here.
*/ */
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { FxChain } from '../../src/core/soundfx/nodes'; import type { FxChain, FxPlan, FxVoice } from '../../src/core/soundfx/nodes';
import { SoundLibrary } from '../../src/viewport/SoundLibrary'; import { SoundLibrary } from '../../src/viewport/SoundLibrary';
const FILE = 'a.wav'; const FILE = 'a.wav';
const OTHER = 'b.wav'; const OTHER = 'b.wav';
/** An AudioParam that remembers where it was sent to glide, and how often. */
interface FakeParam {
value: number;
glides: number[];
setTargetAtTime: (value: number) => void;
}
function param(value: number): FakeParam {
const made: FakeParam = { value, glides: [], setTargetAtTime: (to) => made.glides.push(to) };
return made;
}
interface FakeSource { interface FakeSource {
buffer: unknown; buffer: unknown;
loop: boolean; loop: boolean;
playbackRate: { value: number }; playbackRate: FakeParam;
onended: (() => void) | null; onended: (() => void) | null;
started: boolean; started: boolean;
stopped: boolean; stopped: boolean;
@@ -28,11 +40,11 @@ const sources: FakeSource[] = [];
/** Every `connect` made, as "from > to" by the kind of node – the order a voice is wired in. */ /** Every `connect` made, as "from > to" by the kind of node – the order a voice is wired in. */
const wires: string[] = []; const wires: string[] = [];
/** The level of every gain node made, in the order they were made. */ /** The level of every gain node made, in the order they were made. */
const gains: { value: number }[] = []; const gains: FakeParam[] = [];
const filters: { const filters: {
type: string; type: string;
frequency: { value: number }; frequency: { value: number };
gain: { value: number }; gain: FakeParam;
Q: { value: number }; Q: { value: number };
}[] = []; }[] = [];
@@ -69,6 +81,7 @@ function fakeBuffer(_channels: number, length: number, sampleRate: number) {
function fakeContext() { function fakeContext() {
return { return {
state: 'running', state: 'running',
currentTime: 0,
destination: { kind: 'destination' }, destination: { kind: 'destination' },
resume: () => Promise.resolve(), resume: () => Promise.resolve(),
suspend: () => Promise.resolve(), suspend: () => Promise.resolve(),
@@ -78,7 +91,7 @@ function fakeContext() {
const source: FakeSource = wired('source', { const source: FakeSource = wired('source', {
buffer: null, buffer: null,
loop: false, loop: false,
playbackRate: { value: 1 }, playbackRate: param(1),
onended: null, onended: null,
started: false, started: false,
stopped: false, stopped: false,
@@ -93,7 +106,7 @@ function fakeContext() {
return source; return source;
}, },
createGain: () => { createGain: () => {
const gain = wired('gain', { gain: { value: 1 } }); const gain = wired('gain', { gain: param(1) });
gains.push(gain.gain); gains.push(gain.gain);
return gain; return gain;
}, },
@@ -101,7 +114,7 @@ function fakeContext() {
const filter = wired('filter', { const filter = wired('filter', {
type: '', type: '',
frequency: { value: 0 }, frequency: { value: 0 },
gain: { value: 0 }, gain: param(0),
Q: { value: 0 }, Q: { value: 0 },
}); });
filters.push(filter); filters.push(filter);
@@ -136,11 +149,37 @@ afterEach(() => {
vi.unstubAllGlobals(); vi.unstubAllGlobals();
}); });
/** Plays `file` and lets its bytes arrive, unless `hold` keeps the decode pending. */ /** A chain as the plan of an FX with nothing left open, so a test can say what it wants to hear. */
async function play(library: SoundLibrary, file = FILE, { hold = false } = {}, fx?: FxChain): Promise<void> { function planOf(chain: FxChain): FxPlan {
library.play(file, 1, false, '', fx); const stages = chain.stages.map((stage) =>
stage.kind === 'gain'
? { kind: 'gain' as const, db: { value: 20 * Math.log10(stage.gain) } }
: { ...stage, gain: { value: stage.gain } },
);
if ('mix' in chain) return { mix: { value: chain.mix }, a: planOf(chain.a), b: planOf(chain.b), stages };
return { pitch: chain.pitch === 0 ? [] : [{ value: chain.pitch }], stages };
}
/** Plays `file` and lets its bytes arrive, unless `hold` keeps the decode pending; gives the voice. */
async function play(
library: SoundLibrary,
file = FILE,
{ hold = false, owner = '' } = {},
fx?: FxChain | FxVoice,
): Promise<number> {
const voice = library.play(
file,
1,
false,
owner,
fx && ('plan' in fx ? fx : { plan: planOf(fx), params: {} }),
);
await settle(); await settle();
if (hold) return; if (!hold) await arrive();
return voice;
}
async function arrive(): Promise<void> {
for (const resolve of decodes.splice(0)) resolve(); for (const resolve of decodes.splice(0)) resolve();
await settle(); await settle();
} }
@@ -208,7 +247,7 @@ describe('the sound library', () => {
expect([up!.duration, down!.duration]).toEqual([dry!.duration, dry!.duration]); expect([up!.duration, down!.duration]).toEqual([dry!.duration, dry!.duration]);
}); });
it('mixes two sounds by their shares, a source per pitch, started and stopped together (#316)', async () => { it('mixes two sounds by their shares, a source per way, started and stopped together (#316)', async () => {
const library = new SoundLibrary(); const library = new SoundLibrary();
await play( await play(
library, library,
@@ -231,7 +270,7 @@ describe('the sound library', () => {
'filter > gain', 'filter > gain',
'gain > destination', 'gain > destination',
]); ]);
expect(gains.map((g) => g.value)).toEqual([1, 1, 0.75, 0.5, 0.25]); expect(gains.map((g) => g.value)).toEqual([1, 1, 0.75, expect.closeTo(0.5), 0.25]);
expect(sources.map((s) => s.started)).toEqual([true, true]); expect(sources.map((s) => s.started)).toEqual([true, true]);
expect(library.stats()).toMatchObject({ playing: 1, started: 1, last: { pitches: [12, 0], stages: 2 } }); expect(library.stats()).toMatchObject({ playing: 1, started: 1, last: { pitches: [12, 0], stages: 2 } });
@@ -242,7 +281,7 @@ describe('the sound library', () => {
expect(library.stats()).toMatchObject({ playing: 0, stopped: 1 }); expect(library.stats()).toMatchObject({ playing: 0, stopped: 1 });
}); });
it('starts one source for two ways at the same pitch', async () => { it('gives two ways at the same pitch a source each, so each can glide on its own (#315)', async () => {
const library = new SoundLibrary(); const library = new SoundLibrary();
await play( await play(
library, library,
@@ -255,10 +294,82 @@ describe('the sound library', () => {
stages: [], stages: [],
}, },
); );
expect(sources).toHaveLength(1); expect(sources).toHaveLength(2);
expect(sources[0]!.buffer).toBe(sources[1]!.buffer);
expect(wires.filter((w) => w.startsWith('source'))).toEqual(['source > gain', 'source > gain']); expect(wires.filter((w) => w.startsWith('source'))).toEqual(['source > gain', 'source > gain']);
}); });
describe('a parameter changed while the voice plays (#315)', () => {
/** Pitch on a, a gain on b, the share between them – all three open – and a band after the Mix. */
const open: FxPlan = {
mix: { value: 0.25, param: 'blend', min: 0, max: 1 },
a: { pitch: [{ value: 0, param: 'depth', min: -64, max: 64 }], stages: [] },
b: { pitch: [], stages: [{ kind: 'gain', db: { value: 0, param: 'loud', min: -120, max: 24 } }] },
stages: [{ kind: 'filter', filter: 'lowshelf', frequency: 80, gain: { value: 6 }, q: 1 }],
};
it('glides every knob it moves, a pitch by speed, and leaves the rest alone', async () => {
const library = new SoundLibrary();
const voice = await play(library, FILE, { owner: 'me' }, { plan: open, params: {} });
// Made in the order: the voice, the Mix, share of a, the gain on b, share of b.
const [, , shareA, loud, shareB] = gains;
expect([shareA!.value, loud!.value, shareB!.value]).toEqual([0.75, 1, 0.25]);
expect(library.tune(voice, { depth: 12, loud: -6, blend: 0.5, nothing: 3 })).toBe(true);
expect(sources[0]!.playbackRate.glides).toEqual([2]);
expect(sources[1]!.playbackRate.glides).toEqual([]);
expect(loud!.glides).toEqual([expect.closeTo(10 ** (-6 / 20))]);
expect([shareA!.glides, shareB!.glides]).toEqual([[0.5], [0.5]]);
expect(filters[0]!.gain.glides).toEqual([]);
expect(library.stats().tuned).toBe(1);
// Only what changed is sent again; one not given keeps its value rather than its default.
library.tune(voice, { depth: 12, blend: 1 });
expect(sources[0]!.playbackRate.glides).toEqual([2]);
expect(loud!.glides).toHaveLength(1);
expect([shareA!.glides, shareB!.glides]).toEqual([
[0.5, 0],
[0.5, 1],
]);
});
it('reaches the one voice it is given – a detached one too – and none that has ended', async () => {
const library = new SoundLibrary();
const voices = [
await play(library, FILE, { owner: 'me' }, { plan: open, params: {} }),
await play(library, FILE, { owner: 'me' }, { plan: open, params: {} }),
await play(library, OTHER, { owner: 'me' }, { plan: open, params: {} }),
await play(library, FILE, {}, { plan: open, params: {} }),
];
expect(new Set(voices).size).toBe(4);
expect(voices).not.toContain(0);
library.tune(voices[1]!, { depth: 12 });
library.tune(voices[3]!, { depth: 12 });
// Two sources a voice, the first of each the pitched way.
expect(sources.map((s) => s.playbackRate.glides.length)).toEqual([0, 0, 1, 0, 0, 0, 1, 0]);
library.stop(FILE);
expect(library.tune(voices[1]!, { depth: 3 })).toBe(false);
expect(library.tune(0, { depth: 3 })).toBe(false);
expect(library.stats().tuned).toBe(2);
});
it('starts a voice still waiting on its bytes with the values handed in meanwhile', async () => {
const library = new SoundLibrary();
const voice = await play(
library,
FILE,
{ hold: true, owner: 'me' },
{ plan: open, params: { blend: 0.5 } },
);
expect(library.tune(voice, { depth: 7 })).toBe(true);
await arrive();
expect(library.stats().last?.pitches).toEqual([7, 0]);
expect(gains.map((g) => g.value)).toContain(0.5);
expect(sources[0]!.playbackRate.glides).toEqual([]);
});
});
it('a stop while the bytes are in flight keeps the sound from starting afterwards', async () => { it('a stop while the bytes are in flight keeps the sound from starting afterwards', async () => {
const library = new SoundLibrary(); const library = new SoundLibrary();
await play(library, FILE, { hold: true }); await play(library, FILE, { hold: true });
+67 -1
View File
@@ -1,4 +1,4 @@
import { edge, node } from '../lib/builders.mjs'; import { edge, node, port, variable, written } from '../lib/builders.mjs';
import { LANES, MAKE_OBJECT, pickAssetType } from './lib/scaffold.mjs'; import { LANES, MAKE_OBJECT, pickAssetType } from './lib/scaffold.mjs';
/** /**
@@ -414,6 +414,72 @@ export async function sound(t) {
JSON.stringify(gameParam), JSON.stringify(gameParam),
); );
await stop(); await stop();
// #315: two looping voices of the same sound start at depth 7, the first keeps its number in a
// variable, and `Set sound FX parameters` hands that one voice depth 3 every frame. In a
// real AudioContext the first one's speed glides there, the second one's stays put.
const tuneIds = ['sn-keep', 'sn-play2', 'sn-tick', 'sn-read', 'sn-tune', 'sn-fxp2'];
const tuneEdges = [
'sn-e-keep',
'sn-e-voice',
'sn-e-play2',
'sn-e-fxp3',
'sn-e-tick',
'sn-e-read',
'sn-e-fxp2',
];
await mutate(`
const logic = __gm.state.project.assets.find((a) => a.id === 'sn-logic');
logic.variables = [${JSON.stringify(variable('voice', 'number', 0))}];
const graph = logic.graph;
const play = graph.nodes.find((n) => n.id === 'sn-play');
play.params.loop = 'true';
graph.nodes.push(
${JSON.stringify(node('sn-keep', 'Variables', 520, 0, {}, [written('voice', 'number')]))},
{ ...structuredClone(play), id: 'sn-play2', x: 780 },
${JSON.stringify(node('sn-tick', 'OnUpdate', 0, 300))},
${JSON.stringify(node('sn-read', 'Variables', 0, 420, {}, [port('voice', 'number')]))},
${JSON.stringify(node('sn-tune', 'SetSoundFxParameters', 260, 300))},
${JSON.stringify(
node('sn-fxp2', 'SoundFxParameters', 0, 540, { soundFxAsset: fxId, depth: 3 }, [
{ key: 'depth', kind: 'number' },
]),
)},
);
graph.edges.push(
${JSON.stringify(edge('sn-e-keep', 'sn-play', 'exec', 'sn-keep', 'exec'))},
${JSON.stringify(edge('sn-e-voice', 'sn-play', 'voice', 'sn-keep', 'voice'))},
${JSON.stringify(edge('sn-e-play2', 'sn-keep', 'exec', 'sn-play2', 'exec'))},
${JSON.stringify(edge('sn-e-fxp3', 'sn-fxp', 'params', 'sn-play2', 'fxParams'))},
${JSON.stringify(edge('sn-e-tick', 'sn-tick', 'exec', 'sn-tune', 'exec'))},
${JSON.stringify(edge('sn-e-read', 'sn-read', 'voice', 'sn-tune', 'voice'))},
${JSON.stringify(edge('sn-e-fxp2', 'sn-fxp2', 'params', 'sn-tune', 'fxParams'))},
);`);
const beforeTune = await stats();
await play();
const tunedStats = await waitFor(
stats,
(s) => s.tuned > beforeTune.tuned && s.started >= beforeTune.started + 2,
);
const rate = 2 ** ((3 - 7) / 12);
const glided = (levels) =>
Array.isArray(levels) && levels.length === 2 && Math.abs(levels[0][0] - rate) < 0.01;
const levels = await waitFor(() => json(`__gm.sound.levels(${JSON.stringify(blip.file)})`), glided);
check(
'Set sound FX parameters moves the one voice whose number it holds from depth 7 to 3, not the other',
tunedStats.last?.pitches.join() === '7' &&
tunedStats.last.loop === true &&
glided(levels) &&
Math.abs(levels[0][1] - 10 ** (-6 / 20)) < 0.001 &&
levels[1][0] === 1,
JSON.stringify({ last: tunedStats.last, tuned: tunedStats.tuned, levels, rate }),
);
await stop();
await mutate(`
const logic = __gm.state.project.assets.find((a) => a.id === 'sn-logic');
logic.variables = [];
const graph = logic.graph;
graph.nodes = graph.nodes.filter((n) => !${JSON.stringify(tuneIds)}.includes(n.id));
graph.edges = graph.edges.filter((e) => !${JSON.stringify(tuneEdges)}.includes(e.id));`);
await mutate(` await mutate(`
const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph; const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph;
graph.nodes = graph.nodes.filter((n) => n.id !== 'sn-fxp'); graph.nodes = graph.nodes.filter((n) => n.id !== 'sn-fxp');