Change the sound FX parameters of a sound while it plays #315
Until now a value handed to a sound FX counted from the start of the voice only. An engine whose pitch follows the speed, or a loop that gets quieter, could not be built. Every play of Play sound now gives out the number of its voice, and a new node lets the knobs of that one voice glide to new values while it plays. - Play sound has a new output voice: the number of this one play, or 0 where nothing was played. Kept in a Number variable it reaches the voice from another event. Two plays of one sound are two voices. - New action node Set sound FX parameters (exec, voice, fx params). It reaches exactly the voice it is given, a detached one too. A voice that has ended, or voice 0, is not reached. - AudioSink.play returns the voice number and takes the plan and the values (FxVoice) instead of a finished chain. AudioSink.tune takes a voice number and new values. A name the voice's FX does not have is left out, one not given keeps its value. - SoundLibrary hands out the number at once, before the bytes are there, and never twice. A voice still waiting for its bytes starts with the values handed in meanwhile. - Every knob of a voice moves with setTargetAtTime (30 ms time constant). A knob whose value did not change is not sent again, so a tune every frame costs next to nothing. - A pitch on a playing voice glides by playbackRate. A new WSOLA copy in the middle of a sound is not worked out, so the voice gets shorter or longer while it is moved. On a loop that is not heard. - fxChainOf now gives a chain whose shape depends on the plan alone. A stage fed by a parameter stays at 0 dB, and a Mix fed by a parameter keeps both ways. A typed-in 0 or Mix at 0/1 is still left out by the compiler. - Every way of a voice has a source of its own, also at the same pitch, so each can glide on its own. The pitched copy is shared. - The tab keeps the numbers of what it plays, so moving a parameter field is heard at once. - Test hook __gm.sound.levels(file) reads the knobs of every voice of a file, oldest first, and stats() counts the voices tuned. Tests: - soundfx.test.ts: 29 tests (+2 for the node in a running world: the voice of the first of two plays of one sound, and a play that played nothing) - sound-library.test.ts: 15 tests (+3: glide, reach by number, pending voice) - node-catalog.test.ts: 370 tests (+2, Play sound checks its voice) - sounds + keep-on-scene-load: 31 tests, unchanged - Counter-test: a tune that reaches every voice of the same file makes the reach test fail. - e2e 34 (melektaus-atx): ALL PASSED - 30 checks (+1). In Chrome two looping voices of one sound start at depth 7. The one whose number sits in a variable glides to 0.7938 (2^(-4/12) = 0.7937), the other stays at 1. Both gains stay at 0.5012. - The gate ran before voices got a number (CHECK EXIT 0, 241 files, 3658 tests), not after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
072537b2e8
commit
c7e9769f90
18 files changed
+557
-100
No files matched your search
@@ -240,6 +240,7 @@ export interface NodeContext {
|
||||
* the one case where the rule is in the way: the death sound of an object that destroys itself.
|
||||
* `fx` names a sound FX asset the voice is played through (#311); `''` plays it as it is.
|
||||
* `fxParams` are values for its parameters; one not given is the default the FX declares.
|
||||
* Returns the number of the voice for `tuneSound`, or 0 where nothing is played.
|
||||
*/
|
||||
playSound(
|
||||
assetId: string,
|
||||
@@ -248,9 +249,14 @@ export interface NodeContext {
|
||||
detached: boolean,
|
||||
fx: string,
|
||||
fxParams: FxParams,
|
||||
): void;
|
||||
): number;
|
||||
/** Silences every voice of that sound; `''` silences everything this run started. */
|
||||
stopSound(assetId: string): void;
|
||||
/**
|
||||
* Hands new values to the sound FX parameters of one voice, by the number `playSound` gave it
|
||||
* (#315), so a running sound follows the game. A voice that has ended is not reached.
|
||||
*/
|
||||
tuneSound(voice: number, fxParams: FxParams): void;
|
||||
/**
|
||||
* Asks for another scene, by asset id. The swap happens once the frame is over – a graph must be
|
||||
* able to finish the flow it is in, and half a frame in the old world and half in the new one is
|
||||
|
||||
@@ -59,7 +59,9 @@ export const PlaySound: NodeDef = {
|
||||
'the node only colors it. Each play is a voice of its own: firing this twice in a row is heard ' +
|
||||
'twice, overlapping. The sound belongs to this object and stops when the object is destroyed, ' +
|
||||
'unless "detached" is on. "fx" sends it through a sound FX asset; left empty, it plays as it is. ' +
|
||||
'"fx params" takes values for its parameters from a Sound FX parameters node.',
|
||||
'"fx params" takes values for its parameters from a Sound FX parameters node. "voice" is the ' +
|
||||
'number of this one play, for Set sound FX parameters to change it while it plays; 0 where ' +
|
||||
'nothing was played.',
|
||||
inputs: [
|
||||
exec(),
|
||||
soundPort(),
|
||||
@@ -69,25 +71,26 @@ export const PlaySound: NodeDef = {
|
||||
soundFxPort(),
|
||||
{ key: 'fxParams', label: 'fx params', type: 'fxparams' },
|
||||
],
|
||||
outputs: [exec()],
|
||||
outputs: [exec(), num('voice')],
|
||||
params: [],
|
||||
run: (ctx, inputs) => {
|
||||
const sound = asString(inputs.sound);
|
||||
if (sound)
|
||||
ctx.playSound(
|
||||
const voice = sound
|
||||
? ctx.playSound(
|
||||
sound,
|
||||
asNumber(inputs.volume),
|
||||
loopChoice(inputs.loop),
|
||||
asBoolean(inputs.detached),
|
||||
asString(inputs.fx),
|
||||
fxParamsOf(inputs.fxParams),
|
||||
);
|
||||
return { next: 'exec' };
|
||||
)
|
||||
: 0;
|
||||
return { next: 'exec', outputs: { voice } };
|
||||
},
|
||||
};
|
||||
|
||||
/** The numbers of a parameter bundle; anything else hands in none. */
|
||||
function fxParamsOf(value: Value | undefined): FxParams {
|
||||
export function fxParamsOf(value: Value | undefined): FxParams {
|
||||
if (!isStruct(value)) return {};
|
||||
const params: Record<string, number> = {};
|
||||
for (const [key, item] of Object.entries(value)) if (typeof item === 'number') params[key] = item;
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import type { NodeDef } from '../../NodeRegistry';
|
||||
import { asNumber } from '../../values';
|
||||
import { exec, num } from '../shared';
|
||||
import { fxParamsOf } from './PlaySound';
|
||||
|
||||
/**
|
||||
* Changes the sound FX parameters of a sound that is already playing (#315). It reaches one voice,
|
||||
* by the number its `Play sound` handed out: every play is a sound of its own, and two objects with
|
||||
* the same engine sound – or one object playing it twice – each keep theirs.
|
||||
*/
|
||||
export const SetSoundFxParameters: NodeDef = {
|
||||
type: 'SetSoundFxParameters',
|
||||
label: 'Set sound FX parameters',
|
||||
category: 'action',
|
||||
description:
|
||||
'Changes the sound FX parameters of a sound while it plays: the voice a Play sound handed out ' +
|
||||
'glides to the values from a Sound FX parameters node. Keep the voice in a variable to reach it ' +
|
||||
'from another event. A parameter not given keeps its value, a voice that has ended is not ' +
|
||||
'reached. A pitch changes by speed here, so the sound gets shorter or longer while it is moved.',
|
||||
inputs: [
|
||||
exec(),
|
||||
{ ...num('voice'), inline: false },
|
||||
{ key: 'fxParams', label: 'fx params', type: 'fxparams' },
|
||||
],
|
||||
outputs: [exec()],
|
||||
params: [],
|
||||
run: (ctx, inputs) => {
|
||||
const voice = asNumber(inputs.voice);
|
||||
if (voice > 0) ctx.tuneSound(voice, fxParamsOf(inputs.fxParams));
|
||||
return { next: 'exec' };
|
||||
},
|
||||
};
|
||||
@@ -6,14 +6,14 @@ import { asNumber } from '../../values';
|
||||
/**
|
||||
* Values for the parameters of one sound FX (#311), picked from those its Input declares the way a
|
||||
* Variables node picks variables. Each picked one is a number input starting at the FX's default;
|
||||
* the bundle goes into the `fx params` input of a `Play sound`.
|
||||
* the bundle goes into the `fx params` input of a `Play sound` or a `Set sound FX parameters`.
|
||||
*/
|
||||
export const SoundFxParameters: NodeDef = {
|
||||
type: SOUND_FX_PARAMETERS,
|
||||
label: 'Sound FX parameters',
|
||||
category: 'data',
|
||||
description:
|
||||
'Values for the parameters of a sound FX, for the "fx params" input of a Play sound. Pick the parameters to set; one not picked keeps the default its Input gives it.',
|
||||
'Values for the parameters of a sound FX, for the "fx params" input of a Play sound or a Set sound FX parameters. Pick the parameters to set; one not picked keeps the default its Input gives it, or on a playing sound the value it has.',
|
||||
inputs: [],
|
||||
outputs: [{ key: 'params', label: 'params', type: 'fxparams' }],
|
||||
params: [SOUND_FX_SETTING],
|
||||
|
||||
@@ -23,6 +23,7 @@ import { Raycast } from './action/Raycast';
|
||||
import { RaycastAll } from './action/RaycastAll';
|
||||
import { RestartGame } from './action/RestartGame';
|
||||
import { SetPosition } from './action/SetPosition';
|
||||
import { SetSoundFxParameters } from './action/SetSoundFxParameters';
|
||||
import { SetZ } from './action/SetZ';
|
||||
import { Spawn } from './action/Spawn';
|
||||
import { StopGame } from './action/StopGame';
|
||||
@@ -298,6 +299,7 @@ export const BUILT_IN_NODES: readonly NodeDef[] = [
|
||||
StopAnimation,
|
||||
PlaySound,
|
||||
StopSound,
|
||||
SetSoundFxParameters,
|
||||
SoundFxParameters,
|
||||
LoadScene,
|
||||
Log,
|
||||
|
||||
+19
-10
@@ -50,6 +50,14 @@ export const EMPTY_PLAN: FxPlan = { pitch: [], stages: [] };
|
||||
/** Values for the parameters, by name; a name the plan does not have is left out. */
|
||||
export type FxParams = Readonly<Record<string, number>>;
|
||||
|
||||
/** How a voice goes through its sound FX: the plan and the values it starts with, kept so they can change while it plays (#315). */
|
||||
export interface FxVoice {
|
||||
plan: FxPlan;
|
||||
params: FxParams;
|
||||
}
|
||||
|
||||
export const DRY_VOICE: FxVoice = { plan: EMPTY_PLAN, params: {} };
|
||||
|
||||
/**
|
||||
* A node of a sound FX graph (#311). It never runs: `emit` says what it does to the sounds arriving
|
||||
* at its audio inputs, by input key, given the numbers at its other inputs. Input and Output have no
|
||||
@@ -239,23 +247,24 @@ function amountOf(amount: FxAmount, params: FxParams): number {
|
||||
return Math.max(amount.min ?? -Infinity, Math.min(amount.max ?? Infinity, given));
|
||||
}
|
||||
|
||||
/** A stage fed by a parameter stays at 0 as well: a value handed in later has to find it there. */
|
||||
function stagesOf(plan: FxPlan, params: FxParams): FxStage[] {
|
||||
const stages: FxStage[] = [];
|
||||
for (const stage of plan.stages) {
|
||||
if (stage.kind === 'gain') {
|
||||
const db = amountOf(stage.db, params);
|
||||
if (db !== 0) stages.push({ kind: 'gain', gain: 10 ** (db / 20) });
|
||||
if (db !== 0 || stage.db.param !== undefined) stages.push({ kind: 'gain', gain: 10 ** (db / 20) });
|
||||
continue;
|
||||
}
|
||||
const gain = amountOf(stage.gain, params);
|
||||
if (gain !== 0) stages.push({ ...stage, gain });
|
||||
if (gain !== 0 || stage.gain.param !== undefined) stages.push({ ...stage, gain });
|
||||
}
|
||||
return stages;
|
||||
}
|
||||
|
||||
/**
|
||||
* The chain one voice is sent along: the plan with every parameter filled in. A Mix that ends at 0
|
||||
* or 1 is left out with the sound it silences, so that one is not even started.
|
||||
* The chain one voice is sent along: the plan with every parameter filled in. Its shape depends on
|
||||
* the plan alone, never on the values, so the values of a playing voice can change in place (#315).
|
||||
*/
|
||||
export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain {
|
||||
const stages = stagesOf(plan, params);
|
||||
@@ -263,10 +272,10 @@ export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain {
|
||||
const pitch = plan.pitch.reduce((sum, amount) => sum + amountOf(amount, params), 0);
|
||||
return { pitch, stages };
|
||||
}
|
||||
const mix = amountOf(plan.mix, params);
|
||||
if (mix <= 0 || mix >= 1) {
|
||||
const heard = fxChainOf(mix <= 0 ? plan.a : plan.b, params);
|
||||
return { ...heard, stages: [...heard.stages, ...stages] };
|
||||
}
|
||||
return { mix, a: fxChainOf(plan.a, params), b: fxChainOf(plan.b, params), stages };
|
||||
return {
|
||||
mix: amountOf(plan.mix, params),
|
||||
a: fxChainOf(plan.a, params),
|
||||
b: fxChainOf(plan.b, params),
|
||||
stages,
|
||||
};
|
||||
}
|
||||
@@ -41,6 +41,8 @@ export function installDebugHook(app: Application, state: EditorState, editor: E
|
||||
sound: {
|
||||
stats: () => soundLibrary.stats(),
|
||||
duration: (file: string) => soundLibrary.duration(file),
|
||||
/** Where the knobs of each voice of that file are now, oldest first, a pitch as its speed (#315). */
|
||||
levels: (file: string) => soundLibrary.levels(file),
|
||||
/** Decodes ahead of a check, so a play can be asserted without racing the network. */
|
||||
load: (file: string) => soundLibrary.load(file).then((buffer) => buffer?.duration ?? null),
|
||||
stopAll: () => soundLibrary.stopAll(),
|
||||
|
||||
@@ -3,7 +3,6 @@ import { findAssetOf } from '../../core/model/Project';
|
||||
import { resolveAssetProps } from '../../core/properties/resolve';
|
||||
import { volumeOf } from '../../core/properties/sounds';
|
||||
import { soundFxBuildFor, soundFxParams } from '../../core/soundfx/compile';
|
||||
import { fxChainOf } from '../../core/soundfx/nodes';
|
||||
import { Button, Dropdown, HStack, Label, Menu, NumberField, Stack, Theme } from '../../ui';
|
||||
import { soundLibrary } from '../../viewport/SoundLibrary';
|
||||
import { AssetTabEditor } from '../AssetTabEditor';
|
||||
@@ -40,6 +39,8 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
|
||||
private defaults: Record<string, number> = {};
|
||||
private paramsKey = '';
|
||||
private valuesFor = '';
|
||||
/** The voices the tab started that may still be playing – what a moved field reaches. */
|
||||
private voices: number[] = [];
|
||||
|
||||
constructor(state: EditorState) {
|
||||
super(state, { infoWidth: INFO_WIDTH });
|
||||
@@ -89,8 +90,8 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
|
||||
/** Sets one parameter the header plays with, as typing into its field does. */
|
||||
setPreviewValue(key: string, value: number): void {
|
||||
if (!(key in this.values)) return;
|
||||
this.values[key] = value;
|
||||
this.paramFields.get(key)?.setNumber(value);
|
||||
this.setValue(key, value);
|
||||
}
|
||||
|
||||
/** Plays the picked sound through the graph; a graph that does not compile is not played at all. */
|
||||
@@ -100,11 +101,22 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
|
||||
const plan = asset && soundFxBuildFor(asset).compiled;
|
||||
if (!sound?.file || !plan) return;
|
||||
const volume = volumeOf(resolveAssetProps(this.state.project, sound));
|
||||
soundLibrary.play(sound.file, volume, false, PREVIEW_OWNER, fxChainOf(plan, this.values));
|
||||
const voice = soundLibrary.play(sound.file, volume, false, PREVIEW_OWNER, {
|
||||
plan,
|
||||
params: { ...this.values },
|
||||
});
|
||||
if (voice) this.voices.push(voice);
|
||||
}
|
||||
|
||||
/** A moved field is heard at once on what the tab is playing (#315); an ended voice is let go. */
|
||||
private setValue(key: string, value: number): void {
|
||||
this.values[key] = value;
|
||||
this.voices = this.voices.filter((voice) => soundLibrary.tune(voice, { [key]: value }));
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
soundLibrary.stopOwner(PREVIEW_OWNER);
|
||||
this.voices = [];
|
||||
}
|
||||
|
||||
protected override infoText(asset: SoundFxAsset): string {
|
||||
@@ -152,8 +164,8 @@ export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
|
||||
const field = new NumberField(values[port.key], {
|
||||
width: PARAM_WIDTH,
|
||||
step: 0.1,
|
||||
onInput: (value) => (this.values[port.key] = value),
|
||||
onChange: (value) => (this.values[port.key] = value),
|
||||
onInput: (value) => this.setValue(port.key, value),
|
||||
onChange: (value) => this.setValue(port.key, value),
|
||||
});
|
||||
this.paramFields.set(port.key, field);
|
||||
this.paramBar.add(field);
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { FxChain } from '../../core/soundfx/nodes';
|
||||
import type { FxParams, FxVoice } from '../../core/soundfx/nodes';
|
||||
|
||||
/**
|
||||
* What the runtime needs of the outside world to make a noise – the counterpart of `Input` on the
|
||||
@@ -13,10 +13,17 @@ export interface AudioSink {
|
||||
* Starts one voice. `volume` is final (0–1) and every call overlaps whatever is already playing.
|
||||
* `owner` is the behavior the sound belongs to: when that behavior is destroyed the voice goes
|
||||
* with it, because a sound is something a thing in the world is doing, not a thing of its own.
|
||||
* `''` means the voice has no owner and outlives whatever started it. `fx` is the line of effects
|
||||
* the voice passes on its way out (#311); `DRY` passes none.
|
||||
* `''` means the voice has no owner and outlives whatever started it. `fx` is the sound FX the
|
||||
* voice passes on its way out (#311), with the values of its parameters; `DRY_VOICE` passes none.
|
||||
* Returns the number of the voice, never used twice, or 0 where nothing is going to play.
|
||||
*/
|
||||
play(file: string, volume: number, loop: boolean, owner: string, fx?: FxChain): void;
|
||||
play(file: string, volume: number, loop: boolean, owner: string, fx?: FxVoice): number;
|
||||
/**
|
||||
* Hands new values to the sound FX parameters of one voice, by the number `play` gave it (#315).
|
||||
* A name its FX does not have is left out, one not given keeps its value. A voice that has ended
|
||||
* is not there any more and nothing happens – a detached one is reached like any other.
|
||||
*/
|
||||
tune(voice: number, params: FxParams): void;
|
||||
/** Silences every voice of that file; `''` silences all of them. */
|
||||
stop(file: string): void;
|
||||
/** Silences every voice started by that owner – what destroying a behavior does. */
|
||||
@@ -24,7 +31,8 @@ export interface AudioSink {
|
||||
}
|
||||
|
||||
export const SILENT_AUDIO: AudioSink = {
|
||||
play: () => {},
|
||||
play: () => 0,
|
||||
tune: () => {},
|
||||
stop: () => {},
|
||||
stopOwner: () => {},
|
||||
};
|
||||
@@ -16,7 +16,7 @@ import { hasRenderLayer } from '../../core/properties/PropertySchema';
|
||||
import { resolveAssetProps } from '../../core/properties/resolve';
|
||||
import { loopsOf, volumeOf } from '../../core/properties/sounds';
|
||||
import { soundFxBuildFor } from '../../core/soundfx/compile';
|
||||
import { DRY, fxChainOf, type FxParams } from '../../core/soundfx/nodes';
|
||||
import { DRY_VOICE, type FxParams } from '../../core/soundfx/nodes';
|
||||
import type { Entity } from '../Entity';
|
||||
import { PhysicsSystem } from '../physics/PhysicsSystem';
|
||||
import type { World } from '../World';
|
||||
@@ -172,6 +172,7 @@ export function behaviorContext(
|
||||
const file = assetId === '' ? '' : soundFileOf(project, assetId);
|
||||
if (assetId === '' || file) run.audio.stop(file);
|
||||
},
|
||||
tuneSound: (voice, fxParams) => run.audio.tune(voice, fxParams),
|
||||
spawn: (assetId, position, parent) => world.spawn(assetId, position, resolve(parent) ?? null),
|
||||
addJoint: (assetId, owner, connected, at) => {
|
||||
const on = resolve(owner);
|
||||
@@ -218,7 +219,7 @@ export function behaviorContext(
|
||||
* node's, and its loop flag unless the node overrules it either way. The asset is what a sound
|
||||
* *is* – a node that says nothing therefore plays it as the project describes it. An FX asset that
|
||||
* does not compile plays the sound dry: a broken effect should not silence the game. Its parameters
|
||||
* take the values handed in, the others their defaults.
|
||||
* take the values handed in, the others their defaults. Returns the number of the voice, 0 for none.
|
||||
*/
|
||||
function playSound(
|
||||
{ project, audio }: ContextSources,
|
||||
@@ -228,14 +229,20 @@ function playSound(
|
||||
owner: string,
|
||||
fxId: string,
|
||||
fxParams: FxParams,
|
||||
): void {
|
||||
): number {
|
||||
const asset = findAssetOf(project, assetId, 'sound');
|
||||
if (!asset?.file) return;
|
||||
if (!asset?.file) return 0;
|
||||
const props = resolveAssetProps(project, asset);
|
||||
const level = Math.max(0, Math.min(1, volumeOf(props) * volume));
|
||||
const fx = fxId ? findAssetOf(project, fxId, 'soundfx') : undefined;
|
||||
const plan = fx && soundFxBuildFor(fx).compiled;
|
||||
audio.play(asset.file, level, loop ?? loopsOf(props), owner, plan ? fxChainOf(plan, fxParams) : DRY);
|
||||
return audio.play(
|
||||
asset.file,
|
||||
level,
|
||||
loop ?? loopsOf(props),
|
||||
owner,
|
||||
plan ? { plan, params: fxParams } : DRY_VOICE,
|
||||
);
|
||||
}
|
||||
|
||||
/** Where a ray of that length ends; where it has no direction, where it starts. */
|
||||
|
||||
+143
-31
@@ -1,34 +1,62 @@
|
||||
import { DRY, type FxChain, type FxStage } from '../core/soundfx/nodes';
|
||||
import {
|
||||
DRY_VOICE,
|
||||
fxChainOf,
|
||||
type FxChain,
|
||||
type FxParams,
|
||||
type FxStage,
|
||||
type FxVoice,
|
||||
} from '../core/soundfx/nodes';
|
||||
import { pitchShift } from '../core/soundfx/pitchShift';
|
||||
import type { AudioSink } from '../runtime/systems/Audio';
|
||||
|
||||
const DEFAULT_BASE = '/game-maker/data/sounds';
|
||||
/** How many pitched copies are kept at once. */
|
||||
const SHIFTED_KEPT = 32;
|
||||
/** Seconds a changed parameter takes to get most of the way there: quick, but no click (#315). */
|
||||
const GLIDE = 0.03;
|
||||
|
||||
/**
|
||||
* One number of a voice that can still change while it plays (#315): a level, a share of a Mix, the
|
||||
* gain of a band – or the pitch of a source, which glides by speed from the pitch its copy was made at.
|
||||
*/
|
||||
type Knob =
|
||||
| { kind: 'level'; param: AudioParam; target: number }
|
||||
| { kind: 'pitch'; source: AudioBufferSourceNode; base: number; target: number };
|
||||
|
||||
/**
|
||||
* One sound in flight: its sources, its own gain, the file it came from and who is making it. A
|
||||
* voice has a source per pitch its sound FX plays the file at (#316), started together and as long
|
||||
* as each other.
|
||||
* voice has a source per way through its sound FX (#316), started together and as long as each
|
||||
* other. Its knobs are in the order `levelsOf` lists the numbers of its chain.
|
||||
*/
|
||||
interface Voice {
|
||||
/** What `play` returned for it, the handle a `tune` reaches it by. */
|
||||
id: number;
|
||||
file: string;
|
||||
/** The behavior this voice belongs to; `''` for one nothing owns (see `AudioSink.play`). */
|
||||
owner: string;
|
||||
sources: AudioBufferSourceNode[];
|
||||
gain: GainNode;
|
||||
fx: FxVoice;
|
||||
knobs: Knob[];
|
||||
}
|
||||
|
||||
/** What one voice is wired from: the file, and its sources by pitch so far. */
|
||||
/** What one voice is wired from: the file, and what has been made for it so far. */
|
||||
interface Wiring {
|
||||
context: AudioContext;
|
||||
file: string;
|
||||
buffer: AudioBuffer;
|
||||
loop: boolean;
|
||||
sources: Map<number, AudioBufferSourceNode>;
|
||||
sources: { source: AudioBufferSourceNode; pitch: number }[];
|
||||
knobs: Knob[];
|
||||
stages: number;
|
||||
}
|
||||
|
||||
/** A play still waiting on its bytes: what a `tune` in the meantime writes into. */
|
||||
interface Pending {
|
||||
id: number;
|
||||
fx: FxVoice;
|
||||
}
|
||||
|
||||
/**
|
||||
* Plays uploaded sounds, keyed by file name – the texture library's twin on the audio side. Files
|
||||
* are content addressed, so a decoded buffer can never go stale and nothing is evicted.
|
||||
@@ -52,8 +80,12 @@ export class SoundLibrary implements AudioSink {
|
||||
/** Never retried, so one broken reference cannot spam the network. */
|
||||
private readonly failed = new Set<string>();
|
||||
private voices: Voice[] = [];
|
||||
private pending: Pending[] = [];
|
||||
/** The number the next play is given; 0 stays free for "nothing plays". */
|
||||
private nextId = 1;
|
||||
private started = 0;
|
||||
private stopped = 0;
|
||||
private tuned = 0;
|
||||
/** The last voice actually started, with the values it was started at – what a check reads back. */
|
||||
private last: LastVoice | null = null;
|
||||
/**
|
||||
@@ -73,20 +105,44 @@ export class SoundLibrary implements AudioSink {
|
||||
this.base = base.replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
play(file: string, volume: number, loop: boolean, owner = '', fx: FxChain = DRY): void {
|
||||
if (file === '' || this.failed.has(file)) return;
|
||||
/**
|
||||
* The number is handed out at once, before the bytes are there: a graph can tune a voice in the
|
||||
* same frame it asked for it, and the voice then starts with those values.
|
||||
*/
|
||||
play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE): number {
|
||||
if (file === '' || this.failed.has(file)) return 0;
|
||||
const id = this.nextId++;
|
||||
const buffer = this.buffers.get(file);
|
||||
if (buffer) {
|
||||
this.start(file, buffer, volume, loop, owner, fx);
|
||||
return;
|
||||
this.start(id, file, buffer, volume, loop, owner, fx);
|
||||
return id;
|
||||
}
|
||||
const askedAt = this.cancelsFor(file, owner);
|
||||
const waiting: Pending = { id, fx: { plan: fx.plan, params: { ...fx.params } } };
|
||||
this.pending.push(waiting);
|
||||
void this.load(file).then((loaded) => {
|
||||
this.pending = this.pending.filter((other) => other !== waiting);
|
||||
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
|
||||
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
|
||||
if (loaded && this.cancelsFor(file, owner) === askedAt)
|
||||
this.start(file, loaded, volume, loop, owner, fx);
|
||||
this.start(id, file, loaded, volume, loop, owner, waiting.fx);
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
/** Says whether the voice was still there to be reached – playing, or waiting on its bytes. */
|
||||
tune(id: number, params: FxParams): boolean {
|
||||
const waiting = this.pending.find((other) => other.id === id);
|
||||
if (waiting) {
|
||||
waiting.fx = { plan: waiting.fx.plan, params: { ...waiting.fx.params, ...params } };
|
||||
return true;
|
||||
}
|
||||
const voice = this.voices.find((other) => other.id === id);
|
||||
if (!voice) return false;
|
||||
voice.fx = { plan: voice.fx.plan, params: { ...voice.fx.params, ...params } };
|
||||
this.glide(voice);
|
||||
this.tuned += 1;
|
||||
return true;
|
||||
}
|
||||
|
||||
stop(file: string): void {
|
||||
@@ -123,6 +179,7 @@ export class SoundLibrary implements AudioSink {
|
||||
playing: number;
|
||||
started: number;
|
||||
stopped: number;
|
||||
tuned: number;
|
||||
decoded: number;
|
||||
failed: number;
|
||||
last: LastVoice | null;
|
||||
@@ -131,12 +188,25 @@ export class SoundLibrary implements AudioSink {
|
||||
playing: this.voices.length,
|
||||
started: this.started,
|
||||
stopped: this.stopped,
|
||||
tuned: this.tuned,
|
||||
decoded: this.buffers.size,
|
||||
failed: this.failed.size,
|
||||
last: this.last,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Where the knobs of each voice of that file are right now, oldest first, a pitch as the speed it
|
||||
* plays at – what a check reads to see a change arrive at one voice and not another (test hook).
|
||||
*/
|
||||
levels(file: string): number[][] {
|
||||
return this.voices
|
||||
.filter((voice) => voice.file === file)
|
||||
.map((voice) =>
|
||||
voice.knobs.map((knob) => (knob.kind === 'pitch' ? knob.source.playbackRate : knob.param).value),
|
||||
);
|
||||
}
|
||||
|
||||
/** How long a decoded sound runs, in seconds; null while it has not been loaded (test hook). */
|
||||
duration(file: string): number | null {
|
||||
return this.buffers.get(file)?.duration ?? null;
|
||||
@@ -177,12 +247,13 @@ export class SoundLibrary implements AudioSink {
|
||||
}
|
||||
|
||||
private start(
|
||||
id: number,
|
||||
file: string,
|
||||
buffer: AudioBuffer,
|
||||
volume: number,
|
||||
loop: boolean,
|
||||
owner: string,
|
||||
fx: FxChain,
|
||||
fx: FxVoice,
|
||||
): void {
|
||||
const context = this.audio();
|
||||
// A context starts suspended where the page has not been interacted with; without this the
|
||||
@@ -190,18 +261,18 @@ export class SoundLibrary implements AudioSink {
|
||||
if (context.state === 'suspended') void context.resume();
|
||||
const gain = context.createGain();
|
||||
gain.gain.value = Math.max(0, Math.min(1, volume));
|
||||
const wiring: Wiring = { context, file, buffer, loop, sources: new Map(), stages: 0 };
|
||||
this.wire(wiring, fx).connect(gain).connect(context.destination);
|
||||
const sources = [...wiring.sources.values()];
|
||||
const wiring: Wiring = { context, file, buffer, loop, sources: [], knobs: [], stages: 0 };
|
||||
this.wire(wiring, fxChainOf(fx.plan, fx.params)).connect(gain).connect(context.destination);
|
||||
const sources = wiring.sources.map((made) => made.source);
|
||||
this.last = {
|
||||
file,
|
||||
volume: gain.gain.value,
|
||||
loop,
|
||||
pitches: [...wiring.sources.keys()],
|
||||
pitches: wiring.sources.map((made) => made.pitch),
|
||||
stages: wiring.stages,
|
||||
duration: sources[0]!.buffer!.duration,
|
||||
};
|
||||
const voice: Voice = { file, owner, sources, gain };
|
||||
const voice: Voice = { id, file, owner, sources, gain, fx, knobs: wiring.knobs };
|
||||
let playing = sources.length;
|
||||
for (const source of sources)
|
||||
source.onended = () => {
|
||||
@@ -213,35 +284,60 @@ export class SoundLibrary implements AudioSink {
|
||||
}
|
||||
|
||||
/**
|
||||
* The nodes a chain becomes, up to the one that carries its sound on. Two ways at the same pitch
|
||||
* share a source; a Mix weighs each of its sounds by its share before they are summed.
|
||||
* The nodes a chain becomes, up to the one that carries its sound on. Every way has a source of
|
||||
* its own, so each can glide to its own pitch; a Mix weighs each of its sounds by its share before
|
||||
* they are summed.
|
||||
*/
|
||||
private wire(wiring: Wiring, chain: FxChain): AudioNode {
|
||||
const { context } = wiring;
|
||||
const { context, knobs } = wiring;
|
||||
let tail: AudioNode;
|
||||
if ('mix' in chain) {
|
||||
tail = context.createGain();
|
||||
this.wire(wiring, chain.a)
|
||||
.connect(gainNode(context, 1 - chain.mix))
|
||||
.connect(tail);
|
||||
this.wire(wiring, chain.b).connect(gainNode(context, chain.mix)).connect(tail);
|
||||
const wayA = this.wire(wiring, chain.a);
|
||||
const shareA = gainNode(context, 1 - chain.mix);
|
||||
wayA.connect(shareA).connect(tail);
|
||||
knobs.push({ kind: 'level', param: shareA.gain, target: 1 - chain.mix });
|
||||
const wayB = this.wire(wiring, chain.b);
|
||||
const shareB = gainNode(context, chain.mix);
|
||||
wayB.connect(shareB).connect(tail);
|
||||
knobs.push({ kind: 'level', param: shareB.gain, target: chain.mix });
|
||||
} else tail = this.source(wiring, chain.pitch);
|
||||
for (const stage of chain.stages) tail = tail.connect(fxNode(context, stage));
|
||||
for (const stage of chain.stages) {
|
||||
const [node, param] = fxNode(context, stage);
|
||||
tail = tail.connect(node);
|
||||
knobs.push({ kind: 'level', param, target: stage.gain });
|
||||
}
|
||||
wiring.stages += chain.stages.length;
|
||||
return tail;
|
||||
}
|
||||
|
||||
private source(wiring: Wiring, pitch: number): AudioBufferSourceNode {
|
||||
const held = wiring.sources.get(pitch);
|
||||
if (held) return held;
|
||||
const { context, file, buffer, loop } = wiring;
|
||||
const source = context.createBufferSource();
|
||||
source.buffer = pitch === 0 ? buffer : this.pitched(context, file, buffer, pitch, loop);
|
||||
source.loop = loop;
|
||||
wiring.sources.set(pitch, source);
|
||||
wiring.sources.push({ source, pitch });
|
||||
wiring.knobs.push({ kind: 'pitch', source, base: pitch, target: pitch });
|
||||
return source;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sends every knob of a voice towards what its values say now. A pitch cannot get a new copy in
|
||||
* the middle of a sound, so it glides by speed: the voice gets that much shorter or longer.
|
||||
*/
|
||||
private glide(voice: Voice): void {
|
||||
const levels = levelsOf(fxChainOf(voice.fx.plan, voice.fx.params));
|
||||
const now = this.audio().currentTime;
|
||||
voice.knobs.forEach((knob, i) => {
|
||||
const level = levels[i];
|
||||
if (level === undefined || level === knob.target) return;
|
||||
knob.target = level;
|
||||
if (knob.kind === 'pitch')
|
||||
knob.source.playbackRate.setTargetAtTime(2 ** ((level - knob.base) / 12), now, GLIDE);
|
||||
else knob.param.setTargetAtTime(level, now, GLIDE);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The sound at another pitch and its own length (#312), worked out once per file, pitch and loop
|
||||
* and kept for the next voice. The oldest goes first, so a pitch drawn at random every play cannot
|
||||
@@ -286,7 +382,7 @@ interface LastVoice {
|
||||
file: string;
|
||||
volume: number;
|
||||
loop: boolean;
|
||||
/** Semitones of each source, one per pitch (#316), and how many gain and filter nodes it passed. */
|
||||
/** Semitones of each source, one per way (#316), and how many gain and filter nodes it passed. */
|
||||
pitches: number[];
|
||||
stages: number;
|
||||
/** Seconds of what was played: a pitched voice is as long as the sound itself (#312). */
|
||||
@@ -299,14 +395,30 @@ function gainNode(context: AudioContext, level: number): GainNode {
|
||||
return gain;
|
||||
}
|
||||
|
||||
function fxNode(context: AudioContext, stage: FxStage): AudioNode {
|
||||
if (stage.kind === 'gain') return gainNode(context, stage.gain);
|
||||
/** The node a stage becomes, with the number of it a parameter can move. */
|
||||
function fxNode(context: AudioContext, stage: FxStage): [AudioNode, AudioParam] {
|
||||
if (stage.kind === 'gain') {
|
||||
const gain = gainNode(context, stage.gain);
|
||||
return [gain, gain.gain];
|
||||
}
|
||||
const filter = context.createBiquadFilter();
|
||||
filter.type = stage.filter;
|
||||
filter.frequency.value = stage.frequency;
|
||||
filter.gain.value = stage.gain;
|
||||
filter.Q.value = stage.q;
|
||||
return filter;
|
||||
return [filter, filter.gain];
|
||||
}
|
||||
|
||||
/** The numbers of a chain in the order `wire` makes their knobs: a stage's `gain` is what its node holds. */
|
||||
function levelsOf(chain: FxChain, into: number[] = []): number[] {
|
||||
if ('mix' in chain) {
|
||||
levelsOf(chain.a, into);
|
||||
into.push(1 - chain.mix);
|
||||
levelsOf(chain.b, into);
|
||||
into.push(chain.mix);
|
||||
} else into.push(chain.pitch);
|
||||
for (const stage of chain.stages) into.push(stage.gain);
|
||||
return into;
|
||||
}
|
||||
|
||||
export const soundLibrary = new SoundLibrary();
|
||||
@@ -54,7 +54,8 @@ function keep(f: Fixture, sceneId: string, objectId: string): void {
|
||||
function start(f: Fixture, audio?: AudioSink) {
|
||||
const stoppedOwners: string[] = [];
|
||||
const sink: AudioSink = audio ?? {
|
||||
play: () => {},
|
||||
play: () => 0,
|
||||
tune: () => {},
|
||||
stop: () => {},
|
||||
stopOwner: (owner: string) => stoppedOwners.push(owner),
|
||||
};
|
||||
|
||||
@@ -1940,12 +1940,14 @@ const CASES: Record<string, Case[]> = {
|
||||
name: 'no sound is a no-op that still continues the flow',
|
||||
inputs: { sound: '', volume: 1, loop: '' },
|
||||
next: 'exec',
|
||||
outputs: { voice: 0 },
|
||||
after: (ctx) => expect(ctx.played).toEqual([]),
|
||||
},
|
||||
{
|
||||
name: 'hands the id on with the volume and loop it was given',
|
||||
name: 'hands the id on with the volume and loop it was given, and gives out the voice (#315)',
|
||||
inputs: { sound: 'snd-1', volume: 0.5, loop: 'true' },
|
||||
next: 'exec',
|
||||
outputs: { voice: 1 },
|
||||
// What the two mean – a factor on the asset's volume, an answer that overrules its loop
|
||||
// flag – is the BehaviorSystem's to resolve, since only it holds the project (see
|
||||
// `sounds.test.ts`).
|
||||
@@ -2025,6 +2027,20 @@ const CASES: Record<string, Case[]> = {
|
||||
after: (ctx) => expect(ctx.silenced).toEqual(['']),
|
||||
},
|
||||
],
|
||||
SetSoundFxParameters: [
|
||||
{
|
||||
name: 'hands the numbers of the bundle to the voice it is given',
|
||||
inputs: { voice: 4, fxParams: { depth: 3, label: 'x' } },
|
||||
next: 'exec',
|
||||
after: (ctx) => expect(ctx.tuned).toEqual([{ voice: 4, fxParams: { depth: 3 } }]),
|
||||
},
|
||||
{
|
||||
name: 'voice 0 – nothing was played – reaches nothing, and the flow carries on',
|
||||
inputs: { voice: 0, fxParams: { depth: 3 } },
|
||||
next: 'exec',
|
||||
after: (ctx) => expect(ctx.tuned).toEqual([]),
|
||||
},
|
||||
],
|
||||
Log: [
|
||||
{
|
||||
name: 'logs its message and continues',
|
||||
|
||||
+81
-16
@@ -12,7 +12,14 @@ import type { BehaviorAsset, SoundFxAsset } from '../../src/core/model/Asset';
|
||||
import { isPlaceableType } from '../../src/core/model/AssetTypes';
|
||||
import { assetRefChoices } from '../../src/core/properties/assetRef';
|
||||
import { compileSoundFx, soundFxBuildFor } from '../../src/core/soundfx/compile';
|
||||
import { DRY, EMPTY_PLAN, type FxChain, fxChainOf, type FxPlan } from '../../src/core/soundfx/nodes';
|
||||
import {
|
||||
DRY,
|
||||
EMPTY_PLAN,
|
||||
type FxChain,
|
||||
fxChainOf,
|
||||
type FxPlan,
|
||||
type FxVoice,
|
||||
} from '../../src/core/soundfx/nodes';
|
||||
import { deserializeProject } from '../../src/core/store/serialize';
|
||||
import type { AudioSink } from '../../src/runtime/systems/Audio';
|
||||
import { paletteEntries } from '../../src/editor/graph/paletteEntries';
|
||||
@@ -214,15 +221,15 @@ describe('a Mix in a sound FX graph (#316)', () => {
|
||||
expect(b).toMatchObject({ pitch: 0, stages: [{ kind: 'gain' }, { kind: 'filter' }] });
|
||||
});
|
||||
|
||||
it('takes its share from a parameter, held between 0 and 1', () => {
|
||||
it('takes its share from a parameter, held between 0 and 1, and keeps both ways for it (#315)', () => {
|
||||
const f = forked();
|
||||
f.input.ports = [{ key: 'blend', kind: 'number', default: 0.25 }];
|
||||
f.edit.connect(f.input, 'blend', f.mix, 'mix');
|
||||
const plan = compileSoundFx(f.fx.graph).compiled!;
|
||||
expect(fxChainOf(plan)).toMatchObject({ mix: 0.25 });
|
||||
expect(fxChainOf(plan, { blend: 0.8 })).toMatchObject({ mix: 0.8 });
|
||||
expect(fxChainOf(plan, { blend: -3 })).toMatchObject({ pitch: 12 });
|
||||
expect(fxChainOf(plan, { blend: 3 })).toMatchObject({ pitch: 0 });
|
||||
expect(fxChainOf(plan, { blend: -3 })).toMatchObject({ mix: 0, a: { pitch: 12 }, b: { pitch: 0 } });
|
||||
expect(fxChainOf(plan, { blend: 3 })).toMatchObject({ mix: 1, a: { pitch: 12 }, b: { pitch: 0 } });
|
||||
});
|
||||
|
||||
it('hands a Pitch behind it to both of its sounds', () => {
|
||||
@@ -249,12 +256,20 @@ describe('a Mix in a sound FX graph (#316)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
/** Records what it was asked to play, with the line of effects. */
|
||||
function fakeAudio(): AudioSink & { played: { file: string; fx: FxChain | undefined }[] } {
|
||||
/**
|
||||
* Records what it was asked to play, with the line of effects its values make, and every tune. A
|
||||
* voice is numbered by how many have been played, so the first is 1.
|
||||
*/
|
||||
function fakeAudio(): AudioSink & {
|
||||
played: { file: string; owner: string; fx: FxChain | undefined }[];
|
||||
tuned: { voice: number; params: Readonly<Record<string, number>> }[];
|
||||
} {
|
||||
const sink = {
|
||||
played: [] as { file: string; fx: FxChain | undefined }[],
|
||||
play: (file: string, _volume: number, _loop: boolean, _owner: string, fx?: FxChain) =>
|
||||
sink.played.push({ file, fx }),
|
||||
played: [] as { file: string; owner: string; fx: FxChain | undefined }[],
|
||||
tuned: [] as { voice: number; params: Readonly<Record<string, number>> }[],
|
||||
play: (file: string, _volume: number, _loop: boolean, owner: string, fx?: FxVoice) =>
|
||||
sink.played.push({ file, owner, fx: fx && fxChainOf(fx.plan, fx.params) }),
|
||||
tune: (voice: number, params: Readonly<Record<string, number>>) => sink.tuned.push({ voice, params }),
|
||||
stop: () => {},
|
||||
stopOwner: () => {},
|
||||
};
|
||||
@@ -286,7 +301,7 @@ describe('Play sound with fx', () => {
|
||||
const f = playingScene((s) => s.fx.id);
|
||||
const audio = fakeAudio();
|
||||
startWorld(f, { behaviors: true, audio }).frame();
|
||||
expect(audio.played).toEqual([{ file: FILE, fx: { pitch: -12, stages: [] } }]);
|
||||
expect(audio.played).toMatchObject([{ file: FILE, fx: { pitch: -12, stages: [] } }]);
|
||||
});
|
||||
|
||||
it('plays it dry without an fx, with one that is gone, and with one that does not compile', () => {
|
||||
@@ -305,7 +320,8 @@ describe('Play sound with fx', () => {
|
||||
expect(soundFxBuildFor(broken.fx).compiled).toBeNull();
|
||||
const brokenAudio = fakeAudio();
|
||||
startWorld(broken, { behaviors: true, audio: brokenAudio }).frame();
|
||||
for (const audio of [none, gone, brokenAudio]) expect(audio.played).toEqual([{ file: FILE, fx: DRY }]);
|
||||
for (const audio of [none, gone, brokenAudio])
|
||||
expect(audio.played).toMatchObject([{ file: FILE, fx: DRY }]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -329,11 +345,11 @@ describe('the parameters of a sound FX', () => {
|
||||
expect(pitchOf(fxChainOf(plan, { other: 3 }))).toBe(-12);
|
||||
});
|
||||
|
||||
it('keep a stage that is 0 by default, so a value handed in can open it', () => {
|
||||
it('keep a stage that is 0 by default, so a value handed in can open it, even on a playing voice', () => {
|
||||
const f = withParam('SoundFxGain', 'gain', [{ key: 'loud', kind: 'number' }]);
|
||||
const plan = soundFxBuildFor(f.fx).compiled!;
|
||||
expect(plan.stages).toHaveLength(1);
|
||||
expect(fxChainOf(plan)).toEqual(DRY);
|
||||
expect(fxChainOf(plan)).toEqual({ pitch: 0, stages: [{ kind: 'gain', gain: 1 }] });
|
||||
expect(fxChainOf(plan, { loud: 6 }).stages[0]).toMatchObject({ kind: 'gain' });
|
||||
expect((fxChainOf(plan, { loud: 6 }).stages[0] as { gain: number }).gain).toBeCloseTo(10 ** (6 / 20));
|
||||
});
|
||||
@@ -437,7 +453,9 @@ describe('Sound FX parameters and Play sound', () => {
|
||||
const f = scene({ pick: ['depth'], values: { depth: 7 } });
|
||||
const audio = fakeAudio();
|
||||
startWorld(f, { behaviors: true, audio }).frame();
|
||||
expect(audio.played).toEqual([{ file: FILE, fx: { pitch: 7, stages: [] } }]);
|
||||
expect(audio.played).toMatchObject([
|
||||
{ file: FILE, fx: { pitch: 7, stages: [{ kind: 'gain', gain: 1 }] } },
|
||||
]);
|
||||
});
|
||||
|
||||
it('starts a picked parameter at the default of the FX', () => {
|
||||
@@ -449,7 +467,7 @@ describe('Sound FX parameters and Play sound', () => {
|
||||
]);
|
||||
const audio = fakeAudio();
|
||||
startWorld(f, { behaviors: true, audio }).frame();
|
||||
expect(audio.played[0]!.fx).toEqual({ pitch: -12, stages: [] });
|
||||
expect(audio.played[0]!.fx).toEqual({ pitch: -12, stages: [{ kind: 'gain', gain: 1 }] });
|
||||
});
|
||||
|
||||
it('says so at the node when no FX is picked or it is gone', () => {
|
||||
@@ -475,9 +493,56 @@ describe('Sound FX parameters and Play sound', () => {
|
||||
expect(f.params.params.drop).toBe(7);
|
||||
const audio = fakeAudio();
|
||||
startWorld(f, { behaviors: true, audio }).frame();
|
||||
expect(audio.played[0]!.fx).toEqual({ pitch: 7, stages: [] });
|
||||
expect(audio.played[0]!.fx).toEqual({ pitch: 7, stages: [{ kind: 'gain', gain: 1 }] });
|
||||
f.stack.undo();
|
||||
expect(f.params.ports).toEqual([{ key: 'depth', kind: 'number' }]);
|
||||
expect(f.params.params.depth).toBe(7);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Set sound FX parameters (#315)', () => {
|
||||
/**
|
||||
* Pressing x plays Jump through an FX twice, the first `sound` – then hands `depth` to the voice
|
||||
* the first play gave out.
|
||||
*/
|
||||
function tuning(sound?: string) {
|
||||
const f = makeSpriteScene('Box');
|
||||
const { fx } = makeFx(f);
|
||||
const created = new CreateSoundCommand('Jump', FILE);
|
||||
f.stack.execute(created);
|
||||
const behavior = makeBehavior(f, 'Logic');
|
||||
place(f, behavior.id, { x: 0, y: 0 }, place(f, f.spriteId, { x: 0, y: 0 }));
|
||||
const edit = graphEditor(f, behavior);
|
||||
const first = edit.addNode('PlaySound', { sound: sound ?? created.asset.id, fx: fx.id });
|
||||
const second = edit.addNode('PlaySound', { sound: created.asset.id, fx: fx.id });
|
||||
const set = edit.addNode('SetSoundFxParameters');
|
||||
const params = edit.addNode('SoundFxParameters', { soundFxAsset: fx.id, depth: 3 });
|
||||
params.ports = [{ key: 'depth', kind: 'number' }];
|
||||
edit.wire([
|
||||
[edit.addNode('OnKeyDown', { key: 'x' }), 'exec', first, 'exec'],
|
||||
[first, 'exec', second, 'exec'],
|
||||
[second, 'exec', set, 'exec'],
|
||||
[first, 'voice', set, 'voice'],
|
||||
[params, 'params', set, 'fxParams'],
|
||||
]);
|
||||
return f;
|
||||
}
|
||||
|
||||
it('hands the values to the one voice its Play sound gave out, not to another of the same sound', () => {
|
||||
const audio = fakeAudio();
|
||||
const { input, frame } = startWorld(tuning(), { behaviors: true, audio });
|
||||
input.press('x');
|
||||
frame();
|
||||
expect(audio.played).toHaveLength(2);
|
||||
expect(audio.tuned).toEqual([{ voice: 1, params: { depth: 3 } }]);
|
||||
});
|
||||
|
||||
it('reaches nothing where its Play sound played nothing', () => {
|
||||
const audio = fakeAudio();
|
||||
const { input, frame } = startWorld(tuning('no-such-sound'), { behaviors: true, audio });
|
||||
input.press('x');
|
||||
frame();
|
||||
expect(audio.played).toHaveLength(1);
|
||||
expect(audio.tuned).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -42,6 +42,7 @@ function fakeAudio(): AudioSink & {
|
||||
stoppedOwners: [] as string[],
|
||||
play: (file: string, volume: number, loop: boolean, owner: string) =>
|
||||
sink.played.push({ file, volume, loop, owner }),
|
||||
tune: () => {},
|
||||
stop: (file: string) => sink.stopped.push(file),
|
||||
stopOwner: (owner: string) => sink.stoppedOwners.push(owner),
|
||||
};
|
||||
|
||||
@@ -214,6 +214,7 @@ export type FakeContext = NodeContext & {
|
||||
fxParams: Readonly<Record<string, number>>;
|
||||
}[];
|
||||
silenced: string[];
|
||||
tuned: { voice: number; fxParams: Readonly<Record<string, number>> }[];
|
||||
/** The scenes the context was asked to load, in order. */
|
||||
loaded: string[];
|
||||
/** The time scale and the pause the context holds, as the game nodes left them. */
|
||||
@@ -265,9 +266,12 @@ export function fakeContext(overrides: Partial<NodeContext> = {}): FakeContext {
|
||||
played: [],
|
||||
silenced: [],
|
||||
loaded: [],
|
||||
// The number of a voice is how many have been played: unique, and never 0.
|
||||
playSound: (assetId, volume, loop, detached, fx, fxParams) =>
|
||||
ctx.played.push({ assetId, volume, loop, detached, fx, fxParams }),
|
||||
stopSound: (assetId) => ctx.silenced.push(assetId),
|
||||
tuned: [],
|
||||
tuneSound: (voice, fxParams) => ctx.tuned.push({ voice, fxParams }),
|
||||
loadScene: (sceneId) => ctx.loaded.push(sceneId),
|
||||
spawn: () => null,
|
||||
addJoint: () => null,
|
||||
|
||||
@@ -6,16 +6,28 @@
|
||||
* The device is stubbed – nothing makes a noise in a unit test, and Web Audio does not exist here.
|
||||
*/
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { FxChain } from '../../src/core/soundfx/nodes';
|
||||
import type { FxChain, FxPlan, FxVoice } from '../../src/core/soundfx/nodes';
|
||||
import { SoundLibrary } from '../../src/viewport/SoundLibrary';
|
||||
|
||||
const FILE = 'a.wav';
|
||||
const OTHER = 'b.wav';
|
||||
|
||||
/** An AudioParam that remembers where it was sent to glide, and how often. */
|
||||
interface FakeParam {
|
||||
value: number;
|
||||
glides: number[];
|
||||
setTargetAtTime: (value: number) => void;
|
||||
}
|
||||
|
||||
function param(value: number): FakeParam {
|
||||
const made: FakeParam = { value, glides: [], setTargetAtTime: (to) => made.glides.push(to) };
|
||||
return made;
|
||||
}
|
||||
|
||||
interface FakeSource {
|
||||
buffer: unknown;
|
||||
loop: boolean;
|
||||
playbackRate: { value: number };
|
||||
playbackRate: FakeParam;
|
||||
onended: (() => void) | null;
|
||||
started: boolean;
|
||||
stopped: boolean;
|
||||
@@ -28,11 +40,11 @@ const sources: FakeSource[] = [];
|
||||
/** Every `connect` made, as "from > to" by the kind of node – the order a voice is wired in. */
|
||||
const wires: string[] = [];
|
||||
/** The level of every gain node made, in the order they were made. */
|
||||
const gains: { value: number }[] = [];
|
||||
const gains: FakeParam[] = [];
|
||||
const filters: {
|
||||
type: string;
|
||||
frequency: { value: number };
|
||||
gain: { value: number };
|
||||
gain: FakeParam;
|
||||
Q: { value: number };
|
||||
}[] = [];
|
||||
|
||||
@@ -69,6 +81,7 @@ function fakeBuffer(_channels: number, length: number, sampleRate: number) {
|
||||
function fakeContext() {
|
||||
return {
|
||||
state: 'running',
|
||||
currentTime: 0,
|
||||
destination: { kind: 'destination' },
|
||||
resume: () => Promise.resolve(),
|
||||
suspend: () => Promise.resolve(),
|
||||
@@ -78,7 +91,7 @@ function fakeContext() {
|
||||
const source: FakeSource = wired('source', {
|
||||
buffer: null,
|
||||
loop: false,
|
||||
playbackRate: { value: 1 },
|
||||
playbackRate: param(1),
|
||||
onended: null,
|
||||
started: false,
|
||||
stopped: false,
|
||||
@@ -93,7 +106,7 @@ function fakeContext() {
|
||||
return source;
|
||||
},
|
||||
createGain: () => {
|
||||
const gain = wired('gain', { gain: { value: 1 } });
|
||||
const gain = wired('gain', { gain: param(1) });
|
||||
gains.push(gain.gain);
|
||||
return gain;
|
||||
},
|
||||
@@ -101,7 +114,7 @@ function fakeContext() {
|
||||
const filter = wired('filter', {
|
||||
type: '',
|
||||
frequency: { value: 0 },
|
||||
gain: { value: 0 },
|
||||
gain: param(0),
|
||||
Q: { value: 0 },
|
||||
});
|
||||
filters.push(filter);
|
||||
@@ -136,11 +149,37 @@ afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
/** Plays `file` and lets its bytes arrive, unless `hold` keeps the decode pending. */
|
||||
async function play(library: SoundLibrary, file = FILE, { hold = false } = {}, fx?: FxChain): Promise<void> {
|
||||
library.play(file, 1, false, '', fx);
|
||||
/** A chain as the plan of an FX with nothing left open, so a test can say what it wants to hear. */
|
||||
function planOf(chain: FxChain): FxPlan {
|
||||
const stages = chain.stages.map((stage) =>
|
||||
stage.kind === 'gain'
|
||||
? { kind: 'gain' as const, db: { value: 20 * Math.log10(stage.gain) } }
|
||||
: { ...stage, gain: { value: stage.gain } },
|
||||
);
|
||||
if ('mix' in chain) return { mix: { value: chain.mix }, a: planOf(chain.a), b: planOf(chain.b), stages };
|
||||
return { pitch: chain.pitch === 0 ? [] : [{ value: chain.pitch }], stages };
|
||||
}
|
||||
|
||||
/** Plays `file` and lets its bytes arrive, unless `hold` keeps the decode pending; gives the voice. */
|
||||
async function play(
|
||||
library: SoundLibrary,
|
||||
file = FILE,
|
||||
{ hold = false, owner = '' } = {},
|
||||
fx?: FxChain | FxVoice,
|
||||
): Promise<number> {
|
||||
const voice = library.play(
|
||||
file,
|
||||
1,
|
||||
false,
|
||||
owner,
|
||||
fx && ('plan' in fx ? fx : { plan: planOf(fx), params: {} }),
|
||||
);
|
||||
await settle();
|
||||
if (hold) return;
|
||||
if (!hold) await arrive();
|
||||
return voice;
|
||||
}
|
||||
|
||||
async function arrive(): Promise<void> {
|
||||
for (const resolve of decodes.splice(0)) resolve();
|
||||
await settle();
|
||||
}
|
||||
@@ -208,7 +247,7 @@ describe('the sound library', () => {
|
||||
expect([up!.duration, down!.duration]).toEqual([dry!.duration, dry!.duration]);
|
||||
});
|
||||
|
||||
it('mixes two sounds by their shares, a source per pitch, started and stopped together (#316)', async () => {
|
||||
it('mixes two sounds by their shares, a source per way, started and stopped together (#316)', async () => {
|
||||
const library = new SoundLibrary();
|
||||
await play(
|
||||
library,
|
||||
@@ -231,7 +270,7 @@ describe('the sound library', () => {
|
||||
'filter > gain',
|
||||
'gain > destination',
|
||||
]);
|
||||
expect(gains.map((g) => g.value)).toEqual([1, 1, 0.75, 0.5, 0.25]);
|
||||
expect(gains.map((g) => g.value)).toEqual([1, 1, 0.75, expect.closeTo(0.5), 0.25]);
|
||||
expect(sources.map((s) => s.started)).toEqual([true, true]);
|
||||
expect(library.stats()).toMatchObject({ playing: 1, started: 1, last: { pitches: [12, 0], stages: 2 } });
|
||||
|
||||
@@ -242,7 +281,7 @@ describe('the sound library', () => {
|
||||
expect(library.stats()).toMatchObject({ playing: 0, stopped: 1 });
|
||||
});
|
||||
|
||||
it('starts one source for two ways at the same pitch', async () => {
|
||||
it('gives two ways at the same pitch a source each, so each can glide on its own (#315)', async () => {
|
||||
const library = new SoundLibrary();
|
||||
await play(
|
||||
library,
|
||||
@@ -255,10 +294,82 @@ describe('the sound library', () => {
|
||||
stages: [],
|
||||
},
|
||||
);
|
||||
expect(sources).toHaveLength(1);
|
||||
expect(sources).toHaveLength(2);
|
||||
expect(sources[0]!.buffer).toBe(sources[1]!.buffer);
|
||||
expect(wires.filter((w) => w.startsWith('source'))).toEqual(['source > gain', 'source > gain']);
|
||||
});
|
||||
|
||||
describe('a parameter changed while the voice plays (#315)', () => {
|
||||
/** Pitch on a, a gain on b, the share between them – all three open – and a band after the Mix. */
|
||||
const open: FxPlan = {
|
||||
mix: { value: 0.25, param: 'blend', min: 0, max: 1 },
|
||||
a: { pitch: [{ value: 0, param: 'depth', min: -64, max: 64 }], stages: [] },
|
||||
b: { pitch: [], stages: [{ kind: 'gain', db: { value: 0, param: 'loud', min: -120, max: 24 } }] },
|
||||
stages: [{ kind: 'filter', filter: 'lowshelf', frequency: 80, gain: { value: 6 }, q: 1 }],
|
||||
};
|
||||
|
||||
it('glides every knob it moves, a pitch by speed, and leaves the rest alone', async () => {
|
||||
const library = new SoundLibrary();
|
||||
const voice = await play(library, FILE, { owner: 'me' }, { plan: open, params: {} });
|
||||
// Made in the order: the voice, the Mix, share of a, the gain on b, share of b.
|
||||
const [, , shareA, loud, shareB] = gains;
|
||||
expect([shareA!.value, loud!.value, shareB!.value]).toEqual([0.75, 1, 0.25]);
|
||||
|
||||
expect(library.tune(voice, { depth: 12, loud: -6, blend: 0.5, nothing: 3 })).toBe(true);
|
||||
expect(sources[0]!.playbackRate.glides).toEqual([2]);
|
||||
expect(sources[1]!.playbackRate.glides).toEqual([]);
|
||||
expect(loud!.glides).toEqual([expect.closeTo(10 ** (-6 / 20))]);
|
||||
expect([shareA!.glides, shareB!.glides]).toEqual([[0.5], [0.5]]);
|
||||
expect(filters[0]!.gain.glides).toEqual([]);
|
||||
expect(library.stats().tuned).toBe(1);
|
||||
|
||||
// Only what changed is sent again; one not given keeps its value rather than its default.
|
||||
library.tune(voice, { depth: 12, blend: 1 });
|
||||
expect(sources[0]!.playbackRate.glides).toEqual([2]);
|
||||
expect(loud!.glides).toHaveLength(1);
|
||||
expect([shareA!.glides, shareB!.glides]).toEqual([
|
||||
[0.5, 0],
|
||||
[0.5, 1],
|
||||
]);
|
||||
});
|
||||
|
||||
it('reaches the one voice it is given – a detached one too – and none that has ended', async () => {
|
||||
const library = new SoundLibrary();
|
||||
const voices = [
|
||||
await play(library, FILE, { owner: 'me' }, { plan: open, params: {} }),
|
||||
await play(library, FILE, { owner: 'me' }, { plan: open, params: {} }),
|
||||
await play(library, OTHER, { owner: 'me' }, { plan: open, params: {} }),
|
||||
await play(library, FILE, {}, { plan: open, params: {} }),
|
||||
];
|
||||
expect(new Set(voices).size).toBe(4);
|
||||
expect(voices).not.toContain(0);
|
||||
library.tune(voices[1]!, { depth: 12 });
|
||||
library.tune(voices[3]!, { depth: 12 });
|
||||
// Two sources a voice, the first of each the pitched way.
|
||||
expect(sources.map((s) => s.playbackRate.glides.length)).toEqual([0, 0, 1, 0, 0, 0, 1, 0]);
|
||||
|
||||
library.stop(FILE);
|
||||
expect(library.tune(voices[1]!, { depth: 3 })).toBe(false);
|
||||
expect(library.tune(0, { depth: 3 })).toBe(false);
|
||||
expect(library.stats().tuned).toBe(2);
|
||||
});
|
||||
|
||||
it('starts a voice still waiting on its bytes with the values handed in meanwhile', async () => {
|
||||
const library = new SoundLibrary();
|
||||
const voice = await play(
|
||||
library,
|
||||
FILE,
|
||||
{ hold: true, owner: 'me' },
|
||||
{ plan: open, params: { blend: 0.5 } },
|
||||
);
|
||||
expect(library.tune(voice, { depth: 7 })).toBe(true);
|
||||
await arrive();
|
||||
expect(library.stats().last?.pitches).toEqual([7, 0]);
|
||||
expect(gains.map((g) => g.value)).toContain(0.5);
|
||||
expect(sources[0]!.playbackRate.glides).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
it('a stop while the bytes are in flight keeps the sound from starting afterwards', async () => {
|
||||
const library = new SoundLibrary();
|
||||
await play(library, FILE, { hold: true });
|
||||
|
||||
+67
-1
@@ -1,4 +1,4 @@
|
||||
import { edge, node } from '../lib/builders.mjs';
|
||||
import { edge, node, port, variable, written } from '../lib/builders.mjs';
|
||||
import { LANES, MAKE_OBJECT, pickAssetType } from './lib/scaffold.mjs';
|
||||
|
||||
/**
|
||||
@@ -414,6 +414,72 @@ export async function sound(t) {
|
||||
JSON.stringify(gameParam),
|
||||
);
|
||||
await stop();
|
||||
// #315: two looping voices of the same sound start at depth 7, the first keeps its number in a
|
||||
// variable, and `Set sound FX parameters` hands that one voice depth 3 every frame. In a
|
||||
// real AudioContext the first one's speed glides there, the second one's stays put.
|
||||
const tuneIds = ['sn-keep', 'sn-play2', 'sn-tick', 'sn-read', 'sn-tune', 'sn-fxp2'];
|
||||
const tuneEdges = [
|
||||
'sn-e-keep',
|
||||
'sn-e-voice',
|
||||
'sn-e-play2',
|
||||
'sn-e-fxp3',
|
||||
'sn-e-tick',
|
||||
'sn-e-read',
|
||||
'sn-e-fxp2',
|
||||
];
|
||||
await mutate(`
|
||||
const logic = __gm.state.project.assets.find((a) => a.id === 'sn-logic');
|
||||
logic.variables = [${JSON.stringify(variable('voice', 'number', 0))}];
|
||||
const graph = logic.graph;
|
||||
const play = graph.nodes.find((n) => n.id === 'sn-play');
|
||||
play.params.loop = 'true';
|
||||
graph.nodes.push(
|
||||
${JSON.stringify(node('sn-keep', 'Variables', 520, 0, {}, [written('voice', 'number')]))},
|
||||
{ ...structuredClone(play), id: 'sn-play2', x: 780 },
|
||||
${JSON.stringify(node('sn-tick', 'OnUpdate', 0, 300))},
|
||||
${JSON.stringify(node('sn-read', 'Variables', 0, 420, {}, [port('voice', 'number')]))},
|
||||
${JSON.stringify(node('sn-tune', 'SetSoundFxParameters', 260, 300))},
|
||||
${JSON.stringify(
|
||||
node('sn-fxp2', 'SoundFxParameters', 0, 540, { soundFxAsset: fxId, depth: 3 }, [
|
||||
{ key: 'depth', kind: 'number' },
|
||||
]),
|
||||
)},
|
||||
);
|
||||
graph.edges.push(
|
||||
${JSON.stringify(edge('sn-e-keep', 'sn-play', 'exec', 'sn-keep', 'exec'))},
|
||||
${JSON.stringify(edge('sn-e-voice', 'sn-play', 'voice', 'sn-keep', 'voice'))},
|
||||
${JSON.stringify(edge('sn-e-play2', 'sn-keep', 'exec', 'sn-play2', 'exec'))},
|
||||
${JSON.stringify(edge('sn-e-fxp3', 'sn-fxp', 'params', 'sn-play2', 'fxParams'))},
|
||||
${JSON.stringify(edge('sn-e-tick', 'sn-tick', 'exec', 'sn-tune', 'exec'))},
|
||||
${JSON.stringify(edge('sn-e-read', 'sn-read', 'voice', 'sn-tune', 'voice'))},
|
||||
${JSON.stringify(edge('sn-e-fxp2', 'sn-fxp2', 'params', 'sn-tune', 'fxParams'))},
|
||||
);`);
|
||||
const beforeTune = await stats();
|
||||
await play();
|
||||
const tunedStats = await waitFor(
|
||||
stats,
|
||||
(s) => s.tuned > beforeTune.tuned && s.started >= beforeTune.started + 2,
|
||||
);
|
||||
const rate = 2 ** ((3 - 7) / 12);
|
||||
const glided = (levels) =>
|
||||
Array.isArray(levels) && levels.length === 2 && Math.abs(levels[0][0] - rate) < 0.01;
|
||||
const levels = await waitFor(() => json(`__gm.sound.levels(${JSON.stringify(blip.file)})`), glided);
|
||||
check(
|
||||
'Set sound FX parameters moves the one voice whose number it holds from depth 7 to 3, not the other',
|
||||
tunedStats.last?.pitches.join() === '7' &&
|
||||
tunedStats.last.loop === true &&
|
||||
glided(levels) &&
|
||||
Math.abs(levels[0][1] - 10 ** (-6 / 20)) < 0.001 &&
|
||||
levels[1][0] === 1,
|
||||
JSON.stringify({ last: tunedStats.last, tuned: tunedStats.tuned, levels, rate }),
|
||||
);
|
||||
await stop();
|
||||
await mutate(`
|
||||
const logic = __gm.state.project.assets.find((a) => a.id === 'sn-logic');
|
||||
logic.variables = [];
|
||||
const graph = logic.graph;
|
||||
graph.nodes = graph.nodes.filter((n) => !${JSON.stringify(tuneIds)}.includes(n.id));
|
||||
graph.edges = graph.edges.filter((e) => !${JSON.stringify(tuneEdges)}.includes(e.id));`);
|
||||
await mutate(`
|
||||
const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph;
|
||||
graph.nodes = graph.nodes.filter((n) => n.id !== 'sn-fxp');
|
||||
|
||||
Reference in new issue
Block a user