Play a sound through a graph of gain, pitch and EQ effects #311

A sound could only be played as it was uploaded. A Sound FX asset is
a node graph like a shader: Input, effects, Output. Play sound has a
new input fx that sends the voice through it.

- New asset type soundfx ("Sound FX"), not placeable, with a graph
  and its own tab. The heading in the asset panel is "Sound effects",
  so it does not take the click meant for the + menu entry.
- Nodes in core/soundfx/nodes.ts, category sound, port type audio
  that only fits audio: Input, Output, Gain (dB), Pitch (semitones),
  EQ (80 Hz and 12 kHz shelves, peaks at 250 Hz, 1 kHz and 4 kHz).
- compileSoundFx walks back from the Output with the shared graph
  emitter. An input takes one edge, so the result is one line: a
  pitch and a list of gain and filter stages. A node without input
  or a graph without Output reports a problem on the node.
- Pitch is the playback rate of the source. It applies to the whole
  sound and also changes its length, the node description says so.
- The BehaviorSystem resolves fx and hands the line to the AudioSink.
  An fx that is missing or does not compile plays the sound dry.
- SoundLibrary wires source, stages, volume, destination in that
  order. stats().last now also names pitch and stage count.
- The tab header picks one of the project's sounds and plays it
  through the graph as it stands. Stop there ends only its own voice.

Tests:
- soundfx.test.ts (new, 11), sound-library (9, +2 for the wiring),
  node-catalog (+1 for fx), graph and asset-groups adjusted
- e2e 34: ALL PASSED - 26 checks (+5: + menu opens the tab, palette,
  2 effects, tab Play and game Play through the FX at -12 st)
- Gate on melektaus-atx: typecheck, lint, prettier clean, 239 of 240
  test files green. The red one was sound-library (its fake
  AudioContext had no playbackRate), fixed afterwards, gate not
  re-run in full.
- Counter-check: without the audio rule in portsCompatible the port
  test fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-05 16:45:42 +02:00
1 parent 93c0a80b7d
commit de14f0f9e2
34 files changed
+899 -43

No files matched your search

+2
View File
@@ -101,6 +101,8 @@ export function buildAsset(type: AssetType, name: string, id = newId()): Asset {
return { ...base, type: 'font', file: '' };
case 'sound':
return { ...base, type: 'sound', file: '' };
case 'soundfx':
return { ...base, type: 'soundfx', graph: createEmptyGraph() };
case 'particles':
return { ...base, type: 'particles', graph: createEmptyGraph() };
}
+14 -2
View File
@@ -9,7 +9,17 @@ export type { DeclaredPortKind } from '../properties/kinds';
* `picture` is what a shader's Pass draws (#301), for the nodes that read a picture.
*/
export type PortType =
'exec' | 'number' | 'boolean' | 'string' | 'object' | 'list' | 'struct' | 'vec3' | 'picture' | 'any';
| 'exec'
| 'number'
| 'boolean'
| 'string'
| 'object'
| 'list'
| 'struct'
| 'vec3'
| 'picture'
| 'audio'
| 'any';
export interface PortRef {
node: string;
@@ -172,11 +182,13 @@ export function graphFingerprint(graph: Graph): string {
/**
* A number may feed a colour: the shader compiler spreads it over the three channels. A picture is
* no value a calculation could take, so like `exec` it goes only where a picture is read.
* no value a calculation could take, so like `exec` it goes only where a picture is read – and a
* sound on its way through effects (#311) only where a sound goes in.
*/
export function portsCompatible(from: PortType, to: PortType): boolean {
if (from === 'exec' || to === 'exec') return from === to;
if (from === 'picture' || to === 'picture') return from === to;
if (from === 'audio' || to === 'audio') return from === to;
if (from === 'number' && to === 'vec3') return true;
return from === to || from === 'any' || to === 'any';
}
+2 -1
View File
@@ -237,8 +237,9 @@ export interface NodeContext {
* The voice belongs to the behavior that started it and ends when that behavior is destroyed –
* a sound is something a thing in the world is doing. `detached` gives that up on purpose, for
* the one case where the rule is in the way: the death sound of an object that destroys itself.
* `fx` names a sound FX asset the voice is played through (#311); `''` plays it as it is.
*/
playSound(assetId: string, volume: number, loop: boolean | undefined, detached: boolean): void;
playSound(assetId: string, volume: number, loop: boolean | undefined, detached: boolean, fx: string): void;
/** Silences every voice of that sound; `''` silences everything this run started. */
stopSound(assetId: string): void;
/**
+11 -4
View File
@@ -30,11 +30,12 @@ export interface PortDef {
}
/**
* `state`, `shader` and `particles` are the categories no behavior graph offers: their nodes are the
* states of a state machine, the GLSL of a shader asset and the motion of one particle.
* `state`, `shader`, `particles` and `sound` are the categories no behavior graph offers: their nodes
* are the states of a state machine, the GLSL of a shader asset, the motion of one particle and the
* effects of a sound FX asset.
*/
export type NodeCategory =
'event' | 'flow' | 'data' | 'variable' | 'action' | 'state' | 'shader' | 'particles';
'event' | 'flow' | 'data' | 'variable' | 'action' | 'state' | 'shader' | 'particles' | 'sound';
/**
* How a loop node asks for one of its exec outputs to be run repeatedly: `port` carries the body,
@@ -207,7 +208,13 @@ export function inlineInputs(def: NodeDef): PortDef[] {
function isInlinePort(port: PortDef): boolean {
if (port.inline === false) return false;
if (port.type === 'vec3') return port.field !== undefined;
return port.type !== 'exec' && port.type !== 'object' && port.type !== 'struct' && port.type !== 'picture';
return (
port.type !== 'exec' &&
port.type !== 'object' &&
port.type !== 'struct' &&
port.type !== 'picture' &&
port.type !== 'audio'
);
}
/** Untyped, an input writes the zero of its kind; as its default the node body shows it, not just the name. */
+10 -4
View File
@@ -1,6 +1,6 @@
import type { NodeDef, PortDef } from '../../NodeRegistry';
import { asBoolean, asNumber, asString, type Value } from '../../values';
import { exec, num, soundPort, withDefault } from '../shared';
import { exec, num, soundFxPort, soundPort, withDefault } from '../shared';
/**
* Three answers, not two: a checkbox cannot say "leave it to the asset", and without that a sound
@@ -56,14 +56,20 @@ export const PlaySound: NodeDef = {
"leaves the asset's own setting alone or overrules it, so the asset says what the sound is and " +
'the node only colors it. Each play is a voice of its own: firing this twice in a row is heard ' +
'twice, overlapping. The sound belongs to this object and stops when the object is destroyed, ' +
'unless "detached" is on.',
inputs: [exec(), soundPort(), withDefault(num('volume'), 1), loopPort(), detachedPort()],
'unless "detached" is on. "fx" sends it through a sound FX asset; left empty, it plays as it is.',
inputs: [exec(), soundPort(), withDefault(num('volume'), 1), loopPort(), detachedPort(), soundFxPort()],
outputs: [exec()],
params: [],
run: (ctx, inputs) => {
const sound = asString(inputs.sound);
if (sound)
ctx.playSound(sound, asNumber(inputs.volume), loopChoice(inputs.loop), asBoolean(inputs.detached));
ctx.playSound(
sound,
asNumber(inputs.volume),
loopChoice(inputs.loop),
asBoolean(inputs.detached),
asString(inputs.fx),
);
return { next: 'exec' };
},
};
+2
View File
@@ -146,6 +146,7 @@ import { StateTransition } from './flow/StateTransition';
import { PARTICLE_NODES } from '../../particles/nodes';
import { SHADER_FUNCTION_NODES } from '../../shader/functions';
import { SHADER_NODES } from '../../shader/nodes';
import { SOUND_FX_NODES } from '../../soundfx/nodes';
import { State } from '../stateMachine';
import { Variables } from './variable/Variables';
@@ -314,3 +315,4 @@ registerNode(State);
for (const def of SHADER_NODES) registerNode(def);
for (const def of SHADER_FUNCTION_NODES) registerNode(def);
for (const def of PARTICLE_NODES) registerNode(def);
for (const def of SOUND_FX_NODES) registerNode(def);
+8
View File
@@ -94,6 +94,14 @@ export const soundPort = (): PortDef => ({
field: { key: 'sound', label: 'Sound', kind: 'sound', default: '' },
});
/** The `fx` input of `Play sound` (#311); unconnected, it shows the picker of sound FX assets. */
export const soundFxPort = (): PortDef => ({
key: 'fx',
label: 'fx',
type: 'string',
field: { key: 'fx', label: 'FX', kind: 'asset', default: '', assetTypes: ['soundfx'] },
});
export function binaryMath(type: string, description: string, fn: (a: number, b: number) => number): NodeDef {
return {
type,
+12 -1
View File
@@ -310,6 +310,15 @@ export interface SoundAsset extends AssetBase {
file: string;
}
/**
* What a sound is played through (#311): the graph runs from its `Input` over effect nodes to its
* `Output`, and a `Play sound` naming it sends the voice along that line (see `core/soundfx`).
*/
export interface SoundFxAsset extends AssetBase {
type: 'soundfx';
graph: Graph;
}
/**
* A tile a tilemap paints with: `pictures` names one block per animation step (a texture id for a
* sheet that holds nothing else, a frame id for one block cut out of a sheet), and `props` say
@@ -562,6 +571,7 @@ export type Asset =
| ShaderFunctionAsset
| FontAsset
| SoundAsset
| SoundFxAsset
| ParticlesAsset;
/** The assets whose `variables` are declared by the user – see `requireVariableOwner`. */
@@ -573,7 +583,7 @@ export function isVariableOwner(asset: Asset | undefined): asset is VariableOwne
/** The assets edited as a node graph – what the graph canvas and the graph commands work on. */
export type GraphAsset =
BehaviorAsset | StateMachineAsset | ShaderAsset | ShaderFunctionAsset | ParticlesAsset;
BehaviorAsset | StateMachineAsset | ShaderAsset | ShaderFunctionAsset | ParticlesAsset | SoundFxAsset;
const GRAPH_TYPES: ReadonlySet<AssetType> = new Set([
'behavior',
@@ -581,6 +591,7 @@ const GRAPH_TYPES: ReadonlySet<AssetType> = new Set([
'shader',
'shaderfunction',
'particles',
'soundfx',
]);
export function isGraphAsset(asset: Asset): asset is GraphAsset {
+6 -1
View File
@@ -24,6 +24,7 @@ export const ASSET_TYPES = [
'shaderfunction',
'font',
'sound',
'soundfx',
'particles',
] as const;
@@ -55,6 +56,7 @@ export const ASSET_TYPE_LABELS: Record<AssetType, string> = {
shaderfunction: 'Shader function',
font: 'Font',
sound: 'Sound',
soundfx: 'Sound FX',
particles: 'Particles',
};
@@ -66,7 +68,8 @@ export const ASSET_TYPE_LABELS: Record<AssetType, string> = {
* what a tilemap draws with, so it is placed no more directly than a texture is – what is placed
* is the tilemap. A curve is the way a key or a `Lerp` blends, picked by id like a state machine.
* A struct is a shape that variables and fields take on (#253), nothing that exists on its own.
* A shader function is part of other shaders (#303), worn by nothing itself.
* A shader function is part of other shaders (#303), worn by nothing itself. A sound FX is what a
* sound is played through (#311), heard but never seen.
*/
export type PlaceableAssetType = Exclude<
AssetType,
@@ -83,6 +86,7 @@ export type PlaceableAssetType = Exclude<
| 'shaderfunction'
| 'font'
| 'sound'
| 'soundfx'
| 'variables'
| 'struct'
>;
@@ -101,6 +105,7 @@ const NOT_PLACEABLE: ReadonlySet<AssetType> = new Set([
'shaderfunction',
'font',
'sound',
'soundfx',
'variables',
'struct',
]);
+2
View File
@@ -655,6 +655,8 @@ const ASSET_SCHEMAS: Record<AssetType, PropertySchema> = {
{ key: SOUND_VOLUME_KEY, label: 'Volume', kind: 'number', default: 1, min: 0, max: 1, step: 0.05 },
{ key: SOUND_LOOP_KEY, label: 'Loop', kind: 'boolean', default: false },
],
// The effects are the asset's graph (see SoundFxAsset), not a property.
soundfx: [],
// How many particles and for how long; what one of them does is the asset's graph, whose Property
// nodes add their rows after these – see `schemaForBag`.
particles: [
+67
View File
@@ -0,0 +1,67 @@
import { graphEmitter } from '../graph/emitGraph';
import { type Graph, graphFingerprint } from '../graph/Graph';
import type { SoundFxAsset } from '../model/Asset';
import { nodeDef } from '../graph/NodeRegistry';
import {
DRY,
type FxChain,
isSoundFxNodeDef,
SOUND_FX_INPUT,
SOUND_FX_OUTPUT,
type SoundFxNodeDef,
} from './nodes';
export interface SoundFxBuild {
/** Null while the graph has a problem – a sound played through it is then played dry. */
compiled: FxChain | null;
/** How many nodes between Input and Output the sound passes. */
effects: number;
/** Per node id, what keeps the graph from compiling. */
problems: Record<string, string>;
/** One line for the tab: empty once compiled. */
reason: string;
}
/**
* A sound FX graph to the line a voice is sent along: walked back from the Output to the Input, every
* node on the way adding what it does. An input takes one edge, so the line never forks on its way
* to the Output; a node fed from nowhere is a problem rather than silence.
*/
export function compileSoundFx(graph: Graph): SoundFxBuild {
const walk = graphEmitter<SoundFxNodeDef, FxChain>(graph, {
isDef: isSoundFxNodeDef,
noun: 'sound FX',
// Only reached with every input fed, so whatever is not the Input has its sound arriving.
emit: (def, inputs, node) => ({ out: def.emit(inputs.in ?? DRY, node.params) }),
unfed: (node) => {
walk.problem(node, 'Nothing comes in: connect it to the Input');
return null;
},
});
const output = graph.nodes.find((n) => n.type === SOUND_FX_OUTPUT);
if (!output) return { compiled: null, effects: 0, problems: walk.problems, reason: 'No Output node' };
const chain = walk.input(output, nodeDef(SOUND_FX_OUTPUT)!.inputs[0]!);
const failed = walk.firstProblem();
if (failed !== null || !chain)
return { compiled: null, effects: 0, problems: walk.problems, reason: failed ?? 'Cannot compile' };
const effects = graph.nodes.filter(
(n) => walk.reached(n) && n.type !== SOUND_FX_INPUT && n.type !== SOUND_FX_OUTPUT,
).length;
return { compiled: chain, effects, problems: walk.problems, reason: '' };
}
const builds = new Map<string, { fingerprint: string; build: SoundFxBuild }>();
/**
* The asset's build, compiled again only when what it does has changed – a voice is started far
* more often than its graph is edited, and a moved node changes nothing.
*/
export function soundFxBuildFor(asset: SoundFxAsset): SoundFxBuild {
const fingerprint = graphFingerprint(asset.graph);
const cached = builds.get(asset.id);
if (cached && cached.fingerprint === fingerprint) return cached.build;
const build = compileSoundFx(asset.graph);
builds.set(asset.id, { fingerprint, build });
return build;
}
+147
View File
@@ -0,0 +1,147 @@
import type { NodeDef, PortDef } from '../graph/NodeRegistry';
import type { PropertyBag } from '../model/Properties';
import type { FlatDef } from '../properties/PropertySchema';
/** The filter shapes the EQ uses – Web Audio's names, so the player hands them on as they are. */
export type FxFilter = 'lowshelf' | 'peaking' | 'highshelf';
/** One step a voice passes on its way out: a volume, or one band of an equalizer. */
export type FxStage =
| { kind: 'gain'; gain: number }
| { kind: 'filter'; filter: FxFilter; frequency: number; gain: number; q: number };
/**
* A sound on its way through the graph: how many semitones it is played up or down, and what it
* passes, in order. `gain` is a factor, a filter's `gain` is in dB as Web Audio takes it.
*/
export interface FxChain {
pitch: number;
stages: readonly FxStage[];
}
export const DRY: FxChain = { pitch: 0, stages: [] };
/**
* A node of a sound FX graph (#311). It never runs: `emit` says what it does to the sound arriving
* at its one input. Input and Output have a side less and no effect of their own.
*/
export interface SoundFxNodeDef extends NodeDef {
category: 'sound';
emit(chain: FxChain, params: PropertyBag): FxChain;
}
export function isSoundFxNodeDef(def: NodeDef): def is SoundFxNodeDef {
return def.category === 'sound' && 'emit' in def;
}
export const SOUND_FX_INPUT = 'SoundFxInput';
export const SOUND_FX_OUTPUT = 'SoundFxOutput';
const audioIn: PortDef = { key: 'in', label: 'in', type: 'audio' };
const audioOut: PortDef = { key: 'out', label: 'out', type: 'audio' };
function numberParam(key: string, label: string, min: number, max: number, step: number): FlatDef<'number'> {
return { key, label, kind: 'number', default: 0, min, max, step };
}
/** A param's value held to its row, so a hand-edited project cannot blow a speaker. */
function read(params: PropertyBag, def: FlatDef<'number'>): number {
const value = Number(params[def.key] ?? def.default);
if (!Number.isFinite(value)) return def.default;
return Math.max(def.min ?? -Infinity, Math.min(def.max ?? Infinity, value));
}
export const SoundFxInput: SoundFxNodeDef = {
type: SOUND_FX_INPUT,
label: 'Input',
category: 'sound',
description: 'The sound as the Play sound node starts it, before any effect.',
inputs: [],
outputs: [audioOut],
params: [],
emit: (chain) => chain,
};
export const SoundFxOutput: SoundFxNodeDef = {
type: SOUND_FX_OUTPUT,
label: 'Output',
category: 'sound',
description: 'What is heard: the sound as it arrives here, with every effect on the way.',
inputs: [audioIn],
outputs: [],
params: [],
emit: (chain) => chain,
};
const GAIN = numberParam('gain', 'Gain (dB)', -60, 24, 0.5);
export const SoundFxGain: SoundFxNodeDef = {
type: 'SoundFxGain',
label: 'Gain',
category: 'sound',
description:
'Makes the sound louder or quieter, in decibels: 0 leaves it as it is, -6 is about half as loud, +6 twice. Above 0 it can clip.',
inputs: [audioIn],
outputs: [audioOut],
params: [GAIN],
emit: (chain, params) => {
const db = read(params, GAIN);
if (db === 0) return chain;
return { ...chain, stages: [...chain.stages, { kind: 'gain', gain: 10 ** (db / 20) }] };
},
};
const SEMITONES = numberParam('semitones', 'Semitones', -24, 24, 1);
export const SoundFxPitch: SoundFxNodeDef = {
type: 'SoundFxPitch',
label: 'Pitch',
category: 'sound',
description:
'Plays the sound higher or lower, in semitones: 12 is an octave up. It is the speed the sound is played at, so a higher one is also shorter, and it applies to the whole sound wherever the node sits in the line.',
inputs: [audioIn],
outputs: [audioOut],
params: [SEMITONES],
emit: (chain, params) => ({ ...chain, pitch: chain.pitch + read(params, SEMITONES) }),
};
/** The five bands of the EQ: a shelf at either end and three peaks between them. */
export const EQ_BANDS: readonly { key: string; label: string; filter: FxFilter; frequency: number }[] = [
{ key: 'low', label: '80 Hz', filter: 'lowshelf', frequency: 80 },
{ key: 'lowMid', label: '250 Hz', filter: 'peaking', frequency: 250 },
{ key: 'mid', label: '1 kHz', filter: 'peaking', frequency: 1000 },
{ key: 'highMid', label: '4 kHz', filter: 'peaking', frequency: 4000 },
{ key: 'high', label: '12 kHz', filter: 'highshelf', frequency: 12000 },
];
/** About an octave and a half wide: neighbouring peaks overlap, so no frequency falls between them. */
const PEAK_Q = 1;
const EQ_PARAMS = EQ_BANDS.map((band) => numberParam(band.key, band.label, -24, 24, 0.5));
export const SoundFxEq: SoundFxNodeDef = {
type: 'SoundFxEq',
label: 'EQ',
category: 'sound',
description:
'A five-band equalizer: raises or lowers the lows (80 Hz), low mids (250 Hz), mids (1 kHz), high mids (4 kHz) and highs (12 kHz), each in decibels. A band at 0 does nothing.',
inputs: [audioIn],
outputs: [audioOut],
params: EQ_PARAMS,
emit: (chain, params) => {
const stages: FxStage[] = [];
EQ_BANDS.forEach((band, i) => {
const gain = read(params, EQ_PARAMS[i]!);
if (gain !== 0)
stages.push({ kind: 'filter', filter: band.filter, frequency: band.frequency, gain, q: PEAK_Q });
});
return stages.length === 0 ? chain : { ...chain, stages: [...chain.stages, ...stages] };
},
};
export const SOUND_FX_NODES: readonly SoundFxNodeDef[] = [
SoundFxInput,
SoundFxOutput,
SoundFxGain,
SoundFxPitch,
SoundFxEq,
];
+3
View File
@@ -19,6 +19,7 @@ import {
} from '../graph';
import { isParticleNodeDef } from '../particles/nodes';
import { isShaderNodeDef } from '../shader/nodes';
import { isSoundFxNodeDef } from '../soundfx/nodes';
import { isKeyInterpolation, normalizeKeys } from '../model/animation';
import type {
AnimationKey,
@@ -234,6 +235,8 @@ export function deserializeProject(raw: unknown, fallbackId = ''): Project {
assets.push({ id, type, name, props, graph: shaderGraphs.get(id) ?? createEmptyGraph() });
} else if (type === 'particles') {
assets.push({ id, type, name, props, graph: particleGraphs.get(id) ?? createEmptyGraph() });
} else if (type === 'soundfx') {
assets.push({ id, type, name, props, graph: normalizeGraph(a.graph, isSoundFxNodeDef) });
} else if (type === 'texture') {
assets.push({ id, type, name, props, ...normalizeTexture(a, takenIds) });
} else if (type === 'font') {
+9
View File
@@ -72,6 +72,7 @@ import { type PlayHost, PlaySession, type WorldSnapshot } from './PlaySession';
import { ProjectSession } from './ProjectSession';
import { ParticlesEditor } from './particles/ParticlesEditor';
import { ShaderEditor } from './shader/ShaderEditor';
import { SoundFxEditor } from './soundfx/SoundFxEditor';
import { StateMachineEditor } from './statemachine/StateMachineEditor';
import { installShortcuts } from './shortcuts';
import type { PlayFrom } from '../core/model/playSettings';
@@ -93,6 +94,7 @@ export type CentreTab =
| 'statemachine'
| 'shader'
| 'particles'
| 'soundfx'
| 'game'
| 'texture'
| 'vectorart'
@@ -142,6 +144,7 @@ const PANE_ORDER = [
'statemachine',
'shader',
'particles',
'soundfx',
'texture',
'vectorart',
'pixelart',
@@ -161,6 +164,7 @@ const TAB_LABELS: Record<CentreTab, string> = {
statemachine: 'State machine',
shader: 'Shader',
particles: 'Particles',
soundfx: 'Sound FX',
texture: 'Texture',
vectorart: 'Vector art',
pixelart: 'Pixel art',
@@ -209,6 +213,7 @@ export class Editor implements PlayHost {
readonly stateMachineEditor: StateMachineEditor;
readonly shaderEditor: ShaderEditor;
readonly particlesEditor: ParticlesEditor;
readonly soundFxEditor: SoundFxEditor;
readonly textureEditor: TextureEditor;
readonly vectorArtEditor: VectorArtEditor;
readonly pixelArtEditor: PixelArtEditor;
@@ -294,6 +299,7 @@ export class Editor implements PlayHost {
this.stateMachineEditor = new StateMachineEditor(state);
this.shaderEditor = new ShaderEditor(state);
this.particlesEditor = new ParticlesEditor(state);
this.soundFxEditor = new SoundFxEditor(state);
this.textureEditor = new TextureEditor(state);
this.vectorArtEditor = new VectorArtEditor(state);
this.pixelArtEditor = new PixelArtEditor(state);
@@ -402,6 +408,7 @@ export class Editor implements PlayHost {
statemachine: this.stateMachineEditor,
shader: this.shaderEditor,
particles: this.particlesEditor,
soundfx: this.soundFxEditor,
texture: this.textureEditor,
vectorart: this.vectorArtEditor,
pixelart: this.pixelArtEditor,
@@ -591,6 +598,8 @@ export class Editor implements PlayHost {
return this.shaderEditor.canvas.paste();
case 'particles':
return this.particlesEditor.canvas.paste();
case 'soundfx':
return this.soundFxEditor.canvas.paste();
default:
return false;
}
+1
View File
@@ -9,6 +9,7 @@ export const TAB_OF_ASSET: Partial<Record<AssetType, CentreTab>> = {
shader: 'shader',
shaderfunction: 'shader',
particles: 'particles',
soundfx: 'soundfx',
texture: 'texture',
vectorart: 'vectorart',
pixelart: 'pixelart',
+14
View File
@@ -96,6 +96,20 @@ export function graphHooks({ state, editor }: HookContext) {
startRing: (nodeId: string) => editor.stateMachineEditor.canvas.startRingDrawn(nodeId),
stateMarks: () => editor.stateMachineEditor.canvas.stateMarksDrawn(),
},
soundfx: {
current: () => editor.soundFxEditor.currentAssetId,
info: () => editor.soundFxEditor.shownInfo,
canvasRect: () => editor.soundFxEditor.canvas.canvasRect(),
nodeProblem: (nodeId: string) => editor.soundFxEditor.canvas.nodeProblem(nodeId),
/** The header's listening controls: which sound, and whether Play is offered. */
preview: () => ({
sound: editor.soundFxEditor.previewSoundId,
picker: editor.soundFxEditor.soundPicker.text,
play: editor.soundFxEditor.playButton.enabled,
}),
playAt: () => centreOf(editor.soundFxEditor.playButton),
stopAt: () => centreOf(editor.soundFxEditor.stopButton),
},
particles: {
current: () => editor.particlesEditor.currentAssetId,
info: () => editor.particlesEditor.shownInfo,
+4 -1
View File
@@ -28,6 +28,7 @@ import type { GraphAsset } from '../../core/model/Asset';
import { graphLookup } from '../../core/model/graphLookup';
import { particleLibrary } from '../../viewport/ParticleLibrary';
import { shaderLibrary } from '../../viewport/ShaderLibrary';
import { soundFxBuildFor } from '../../core/soundfx/compile';
import { findAsset, findGraphAsset } from '../../core/model/Project';
import { type Trace, TRACE_FADE } from '../../runtime/systems/BehaviorSystem';
import { ViewTween } from '../viewTween';
@@ -305,7 +306,9 @@ export class GraphCanvas extends Widget {
? shaderLibrary.checkFor(asset, this.state.project).problems
: asset.type === 'particles'
? particleLibrary.buildFor(asset).problems
: null;
: asset.type === 'soundfx'
? soundFxBuildFor(asset).problems
: null;
for (const node of asset.graph.nodes) {
const resolved = resolveDef(node, asset.graph, lookup);
if (!resolved) continue;
+4
View File
@@ -67,6 +67,8 @@ function categoryColor(category: NodeCategory): number {
return c.categoryShader;
case 'particles':
return c.categoryParticles;
case 'sound':
return c.categorySound;
}
}
@@ -91,6 +93,8 @@ export function portColor(type: PortType): number {
return c.portVec3;
case 'picture':
return c.portPicture;
case 'audio':
return c.portAudio;
case 'any':
return c.portAny;
}
+10 -2
View File
@@ -41,6 +41,7 @@ import {
import { nextParticlePropertyName, PARTICLE_NODES, PARTICLE_OUTPUT } from '../../core/particles/nodes';
import { offeredIn, ShaderFunctionInput, ShaderFunctionOutput } from '../../core/shader/functions';
import { SHADER_NODES, SHADER_OUTPUT, SHADER_VERTEX_OUTPUT } from '../../core/shader/nodes';
import { SOUND_FX_INPUT, SOUND_FX_NODES, SOUND_FX_OUTPUT } from '../../core/soundfx/nodes';
/**
* One row of the palette. `def` is the definition the node would actually have once placed, so the
@@ -125,7 +126,8 @@ function shaderEntries(project: Project, asset: ShaderAsset | ShaderFunctionAsse
/**
* The node types offerable in `asset`'s graph. A state machine holds states and nothing else, each
* new one named by its number; a shader holds shader nodes (`shaderEntries`). In a behavior, every other behavior appears under its own name instead of behind a
* new one named by its number; a shader holds shader nodes (`shaderEntries`), a sound FX its effects
* with one Input and one Output. In a behavior, every other behavior appears under its own name instead of behind a
* generic "Run behavior" – and once more as the event "On <name>" if it has an End to be heard
* through (#294) –, and the two terminals drop out once the graph has them.
*/
@@ -135,6 +137,12 @@ export function paletteEntries(project: Project, asset: GraphAsset): PaletteEntr
return [{ id: State.type, hint: State.category, def: State, params }];
}
if (asset.type === 'shader' || asset.type === 'shaderfunction') return shaderEntries(project, asset);
if (asset.type === 'soundfx') {
const placed = new Set(asset.graph.nodes.map((n) => n.type));
return SOUND_FX_NODES.filter(
(def) => !(def.type === SOUND_FX_INPUT || def.type === SOUND_FX_OUTPUT) || !placed.has(def.type),
).map((def) => ({ id: def.type, hint: def.category, def, params: {} }));
}
if (asset.type === 'particles') {
const hasOutput = asset.graph.nodes.some((n) => n.type === PARTICLE_OUTPUT);
const taken = new Set(asset.graph.nodes.filter(isParticlePropertyNode).map(propertyNameOf));
@@ -147,7 +155,7 @@ export function paletteEntries(project: Project, asset: GraphAsset): PaletteEntr
const entries: PaletteEntry[] = [];
for (const def of allNodeDefs()) {
if (namesBehavior(def.type) || def.category === 'state') continue;
if (def.category === 'shader' || def.category === 'particles') continue;
if (def.category === 'shader' || def.category === 'particles' || def.category === 'sound') continue;
if (isTerminalType(def.type) && terminalNode(asset.graph, def.type)) continue;
entries.push({ id: def.type, hint: def.category, def, params: {} });
if (def.type === FIND) entries.push(filterEntry(def));
+2
View File
@@ -34,6 +34,8 @@ export const ASSET_GROUP_LABELS: Record<AssetType, string> = {
shaderfunction: 'Shader functions',
font: 'Fonts',
sound: 'Sounds',
// Not "Sound FX": that is the `+` menu entry (see Vector art).
soundfx: 'Sound effects',
particles: 'Particles',
};
+2
View File
@@ -24,6 +24,7 @@ import scene from './icons/scene.svg?raw';
import shader from './icons/shader.svg?raw';
import shaderfunction from './icons/shaderfunction.svg?raw';
import sound from './icons/sound.svg?raw';
import soundfx from './icons/soundfx.svg?raw';
import sprite from './icons/sprite.svg?raw';
import statemachine from './icons/statemachine.svg?raw';
import struct from './icons/struct.svg?raw';
@@ -53,6 +54,7 @@ export const ASSET_ICON_SVG: Record<AssetType, string> = {
shader,
shaderfunction,
sound,
soundfx,
sprite,
statemachine,
struct,
+7
View File
@@ -0,0 +1,7 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="-8 -8 16 16" width="16" height="16">
<!-- Three faders of an equalizer: what a sound is played through. -->
<path d="M-4.5 -6.5 V6.5 M0 -6.5 V6.5 M4.5 -6.5 V6.5" fill="none" stroke="currentColor" stroke-width="1.25" stroke-linecap="round" />
<rect x="-6.5" y="0.5" width="4" height="2.5" rx="0.8" fill="currentColor" />
<rect x="-2" y="-4" width="4" height="2.5" rx="0.8" fill="currentColor" />
<rect x="2.5" y="-1" width="4" height="2.5" rx="0.8" fill="currentColor" />
</svg>

After

Width:  |  Height:  |  Size: 537 B

+107
View File
@@ -0,0 +1,107 @@
import type { Asset, SoundFxAsset } from '../../core/model/Asset';
import { findAssetOf } from '../../core/model/Project';
import { resolveAssetProps } from '../../core/properties/resolve';
import { volumeOf } from '../../core/properties/sounds';
import { soundFxBuildFor } from '../../core/soundfx/compile';
import { Button, Dropdown, Menu } from '../../ui';
import { soundLibrary } from '../../viewport/SoundLibrary';
import { AssetTabEditor } from '../AssetTabEditor';
import { GraphCanvas } from '../graph/GraphCanvas';
import { SnapToggle } from '../graph/SnapToggle';
import { TOOLBAR_ICON_SVG } from '../panels/toolbarIcons';
import type { EditorState } from '../state';
/** Room kept for "N effects", or for the reason the graph does not compile. */
const INFO_WIDTH = 220;
const SOUND_WIDTH = 140;
/** Whose voices the tab's Stop silences: its own, never a running game's. */
const PREVIEW_OWNER = 'soundfx-preview';
/**
* The nodes of one sound FX asset (#311), and in the header a way to hear them: a sound of the
* project played through the graph as it stands. Which sound is the tab's, not the asset's – it is
* only there to listen with.
*/
export class SoundFxEditor extends AssetTabEditor<SoundFxAsset, GraphCanvas> {
readonly snap = new SnapToggle(() => this.canvas);
/** Public for the check scripts. */
readonly soundPicker: Dropdown;
readonly playButton: Button;
readonly stopButton: Button;
private soundId = '';
constructor(state: EditorState) {
super(state, { infoWidth: INFO_WIDTH });
this.soundPicker = new Dropdown('No sound', {
width: SOUND_WIDTH,
onClick: () => this.openSoundPicker(),
});
this.playButton = new Button('Play', { icon: TOOLBAR_ICON_SVG.play, onClick: () => this.play() });
this.stopButton = new Button('Stop', { icon: TOOLBAR_ICON_SVG.stop, onClick: () => this.stop() });
this.mount(new GraphCanvas(state, 'sound FX'), [
...this.snap.widgets,
this.soundPicker,
this.playButton,
this.stopButton,
]);
state.on('load', () => {
this.soundId = '';
});
}
override selectedId(): string {
return this.state.selectedSoundFxId ?? '';
}
override isMine(asset: Asset): asset is SoundFxAsset {
return asset.type === 'soundfx';
}
/** The sound the header plays: the one picked, else the project's first. */
get previewSoundId(): string {
const sounds = this.state.project.assets.filter((a) => a.type === 'sound');
return sounds.some((a) => a.id === this.soundId) ? this.soundId : (sounds[0]?.id ?? '');
}
pickSound(id: string): void {
this.soundId = id;
this.refresh();
}
/** Plays the picked sound through the graph; a graph that does not compile is not played at all. */
play(): void {
const asset = this.asset();
const sound = findAssetOf(this.state.project, this.previewSoundId, 'sound');
const chain = asset && soundFxBuildFor(asset).compiled;
if (!sound?.file || !chain) return;
const volume = volumeOf(resolveAssetProps(this.state.project, sound));
soundLibrary.play(sound.file, volume, false, PREVIEW_OWNER, chain);
}
stop(): void {
soundLibrary.stopOwner(PREVIEW_OWNER);
}
protected override infoText(asset: SoundFxAsset): string {
const build = soundFxBuildFor(asset);
if (!build.compiled) return build.reason;
return `${build.effects} ${build.effects === 1 ? 'effect' : 'effects'}`;
}
protected override syncExtras(asset: SoundFxAsset | null): void {
this.snap.setEnabled(Boolean(asset));
const sound = findAssetOf(this.state.project, this.previewSoundId, 'sound');
this.soundPicker.setText(sound?.name ?? 'No sound');
this.soundPicker.setEnabled(Boolean(asset && sound));
this.playButton.setEnabled(Boolean(asset && sound?.file && soundFxBuildFor(asset).compiled));
this.stopButton.setEnabled(Boolean(asset));
}
private openSoundPicker(): void {
const current = this.previewSoundId;
const items = this.state.project.assets
.filter((a) => a.type === 'sound')
.map((a) => ({ label: a.name, current: a.id === current, onSelect: () => this.pickSound(a.id) }));
if (items.length > 0) new Menu(items).openBelow(this.soundPicker);
}
}
+5
View File
@@ -810,6 +810,11 @@ export class EditorState {
return this.selectedGraphId('particles');
}
/** The sound FX asset the selection refers to: the asset, or one of its nodes. */
get selectedSoundFxId(): string | null {
return this.selectedGraphId('soundfx');
}
private selectedGraphId(type: GraphAsset['type']): string | null {
const s = this.selection;
if (!s) return null;
+5 -2
View File
@@ -1,3 +1,5 @@
import type { FxChain } from '../../core/soundfx/nodes';
/**
* What the runtime needs of the outside world to make a noise – the counterpart of `Input` on the
* way out. The browser side of it is `viewport/SoundLibrary`; a test (and a run started before
@@ -11,9 +13,10 @@ export interface AudioSink {
* Starts one voice. `volume` is final (0–1) and every call overlaps whatever is already playing.
* `owner` is the behavior the sound belongs to: when that behavior is destroyed the voice goes
* with it, because a sound is something a thing in the world is doing, not a thing of its own.
* `''` means the voice has no owner and outlives whatever started it.
* `''` means the voice has no owner and outlives whatever started it. `fx` is the line of effects
* the voice passes on its way out (#311); `DRY` passes none.
*/
play(file: string, volume: number, loop: boolean, owner: string): void;
play(file: string, volume: number, loop: boolean, owner: string, fx?: FxChain): void;
/** Silences every voice of that file; `''` silences all of them. */
stop(file: string): void;
/** Silences every voice started by that owner – what destroying a behavior does. */
+9 -4
View File
@@ -15,6 +15,8 @@ import { soundFileOf } from '../../core/model/sound';
import { hasRenderLayer } from '../../core/properties/PropertySchema';
import { resolveAssetProps } from '../../core/properties/resolve';
import { loopsOf, volumeOf } from '../../core/properties/sounds';
import { soundFxBuildFor } from '../../core/soundfx/compile';
import { DRY } from '../../core/soundfx/nodes';
import type { Entity } from '../Entity';
import { PhysicsSystem } from '../physics/PhysicsSystem';
import type { World } from '../World';
@@ -162,8 +164,8 @@ export function behaviorContext(
// eslint-disable-next-line no-console
console.log(`[behavior ${entity.name}]`, message);
},
playSound: (assetId, volume, loop, detached) =>
playSound(run, assetId, volume, loop, detached ? '' : entity.id),
playSound: (assetId, volume, loop, detached, fx) =>
playSound(run, assetId, volume, loop, detached ? '' : entity.id, fx),
// An id nothing resolves stops nothing, rather than silencing the whole run by accident;
// only the empty id means "everything", which is what the node promises.
stopSound: (assetId) => {
@@ -214,7 +216,8 @@ export function behaviorContext(
/**
* What the graph asked for, as the sink understands it: the asset's own volume scaled by the
* node's, and its loop flag unless the node overrules it either way. The asset is what a sound
* *is* – a node that says nothing therefore plays it as the project describes it.
* *is* – a node that says nothing therefore plays it as the project describes it. An FX asset that
* does not compile plays the sound dry: a broken effect should not silence the game.
*/
function playSound(
{ project, audio }: ContextSources,
@@ -222,12 +225,14 @@ function playSound(
volume: number,
loop: boolean | undefined,
owner: string,
fxId: string,
): void {
const asset = findAssetOf(project, assetId, 'sound');
if (!asset?.file) return;
const props = resolveAssetProps(project, asset);
const level = Math.max(0, Math.min(1, volumeOf(props) * volume));
audio.play(asset.file, level, loop ?? loopsOf(props), owner);
const fx = fxId ? findAssetOf(project, fxId, 'soundfx') : undefined;
audio.play(asset.file, level, loop ?? loopsOf(props), owner, (fx && soundFxBuildFor(fx).compiled) || DRY);
}
/** Where a ray of that length ends; where it has no direction, where it starts. */
+2
View File
@@ -69,6 +69,7 @@ export const Theme = {
categoryState: 0x8e5ad6,
categoryShader: 0x1f8f8f,
categoryParticles: 0xc26a1f,
categorySound: 0xb0457a,
portExec: 0xffffff,
portNumber: 0x6fcf97,
portBoolean: 0xe57373,
@@ -78,6 +79,7 @@ export const Theme = {
portStruct: 0xd4c25a,
portVec3: 0xff9ad5,
portPicture: 0x9b8cff,
portAudio: 0xf0c060,
portAny: 0xbdbdbd,
/** Ring around the port a running edge drag would connect to. */
portTarget: 0xffffff,
+44 -8
View File
@@ -1,3 +1,4 @@
import { DRY, type FxChain, type FxStage } from '../core/soundfx/nodes';
import type { AudioSink } from '../runtime/systems/Audio';
const DEFAULT_BASE = '/game-maker/data/sounds';
@@ -35,7 +36,7 @@ export class SoundLibrary implements AudioSink {
private started = 0;
private stopped = 0;
/** The last voice actually started, with the values it was started at – what a check reads back. */
private last: { file: string; volume: number; loop: boolean } | null = null;
private last: LastVoice | null = null;
/**
* How often a stop has reached each file, and how often one has silenced everything. Counters
* rather than a "cancelled" flag: what a play has to know is whether a stop happened *after it
@@ -53,18 +54,19 @@ export class SoundLibrary implements AudioSink {
this.base = base.replace(/\/+$/, '');
}
play(file: string, volume: number, loop: boolean, owner = ''): void {
play(file: string, volume: number, loop: boolean, owner = '', fx: FxChain = DRY): void {
if (file === '' || this.failed.has(file)) return;
const buffer = this.buffers.get(file);
if (buffer) {
this.start(file, buffer, volume, loop, owner);
this.start(file, buffer, volume, loop, owner, fx);
return;
}
const askedAt = this.cancelsFor(file, owner);
void this.load(file).then((loaded) => {
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
if (loaded && this.cancelsFor(file, owner) === askedAt) this.start(file, loaded, volume, loop, owner);
if (loaded && this.cancelsFor(file, owner) === askedAt)
this.start(file, loaded, volume, loop, owner, fx);
});
}
@@ -104,7 +106,7 @@ export class SoundLibrary implements AudioSink {
stopped: number;
decoded: number;
failed: number;
last: { file: string; volume: number; loop: boolean } | null;
last: LastVoice | null;
} {
return {
playing: this.voices.length,
@@ -155,7 +157,14 @@ export class SoundLibrary implements AudioSink {
this.stopped += going.length;
}
private start(file: string, buffer: AudioBuffer, volume: number, loop: boolean, owner: string): void {
private start(
file: string,
buffer: AudioBuffer,
volume: number,
loop: boolean,
owner: string,
fx: FxChain,
): void {
const context = this.audio();
// A context starts suspended where the page has not been interacted with; without this the
// voice would run silently and end as if it had played.
@@ -163,10 +172,13 @@ export class SoundLibrary implements AudioSink {
const source = context.createBufferSource();
source.buffer = buffer;
source.loop = loop;
source.playbackRate.value = 2 ** (fx.pitch / 12);
const gain = context.createGain();
gain.gain.value = Math.max(0, Math.min(1, volume));
this.last = { file, volume: gain.gain.value, loop };
source.connect(gain).connect(context.destination);
this.last = { file, volume: gain.gain.value, loop, pitch: fx.pitch, stages: fx.stages.length };
let tail: AudioNode = source;
for (const stage of fx.stages) tail = tail.connect(fxNode(context, stage));
tail.connect(gain).connect(context.destination);
const voice: Voice = { file, owner, source, gain };
source.onended = () => {
this.voices = this.voices.filter((other) => other !== voice);
@@ -186,4 +198,28 @@ export class SoundLibrary implements AudioSink {
}
}
/** What `stats` says about the last voice: enough for a check to tell which effects it went through. */
interface LastVoice {
file: string;
volume: number;
loop: boolean;
/** Semitones, and how many gain and filter nodes it passed. */
pitch: number;
stages: number;
}
function fxNode(context: AudioContext, stage: FxStage): AudioNode {
if (stage.kind === 'gain') {
const gain = context.createGain();
gain.gain.value = stage.gain;
return gain;
}
const filter = context.createBiquadFilter();
filter.type = stage.filter;
filter.frequency.value = stage.frequency;
filter.gain.value = stage.gain;
filter.Q.value = stage.q;
return filter;
}
export const soundLibrary = new SoundLibrary();