From 8a4e4a676263b5e6004018bfe581977c9c3c7cc4 Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Tue, 6 Oct 2026 10:07:40 +0200 Subject: [PATCH] Play sounds as notes of an instrument asset #331 A sound could only be played as it was, one at a time. An instrument is a node graph that makes a note a sound: Notes nodes bring the note in, Samples play sound assets for it, the effects of a sound FX shape it and the Output sums the layers. Its tab plays it from a keyboard, a behavior with Play note. It is what patterns and songs (#330) will play. - New asset type instrument, under the heading Music, not placeable. A new one starts as Notes into a Sample into the Output. - Notes are counted as MIDI does: 60 is C4, names like C#4, sharps only, the lowest C-1. Notes has From and To, outputs any, note, frequency and velocity, and with Single notes a trigger per note, each with an optional name for a drum kit. - New port type trigger, only into the trigger of a source. Inputs can take any number of edges now (PortDef.many): a trigger, and the Output, which plays everything into it side by side. - Sample plays a sound asset. Follow note shifts it by note - root through the pitch worklet of #317, so it keeps its length. Gate fades it out at the end of the note, One-shot plays through. Loop runs between Loop start and Loop end, also in the worklet. - Parameters declares the instrument's own parameters with a default. Sound FX sends the sound through a sound FX asset, its parameters as inputs, the sound set in where its Input was. - compileInstrument walks the graph once per note. A trigger is on only for the notes it stands for, a silent source silences what it alone feeds, a Mix with a silent side is the other side. velocity and the parameters stay open for the voice. Builds are cached per note until the graph or a sound or FX it names changes. - The player takes a note as one voice of several layers, each way with its own file and volume. release fades the gated ways out, one-shots play on. playNote can release after a duration. - Play note plays one note with velocity, volume, duration and detached, and hands out voice for Set sound FX parameters and Stop sound. Velocity is always loudness. Instrument parameters picks values for the instrument's parameters like Sound FX parameters does for an FX, renames follow. - The tab has the graph, a header field per parameter, Stop, and 128 keys below. Keys without sound are grey. A key plays while it is held. Focused, the computer keyboard plays as in a tracker by KeyboardEvent.code, the arrows move the octave. Findings: - A label of the keys set fill on the shared small text style and darkened every small label of the editor. Caught in a screenshot before the commit, the keys have a style of their own now. - Suite task 523 on cf6799a failed two checks of e2e 34 that timed audio by the page's clock: a 2.95 s echo tail was gone after 1 s, and Quad in was further along than 0.7 s allow. Both time on the audio clock now through a new hook __gm.sound.clock(). Tests: - instrument.test.ts (new, 22): notes, asset, palette, round trip, compile per note, drum kit, zones, Mix, velocity and parameters, Sound FX node with rename, problems, cache, Play note, the keys - sound-library +7 (layers, release, duration, early release, loop region in buffer and worklet), pitch-player +2 (loop region), node-catalog +4, graph adjusted. 582 tests in the touched files. - e2e 4 and 34 on melektaus-atx: ALL PASSED, 34 with 44 checks (+7) - Gate on melektaus-atx: CHECK EXIT 0, 243 test files, 3750 tests - Counter-checks: any always on, release ending one-shots, Sound FX inputs ignored, loop region ignored. 3, 2, 1 and 1 tests failed. - Browser probe on stein2: keys, held key, Single notes names and the header looked at in screenshots. Co-Authored-By: Claude Opus 5.5 --- src/core/commands/assetCommands.ts | 3 + src/core/commands/graphCommands.ts | 4 +- src/core/commands/portCommands.ts | 20 +- src/core/graph/Graph.ts | 7 +- src/core/graph/NodeContext.ts | 15 + src/core/graph/NodeRegistry.ts | 10 +- src/core/graph/dropTarget.ts | 12 +- src/core/graph/emitGraph.ts | 33 +- src/core/graph/lookup.ts | 5 +- src/core/graph/nested.ts | 55 +- src/core/graph/nodes/action/PlayNote.ts | 63 +++ .../nodes/data/InstrumentParameterValues.ts | 37 ++ src/core/graph/nodes/index.ts | 6 + src/core/instrument/compile.ts | 320 ++++++++++++ src/core/instrument/nodes.ts | 246 +++++++++ src/core/instrument/notes.ts | 37 ++ src/core/model/Asset.ts | 19 +- src/core/model/AssetTypes.ts | 7 +- src/core/model/graphLookup.ts | 10 +- src/core/properties/PropertySchema.ts | 2 + src/core/soundfx/nodes.ts | 25 +- src/core/soundfx/pitchPlayer.ts | 21 +- src/core/store/serialize.ts | 4 + src/editor/Editor.ts | 9 + src/editor/assetTabs.ts | 1 + src/editor/debug/graphHooks.ts | 21 + src/editor/debugHook.ts | 2 + src/editor/graph/GraphCanvas.ts | 17 +- src/editor/graph/NodeView.ts | 2 + src/editor/graph/paletteEntries.ts | 12 +- src/editor/inspector/portSection.ts | 35 +- src/editor/instrument/InstrumentEditor.ts | 185 +++++++ src/editor/instrument/PianoKeyboard.ts | 185 +++++++ src/editor/instrument/keys.ts | 106 ++++ src/editor/panels/assetGroups.ts | 2 + src/editor/panels/assetIcons.ts | 2 + src/editor/panels/icons/instrument.svg | 7 + src/editor/state.ts | 5 + src/runtime/systems/Audio.ts | 23 +- src/runtime/systems/behaviorContext.ts | 33 ++ src/ui/Theme.ts | 1 + src/viewport/SoundLibrary.ts | 237 +++++++-- src/viewport/pitchWorklet.ts | 9 +- tests/core/graph.test.ts | 6 +- tests/core/instrument.test.ts | 485 ++++++++++++++++++ tests/core/keep-on-scene-load.test.ts | 2 + tests/core/node-catalog.test.ts | 43 ++ tests/core/pitch-player.test.ts | 25 +- tests/core/soundfx.test.ts | 2 + tests/core/sounds.test.ts | 2 + tests/helpers.ts | 13 + tests/viewport/sound-library.test.ts | 106 ++++ tools/e2e/04-assets.mjs | 2 +- tools/e2e/34-sound.mjs | 163 +++++- 54 files changed, 2561 insertions(+), 143 deletions(-) create mode 100644 src/core/graph/nodes/action/PlayNote.ts create mode 100644 src/core/graph/nodes/data/InstrumentParameterValues.ts create mode 100644 src/core/instrument/compile.ts create mode 100644 src/core/instrument/nodes.ts create mode 100644 src/core/instrument/notes.ts create mode 100644 src/editor/instrument/InstrumentEditor.ts create mode 100644 src/editor/instrument/PianoKeyboard.ts create mode 100644 src/editor/instrument/keys.ts create mode 100644 src/editor/panels/icons/instrument.svg create mode 100644 tests/core/instrument.test.ts diff --git a/src/core/commands/assetCommands.ts b/src/core/commands/assetCommands.ts index 8e9bcb5f..3925559e 100644 --- a/src/core/commands/assetCommands.ts +++ b/src/core/commands/assetCommands.ts @@ -1,5 +1,6 @@ import { createEmptyGraph } from '../graph/Graph'; import { newId } from '../ids'; +import { starterInstrumentGraph } from '../instrument/nodes'; import type { Asset, FontAsset, @@ -103,6 +104,8 @@ export function buildAsset(type: AssetType, name: string, id = newId()): Asset { return { ...base, type: 'sound', file: '' }; case 'soundfx': return { ...base, type: 'soundfx', graph: createEmptyGraph() }; + case 'instrument': + return { ...base, type: 'instrument', graph: starterInstrumentGraph() }; case 'particles': return { ...base, type: 'particles', graph: createEmptyGraph() }; } diff --git a/src/core/commands/graphCommands.ts b/src/core/commands/graphCommands.ts index dc94f8a8..f7eebfae 100644 --- a/src/core/commands/graphCommands.ts +++ b/src/core/commands/graphCommands.ts @@ -10,7 +10,7 @@ import { import type { GraphLookup } from '../graph/lookup'; import { isTerminalType, resolveDef, terminalNode } from '../graph/nested'; import { nextTransitionName, parallelTransitions } from '../graph/stateMachine'; -import { nodeDef } from '../graph/NodeRegistry'; +import { nodeDef, takesManyEdges } from '../graph/NodeRegistry'; import { graphLookup } from '../model/graphLookup'; import { newId } from '../ids'; import type { PropertyBag, PropertyValue } from '../model/Properties'; @@ -229,7 +229,7 @@ export class ConnectCommand implements Command { throw new CommandError('Already connected'); this.edge.label = nextTransitionName(graph, this.edge); } - const occupied = inp.type === 'exec' ? [] : edgesInto(graph, this.edge.to.node, this.edge.to.port); + const occupied = takesManyEdges(inp) ? [] : edgesInto(graph, this.edge.to.node, this.edge.to.port); this.replaced = removeWhere(graph.edges, (e) => occupied.includes(e)); graph.edges.push(this.edge); } diff --git a/src/core/commands/portCommands.ts b/src/core/commands/portCommands.ts index 88ccd411..32fd5e4b 100644 --- a/src/core/commands/portCommands.ts +++ b/src/core/commands/portCommands.ts @@ -1,5 +1,6 @@ import { BEHAVIOR_PARAM, + declaresParameters, declaresPropertyPorts, declaresSoundFxPorts, declaresVariablePorts, @@ -17,8 +18,8 @@ import { namesBehavior, OBJECT_BEHAVIOR_PARAM, propertyTargetOf, + INSTRUMENT_SOUND_FX, SHADER_CALL, - SOUND_FX_INPUT, soundFxSourceOf, variablesSourceOf, } from '../graph'; @@ -345,11 +346,14 @@ export class RenamePortCommand extends RenameCommand { if (ports.some((p) => p.key === to)) throw new CommandError(`"${to}" already exists`); node.ports = ports.map((p) => (p.key === from ? { ...p, key: to } : p)); for (const site of sites) carryPort(site, from, to); - if (node.type === SOUND_FX_INPUT) renameSoundFxParamPorts(project, this.assetId, from, to); + if (declaresParameters(node.type)) renameSoundFxParamPorts(project, this.assetId, from, to); } } -/** The default of a sound FX parameter (#311): what it is wherever no `Play sound` hands a value in. */ +/** + * The default of a sound FX parameter (#311) or an instrument's (#331): what it is wherever no + * `Play sound` or `Play note` hands a value in. + */ export class SetPortDefaultCommand implements Command { readonly label: string; private was: number | undefined; @@ -365,7 +369,8 @@ export class SetPortDefaultCommand implements Command { execute(project: Project): void { const node = requireNode(project, this.assetId, this.nodeId); - if (node.type !== SOUND_FX_INPUT) throw new CommandError('Only a sound FX Input has defaults'); + if (!declaresParameters(node.type)) + throw new CommandError('Only a sound FX Input or an instrument Parameters node has defaults'); const port = (node.ports ?? []).find((p) => p.key === this.key); if (!port) throw new CommandError(`Port "${this.key}" not found`); if (!Number.isFinite(this.value)) throw new CommandError('A default is a number'); @@ -386,13 +391,18 @@ export class SetPortDefaultCommand implements Command { } } -/** A renamed parameter of a sound FX, followed into every node that sets it. */ +/** + * A renamed parameter of a sound FX or an instrument, followed into every node that sets it – and + * into the Sound FX nodes of instruments (#331), whose inputs are the FX's parameters themselves. + */ function renameSoundFxParamPorts(project: Project, fxId: string, from: string, to: string): void { for (const asset of project.assets) { if (!isGraphAsset(asset)) continue; for (const node of asset.graph.nodes) if (declaresSoundFxPorts(node.type) && soundFxSourceOf(node.params) === fxId) renamePortOn(asset.graph, node, from, to); + else if (node.type === INSTRUMENT_SOUND_FX && node.params.fx === fxId) + carryPort({ graph: asset.graph, nodeId: node.id, asInput: true }, from, to); } } diff --git a/src/core/graph/Graph.ts b/src/core/graph/Graph.ts index 569980b6..df1926eb 100644 --- a/src/core/graph/Graph.ts +++ b/src/core/graph/Graph.ts @@ -7,7 +7,8 @@ export type { DeclaredPortKind } from '../properties/kinds'; * `exec` carries control flow, the others values; `any` accepts every value type. `vec3` is a * colour inside a shader graph, where the values are GLSL and no behavior node ever sees one; * `picture` is what a shader's Pass draws (#301), for the nodes that read a picture. `fxparams` is - * the values a `Play sound` hands to the parameters of its sound FX (#311). + * the values a `Play sound` hands to the parameters of its sound FX (#311). `trigger` is a note + * arriving at a source of an instrument (#331): on for the notes it stands for, off for all others. */ export type PortType = | 'exec' @@ -21,6 +22,7 @@ export type PortType = | 'picture' | 'audio' | 'fxparams' + | 'trigger' | 'any'; export interface PortRef { @@ -159,7 +161,7 @@ export function findNode(graph: Graph, nodeId: string): GraphNode | undefined { return graph.nodes.find((n) => n.id === nodeId); } -/** Edges arriving at one input port (data inputs allow at most one, exec inputs any number). */ +/** Edges arriving at one input port (data inputs allow at most one, exec and `many` inputs any number). */ export function edgesInto(graph: Graph, nodeId: string, port: string): GraphEdge[] { return graph.edges.filter((e) => e.to.node === nodeId && e.to.port === port); } @@ -195,6 +197,7 @@ export function portsCompatible(from: PortType, to: PortType): boolean { if (from === 'picture' || to === 'picture') return from === to; if (from === 'audio' || to === 'audio') return from === to; if (from === 'fxparams' || to === 'fxparams') return from === to; + if (from === 'trigger' || to === 'trigger') return from === to; if (from === 'number' && to === 'vec3') return true; return from === to || from === 'any' || to === 'any'; } diff --git a/src/core/graph/NodeContext.ts b/src/core/graph/NodeContext.ts index e3db4d48..0c68f407 100644 --- a/src/core/graph/NodeContext.ts +++ b/src/core/graph/NodeContext.ts @@ -250,6 +250,21 @@ export interface NodeContext { fx: string, fxParams: FxParams, ): number; + /** + * Plays one note of an instrument asset (#331): `note` as MIDI counts it (60 is C4), struck with + * `velocity` (0 to 1, how loud it is next to `volume`) and released after `duration` seconds, or + * held until it is stopped where that is 0. Owned like a `playSound` voice, `instrumentParams` are + * values for the instrument's parameters. Returns the number of the voice, 0 where nothing plays. + */ + playNote( + instrumentId: string, + note: number, + velocity: number, + volume: number, + duration: number, + detached: boolean, + instrumentParams: FxParams, + ): number; /** * How many seconds one run of that sound asset lasts, so a change can span it; 0 where it does not * exist or has not been loaded yet. diff --git a/src/core/graph/NodeRegistry.ts b/src/core/graph/NodeRegistry.ts index c81656f6..56d45c2a 100644 --- a/src/core/graph/NodeRegistry.ts +++ b/src/core/graph/NodeRegistry.ts @@ -27,6 +27,8 @@ export interface PortDef { paintType?: PortType; /** On a port writing a declared variable: what the written value is held to (#271). */ typing?: ValueTyping; + /** An input taking any number of edges, like an exec input: the layers of an instrument (#331). */ + many?: true; } /** @@ -197,6 +199,11 @@ export function isEventNode(def: NodeDef): boolean { return def.category === 'event'; } +/** Whether an input keeps the edges it has when another one is connected to it. */ +export function takesManyEdges(port: PortDef): boolean { + return port.type === 'exec' || port.many === true; +} + export function hasExecInput(def: NodeDef): boolean { return def.inputs.some((p) => p.type === 'exec'); } @@ -219,7 +226,8 @@ function isInlinePort(port: PortDef): boolean { port.type !== 'struct' && port.type !== 'picture' && port.type !== 'audio' && - port.type !== 'fxparams' + port.type !== 'fxparams' && + port.type !== 'trigger' ); } diff --git a/src/core/graph/dropTarget.ts b/src/core/graph/dropTarget.ts index 1d189dec..6b1ac934 100644 --- a/src/core/graph/dropTarget.ts +++ b/src/core/graph/dropTarget.ts @@ -1,7 +1,7 @@ import { edgesInto, findNode, type Graph, type PortType, portsCompatible } from './Graph'; import type { GraphLookup } from './lookup'; import { resolveDef } from './nested'; -import type { NodeDef } from './NodeRegistry'; +import { type NodeDef, takesManyEdges } from './NodeRegistry'; /** One end of an edge being dragged: a port on a node, seen from its own side. */ export interface PortSide { @@ -44,9 +44,9 @@ export function dropTargetPort( const fromType = portTypeOf(graph, from, lookup); if (!def || fromType === undefined) return null; const candidates = compatiblePorts(def, fromType, from.isOutput) - .map((p) => ({ side: { nodeId, port: p.key, isOutput: !from.isOutput }, type: p.type })) + .map((p) => ({ side: { nodeId, port: p.key, isOutput: !from.isOutput }, many: takesManyEdges(p) })) .filter((c) => !linked(graph, from, c.side)); - const free = candidates.find((c) => isFree(graph, c.side, c.type)); + const free = candidates.find((c) => isFree(graph, c.side, c.many)); return (free ?? candidates.at(-1))?.side ?? null; } @@ -73,9 +73,9 @@ function compatiblePorts(def: NodeDef, fromType: PortType, fromIsOutput: boolean ); } -/** Only a data input is exclusive – exec inputs and every output take any number of edges. */ -function isFree(graph: Graph, side: PortSide, type: PortType): boolean { - if (side.isOutput || type === 'exec') return true; +/** Only a data input is exclusive – exec and `many` inputs and every output take any number of edges. */ +function isFree(graph: Graph, side: PortSide, many: boolean): boolean { + if (side.isOutput || many) return true; return edgesInto(graph, side.nodeId, side.port).length === 0; } diff --git a/src/core/graph/emitGraph.ts b/src/core/graph/emitGraph.ts index efd90df5..d08d9ae6 100644 --- a/src/core/graph/emitGraph.ts +++ b/src/core/graph/emitGraph.ts @@ -1,4 +1,4 @@ -import { edgesInto, findNode, type Graph, type GraphNode } from './Graph'; +import { edgesInto, findNode, type Graph, type GraphEdge, type GraphNode } from './Graph'; import { nodeDef, type NodeDef, type PortDef } from './NodeRegistry'; export interface EmitRules { @@ -16,6 +16,13 @@ export interface EmitRules { * reporting a problem). Undefined for every other node. */ cut?: (def: D, node: GraphNode) => Record | null | undefined; + /** + * What an input taking any number of edges (`PortDef.many`) gets from the values of all of them, + * in edge order. Without it such an input reads its first edge like any other. + */ + gather?: (node: GraphNode, port: PortDef, values: T[]) => T | null; + /** The definition as it applies to this node, for a node whose ports are derived; else the registered one. */ + resolve?: (node: GraphNode, def: D) => D; } export interface GraphEmitter { @@ -56,11 +63,12 @@ export function graphEmitter(graph: Graph, rules: EmitRule problem(node, 'Feeds itself'); return null; } - const def = nodeDef(node.type); - if (!def || !rules.isDef(def)) { + const registered = nodeDef(node.type); + if (!registered || !rules.isDef(registered)) { problem(node, `Not a ${rules.noun} node`); return null; } + const def = rules.resolve ? rules.resolve(node, registered) : registered; const cut = rules.cut?.(def, node); if (cut !== undefined) { if (cut) emitted.set(node.id, cut); @@ -81,14 +89,21 @@ export function graphEmitter(graph: Graph, rules: EmitRule return outputs; }; + const arriving = (node: GraphNode, port: PortDef, edge: GraphEdge): T | null => { + const source = findNode(graph, edge.from.node); + const value = (source && emitNode(source))?.[edge.from.port]; + if (!value) return null; + return rules.conform ? rules.conform(node, port, value) : value; + }; + const input = (node: GraphNode, port: PortDef): T | null => { - const edge = edgesInto(graph, node.id, port.key)[0]; - if (edge) { - const source = findNode(graph, edge.from.node); - const value = (source && emitNode(source))?.[edge.from.port]; - if (!value) return null; - return rules.conform ? rules.conform(node, port, value) : value; + const edges = edgesInto(graph, node.id, port.key); + if (port.many && rules.gather && edges.length > 0) { + const values = edges.map((edge) => arriving(node, port, edge)); + return values.every((value): value is T => value !== null) ? rules.gather(node, port, values) : null; } + const edge = edges[0]; + if (edge) return arriving(node, port, edge); const def = nodeDef(node.type); return rules.unfed(node, port, def && rules.isDef(def) ? def : undefined); }; diff --git a/src/core/graph/lookup.ts b/src/core/graph/lookup.ts index 8a6d1639..b26713a4 100644 --- a/src/core/graph/lookup.ts +++ b/src/core/graph/lookup.ts @@ -17,7 +17,10 @@ export interface ShaderFunction { graph: Graph; } -/** A sound FX as a node naming it sees it: the parameters its Input declares (#311). */ +/** + * A sound FX as a node naming it sees it: the parameters its Input declares (#311). An instrument + * looks the same from outside, with the parameters of its Parameters node (#331). + */ export interface SoundFxShape { name: string; params: readonly DeclaredPort[]; diff --git a/src/core/graph/nested.ts b/src/core/graph/nested.ts index 3f54ac6a..2f16cfe3 100644 --- a/src/core/graph/nested.ts +++ b/src/core/graph/nested.ts @@ -33,6 +33,17 @@ export const SOUND_FX_INPUT = 'SoundFxInput'; export const SOUND_FX_OUTPUT = 'SoundFxOutput'; export const SOUND_FX_PARAMETERS = 'SoundFxParameters'; const PLAY_SOUND = 'PlaySound'; +/** + * The nodes of an instrument's graph that are its own (#331), and the behavior node naming its + * parameters. The effects it shares with a sound FX keep their types. + */ +export const INSTRUMENT_NOTES = 'InstrumentNotes'; +export const INSTRUMENT_SAMPLE = 'InstrumentSample'; +export const INSTRUMENT_SOUND_FX = 'InstrumentSoundFx'; +export const INSTRUMENT_PARAMETERS = 'InstrumentParameters'; +export const INSTRUMENT_OUTPUT = 'InstrumentOutput'; +export const INSTRUMENT_PARAMETER_VALUES = 'InstrumentParameterValues'; +const PLAY_NOTE = 'PlayNote'; export const BEHAVIOR_PARAM = 'behavior'; export const FUNCTION_PARAM = 'function'; @@ -48,6 +59,14 @@ export const SOUND_FX_SETTING: PropertyDef = { assetTypes: ['soundfx'], }; +/** Whose parameters an `Instrument parameters` node sets: the same key, another kind of asset. */ +export const INSTRUMENT_SETTING: PropertyDef = { + ...SOUND_FX_SETTING, + label: 'Instrument', + assetTypes: ['instrument'], +}; + +/** The sound FX – or the instrument (#331) – whose parameters a node names. */ export function soundFxSourceOf(params: PropertyBag): string { return textOf(params[SOUND_FX_PARAM] ?? ''); } @@ -90,9 +109,14 @@ export function declaresPropertyPorts(type: string): boolean { return type === PROPERTIES; } -/** Nodes whose declared ports are parameters of the sound FX they name. */ +/** Nodes whose declared ports are parameters of the sound FX or the instrument they name. */ export function declaresSoundFxPorts(type: string): boolean { - return type === SOUND_FX_PARAMETERS; + return type === SOUND_FX_PARAMETERS || type === INSTRUMENT_PARAMETER_VALUES; +} + +/** The node declaring the parameters of a sound FX or an instrument, each with a default. */ +export function declaresParameters(type: string): boolean { + return type === SOUND_FX_INPUT || type === INSTRUMENT_PARAMETERS; } /** Identifier-like, and not `exec`: that key belongs to the fixed flow port of every terminal node. */ @@ -153,7 +177,9 @@ export function isTerminalType(type: string): boolean { type === SHADER_FUNCTION_INPUT || type === SHADER_FUNCTION_OUTPUT || type === SOUND_FX_INPUT || - type === SOUND_FX_OUTPUT + type === SOUND_FX_OUTPUT || + type === INSTRUMENT_PARAMETERS || + type === INSTRUMENT_OUTPUT ); } @@ -381,7 +407,8 @@ export function ghostProblem(node: GraphNode, graph: Graph, def: NodeDef): strin export function callProblem(node: GraphNode, lookup: GraphLookup, ownerId = ''): string | null { if (node.type === ON_BEHAVIOR) return listenProblem(node, lookup, ownerId); if (node.type === SHADER_CALL) return functionCallProblem(node, lookup, ownerId); - if (node.type === SOUND_FX_PARAMETERS) return soundFxParametersProblem(node, lookup); + if (declaresSoundFxPorts(node.type)) return soundFxParametersProblem(node, lookup); + if (node.type === INSTRUMENT_SOUND_FX) return nestedFxProblem(node, lookup); if (node.type !== RUN_BEHAVIOR) return easingProblem(node, lookup); const id = textOf(node.params[BEHAVIOR_PARAM] ?? ''); if (!id) return 'No behavior picked'; @@ -427,7 +454,15 @@ function functionCallProblem(node: GraphNode, lookup: GraphLookup, ownerId: stri } function soundFxParametersProblem(node: GraphNode, lookup: GraphLookup): string | null { + const noun = node.type === INSTRUMENT_PARAMETER_VALUES ? 'instrument' : 'sound FX'; const id = soundFxSourceOf(node.params); + if (!id) return `No ${noun} picked`; + return lookup.soundFx(id) ? null : `That ${noun} is gone`; +} + +/** A Sound FX node in an instrument (#331) without its FX lets the sound through untouched. */ +function nestedFxProblem(node: GraphNode, lookup: GraphLookup): string | null { + const id = textOf(node.params.fx ?? ''); if (!id) return 'No sound FX picked'; return lookup.soundFx(id) ? null : 'That sound FX is gone'; } @@ -438,13 +473,17 @@ function soundFxParametersProblem(node: GraphNode, lookup: GraphLookup): string * are picked in place – one coming over an edge may be either. */ export function soundFxParamsMismatch(node: GraphNode, graph: Graph): string | null { - if (node.type !== PLAY_SOUND || edgesInto(graph, node.id, 'fx').length > 0) return null; - const played = textOf(node.params.fx ?? ''); + const playsNote = node.type === PLAY_NOTE; + if (node.type !== PLAY_SOUND && !playsNote) return null; + const pick = playsNote ? 'instrument' : 'fx'; + if (edgesInto(graph, node.id, pick).length > 0) return null; + const played = textOf(node.params[pick] ?? ''); const edge = edgesInto(graph, node.id, 'fxParams')[0]; const from = edge && findNode(graph, edge.from.node); - if (!played || from?.type !== SOUND_FX_PARAMETERS) return null; + if (!played || !from || !declaresSoundFxPorts(from.type)) return null; const named = soundFxSourceOf(from.params); - return named && named !== played ? 'The parameters are for another sound FX' : null; + if (!named || named === played) return null; + return playsNote ? 'The parameters are for another instrument' : 'The parameters are for another sound FX'; } /** The ids the nodes of one type in that graph name in `param`, picked or not found alike. */ diff --git a/src/core/graph/nodes/action/PlayNote.ts b/src/core/graph/nodes/action/PlayNote.ts new file mode 100644 index 00000000..208049b8 --- /dev/null +++ b/src/core/graph/nodes/action/PlayNote.ts @@ -0,0 +1,63 @@ +import { MIDDLE_C } from '../../../instrument/notes'; +import type { NodeDef } from '../../NodeRegistry'; +import { asBoolean, asNumber, asString } from '../../values'; +import { exec, num, withDefault } from '../shared'; +import { fxParamsOf } from './PlaySound'; + +export const PlayNote: NodeDef = { + type: 'PlayNote', + label: 'Play note', + category: 'action', + description: + 'Plays one note of an instrument asset. "note" is its number as MIDI counts it: 60 is C4, 61 C#4, 72 C5. "velocity" is how hard it is struck, 0 to 1 – it makes the note quieter and is handed to what the instrument does with it. "duration" is how many seconds until the note is released; at 0 it is held until a Stop sound ends it. A note belongs to this object like a sound and stops when the object is destroyed, unless "detached" is on. "params" takes values for the parameters of the instrument from an Instrument parameters node. "voice" is the number of this note, for Set sound FX parameters and Stop sound; 0 where the instrument plays nothing for it.', + inputs: [ + exec(), + { + key: 'instrument', + label: 'instrument', + type: 'string', + field: { + key: 'instrument', + label: 'Instrument', + kind: 'asset', + default: '', + assetTypes: ['instrument'], + }, + }, + { + key: 'note', + label: 'note', + type: 'number', + default: MIDDLE_C, + field: { key: 'note', label: 'Note', kind: 'number', default: MIDDLE_C, min: 0, max: 127, step: 1 }, + }, + withDefault(num('velocity'), 1), + withDefault(num('volume'), 1), + withDefault(num('duration'), 0), + { + key: 'detached', + label: 'detached', + type: 'boolean', + default: false, + field: { key: 'detached', label: 'Detached (outlives this object)', kind: 'boolean', default: false }, + }, + { key: 'fxParams', label: 'params', type: 'fxparams' }, + ], + outputs: [exec(), num('voice')], + params: [], + run: (ctx, inputs) => { + const instrument = asString(inputs.instrument); + const voice = instrument + ? ctx.playNote( + instrument, + asNumber(inputs.note), + asNumber(inputs.velocity), + asNumber(inputs.volume), + asNumber(inputs.duration), + asBoolean(inputs.detached), + fxParamsOf(inputs.fxParams), + ) + : 0; + return { next: 'exec', outputs: { voice } }; + }, +}; diff --git a/src/core/graph/nodes/data/InstrumentParameterValues.ts b/src/core/graph/nodes/data/InstrumentParameterValues.ts new file mode 100644 index 00000000..c46a87d0 --- /dev/null +++ b/src/core/graph/nodes/data/InstrumentParameterValues.ts @@ -0,0 +1,37 @@ +import type { StructValue } from '../../../model/Properties'; +import { INSTRUMENT_PARAMETER_VALUES, INSTRUMENT_SETTING, soundFxSourceOf } from '../../nested'; +import type { NodeDef } from '../../NodeRegistry'; +import { asNumber } from '../../values'; + +/** + * Values for the parameters of one instrument (#331), picked like those of a sound FX: each one picked + * is a number input starting at the instrument's default. The bundle goes into the `params` input of + * a `Play note`, or the `fx params` of a `Set sound FX parameters` that changes a playing note. + */ +export const InstrumentParameterValues: NodeDef = { + type: INSTRUMENT_PARAMETER_VALUES, + label: 'Instrument parameters', + category: 'data', + description: + 'Values for the parameters of an instrument, for the "params" input of a Play note or the "fx params" of a Set sound FX parameters. Pick the parameters to set; one not picked keeps the default its Parameters node gives it, or on a playing note the value it has.', + inputs: [], + outputs: [{ key: 'params', label: 'params', type: 'fxparams' }], + params: [INSTRUMENT_SETTING], + declaresPorts: 'inputs', + portKinds: ['number'], + declaredPort: (port, node, _graph, lookup) => { + const declared = lookup.soundFx(soundFxSourceOf(node.params))?.params.find((p) => p.key === port.key); + const value = declared?.default ?? 0; + return { + key: port.key, + label: port.key, + type: 'number', + default: value, + field: { key: port.key, label: port.key, kind: 'number', default: value }, + }; + }, + run: (_ctx, inputs, _params, def) => { + const values: StructValue = Object.fromEntries(def.inputs.map((p) => [p.key, asNumber(inputs[p.key])])); + return { outputs: { params: values } }; + }, +}; diff --git a/src/core/graph/nodes/index.ts b/src/core/graph/nodes/index.ts index 217ff5ab..f13af91a 100644 --- a/src/core/graph/nodes/index.ts +++ b/src/core/graph/nodes/index.ts @@ -24,6 +24,7 @@ import { RaycastAll } from './action/RaycastAll'; import { RestartGame } from './action/RestartGame'; import { SetPosition } from './action/SetPosition'; import { SetSoundFxParameters } from './action/SetSoundFxParameters'; +import { PlayNote } from './action/PlayNote'; import { SetZ } from './action/SetZ'; import { Spawn } from './action/Spawn'; import { StopGame } from './action/StopGame'; @@ -91,6 +92,7 @@ import { Owner } from './data/Owner'; import { PackStruct } from './data/PackStruct'; import { Properties } from './data/Properties'; import { SoundFxParameters } from './data/SoundFxParameters'; +import { InstrumentParameterValues } from './data/InstrumentParameterValues'; import { Parent } from './data/Parent'; import { Random } from './data/Random'; import { RenderLayerOf } from './data/RenderLayerOf'; @@ -150,6 +152,7 @@ import { PARTICLE_NODES } from '../../particles/nodes'; import { SHADER_FUNCTION_NODES } from '../../shader/functions'; import { SHADER_NODES } from '../../shader/nodes'; import { SOUND_FX_NODES } from '../../soundfx/nodes'; +import { INSTRUMENT_NODES } from '../../instrument/nodes'; import { State } from '../stateMachine'; import { Variables } from './variable/Variables'; @@ -303,6 +306,8 @@ export const BUILT_IN_NODES: readonly NodeDef[] = [ StopSound, SetSoundFxParameters, SoundFxParameters, + PlayNote, + InstrumentParameterValues, LoadScene, Log, StopGame, @@ -322,3 +327,4 @@ for (const def of SHADER_NODES) registerNode(def); for (const def of SHADER_FUNCTION_NODES) registerNode(def); for (const def of PARTICLE_NODES) registerNode(def); for (const def of SOUND_FX_NODES) registerNode(def); +for (const def of INSTRUMENT_NODES) registerNode(def); diff --git a/src/core/instrument/compile.ts b/src/core/instrument/compile.ts new file mode 100644 index 00000000..40275f03 --- /dev/null +++ b/src/core/instrument/compile.ts @@ -0,0 +1,320 @@ +import { graphEmitter } from '../graph/emitGraph'; +import { type DeclaredPort, type Graph, graphFingerprint, type GraphNode } from '../graph/Graph'; +import type { GraphLookup } from '../graph/lookup'; +import { resolveDef, terminalNode } from '../graph/nested'; +import { inlineValue, type NodeDef } from '../graph/NodeRegistry'; +import { asNumber } from '../graph/values'; +import type { InstrumentAsset } from '../model/Asset'; +import { graphLookup } from '../model/graphLookup'; +import { textOf } from '../model/Properties'; +import { findAssetOf, type Project } from '../model/Project'; +import { resolveAssetProps } from '../properties/resolve'; +import { volumeOf } from '../properties/sounds'; +import { soundFxBuildFor } from '../soundfx/compile'; +import { + type FxAmount, + type FxPlan, + type FxStagePlan, + isSoundFxNodeDef, + SOUND_FX_INPUT, + SOUND_FX_OUTPUT, + SoundFxMix, + type SoundFxNodeDef, + withPitch, + withStages, +} from '../soundfx/nodes'; +import { + INSTRUMENT_NOTES, + INSTRUMENT_OUTPUT, + INSTRUMENT_PARAMETERS, + INSTRUMENT_SAMPLE, + INSTRUMENT_SOUND_FX, + isInstrumentNodeDef, + noteOutputKey, + noteParam, + notesRange, + singleNotes, + VELOCITY_PARAM, +} from './nodes'; +import { MIDDLE_C, noteFrequency } from './notes'; + +/** + * What an instrument's graph reaches outside itself: the sounds its Samples play and the sound FX + * its Sound FX nodes send them through. `key` changes whenever one of them does, so a cached note is + * compiled again when the sound or the FX it was made of is edited. + */ +export interface InstrumentSources { + lookup: GraphLookup; + sound(assetId: string): { file: string; volume: number } | null; + soundFx(assetId: string): FxPlan | null; + key(assetId: string): string; +} + +export function instrumentSourcesOf(project: Project): InstrumentSources { + return { + lookup: graphLookup(project), + sound: (assetId) => { + const asset = findAssetOf(project, assetId, 'sound'); + return asset ? { file: asset.file, volume: volumeOf(resolveAssetProps(project, asset)) } : null; + }, + soundFx: (assetId) => { + const asset = findAssetOf(project, assetId, 'soundfx'); + return asset ? soundFxBuildFor(asset).compiled : null; + }, + key: (assetId) => { + const sound = findAssetOf(project, assetId, 'sound'); + if (sound) return `${sound.file}@${volumeOf(resolveAssetProps(project, sound))}`; + const fx = findAssetOf(project, assetId, 'soundfx'); + return fx ? graphFingerprint(fx.graph) : ''; + }, + }; +} + +export interface InstrumentBuild { + /** What the note is made of, each played on its own and summed; empty is silence. */ + layers: readonly FxPlan[]; + /** Per node id, what keeps the graph from compiling. */ + problems: Record; + /** One line for the tab: empty once compiled. */ + reason: string; +} + +/** + * What travels along an edge of an instrument: sounds (none where nothing plays for this note, more + * than one only into the Output), a number, or whether a trigger is on for this note. + */ +type Value = { plans: FxPlan[] } | { amount: FxAmount } | { on: boolean }; + +const SILENT: Value = { plans: [] }; + +type Def = SoundFxNodeDef | (NodeDef & { instrument: true }); + +function isInstrumentDef(def: NodeDef): def is Def { + if (isInstrumentNodeDef(def)) return true; + return isSoundFxNodeDef(def) && def.type !== SOUND_FX_INPUT && def.type !== SOUND_FX_OUTPUT; +} + +/** The parameters the Parameters node declares, in their order, each with its default. */ +export function instrumentParams(graph: Graph): readonly DeclaredPort[] { + return terminalNode(graph, INSTRUMENT_PARAMETERS)?.ports ?? []; +} + +/** + * One note of an instrument (#331): the graph walked back from the Output, with the triggers that + * stand for this note on and every other one off. A source whose trigger is off is silence, and so + * is everything it alone feeds; a Mix with one silent side is its other side. `note` and `frequency` + * are numbers here, `velocity` and the parameters stay open for every voice to fill in. + */ +export function compileInstrument(graph: Graph, note: number, sources: InstrumentSources): InstrumentBuild { + const walk = graphEmitter(graph, { + isDef: isInstrumentDef, + noun: 'instrument', + resolve: (node, def) => + def.derivePorts ? ((resolveDef(node, graph, sources.lookup) as Def | undefined) ?? def) : def, + emit: (def, inputs, node) => emitNode(def, inputs, node), + unfed: (node, port) => { + if (port.type === 'number') return { amount: { value: asNumber(inlineValue(node, port)) } }; + if (port.type === 'audio' && port.many) return SILENT; + walk.problem( + node, + port.type === 'trigger' + ? 'Nothing triggers it: connect it to a Notes node' + : 'Nothing comes in: connect it to a Sample', + ); + return null; + }, + conform: (node, port, value) => { + const fits = + port.type === 'audio' + ? 'plans' in value + : port.type === 'trigger' + ? 'on' in value + : 'amount' in value; + if (fits) return value; + walk.problem( + node, + `"${port.label}" takes ${port.type === 'audio' ? 'a sound' : port.type === 'trigger' ? 'a trigger' : 'a number'}`, + ); + return null; + }, + gather: (_node, port, values) => + port.type === 'trigger' + ? { on: values.some((v) => 'on' in v && v.on) } + : { plans: values.flatMap((v) => ('plans' in v ? v.plans : [])) }, + }); + + function emitNode(def: Def, inputs: Record, node: GraphNode): Record { + const plansAt = (key: string) => { + const value = inputs[key]; + return value && 'plans' in value ? value.plans : []; + }; + const amounts: Record = {}; + for (const [key, value] of Object.entries(inputs)) if ('amount' in value) amounts[key] = value.amount; + switch (node.type) { + case INSTRUMENT_NOTES: + return notesOutputs(node, note); + case INSTRUMENT_PARAMETERS: + return Object.fromEntries( + (node.ports ?? []).map((port) => [ + port.key, + { amount: { value: port.default ?? 0, param: port.key } }, + ]), + ); + case INSTRUMENT_SAMPLE: { + const plan = sampleOf(node, note); + const trigger = inputs.trigger; + return { out: plan && trigger && 'on' in trigger && trigger.on ? { plans: [plan] } : SILENT }; + } + case INSTRUMENT_SOUND_FX: { + const fx = sources.soundFx(textOf(node.params.fx ?? '')); + const arriving = plansAt('in'); + return { out: { plans: fx ? arriving.map((plan) => throughFx(fx, plan, amounts)) : arriving } }; + } + case INSTRUMENT_OUTPUT: + return { out: { plans: plansAt('in') } }; + } + if (!isSoundFxNodeDef(def)) return {}; + if (def.type === SoundFxMix.type) { + const [a] = plansAt('a'); + const [b] = plansAt('b'); + if (!a || !b) return { out: { plans: a ? [a] : b ? [b] : [] } }; + return { out: { plans: [def.emit({ a, b }, amounts, node.params)] } }; + } + const [arriving] = plansAt('in'); + return { out: arriving ? { plans: [def.emit({ in: arriving }, amounts, node.params)] } : SILENT }; + } + + /** A Sample's sound, or null after saying why it has none – also where its trigger is off. */ + function sampleOf(node: GraphNode, played: number): FxPlan | null { + const id = textOf(node.params.sound ?? ''); + const sound = id ? sources.sound(id) : null; + if (!id) walk.problem(node, 'No sound picked'); + else if (!sound) walk.problem(node, 'That sound is gone'); + else if (!sound.file) walk.problem(node, 'The sound has no file'); + if (!sound?.file) return null; + const follows = node.params.followNote !== false; + const shift = follows ? played - noteParam(node.params, 'root', MIDDLE_C) : 0; + const loop = node.params.loop === true; + return { + pitch: shift === 0 ? [] : [{ value: shift }], + stages: [], + source: { + file: sound.file, + volume: sound.volume, + loop, + loopStart: loop ? Math.max(0, asNumber(node.params.loopStart ?? 0)) : 0, + loopEnd: loop ? Math.max(0, asNumber(node.params.loopEnd ?? 0)) : 0, + gate: textOf(node.params.mode ?? 'gate') !== 'oneshot', + }, + }; + } + + const output = graph.nodes.find((n) => n.type === INSTRUMENT_OUTPUT); + if (!output) return { layers: [], problems: walk.problems, reason: 'No Output node' }; + const arrived = walk.input(output, { key: 'in', label: 'in', type: 'audio', many: true }); + const failed = walk.firstProblem(); + if (failed !== null || !arrived || !('plans' in arrived)) + return { layers: [], problems: walk.problems, reason: failed ?? 'Cannot compile' }; + return { layers: arrived.plans, problems: walk.problems, reason: '' }; +} + +/** The outputs of a Notes node for one note: its triggers, and the note as numbers. */ +function notesOutputs(node: GraphNode, note: number): Record { + const { from, to } = notesRange(node.params); + const inside = note >= from && note <= to; + const outputs: Record = { + any: { on: inside }, + note: { amount: { value: note } }, + frequency: { amount: { value: noteFrequency(note) } }, + velocity: { amount: { value: 1, param: VELOCITY_PARAM, min: 0, max: 1 } }, + }; + if (singleNotes(node.params)) + for (let at = from; at <= to; at++) outputs[noteOutputKey(at)] = { on: at === note }; + return outputs; +} + +/** + * A sound sent through a sound FX: the plan of the FX with the sound set in where its Input was, and + * each of its parameters bound to what arrives at the input of that name – held to the FX's row, as + * a value handed to the FX itself would be. One left open keeps its default. + */ +function throughFx(fx: FxPlan, sound: FxPlan, given: Readonly>): FxPlan { + const bind = (amount: FxAmount): FxAmount => { + if (amount.param === undefined) return amount; + const fed = given[amount.param]; + if (!fed) return amount; + const value = Math.max(amount.min ?? -Infinity, Math.min(amount.max ?? Infinity, fed.value)); + return fed.param === undefined + ? { value } + : { value, param: fed.param, min: amount.min, max: amount.max }; + }; + const stages = fx.stages.map((stage) => bindStage(stage, bind)); + if ('mix' in fx) + return { + mix: bind(fx.mix), + a: throughFx(fx.a, sound, given), + b: throughFx(fx.b, sound, given), + stages, + }; + const pitched = fx.pitch.reduce((plan, semitones) => withPitch(plan, bind(semitones)), sound); + return withStages(pitched, stages); +} + +function bindStage(stage: FxStagePlan, bind: (amount: FxAmount) => FxAmount): FxStagePlan { + if (stage.kind === 'gain') return { ...stage, db: bind(stage.db) }; + if (stage.kind === 'filter') return { ...stage, gain: bind(stage.gain) }; + return { + ...stage, + time: bind(stage.time), + feedback: bind(stage.feedback), + damping: bind(stage.damping), + mix: bind(stage.mix), + }; +} + +const builds = new Map }>(); + +/** + * One note of the asset, compiled again only when what it is made of has changed: the graph, or a + * sound or sound FX it names. A note is played far more often than its instrument is edited. + */ +export function instrumentBuildFor( + asset: InstrumentAsset, + note: number, + sources: InstrumentSources, +): InstrumentBuild { + const named = asset.graph.nodes + .map((node) => textOf(node.params[node.type === INSTRUMENT_SOUND_FX ? 'fx' : 'sound'] ?? '')) + .filter((id) => id !== ''); + const fingerprint = `${graphFingerprint(asset.graph)}|${named.map((id) => sources.key(id)).join('|')}`; + let cached = builds.get(asset.id); + if (!cached || cached.fingerprint !== fingerprint) { + cached = { fingerprint, notes: new Map() }; + builds.set(asset.id, cached); + } + let build = cached.notes.get(note); + if (!build) { + build = compileInstrument(asset.graph, note, sources); + cached.notes.set(note, build); + } + return build; +} + +/** + * The instrument's problems, the same for every note: the walk from the Output reaches every node + * whatever is played, only the triggers differ. + */ +export function instrumentCheck(asset: InstrumentAsset, sources: InstrumentSources): InstrumentBuild { + return instrumentBuildFor(asset, MIDDLE_C, sources); +} + +/** Which notes of the keyboard make a sound at all: those inside a Notes node's range. */ +export function playedNotes(graph: Graph): Set { + const notes = new Set(); + for (const node of graph.nodes) { + if (node.type !== INSTRUMENT_NOTES) continue; + const { from, to } = notesRange(node.params); + for (let note = from; note <= to; note++) notes.add(note); + } + return notes; +} diff --git a/src/core/instrument/nodes.ts b/src/core/instrument/nodes.ts new file mode 100644 index 00000000..ac730025 --- /dev/null +++ b/src/core/instrument/nodes.ts @@ -0,0 +1,246 @@ +import type { DeclaredPort, Graph } from '../graph/Graph'; +import { newId } from '../ids'; +import { + INSTRUMENT_NOTES, + INSTRUMENT_OUTPUT, + INSTRUMENT_PARAMETERS, + INSTRUMENT_SAMPLE, + INSTRUMENT_SOUND_FX, +} from '../graph/nested'; +import type { NodeDef, PortDef } from '../graph/NodeRegistry'; +import { type PropertyBag, textOf } from '../model/Properties'; +import type { PropertyDef } from '../properties/PropertySchema'; +import { soundFxParamPort } from '../soundfx/nodes'; +import { HIGHEST_NOTE, heldNote, LOWEST_NOTE, MIDDLE_C, NOTE_OPTIONS, noteName } from './notes'; + +export { INSTRUMENT_NOTES, INSTRUMENT_OUTPUT, INSTRUMENT_PARAMETERS, INSTRUMENT_SAMPLE, INSTRUMENT_SOUND_FX }; + +/** + * A node only an instrument's graph has (#331). Like a sound FX node it never runs: the instrument's + * compiler (`compile.ts`) reads what it stands for, once per note. + */ +export interface InstrumentNodeDef extends NodeDef { + category: 'sound'; + instrument: true; +} + +export function isInstrumentNodeDef(def: NodeDef): def is InstrumentNodeDef { + return def.category === 'sound' && 'instrument' in def; +} + +/** + * The value of an instrument's `velocity` while its voice is open: not a name a parameter can have, + * so it never meets one, and a `Set sound FX parameters` cannot reach it by accident. + */ +export const VELOCITY_PARAM = '@velocity'; + +const triggerIn: PortDef = { key: 'trigger', label: 'trigger', type: 'trigger', many: true }; +const audioIn: PortDef = { key: 'in', label: 'in', type: 'audio' }; +const audioOut: PortDef = { key: 'out', label: 'out', type: 'audio' }; + +function noteRow(key: string, label: string, note: number): PropertyDef { + return { key, label, kind: 'enum', default: String(note), options: NOTE_OPTIONS }; +} + +/** A note setting as a number, held to the keyboard. */ +export function noteParam(params: PropertyBag, key: string, fallback: number): number { + const text = textOf(params[key] ?? ''); + return text === '' ? fallback : heldNote(Number(text)); +} + +const FROM = noteRow('from', 'From', MIDDLE_C); +const TO = noteRow('to', 'To', MIDDLE_C + 11); +const SINGLE = { key: 'singleNotes', label: 'Single notes', kind: 'boolean' as const, default: false }; + +/** Where a note's own name is kept: `name60` for C4. */ +export const noteNameKey = (note: number) => `name${note}`; +/** The output a single note leaves through: `n60` for C4, whatever it is called. */ +export const noteOutputKey = (note: number) => `n${note}`; + +const NAME_ROWS: PropertyDef[] = Array.from({ length: HIGHEST_NOTE - LOWEST_NOTE + 1 }, (_, i) => ({ + key: noteNameKey(LOWEST_NOTE + i), + label: `Name of ${noteName(LOWEST_NOTE + i)}`, + kind: 'text', + default: '', +})); +const NAME_KEYS = NAME_ROWS.map((row) => row.key); + +/** The lowest and the highest note of a Notes node, whichever way round they were picked. */ +export function notesRange(params: PropertyBag): { from: number; to: number } { + const a = noteParam(params, FROM.key, MIDDLE_C); + const b = noteParam(params, TO.key, MIDDLE_C + 11); + return { from: Math.min(a, b), to: Math.max(a, b) }; +} + +export function singleNotes(params: PropertyBag): boolean { + return params[SINGLE.key] === true; +} + +/** What a note is called on its output and in a piano roll: its own name, else its note name. */ +export function noteLabel(params: PropertyBag, note: number): string { + return textOf(params[noteNameKey(note)] ?? '').trim() || noteName(note); +} + +export const InstrumentNotes: InstrumentNodeDef = { + type: INSTRUMENT_NOTES, + label: 'Notes', + category: 'sound', + instrument: true, + description: + 'Where the notes come in. "any" triggers what it is connected to for every note from "from" to "to"; "note" is the number of the note played (60 is C4), "frequency" its pitch in Hz and "velocity" how hard it is struck, 0 to 1. With "single notes" there is a trigger output for each note in the range as well, for a drum kit, each with a name of its own if you give it one. Several Notes nodes split the keyboard into zones; where two overlap, both play.', + inputs: [], + outputs: [ + { key: 'any', label: 'any', type: 'trigger' }, + { key: 'note', label: 'note', type: 'number' }, + { key: 'frequency', label: 'frequency', type: 'number' }, + { key: 'velocity', label: 'velocity', type: 'number' }, + ], + params: [FROM, TO, SINGLE, ...NAME_ROWS], + // The node shows the names on its outputs already. + hiddenParams: () => NAME_KEYS, + idleParams: (params) => { + if (!singleNotes(params)) return NAME_KEYS; + const { from, to } = notesRange(params); + return NAME_KEYS.filter((_, i) => LOWEST_NOTE + i < from || LOWEST_NOTE + i > to); + }, + derivePorts: ({ node }) => { + if (!singleNotes(node.params)) return {}; + const { from, to } = notesRange(node.params); + const outputs: PortDef[] = []; + for (let note = from; note <= to; note++) + outputs.push({ key: noteOutputKey(note), label: noteLabel(node.params, note), type: 'trigger' }); + return { outputs }; + }, +}; + +const SOUND: PropertyDef = { key: 'sound', label: 'Sound', kind: 'sound', default: '' }; +const ROOT = noteRow('root', 'Root', MIDDLE_C); +const FOLLOW = { key: 'followNote', label: 'Follow note', kind: 'boolean' as const, default: true }; +export const SAMPLE_MODES = [ + { value: 'gate', label: 'Gate' }, + { value: 'oneshot', label: 'One-shot' }, +]; +const MODE: PropertyDef = { + key: 'mode', + label: 'Mode', + kind: 'enum', + default: 'gate', + options: SAMPLE_MODES, +}; +const LOOP = { key: 'loop', label: 'Loop', kind: 'boolean' as const, default: false }; +const LOOP_START: PropertyDef = { + key: 'loopStart', + label: 'Loop start (s)', + kind: 'number', + default: 0, + min: 0, + step: 0.01, +}; +const LOOP_END: PropertyDef = { + key: 'loopEnd', + label: 'Loop end (s)', + kind: 'number', + default: 0, + min: 0, + step: 0.01, +}; + +export const InstrumentSample: InstrumentNodeDef = { + type: INSTRUMENT_SAMPLE, + label: 'Sample', + category: 'sound', + instrument: true, + description: + 'Plays a sound asset when a note triggers it. With "follow note" it is played higher or lower by the distance from "root" – the note it sounds as it is – and keeps its length. Off, every note plays it as it is, as for a drum. "Gate" ends it with the note, faded out in a few milliseconds; "One-shot" plays it through whatever the note does. "Loop" repeats it, between "loop start" and "loop end" in seconds where they are set (0 is the end of the sound), until the note ends – or, as a one-shot, until it is stopped. The volume of the sound asset counts.', + inputs: [triggerIn], + outputs: [audioOut], + params: [SOUND, ROOT, FOLLOW, MODE, LOOP, LOOP_START, LOOP_END], + idleParams: (params) => [ + ...(params[FOLLOW.key] === false ? [ROOT.key] : []), + ...(params[LOOP.key] === true ? [] : [LOOP_START.key, LOOP_END.key]), + ], +}; + +export const InstrumentParameters: InstrumentNodeDef = { + type: INSTRUMENT_PARAMETERS, + label: 'Parameters', + category: 'sound', + instrument: true, + description: + 'The parameters of the instrument: every one declared here is a number output with a default. A Play note can hand in other values through an Instrument parameters node, and a Set sound FX parameters can change them on a note while it plays.', + inputs: [], + outputs: [], + params: [], + declaresPorts: 'outputs', + portKinds: ['number'], + declaredPort: (port: DeclaredPort) => soundFxParamPort(port), +}; + +const FX: PropertyDef = { key: 'fx', label: 'Sound FX', kind: 'asset', default: '', assetTypes: ['soundfx'] }; + +export const InstrumentSoundFx: InstrumentNodeDef = { + type: INSTRUMENT_SOUND_FX, + label: 'Sound FX', + category: 'sound', + instrument: true, + description: + "Sends the sound through a sound FX asset, as if it were the sound at the FX's Input. Each parameter of the FX is an input here; one left open keeps the FX's default. A Mix in the FX plays the sound on both of its ways.", + inputs: [audioIn], + outputs: [audioOut], + params: [FX], + derivePorts: ({ node, lookup }) => { + const fx = lookup.soundFx(textOf(node.params[FX.key] ?? '')); + if (!fx) return {}; + return { + label: fx.name, + inputs: fx.params.map((port) => ({ + key: port.key, + label: port.key, + type: 'number' as const, + default: port.default ?? 0, + field: { key: port.key, label: port.key, kind: 'number' as const, default: port.default ?? 0 }, + })), + }; + }, +}; + +export const InstrumentOutput: InstrumentNodeDef = { + type: INSTRUMENT_OUTPUT, + label: 'Output', + category: 'sound', + instrument: true, + description: + 'What is heard of a note. It takes any number of sounds and plays them together – the layers of the instrument.', + inputs: [{ ...audioIn, many: true }], + outputs: [], + params: [], +}; + +export const INSTRUMENT_NODES: readonly InstrumentNodeDef[] = [ + InstrumentNotes, + InstrumentSample, + InstrumentParameters, + InstrumentSoundFx, + InstrumentOutput, +]; + +/** + * What a new instrument starts as: every note of an octave from C4 plays one Sample, which waits for + * its sound to be picked. + */ +export function starterInstrumentGraph(): Graph { + const notes = newId(); + const sample = newId(); + const output = newId(); + return { + nodes: [ + { id: notes, type: INSTRUMENT_NOTES, x: 0, y: 0, params: {} }, + { id: sample, type: INSTRUMENT_SAMPLE, x: 220, y: 0, params: {} }, + { id: output, type: INSTRUMENT_OUTPUT, x: 460, y: 0, params: {} }, + ], + edges: [ + { id: newId(), from: { node: notes, port: 'any' }, to: { node: sample, port: 'trigger' } }, + { id: newId(), from: { node: sample, port: 'out' }, to: { node: output, port: 'in' } }, + ], + }; +} diff --git a/src/core/instrument/notes.ts b/src/core/instrument/notes.ts new file mode 100644 index 00000000..b777cd28 --- /dev/null +++ b/src/core/instrument/notes.ts @@ -0,0 +1,37 @@ +/** + * Notes as an instrument counts them (#331): by number as MIDI does, 0 the lowest, 60 the middle C + * and 69 the A at 440 Hz. A name has the octave after it without a dash (`C4`, `C#4`), sharps only, + * so the lowest is `C-1`. + */ +export const LOWEST_NOTE = 0; +export const HIGHEST_NOTE = 127; +export const MIDDLE_C = 60; + +const NAMES = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B']; + +export function noteName(note: number): string { + const n = Math.round(note); + return `${NAMES[((n % 12) + 12) % 12]}${Math.floor(n / 12) - 1}`; +} + +/** The frequency of a note in Hz, a fractional one between its neighbours. */ +export function noteFrequency(note: number): number { + return 440 * 2 ** ((note - 69) / 12); +} + +/** A note as a whole number in range: what a pattern, a setting or a `Play note` hands in. */ +export function heldNote(value: number): number { + if (!Number.isFinite(value)) return MIDDLE_C; + return Math.max(LOWEST_NOTE, Math.min(HIGHEST_NOTE, Math.round(value))); +} + +/** Every note, for a setting that picks one by name. */ +export const NOTE_OPTIONS: { value: string; label: string }[] = Array.from( + { length: HIGHEST_NOTE - LOWEST_NOTE + 1 }, + (_, i) => ({ value: String(LOWEST_NOTE + i), label: noteName(LOWEST_NOTE + i) }), +); + +/** Whether a note is a black key. */ +export function isSharp(note: number): boolean { + return NAMES[((Math.round(note) % 12) + 12) % 12]!.length > 1; +} diff --git a/src/core/model/Asset.ts b/src/core/model/Asset.ts index fa36f344..2d4216e5 100644 --- a/src/core/model/Asset.ts +++ b/src/core/model/Asset.ts @@ -319,6 +319,15 @@ export interface SoundFxAsset extends AssetBase { graph: Graph; } +/** + * What makes a note a sound (#331): the graph runs from its `Notes` nodes over sources such as + * `Sample` and the effects of a sound FX to its `Output`, compiled once per note (see `core/instrument`). + */ +export interface InstrumentAsset extends AssetBase { + type: 'instrument'; + graph: Graph; +} + /** * A tile a tilemap paints with: `pictures` names one block per animation step (a texture id for a * sheet that holds nothing else, a frame id for one block cut out of a sheet), and `props` say @@ -572,6 +581,7 @@ export type Asset = | FontAsset | SoundAsset | SoundFxAsset + | InstrumentAsset | ParticlesAsset; /** The assets whose `variables` are declared by the user – see `requireVariableOwner`. */ @@ -583,7 +593,13 @@ export function isVariableOwner(asset: Asset | undefined): asset is VariableOwne /** The assets edited as a node graph – what the graph canvas and the graph commands work on. */ export type GraphAsset = - BehaviorAsset | StateMachineAsset | ShaderAsset | ShaderFunctionAsset | ParticlesAsset | SoundFxAsset; + | BehaviorAsset + | StateMachineAsset + | ShaderAsset + | ShaderFunctionAsset + | ParticlesAsset + | SoundFxAsset + | InstrumentAsset; const GRAPH_TYPES: ReadonlySet = new Set([ 'behavior', @@ -592,6 +608,7 @@ const GRAPH_TYPES: ReadonlySet = new Set([ 'shaderfunction', 'particles', 'soundfx', + 'instrument', ]); export function isGraphAsset(asset: Asset): asset is GraphAsset { diff --git a/src/core/model/AssetTypes.ts b/src/core/model/AssetTypes.ts index a333b4f5..243f21b0 100644 --- a/src/core/model/AssetTypes.ts +++ b/src/core/model/AssetTypes.ts @@ -25,6 +25,7 @@ export const ASSET_TYPES = [ 'font', 'sound', 'soundfx', + 'instrument', 'particles', ] as const; @@ -57,6 +58,7 @@ export const ASSET_TYPE_LABELS: Record = { font: 'Font', sound: 'Sound', soundfx: 'Sound FX', + instrument: 'Instrument', particles: 'Particles', }; @@ -69,7 +71,8 @@ export const ASSET_TYPE_LABELS: Record = { * is the tilemap. A curve is the way a key or a `Lerp` blends, picked by id like a state machine. * A struct is a shape that variables and fields take on (#253), nothing that exists on its own. * A shader function is part of other shaders (#303), worn by nothing itself. A sound FX is what a - * sound is played through (#311), heard but never seen. + * sound is played through (#311), heard but never seen, and an instrument what a note is played + * with (#331). */ export type PlaceableAssetType = Exclude< AssetType, @@ -87,6 +90,7 @@ export type PlaceableAssetType = Exclude< | 'font' | 'sound' | 'soundfx' + | 'instrument' | 'variables' | 'struct' >; @@ -106,6 +110,7 @@ const NOT_PLACEABLE: ReadonlySet = new Set([ 'font', 'sound', 'soundfx', + 'instrument', 'variables', 'struct', ]); diff --git a/src/core/model/graphLookup.ts b/src/core/model/graphLookup.ts index f89949fb..07278568 100644 --- a/src/core/model/graphLookup.ts +++ b/src/core/model/graphLookup.ts @@ -1,5 +1,5 @@ import type { GraphLookup } from '../graph/lookup'; -import { SOUND_FX_INPUT, terminalNode } from '../graph/nested'; +import { INSTRUMENT_PARAMETERS, SOUND_FX_INPUT, terminalNode } from '../graph/nested'; import { startStateOf, statesOf, transitionsOf } from '../graph/stateMachine'; import { pictureSize } from '../properties/aspect'; import { resolveAssetProps } from '../properties/resolve'; @@ -36,10 +36,10 @@ export function graphLookup(project: Project): GraphLookup { }, shaderFunction: shaderFunctionsOf(project), soundFx: (assetId) => { - const asset = findAssetOf(project, assetId, 'soundfx'); - return asset - ? { name: asset.name, params: terminalNode(asset.graph, SOUND_FX_INPUT)?.ports ?? [] } - : null; + const asset = findAssetOf(project, assetId, 'soundfx') ?? findAssetOf(project, assetId, 'instrument'); + if (!asset) return null; + const declaring = asset.type === 'soundfx' ? SOUND_FX_INPUT : INSTRUMENT_PARAMETERS; + return { name: asset.name, params: terminalNode(asset.graph, declaring)?.ports ?? [] }; }, machine: (assetId) => { const asset = findAssetOf(project, assetId, 'statemachine'); diff --git a/src/core/properties/PropertySchema.ts b/src/core/properties/PropertySchema.ts index b756bf3d..0227bb74 100644 --- a/src/core/properties/PropertySchema.ts +++ b/src/core/properties/PropertySchema.ts @@ -657,6 +657,8 @@ const ASSET_SCHEMAS: Record = { ], // The effects are the asset's graph (see SoundFxAsset), not a property. soundfx: [], + // What a note sounds like is the asset's graph (see InstrumentAsset), not a property. + instrument: [], // How many particles and for how long; what one of them does is the asset's graph, whose Property // nodes add their rows after these – see `schemaForBag`. particles: [ diff --git a/src/core/soundfx/nodes.ts b/src/core/soundfx/nodes.ts index 9c23bf86..5eddebbf 100644 --- a/src/core/soundfx/nodes.ts +++ b/src/core/soundfx/nodes.ts @@ -27,6 +27,20 @@ export interface FxDelay { pingPong: boolean; } +/** + * Where a way of an instrument's note gets its sound (#331): a file of its own, how loud and how it + * loops, and whether it ends with the note (`gate`) or plays through. A way of a sound FX has none – + * it plays the sound the voice was started with. `loopEnd` 0 is the end of the sound. + */ +export interface FxSource { + file: string; + volume: number; + loop: boolean; + loopStart: number; + loopEnd: number; + gate: boolean; +} + /** * A sound on its way through the graph: how many semitones it is played up or down, and what it * passes, in order. `gain` is a factor, a filter's `gain` is in dB as Web Audio takes it. `shifted` @@ -35,7 +49,7 @@ export interface FxDelay { * of what goes on through `stages`. */ export type FxChain = - | { pitch: number; shifted: boolean; stages: readonly FxStage[] } + | { pitch: number; shifted: boolean; stages: readonly FxStage[]; source?: FxSource } | { mix: number; a: FxChain; b: FxChain; stages: readonly FxStage[] }; export const DRY: FxChain = { pitch: 0, shifted: false, stages: [] }; @@ -66,7 +80,7 @@ export type FxStagePlan = /** A chain with its parameters still open: compiled once, filled in for every voice by `fxChainOf`. */ export type FxPlan = - | { pitch: readonly FxAmount[]; stages: readonly FxStagePlan[] } + | { pitch: readonly FxAmount[]; stages: readonly FxStagePlan[]; source?: FxSource } | { mix: FxAmount; a: FxPlan; b: FxPlan; stages: readonly FxStagePlan[] }; export const EMPTY_PLAN: FxPlan = { pitch: [], stages: [] }; @@ -130,12 +144,12 @@ const isNone = (amount: FxAmount) => amount.param === undefined && amount.value const arriving = (sounds: Readonly>) => sounds.in ?? EMPTY_PLAN; -function withStages(plan: FxPlan, stages: readonly FxStagePlan[]): FxPlan { +export function withStages(plan: FxPlan, stages: readonly FxStagePlan[]): FxPlan { return stages.length === 0 ? plan : { ...plan, stages: [...plan.stages, ...stages] }; } /** A pitch is made where the sound is read, so behind a Mix it goes to both sounds. */ -function withPitch(plan: FxPlan, semitones: FxAmount): FxPlan { +export function withPitch(plan: FxPlan, semitones: FxAmount): FxPlan { if ('mix' in plan) return { ...plan, a: withPitch(plan.a, semitones), b: withPitch(plan.b, semitones) }; return { ...plan, pitch: [...plan.pitch, semitones] }; } @@ -352,7 +366,8 @@ export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain { const stages = stagesOf(plan, params); if (!('mix' in plan)) { const pitch = plan.pitch.reduce((sum, amount) => sum + amountOf(amount, params), 0); - return { pitch, shifted: plan.pitch.length > 0, stages }; + const chain = { pitch, shifted: plan.pitch.length > 0, stages }; + return plan.source ? { ...chain, source: plan.source } : chain; } return { mix: amountOf(plan.mix, params), diff --git a/src/core/soundfx/pitchPlayer.ts b/src/core/soundfx/pitchPlayer.ts index 4f34ed3e..0125732b 100644 --- a/src/core/soundfx/pitchPlayer.ts +++ b/src/core/soundfx/pitchPlayer.ts @@ -57,16 +57,19 @@ export class PitchPlayer { private primed = false; /** How many samples of the output the sound lasts; a looping one never ends. */ private readonly end: number; + /** Where a loop runs in the samples of the sound; null for a sound that does not loop. */ + private readonly loop: { start: number; end: number } | null; /** * `channels` are the samples of the sound, `step` how many of them pass for one sample of the * output (the sound's rate over the output's) and `rate` the output's rate, which sets the frame. + * `loop` repeats the whole sound, or the samples from `start` up to `end` (#331) once it gets there. */ constructor( private readonly channels: readonly Float32Array[], private readonly step: number, rate: number, - private readonly loop: boolean, + loop: boolean | { start: number; end: number }, ) { // About 43 ms: a frame holds a few periods of a low note. this.size = 2 ** Math.round(Math.log2(rate * 0.043)); @@ -91,6 +94,11 @@ export class PitchPlayer { this.frameAt = -this.size; const length = channels[0]?.length ?? 0; this.end = loop ? Infinity : Math.ceil(length / step); + const region = loop === true ? { start: 0, end: length } : loop || null; + this.loop = + region && region.start >= 0 && region.end <= length && region.start < region.end + ? region + : region && { start: 0, end: length }; } /** @@ -256,15 +264,16 @@ export class PitchPlayer { private readFrame(c: number, from: number): Float64Array { const { reading, size, step, loop } = this; const samples = this.channels[c]!; - const length = samples.length; + const end = loop ? loop.end : samples.length; for (let j = 0; j < size; j++) { let position = (from + j) * step; - if (position < 0 || position >= length) { - if (!loop) { + if (position < 0 || position >= end) { + if (!loop || position < 0) { reading[j] = 0; continue; } - position -= Math.floor(position / length) * length; + const span = loop.end - loop.start; + position = loop.start + ((position - loop.start) % span); } const below = Math.floor(position); const fraction = position - below; @@ -272,7 +281,7 @@ export class PitchPlayer { reading[j] = samples[below]!; continue; } - const next = below + 1 < length ? samples[below + 1]! : loop ? samples[0]! : 0; + const next = below + 1 < end ? samples[below + 1]! : loop ? samples[loop.start]! : 0; reading[j] = samples[below]! * (1 - fraction) + next * fraction; } return reading; diff --git a/src/core/store/serialize.ts b/src/core/store/serialize.ts index b47e8782..2bc2026c 100644 --- a/src/core/store/serialize.ts +++ b/src/core/store/serialize.ts @@ -20,6 +20,7 @@ import { import { isParticleNodeDef } from '../particles/nodes'; import { isShaderNodeDef } from '../shader/nodes'; import { isSoundFxNodeDef } from '../soundfx/nodes'; +import { isInstrumentNodeDef } from '../instrument/nodes'; import { isKeyInterpolation, normalizeKeys } from '../model/animation'; import type { AnimationKey, @@ -237,6 +238,9 @@ export function deserializeProject(raw: unknown, fallbackId = ''): Project { assets.push({ id, type, name, props, graph: particleGraphs.get(id) ?? createEmptyGraph() }); } else if (type === 'soundfx') { assets.push({ id, type, name, props, graph: normalizeGraph(a.graph, isSoundFxNodeDef) }); + } else if (type === 'instrument') { + const keep = (def: NodeDef) => isInstrumentNodeDef(def) || isSoundFxNodeDef(def); + assets.push({ id, type, name, props, graph: normalizeGraph(a.graph, keep) }); } else if (type === 'texture') { assets.push({ id, type, name, props, ...normalizeTexture(a, takenIds) }); } else if (type === 'font') { diff --git a/src/editor/Editor.ts b/src/editor/Editor.ts index 1762ed03..6f48ffaf 100644 --- a/src/editor/Editor.ts +++ b/src/editor/Editor.ts @@ -73,6 +73,7 @@ import { ProjectSession } from './ProjectSession'; import { ParticlesEditor } from './particles/ParticlesEditor'; import { ShaderEditor } from './shader/ShaderEditor'; import { SoundFxEditor } from './soundfx/SoundFxEditor'; +import { InstrumentEditor } from './instrument/InstrumentEditor'; import { StateMachineEditor } from './statemachine/StateMachineEditor'; import { installShortcuts } from './shortcuts'; import type { PlayFrom } from '../core/model/playSettings'; @@ -95,6 +96,7 @@ export type CentreTab = | 'shader' | 'particles' | 'soundfx' + | 'instrument' | 'game' | 'texture' | 'vectorart' @@ -145,6 +147,7 @@ const PANE_ORDER = [ 'shader', 'particles', 'soundfx', + 'instrument', 'texture', 'vectorart', 'pixelart', @@ -165,6 +168,7 @@ const TAB_LABELS: Record = { shader: 'Shader', particles: 'Particles', soundfx: 'Sound FX', + instrument: 'Instrument', texture: 'Texture', vectorart: 'Vector art', pixelart: 'Pixel art', @@ -214,6 +218,7 @@ export class Editor implements PlayHost { readonly shaderEditor: ShaderEditor; readonly particlesEditor: ParticlesEditor; readonly soundFxEditor: SoundFxEditor; + readonly instrumentEditor: InstrumentEditor; readonly textureEditor: TextureEditor; readonly vectorArtEditor: VectorArtEditor; readonly pixelArtEditor: PixelArtEditor; @@ -300,6 +305,7 @@ export class Editor implements PlayHost { this.shaderEditor = new ShaderEditor(state); this.particlesEditor = new ParticlesEditor(state); this.soundFxEditor = new SoundFxEditor(state); + this.instrumentEditor = new InstrumentEditor(state); this.textureEditor = new TextureEditor(state); this.vectorArtEditor = new VectorArtEditor(state); this.pixelArtEditor = new PixelArtEditor(state); @@ -409,6 +415,7 @@ export class Editor implements PlayHost { shader: this.shaderEditor, particles: this.particlesEditor, soundfx: this.soundFxEditor, + instrument: this.instrumentEditor, texture: this.textureEditor, vectorart: this.vectorArtEditor, pixelart: this.pixelArtEditor, @@ -600,6 +607,8 @@ export class Editor implements PlayHost { return this.particlesEditor.canvas.paste(); case 'soundfx': return this.soundFxEditor.canvas.paste(); + case 'instrument': + return this.instrumentEditor.canvas.paste(); default: return false; } diff --git a/src/editor/assetTabs.ts b/src/editor/assetTabs.ts index e10936e8..8ea68197 100644 --- a/src/editor/assetTabs.ts +++ b/src/editor/assetTabs.ts @@ -10,6 +10,7 @@ export const TAB_OF_ASSET: Partial> = { shaderfunction: 'shader', particles: 'particles', soundfx: 'soundfx', + instrument: 'instrument', texture: 'texture', vectorart: 'vectorart', pixelart: 'pixelart', diff --git a/src/editor/debug/graphHooks.ts b/src/editor/debug/graphHooks.ts index 2eeb026c..a5ded3d3 100644 --- a/src/editor/debug/graphHooks.ts +++ b/src/editor/debug/graphHooks.ts @@ -113,6 +113,27 @@ export function graphHooks({ state, editor }: HookContext) { params: () => editor.soundFxEditor.previewValues, setParam: (key: string, value: number) => editor.soundFxEditor.setPreviewValue(key, value), }, + instrument: { + current: () => editor.instrumentEditor.currentAssetId, + info: () => editor.instrumentEditor.shownInfo, + nodeAt: (nodeId: string) => editor.instrumentEditor.canvas.nodeGlobal(nodeId), + nodeLabel: (nodeId: string) => editor.instrumentEditor.canvas.nodeLabel(nodeId), + nodeOutputs: (nodeId: string) => editor.instrumentEditor.canvas.nodeOutputs(nodeId), + nodeProblem: (nodeId: string) => editor.instrumentEditor.canvas.nodeProblem(nodeId), + portAt: (nodeId: string, port: string, isOutput: boolean) => + editor.instrumentEditor.canvas.portGlobal(nodeId, port, isOutput), + canvasRect: () => editor.instrumentEditor.canvas.canvasRect(), + /** The keys: which make a sound, which are held, where one is, and the computer keyboard's octave. */ + keys: () => ({ + sounding: editor.instrumentEditor.soundingNotes(), + held: editor.instrumentEditor.keyboard.heldNotes, + octave: editor.instrumentEditor.keyboard.bottomOctave, + }), + keyAt: (note: number) => editor.instrumentEditor.keyboard.keyCentre(note), + stopAt: () => centreOf(editor.instrumentEditor.stopButton), + params: () => editor.instrumentEditor.previewValues, + setParam: (key: string, value: number) => editor.instrumentEditor.setPreviewValue(key, value), + }, particles: { current: () => editor.particlesEditor.currentAssetId, info: () => editor.particlesEditor.shownInfo, diff --git a/src/editor/debugHook.ts b/src/editor/debugHook.ts index ea6f24c0..dcbea97a 100644 --- a/src/editor/debugHook.ts +++ b/src/editor/debugHook.ts @@ -45,6 +45,8 @@ export function installDebugHook(app: Application, state: EditorState, editor: E /** Where the knobs of each voice of that file are now, oldest first, a pitch as its speed (#315). */ levels: (file: string) => soundLibrary.levels(file), pitchWorkletUrl: () => soundLibrary.pitchWorkletUrl(), + /** Seconds on the audio clock: the time a voice really runs on. */ + clock: () => soundLibrary.clock(), /** Decodes ahead of a check, so a play can be asserted without racing the network. */ load: (file: string) => soundLibrary.load(file).then((buffer) => buffer?.duration ?? null), stopAll: () => soundLibrary.stopAll(), diff --git a/src/editor/graph/GraphCanvas.ts b/src/editor/graph/GraphCanvas.ts index b4458ea5..4fb29e38 100644 --- a/src/editor/graph/GraphCanvas.ts +++ b/src/editor/graph/GraphCanvas.ts @@ -21,6 +21,7 @@ import { namesBehavior, RUN_BEHAVIOR, SHADER_CALL, + INSTRUMENT_SOUND_FX, callsFunction, soundFxParamsMismatch, startStateOf, @@ -30,6 +31,7 @@ import { graphLookup } from '../../core/model/graphLookup'; import { particleLibrary } from '../../viewport/ParticleLibrary'; import { shaderLibrary } from '../../viewport/ShaderLibrary'; import { soundFxBuildFor } from '../../core/soundfx/compile'; +import { instrumentCheck, instrumentSourcesOf } from '../../core/instrument/compile'; import { findAsset, findGraphAsset } from '../../core/model/Project'; import { type Trace, TRACE_FADE } from '../../runtime/systems/BehaviorSystem'; import { ViewTween } from '../viewTween'; @@ -309,7 +311,9 @@ export class GraphCanvas extends Widget { ? particleLibrary.buildFor(asset).problems : asset.type === 'soundfx' ? soundFxBuildFor(asset).problems - : null; + : asset.type === 'instrument' + ? instrumentCheck(asset, instrumentSourcesOf(this.state.project)).problems + : null; for (const node of asset.graph.nodes) { const resolved = resolveDef(node, asset.graph, lookup); if (!resolved) continue; @@ -325,7 +329,9 @@ export class GraphCanvas extends Widget { } const problem = compilerProblems?.[node.id] ?? - (!compilerProblems || node.type === SHADER_CALL ? callProblem(node, lookup, asset.id) : null) ?? + (!compilerProblems || node.type === SHADER_CALL || node.type === INSTRUMENT_SOUND_FX + ? callProblem(node, lookup, asset.id) + : null) ?? soundFxParamsMismatch(node, asset.graph) ?? ghostProblem(node, asset.graph, resolved) ?? blockProblem(asset.graph, node); @@ -656,9 +662,10 @@ export class GraphCanvas extends Widget { } private openCalledBehavior(node: GraphNode): boolean { - if (node.type === SHADER_CALL) { - const id = textOf(node.params[FUNCTION_PARAM] ?? ''); - if (findAsset(this.state.project, id)?.type !== 'shaderfunction') return false; + if (node.type === SHADER_CALL || node.type === INSTRUMENT_SOUND_FX) { + const shader = node.type === SHADER_CALL; + const id = textOf(node.params[shader ? FUNCTION_PARAM : 'fx'] ?? ''); + if (findAsset(this.state.project, id)?.type !== (shader ? 'shaderfunction' : 'soundfx')) return false; this.state.openAsset(id); return true; } diff --git a/src/editor/graph/NodeView.ts b/src/editor/graph/NodeView.ts index 37a92ab0..cdcc52c0 100644 --- a/src/editor/graph/NodeView.ts +++ b/src/editor/graph/NodeView.ts @@ -97,6 +97,8 @@ export function portColor(type: PortType): number { return c.portAudio; case 'fxparams': return c.portFxParams; + case 'trigger': + return c.portTrigger; case 'any': return c.portAny; } diff --git a/src/editor/graph/paletteEntries.ts b/src/editor/graph/paletteEntries.ts index 2a039ba7..065131fb 100644 --- a/src/editor/graph/paletteEntries.ts +++ b/src/editor/graph/paletteEntries.ts @@ -42,6 +42,7 @@ import { nextParticlePropertyName, PARTICLE_NODES, PARTICLE_OUTPUT } from '../.. import { offeredIn, ShaderFunctionInput, ShaderFunctionOutput } from '../../core/shader/functions'; import { SHADER_NODES, SHADER_OUTPUT, SHADER_VERTEX_OUTPUT } from '../../core/shader/nodes'; import { SOUND_FX_INPUT, SOUND_FX_NODES, SOUND_FX_OUTPUT } from '../../core/soundfx/nodes'; +import { INSTRUMENT_NODES } from '../../core/instrument/nodes'; /** * One row of the palette. `def` is the definition the node would actually have once placed, so the @@ -127,7 +128,7 @@ function shaderEntries(project: Project, asset: ShaderAsset | ShaderFunctionAsse /** * The node types offerable in `asset`'s graph. A state machine holds states and nothing else, each * new one named by its number; a shader holds shader nodes (`shaderEntries`), a sound FX its effects - * with one Input and one Output. In a behavior, every other behavior appears under its own name instead of behind a + * with one Input and one Output, an instrument its own nodes and the same effects. In a behavior, every other behavior appears under its own name instead of behind a * generic "Run behavior" – and once more as the event "On " if it has an End to be heard * through (#294) –, and the two terminals drop out once the graph has them. */ @@ -143,6 +144,15 @@ export function paletteEntries(project: Project, asset: GraphAsset): PaletteEntr (def) => !(def.type === SOUND_FX_INPUT || def.type === SOUND_FX_OUTPUT) || !placed.has(def.type), ).map((def) => ({ id: def.type, hint: def.category, def, params: {} })); } + if (asset.type === 'instrument') { + const placed = new Set(asset.graph.nodes.map((n) => n.type)); + const effects = SOUND_FX_NODES.filter( + (def) => def.type !== SOUND_FX_INPUT && def.type !== SOUND_FX_OUTPUT, + ); + return [...INSTRUMENT_NODES, ...effects] + .filter((def) => !isTerminalType(def.type) || !placed.has(def.type)) + .map((def) => ({ id: def.type, hint: def.category, def, params: {} })); + } if (asset.type === 'particles') { const hasOutput = asset.graph.nodes.some((n) => n.type === PARTICLE_OUTPUT); const taken = new Set(asset.graph.nodes.filter(isParticlePropertyNode).map(propertyNameOf)); diff --git a/src/editor/inspector/portSection.ts b/src/editor/inspector/portSection.ts index 922c5c74..25b75842 100644 --- a/src/editor/inspector/portSection.ts +++ b/src/editor/inspector/portSection.ts @@ -13,7 +13,10 @@ import { import { type DeclaredPort, type GraphNode, + declaresParameters, declaresSoundFxPorts, + INSTRUMENT_PARAMETER_VALUES, + INSTRUMENT_SETTING, isEntryTerminal, isPortOf, isTerminalNode, @@ -28,7 +31,6 @@ import { SOUND_FX_INPUT, SOUND_FX_SETTING, soundFxSourceOf, - terminalNode, VARIABLES_SETTING, variablesSourceOf, propertyTargetOf, @@ -37,6 +39,7 @@ import { targetProperties, } from '../../core/graph'; import { isPrivateVariable } from '../../core/model/Asset'; +import { graphLookup } from '../../core/model/graphLookup'; import { findAssetOf, findGraphAsset, type Project } from '../../core/model/Project'; import { textOf } from '../../core/model/Properties'; import { assetRefChoices, NO_ASSET_LABEL } from '../../core/properties/assetRef'; @@ -98,10 +101,12 @@ export function buildVariablePortSection( }); } -/** The parameters the Input of the sound FX a `Sound FX parameters` node names declares (#311). */ +/** + * The parameters the sound FX a `Sound FX parameters` node names declares at its Input (#311), or + * the instrument an `Instrument parameters` node names at its Parameters node (#331). + */ function soundFxChoices(project: Project, node: GraphNode): PropertyDef[] { - const fx = findAssetOf(project, soundFxSourceOf(node.params), 'soundfx'); - const ports = fx ? (terminalNode(fx.graph, SOUND_FX_INPUT)?.ports ?? []) : []; + const ports = graphLookup(project).soundFx(soundFxSourceOf(node.params))?.params ?? []; return ports.map((p) => ({ key: p.key, label: p.key, kind: 'number', default: p.default ?? 0 })); } @@ -112,18 +117,20 @@ export function buildSoundFxPortSection( node: GraphNode, addSetting: (def: PropertyDef) => void, ): void { - addSetting(SOUND_FX_SETTING); + const ofInstrument = node.type === INSTRUMENT_PARAMETER_VALUES; + addSetting(ofInstrument ? INSTRUMENT_SETTING : SOUND_FX_SETTING); + const noun = ofInstrument ? 'instrument' : 'sound FX'; const id = soundFxSourceOf(node.params); - const fx = id ? findAssetOf(host.state.project, id, 'soundfx') : undefined; + const fx = id ? graphLookup(host.state.project).soundFx(id) : null; buildPickedPortSection(host, assetId, node, { noun: 'Parameter', choices: soundFxChoices(host.state.project, node), - owner: fx ? `"${fx.name}"` : 'a sound FX', + owner: fx ? `"${fx.name}"` : `a ${noun}`, empty: id ? fx - ? `"${fx.name}" declares no parameters yet. Add them at its Input.` - : 'That sound FX is gone.' - : 'Pick a sound FX first.', + ? `"${fx.name}" declares no parameters yet. Add them at its ${ofInstrument ? 'Parameters node' : 'Input'}.` + : `That ${noun} is gone.` + : `Pick ${ofInstrument ? 'an instrument' : 'a sound FX'} first.`, }); } @@ -276,7 +283,7 @@ export function buildPortSection( typeName: string | null = null, ): void { const onStart = isEntryTerminal(node.type); - const ofSoundFx = node.type === SOUND_FX_INPUT; + const ofSoundFx = declaresParameters(node.type); const noun = ofSoundFx ? 'Parameter' : isTerminalNode(node) ? (onStart ? 'Input' : 'Output') : 'Field'; const ofFunction = node.type === SHADER_FUNCTION_INPUT || node.type === SHADER_FUNCTION_OUTPUT; const base = ofSoundFx ? 'param' : noun === 'Field' ? 'field' : onStart ? 'in' : 'out'; @@ -317,7 +324,11 @@ export function buildPortSection( noun === 'Field' ? 'No fields yet. Every field added here becomes a port of the struct.' : ofSoundFx - ? 'No parameters yet. Every parameter added here is a number with a default, and a Sound FX parameters node can set it when the sound is played.' + ? `No parameters yet. Every parameter added here is a number with a default, and ${ + node.type === SOUND_FX_INPUT + ? 'a Sound FX parameters node can set it when the sound' + : 'an Instrument parameters node can set it when a note' + } is played.` : ofFunction ? `No ${plural(noun)} yet. Every ${noun.toLowerCase()} added here, a number or a color, becomes a port on the nodes calling this function.` : onStart diff --git a/src/editor/instrument/InstrumentEditor.ts b/src/editor/instrument/InstrumentEditor.ts new file mode 100644 index 00000000..294c6a10 --- /dev/null +++ b/src/editor/instrument/InstrumentEditor.ts @@ -0,0 +1,185 @@ +import { + instrumentBuildFor, + instrumentCheck, + instrumentParams, + instrumentSourcesOf, + playedNotes, +} from '../../core/instrument/compile'; +import { noteName } from '../../core/instrument/notes'; +import type { Asset, InstrumentAsset } from '../../core/model/Asset'; +import { playNote } from '../../runtime/systems/behaviorContext'; +import { Button, HStack, Label, NumberField, Stack, Theme, VStack, type Widget } from '../../ui'; +import { soundLibrary } from '../../viewport/SoundLibrary'; +import { AssetTabEditor } from '../AssetTabEditor'; +import { GraphCanvas } from '../graph/GraphCanvas'; +import { SnapToggle } from '../graph/SnapToggle'; +import { TOOLBAR_ICON_SVG } from '../panels/toolbarIcons'; +import type { EditorState } from '../state'; +import { PianoKeyboard } from './PianoKeyboard'; + +/** Room kept for "N notes", or for the reason the graph does not compile. */ +const INFO_WIDTH = 220; +const PARAM_WIDTH = 48; +/** Whose voices the tab's Stop silences: its own, never a running game's. */ +const PREVIEW_OWNER = 'instrument-preview'; + +/** + * The nodes of one instrument (#331), and below them its keys: a note is heard while a key is held, + * by the pointer or by the computer keyboard. The header holds a value for each of its parameters – + * the tab's own, never written into the asset – and a Stop for everything the tab still plays. + */ +export class InstrumentEditor extends AssetTabEditor { + readonly snap = new SnapToggle(() => this.canvas); + /** Public for the check scripts. */ + readonly keyboard = new PianoKeyboard(); + readonly stopButton: Button; + private readonly octaveLabel = new Label('', { style: 'small' }); + private readonly paramBar = new HStack({ gap: Theme.gap, align: 'center' }); + private readonly paramFields = new Map(); + private values: Record = {}; + private defaults: Record = {}; + private paramsKey = ''; + private valuesFor = ''; + /** The voices the tab started that may still be playing – what a moved field reaches. */ + private voices: number[] = []; + + constructor(state: EditorState) { + super(state, { infoWidth: INFO_WIDTH }); + this.stopButton = new Button('Stop', { icon: TOOLBAR_ICON_SVG.stop, onClick: () => this.stop() }); + this.keyboard.onNoteOn = (note) => this.playNote(note); + this.keyboard.onNoteOff = (voice) => soundLibrary.release(voice); + this.keyboard.onOctave = () => this.syncOctave(); + this.mount(new GraphCanvas(state, 'instrument'), [ + ...this.snap.widgets, + this.stopButton, + this.octaveLabel, + this.paramBar, + ]); + this.syncOctave(); + state.on('load', () => { + this.valuesFor = ''; + }); + } + + override selectedId(): string { + return this.state.selectedInstrumentId ?? ''; + } + + override isMine(asset: Asset): asset is InstrumentAsset { + return asset.type === 'instrument'; + } + + protected override body(canvas: GraphCanvas): Widget { + const column = new VStack(); + column.add(canvas, 1); + column.add(this.keyboard); + return column; + } + + /** What the header's parameter fields are set to – the check scripts read this. */ + get previewValues(): Record { + return { ...this.values }; + } + + setPreviewValue(key: string, value: number): void { + if (!(key in this.values)) return; + this.paramFields.get(key)?.setNumber(value); + this.setValue(key, value); + } + + /** Starts one note as the keys do, held until it is released; gives back its voice. */ + playNote(note: number): number { + const asset = this.asset(); + if (!asset) return 0; + const struck = { note, velocity: 1, volume: 1, duration: 0 }; + const voice = playNote( + { project: this.state.project, audio: soundLibrary }, + asset.id, + struck, + PREVIEW_OWNER, + { ...this.values }, + ); + if (voice) this.voices.push(voice); + return voice; + } + + stop(): void { + this.keyboard.releaseAll(); + soundLibrary.stopOwner(PREVIEW_OWNER); + this.voices = []; + } + + /** The notes the keys show as making a sound (test hook). */ + soundingNotes(): number[] { + const asset = this.asset(); + if (!asset) return []; + const sources = instrumentSourcesOf(this.state.project); + return [...playedNotes(asset.graph)] + .filter((note) => instrumentBuildFor(asset, note, sources).layers.length > 0) + .sort((a, b) => a - b); + } + + private setValue(key: string, value: number): void { + this.values[key] = value; + this.voices = this.voices.filter((voice) => soundLibrary.tune(voice, { [key]: value })); + } + + protected override infoText(asset: InstrumentAsset): string { + const build = instrumentCheck(asset, instrumentSourcesOf(this.state.project)); + if (build.reason) return build.reason; + const count = this.soundingNotes().length; + return `${count} ${count === 1 ? 'note' : 'notes'}`; + } + + protected override syncExtras(asset: InstrumentAsset | null): void { + this.snap.setEnabled(Boolean(asset)); + this.stopButton.setEnabled(Boolean(asset)); + this.keyboard.setEnabled(Boolean(asset)); + if (!asset) this.keyboard.releaseAll(); + this.keyboard.setSounding(new Set(this.soundingNotes())); + this.syncParams(asset); + } + + private syncOctave(): void { + const octave = this.keyboard.bottomOctave; + this.octaveLabel.setText(`Keys ${noteName((octave + 1) * 12)} to ${noteName((octave + 3) * 12 + 4)}`); + } + + /** As in the Sound FX tab: another asset starts from its defaults, an edit keeps a moved value. */ + private syncParams(asset: InstrumentAsset | null): void { + const params = asset ? instrumentParams(asset.graph) : []; + const same = (asset?.id ?? '') === this.valuesFor; + this.valuesFor = asset?.id ?? ''; + const values: Record = {}; + const defaults: Record = {}; + for (const port of params) { + const fallback = port.default ?? 0; + const kept = same && this.defaults[port.key] === fallback ? this.values[port.key] : undefined; + values[port.key] = kept ?? fallback; + defaults[port.key] = fallback; + } + this.values = values; + this.defaults = defaults; + + const key = params.map((p) => p.key).join('\u0000'); + if (key !== this.paramsKey) { + this.paramsKey = key; + for (const child of this.paramBar.removeChildren()) child.destroy({ children: true }); + this.paramFields.clear(); + for (const port of params) { + this.paramBar.add(new Label(port.key, { style: 'small' })); + const field = new NumberField(values[port.key], { + width: PARAM_WIDTH, + step: 0.1, + onInput: (value) => this.setValue(port.key, value), + onChange: (value) => this.setValue(port.key, value), + }); + this.paramFields.set(port.key, field); + this.paramBar.add(field); + } + if (this.paramBar.parent instanceof Stack) this.paramBar.parent.relayout(); + this.paramBar.relayout(); + } + for (const [name, field] of this.paramFields) field.setNumber(this.values[name] ?? 0); + } +} diff --git a/src/editor/instrument/PianoKeyboard.ts b/src/editor/instrument/PianoKeyboard.ts new file mode 100644 index 00000000..9ff533cb --- /dev/null +++ b/src/editor/instrument/PianoKeyboard.ts @@ -0,0 +1,185 @@ +import { type FederatedPointerEvent, Graphics, Text } from 'pixi.js'; +import { noteName } from '../../core/instrument/notes'; +import { textStyle, Theme, ui, Widget } from '../../ui'; +import { + DEFAULT_OCTAVE, + HIGHEST_OCTAVE, + type KeyRect, + keyboardLayout, + LOWEST_OCTAVE, + noteAt, + octaveStep, + trackerNote, +} from './keys'; + +const HEIGHT = 64; +const WHITE = 0xe8e8e8; +const BLACK = 0x22252b; +const MUTED_WHITE = 0x6b6f78; +const MUTED_BLACK = 0x3a3e46; +/** Dark, to stand on a white key – its own style, since the shared ones are cached for everyone. */ +const KEY_LABEL = textStyle('small').clone(); +KEY_LABEL.fill = 0x30343c; + +/** + * The keys of an instrument's tab (#331): every note there is, the ones the instrument plays nothing + * for greyed out, a pressed one lit. A key sounds while the pointer holds it; with the keyboard + * focused, the computer keyboard plays it as a tracker does, and the arrows move its octave. + */ +export class PianoKeyboard extends Widget { + private readonly keysLayer = new Graphics(); + private readonly labels: Text[] = []; + private keys: KeyRect[] = []; + private sounding: ReadonlySet = new Set(); + /** What each held note is held by – the pointer, or a key by its code – and the voice it started. */ + private readonly held = new Map(); + private octave = DEFAULT_OCTAVE; + private readonly onKeyUp = (event: KeyboardEvent) => this.up(event.code); + /** Starts a note and gives back its voice, 0 for none. */ + onNoteOn: (note: number) => number = () => 0; + onNoteOff: (voice: number) => void = () => {}; + onOctave: (octave: number) => void = () => {}; + + constructor() { + super(); + this.focusable = true; + this.minHeight = HEIGHT; + this.cursor = Theme.hoverCursor; + this.addChild(this.keysLayer); + this.on('pointerdown', (e: FederatedPointerEvent) => { + ui().focus.focus(this); + this.press('pointer', this.noteUnder(e)); + e.stopPropagation(); + }); + this.on('globalpointermove', (e: FederatedPointerEvent) => { + const holding = this.held.get('pointer'); + if (!holding) return; + const note = this.noteUnder(e); + if (note !== holding.note) this.press('pointer', note); + }); + this.onPointerRelease(() => this.up('pointer')); + } + + /** The notes the instrument makes a sound for; the others are drawn greyed out. */ + setSounding(notes: ReadonlySet): void { + this.sounding = notes; + this.invalidate(); + } + + /** The notes held right now, lowest first (test hook). */ + get heldNotes(): number[] { + return [...this.held.values()].map((h) => h.note).sort((a, b) => a - b); + } + + get bottomOctave(): number { + return this.octave; + } + + /** Where a note's key is on screen, its middle (test hook). */ + keyCentre(note: number): { x: number; y: number } | null { + const key = this.keys.find((k) => k.note === note); + if (!key) return null; + const at = this.getGlobalPosition(); + return { x: at.x + key.x + key.w / 2, y: at.y + key.y + key.h * (key.black ? 0.5 : 0.85) }; + } + + /** Lets go of every note, as losing the focus or leaving the tab does. */ + releaseAll(): void { + for (const holder of [...this.held.keys()]) this.up(holder); + } + + override onFocus(): void { + super.onFocus(); + window.addEventListener('keyup', this.onKeyUp); + } + + override onBlur(): void { + super.onBlur(); + window.removeEventListener('keyup', this.onKeyUp); + this.releaseAll(); + } + + override onKeyDown(event: KeyboardEvent): boolean { + if (event.ctrlKey || event.metaKey || event.altKey) return false; + const step = octaveStep(event.code); + if (step !== 0) { + this.octave = Math.max(LOWEST_OCTAVE, Math.min(HIGHEST_OCTAVE, this.octave + step)); + this.onOctave(this.octave); + this.invalidate(); + return true; + } + const note = trackerNote(event.code, this.octave); + if (note === null) return false; + // A held key repeats; the note it started goes on until it is let go. + if (!event.repeat && !this.held.has(event.code)) this.press(event.code, note); + return true; + } + + override destroy(options?: Parameters[0]): void { + window.removeEventListener('keyup', this.onKeyUp); + super.destroy(options); + } + + override layout(): void { + this.keys = keyboardLayout(this.w, this.h); + } + + override redraw(): void { + const g = this.keysLayer; + g.clear(); + const lit = new Set(this.heldNotes); + const c = Theme.colors; + for (const key of this.keys) { + const sounds = this.sounding.has(key.note); + const fill = lit.has(key.note) + ? c.accent + : key.black + ? sounds + ? BLACK + : MUTED_BLACK + : sounds + ? WHITE + : MUTED_WHITE; + g.rect(key.x, key.y, key.w, key.h).fill(fill).stroke({ width: 1, color: c.border }); + } + // The octave the bottom row of the computer keyboard plays, underlined. + const from = this.keys.find((k) => k.note === (this.octave + 1) * 12); + const to = this.keys.find((k) => k.note === Math.min(127, (this.octave + 2) * 12 + 16)); + if (from && to && this.focused) g.rect(from.x, this.h - 3, to.x + to.w - from.x, 3).fill(c.accent); + this.drawLabels(); + } + + /** A name on every C, where there is room for one. */ + private drawLabels(): void { + for (const label of this.labels.splice(0)) label.destroy(); + const cs = this.keys.filter((k) => !k.black && k.note % 12 === 0); + if (cs.length === 0 || (cs[0]?.w ?? 0) * 7 < 28) return; + for (const key of cs) { + const label = new Text({ text: noteName(key.note), style: KEY_LABEL }); + label.x = key.x + 1; + label.y = key.h - label.height - 2; + this.labels.push(label); + this.addChild(label); + } + } + + private noteUnder(e: FederatedPointerEvent): number | null { + const at = this.toLocal(e.global); + return noteAt(this.keys, at.x, Math.min(at.y, this.h - 1)); + } + + private press(holder: string, note: number | null): void { + this.up(holder); + if (note === null) return; + this.held.set(holder, { note, voice: this.onNoteOn(note) }); + this.invalidate(); + } + + private up(holder: string): void { + const holding = this.held.get(holder); + if (!holding) return; + this.held.delete(holder); + if (holding.voice) this.onNoteOff(holding.voice); + this.invalidate(); + } +} diff --git a/src/editor/instrument/keys.ts b/src/editor/instrument/keys.ts new file mode 100644 index 00000000..4d8f31a9 --- /dev/null +++ b/src/editor/instrument/keys.ts @@ -0,0 +1,106 @@ +import { HIGHEST_NOTE, isSharp, LOWEST_NOTE } from '../../core/instrument/notes'; + +/** One key as drawn: where it is, and whether it is a black one lying over the white ones. */ +export interface KeyRect { + note: number; + x: number; + y: number; + w: number; + h: number; + black: boolean; +} + +/** Which white key in its octave a natural note is: C 0, D 1 … B 6. */ +const WHITE_STEP = [0, -1, 1, -1, 2, 3, -1, 4, -1, 5, -1, 6]; +const BLACK_WIDTH = 0.6; +const BLACK_HEIGHT = 0.62; + +function whiteIndex(note: number): number { + return Math.floor(note / 12) * 7 + WHITE_STEP[note % 12]!; +} + +const WHITE_KEYS = whiteIndex(HIGHEST_NOTE) + 1; + +/** + * Every key of the 128 notes across `w` by `h`: the white keys side by side, each black key over the + * gap after the white one below it. White first, so a black one is drawn over them. + */ +export function keyboardLayout(w: number, h: number): KeyRect[] { + const white = w / WHITE_KEYS; + const keys: KeyRect[] = []; + for (let note = LOWEST_NOTE; note <= HIGHEST_NOTE; note++) + if (!isSharp(note)) keys.push({ note, x: whiteIndex(note) * white, y: 0, w: white, h, black: false }); + for (let note = LOWEST_NOTE; note <= HIGHEST_NOTE; note++) { + if (!isSharp(note)) continue; + const gap = (whiteIndex(note - 1) + 1) * white; + const width = white * BLACK_WIDTH; + keys.push({ note, x: gap - width / 2, y: 0, w: width, h: h * BLACK_HEIGHT, black: true }); + } + return keys; +} + +/** The note under a point, a black key before the white one it lies on; null off the keys. */ +export function noteAt(keys: readonly KeyRect[], x: number, y: number): number | null { + const inside = (k: KeyRect) => x >= k.x && x < k.x + k.w && y >= k.y && y < k.y + k.h; + return (keys.find((k) => k.black && inside(k)) ?? keys.find((k) => !k.black && inside(k)))?.note ?? null; +} + +/** + * The computer keyboard as a tracker has it, by where a key is rather than what it says, so QWERTZ and + * QWERTY play alike: the bottom row is one octave from C, the row above it the black keys, the row + * with Q the octave after, its black keys on the digits. + */ +const TRACKER_KEYS: Readonly> = { + KeyZ: 0, + KeyS: 1, + KeyX: 2, + KeyD: 3, + KeyC: 4, + KeyV: 5, + KeyG: 6, + KeyB: 7, + KeyH: 8, + KeyN: 9, + KeyJ: 10, + KeyM: 11, + Comma: 12, + KeyL: 13, + Period: 14, + Semicolon: 15, + Slash: 16, + KeyQ: 12, + Digit2: 13, + KeyW: 14, + Digit3: 15, + KeyE: 16, + KeyR: 17, + Digit5: 18, + KeyT: 19, + Digit6: 20, + KeyY: 21, + Digit7: 22, + KeyU: 23, + KeyI: 24, + Digit9: 25, + KeyO: 26, + Digit0: 27, + KeyP: 28, +}; + +/** The octave the bottom row starts in at first: Z is C3, Q is C4. */ +export const DEFAULT_OCTAVE = 3; +export const LOWEST_OCTAVE = -1; +export const HIGHEST_OCTAVE = 8; + +/** The note a key plays with the bottom row starting at C of `octave`; null for no note key. */ +export function trackerNote(code: string, octave: number): number | null { + const step = TRACKER_KEYS[code]; + if (step === undefined) return null; + const note = (octave + 1) * 12 + step; + return note >= LOWEST_NOTE && note <= HIGHEST_NOTE ? note : null; +} + +/** Which way a key moves the octave: the arrows left and right. */ +export function octaveStep(code: string): -1 | 1 | 0 { + return code === 'ArrowLeft' ? -1 : code === 'ArrowRight' ? 1 : 0; +} diff --git a/src/editor/panels/assetGroups.ts b/src/editor/panels/assetGroups.ts index 3fc33a7f..64a331e0 100644 --- a/src/editor/panels/assetGroups.ts +++ b/src/editor/panels/assetGroups.ts @@ -36,6 +36,8 @@ export const ASSET_GROUP_LABELS: Record = { sound: 'Sounds', // Not "Sound FX": that is the `+` menu entry (see Vector art). soundfx: 'Sound effects', + // Where patterns and songs join it (#330). + instrument: 'Music', particles: 'Particles', }; diff --git a/src/editor/panels/assetIcons.ts b/src/editor/panels/assetIcons.ts index 4c33ef43..308c80d8 100644 --- a/src/editor/panels/assetIcons.ts +++ b/src/editor/panels/assetIcons.ts @@ -14,6 +14,7 @@ import font from './icons/font.svg?raw'; import game from './icons/game.svg?raw'; import group from './icons/group.svg?raw'; import list from './icons/list.svg?raw'; +import instrument from './icons/instrument.svg?raw'; import joint from './icons/joint.svg?raw'; import palette from './icons/palette.svg?raw'; import particles from './icons/particles.svg?raw'; @@ -45,6 +46,7 @@ export const ASSET_ICON_SVG: Record = { container, curve, font, + instrument, joint, palette, particles, diff --git a/src/editor/panels/icons/instrument.svg b/src/editor/panels/icons/instrument.svg new file mode 100644 index 00000000..7e21d339 --- /dev/null +++ b/src/editor/panels/icons/instrument.svg @@ -0,0 +1,7 @@ + + + + + + + diff --git a/src/editor/state.ts b/src/editor/state.ts index 40eecea5..18a065d7 100644 --- a/src/editor/state.ts +++ b/src/editor/state.ts @@ -815,6 +815,11 @@ export class EditorState { return this.selectedGraphId('soundfx'); } + /** The instrument the selection refers to: the asset, or one of its nodes. */ + get selectedInstrumentId(): string | null { + return this.selectedGraphId('instrument'); + } + private selectedGraphId(type: GraphAsset['type']): string | null { const s = this.selection; if (!s) return null; diff --git a/src/runtime/systems/Audio.ts b/src/runtime/systems/Audio.ts index 9c976548..52f89df7 100644 --- a/src/runtime/systems/Audio.ts +++ b/src/runtime/systems/Audio.ts @@ -1,5 +1,5 @@ import type { EaseShare } from '../../core/model/animation'; -import type { FxParams, FxVoice } from '../../core/soundfx/nodes'; +import type { FxParams, FxPlan, FxVoice } from '../../core/soundfx/nodes'; /** * What the runtime needs of the outside world to make a noise – the counterpart of `Input` on the @@ -19,6 +19,25 @@ export interface AudioSink { * Returns the number of the voice, never used twice, or 0 where nothing is going to play. */ play(file: string, volume: number, loop: boolean, owner: string, fx?: FxVoice): number; + /** + * Starts one note of an instrument (#331), as `play` starts a sound: `layers` are what the note is + * made of, played side by side as one voice, each way reading the file of its own source. `name` + * stands for the instrument. `params` fill in the parameters of the plans, its velocity among them. + * A `duration` above 0 releases it that many seconds after it starts; 0 holds it. + */ + playNote( + name: string, + layers: readonly FxPlan[], + params: FxParams, + volume: number, + owner: string, + duration: number, + ): number; + /** + * The end of a note (#331): its ways that end with the note fade out, those that play through + * go on. Nothing happens for a voice that is no note or not there. + */ + release(voice: number): void; /** * Hands new values to the sound FX parameters of one voice, by the number `play` gave it (#315). * A name its FX does not have is left out, one not given keeps its value. A voice that has ended @@ -51,6 +70,8 @@ export interface AudioSink { export const SILENT_AUDIO: AudioSink = { play: () => 0, + playNote: () => 0, + release: () => {}, tune: () => {}, length: () => 0, tail: () => 0, diff --git a/src/runtime/systems/behaviorContext.ts b/src/runtime/systems/behaviorContext.ts index daed6579..4dc99caa 100644 --- a/src/runtime/systems/behaviorContext.ts +++ b/src/runtime/systems/behaviorContext.ts @@ -16,6 +16,9 @@ import { hasRenderLayer } from '../../core/properties/PropertySchema'; import { resolveAssetProps } from '../../core/properties/resolve'; import { loopsOf, volumeOf } from '../../core/properties/sounds'; import { soundFxBuildFor } from '../../core/soundfx/compile'; +import { instrumentBuildFor, instrumentSourcesOf } from '../../core/instrument/compile'; +import { VELOCITY_PARAM } from '../../core/instrument/nodes'; +import { heldNote } from '../../core/instrument/notes'; import { DRY_VOICE, type FxParams } from '../../core/soundfx/nodes'; import type { Entity } from '../Entity'; import { PhysicsSystem } from '../physics/PhysicsSystem'; @@ -166,6 +169,8 @@ export function behaviorContext( }, playSound: (assetId, volume, loop, detached, fx, fxParams) => playSound(run, assetId, volume, loop, detached ? '' : entity.id, fx, fxParams), + playNote: (instrumentId, note, velocity, volume, duration, detached, params) => + playNote(run, instrumentId, { note, velocity, volume, duration }, detached ? '' : entity.id, params), soundLength: (assetId) => { const file = soundFileOf(project, assetId); return file ? run.audio.length(file) : 0; @@ -251,6 +256,34 @@ function playSound( ); } +/** + * One note of an instrument (#331), as the sink understands it: the layers the instrument makes of + * it, the velocity as loudness and as the value of its `velocity`, and the voice's volume held to + * 0..1 like a sound's. An instrument that plays nothing for that note plays nothing at all. + */ +export function playNote( + { project, audio }: Pick, + instrumentId: string, + struck: { note: number; velocity: number; volume: number; duration: number }, + owner: string, + params: FxParams, +): number { + const asset = findAssetOf(project, instrumentId, 'instrument'); + if (!asset) return 0; + const { layers } = instrumentBuildFor(asset, heldNote(struck.note), instrumentSourcesOf(project)); + if (layers.length === 0) return 0; + const velocity = Number.isFinite(struck.velocity) ? Math.max(0, Math.min(1, struck.velocity)) : 1; + const level = Math.max(0, Math.min(1, struck.volume * velocity)); + const duration = Number.isFinite(struck.duration) ? Math.max(0, struck.duration) : 0; + const values = { ...params, [VELOCITY_PARAM]: velocity }; + return audio.playNote(instrumentVoiceName(instrumentId), layers, values, level, owner, duration); +} + +/** What a note's voice is named by, where a sound's is named by its file. */ +export function instrumentVoiceName(instrumentId: string): string { + return `instrument:${instrumentId}`; +} + /** Where a ray of that length ends; where it has no direction, where it starts. */ function pointAlong(from: Vec2, direction: Vec2, length: number): Vec2 { const d = Math.hypot(direction.x, direction.y); diff --git a/src/ui/Theme.ts b/src/ui/Theme.ts index a3d939a9..36913d00 100644 --- a/src/ui/Theme.ts +++ b/src/ui/Theme.ts @@ -81,6 +81,7 @@ export const Theme = { portPicture: 0x9b8cff, portAudio: 0xf0c060, portFxParams: 0xd0629b, + portTrigger: 0xb5e853, portAny: 0xbdbdbd, /** Ring around the port a running edge drag would connect to. */ portTarget: 0xffffff, diff --git a/src/viewport/SoundLibrary.ts b/src/viewport/SoundLibrary.ts index ecba8e1e..4322f591 100644 --- a/src/viewport/SoundLibrary.ts +++ b/src/viewport/SoundLibrary.ts @@ -48,22 +48,28 @@ interface Source { stop(): void; ended(then: () => void): void; fade: GainNode | null; + /** Ends with the note (#331): a release fades it out, where everything else plays on. */ + gate: boolean; } /** * One sound in flight: its sources, its own gain, the file it came from and who is making it. A * voice has a source per way through its sound FX (#316), started together and as long as each - * other. Its knobs are in the order `levelsOf` lists the numbers of its chain. + * other. A note of an instrument (#331) is one voice too, its layers side by side, each way with a + * file of its own. Its knobs are in the order `levelsOf` lists the numbers of its chains. */ interface Voice { /** What `play` returned for it, the handle a `tune` reaches it by. */ id: number; + /** The file it plays, or for a note what `playNote` was given to name its instrument by. */ file: string; /** The behavior this voice belongs to; `''` for one nothing owns (see `AudioSink.play`). */ owner: string; sources: Source[]; + /** Calls the end of one source in, once however it comes: by itself or by a release. */ + finish: ((source: Source) => void) | null; gain: GainNode; - fx: FxVoice; + fx: Layered; knobs: Knob[]; /** Whether its FX can ring on after the sources end (#324), and whether that is all it does now. */ rings: boolean; @@ -72,11 +78,26 @@ interface Voice { quiet: (() => void) | null; } -/** What one voice is wired from: the file, and what has been made for it so far. */ +/** + * The plans of a voice and the values of their parameters: one plan for a sound played through its + * FX, one per layer for a note (#331). + */ +interface Layered { + layers: readonly FxPlan[]; + params: FxParams; +} + +/** One way of a chain: what a source plays. */ +type Way = Exclude; + +/** + * What one voice is wired from: its buffers by file, the file and loop a way without a source of + * its own plays (the sound a `play` was given), and what has been made for it so far. + */ interface Wiring { context: AudioContext; + buffers: ReadonlyMap; file: string; - buffer: AudioBuffer; loop: boolean; rings: boolean; sources: Source[]; @@ -85,16 +106,32 @@ interface Wiring { stages: number; } +/** Everything a voice is started with, the same whether it waits on its bytes first or not. */ +interface Start { + id: number; + file: string; + /** The files its ways read, every one of them loaded before it starts. */ + files: readonly string[]; + volume: number; + loop: boolean; + owner: string; + fx: Layered; + /** Seconds until a note is released (#331); 0 holds it until it is released or stopped. */ + duration: number; +} + /** * A play still waiting on its bytes: what a quick `tune` in the meantime writes into, and the timed * ones, which have to start with the voice rather than run out before it is heard. */ interface Pending { id: number; - fx: FxVoice; + fx: Layered; timed: { params: FxParams; time: number; ease: EaseShare }[]; /** Stopped by its number before it started. */ stopped: boolean; + /** A note released before it started: it starts and is released at once, so a one-shot is heard. */ + released: boolean; } /** @@ -159,29 +196,82 @@ export class SoundLibrary implements AudioSink { */ play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE): number { if (file === '' || this.failed.has(file)) return 0; + const layered = { layers: [fx.plan], params: fx.params }; + return this.begin({ id: 0, file, files: [file], volume, loop, owner, fx: layered, duration: 0 }); + } + + /** + * One note of an instrument (#331): its layers played side by side as one voice, each way with its + * own file. `name` stands for the instrument where a file would stand – what `levels` finds it by. + * A `duration` above 0 releases it that many seconds after it starts. + */ + playNote( + name: string, + layers: readonly FxPlan[], + params: FxParams, + volume: number, + owner = '', + duration = 0, + ): number { + const files = [...new Set(layers.flatMap(filesOf))].filter( + (file) => file !== '' && !this.failed.has(file), + ); + if (files.length === 0) return 0; + const fx = { layers, params }; + return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration }); + } + + /** + * The end of a note (#331): the ways that end with it fade out, the others play on, and the voice + * goes once they have all ended – or rings on first where it has echoes. + */ + release(id: number): void { + const waiting = this.pending.find((other) => other.id === id); + if (waiting) waiting.released = true; + const voice = this.voices.find((other) => other.id === id); + if (!voice) return; + const context = this.audio(); + const gated = voice.sources.filter((source) => source.gate); + for (const source of gated) if (source.fade) fadeOut(source.fade.gain, context.currentTime); + if (gated.length === 0) return; + this.after(context.currentTime + FADE, () => { + for (const source of gated) { + source.stop(); + voice.finish?.(source); + } + }); + } + + private begin(start: Start): number { const id = this.nextId++; - const buffer = this.buffers.get(file); - const pitching = pitches(fx.plan) ? this.pitchWorklet() : null; - if (buffer && (!pitching || this.canPitch !== null)) { - this.start(id, file, buffer, volume, loop, owner, fx); + const asked: Start = { ...start, id }; + const loaded = start.files.every((file) => this.buffers.has(file)); + const pitching = start.fx.layers.some(pitches) ? this.pitchWorklet() : null; + if (loaded && (!pitching || this.canPitch !== null)) { + this.start(asked); return id; } - const askedAt = this.cancelsFor(file, owner); + const askedAt = this.cancelsFor(start.file, start.owner); const waiting: Pending = { id, - fx: { plan: fx.plan, params: { ...fx.params } }, + fx: { layers: start.fx.layers, params: { ...start.fx.params } }, timed: [], stopped: false, + released: false, }; this.pending.push(waiting); - void Promise.all([this.load(file), pitching]).then(([loaded]) => { - this.pending = this.pending.filter((other) => other !== waiting); - // A `Stop sound` – or the destruction of the behavior that asked – that happened while the - // bytes were in flight has to win, or a sound the run already ended would start afterwards. - if (!loaded || waiting.stopped || this.cancelsFor(file, owner) !== askedAt) return; - this.start(id, file, loaded, volume, loop, owner, waiting.fx); - for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease); - }); + void Promise.all([Promise.all(start.files.map((file) => this.load(file))), pitching]).then( + ([buffers]) => { + this.pending = this.pending.filter((other) => other !== waiting); + // A `Stop sound` – or the destruction of the behavior that asked – that happened while the + // bytes were in flight has to win, or a sound the run already ended would start afterwards. + if (buffers.every((buffer) => !buffer) || waiting.stopped) return; + if (this.cancelsFor(start.file, start.owner) !== askedAt) return; + this.start({ ...asked, fx: waiting.fx }); + for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease); + if (waiting.released) this.release(id); + }, + ); return id; } @@ -192,12 +282,12 @@ export class SoundLibrary implements AudioSink { // A quick change is where the voice starts; a timed one – and anything after it, so the order // holds – runs from the moment it is heard. if (time > 0 || waiting.timed.length > 0) waiting.timed.push({ params, time, ease }); - else waiting.fx = { plan: waiting.fx.plan, params: { ...waiting.fx.params, ...params } }; + else waiting.fx = { layers: waiting.fx.layers, params: { ...waiting.fx.params, ...params } }; return true; } const voice = this.voices.find((other) => other.id === id); if (!voice) return false; - voice.fx = { plan: voice.fx.plan, params: { ...voice.fx.params, ...params } }; + voice.fx = { layers: voice.fx.layers, params: { ...voice.fx.params, ...params } }; this.glide(voice, time, ease); // A longer feedback makes for a longer tail: it is counted again from where the change ends. if (voice.ringing) this.ringFor(voice, Math.max(0, time) + this.tailOf(voice.fx)); @@ -289,6 +379,14 @@ export class SoundLibrary implements AudioSink { .map((voice) => voice.knobs.map((knob) => knob.param?.value ?? knob.target)); } + /** + * Seconds on the audio clock, 0 before there is one (test hook): what a check times a voice by, + * since that clock and the page's need not run alike. + */ + clock(): number { + return this.context?.currentTime ?? 0; + } + /** Where the pitch worklet is served from, for a check that runs it in an offline context (test hook). */ pitchWorkletUrl(): string { return pitchWorklet; @@ -347,7 +445,7 @@ export class SoundLibrary implements AudioSink { this.stopped += going.filter((voice) => !(letRing && voice.ringing)).length; for (const voice of going) { if (letRing && voice.rings) { - if (!voice.ringing) this.release(voice); + if (!voice.ringing) this.ringOut(voice); continue; } this.cut(voice); @@ -369,7 +467,7 @@ export class SoundLibrary implements AudioSink { } /** Fades the sources of a voice with echoes out and lets the echoes ring on (#324). */ - private release(voice: Voice): void { + private ringOut(voice: Voice): void { voice.ringing = true; const context = this.audio(); for (const source of voice.sources) if (source.fade) fadeOut(source.fade.gain, context.currentTime); @@ -410,8 +508,12 @@ export class SoundLibrary implements AudioSink { this.voices = this.voices.filter((other) => other !== voice); } - private tailOf(fx: FxVoice): number { - return fxRings(fx.plan) ? fxTailOf(fxChainOf(fx.plan, fx.params)) : 0; + /** The longest tail of its layers: they are heard side by side. */ + private tailOf(fx: Layered): number { + return fx.layers.reduce( + (longest, plan) => (fxRings(plan) ? Math.max(longest, fxTailOf(fxChainOf(plan, fx.params))) : longest), + 0, + ); } /** @@ -436,26 +538,18 @@ export class SoundLibrary implements AudioSink { }; } - private start( - id: number, - file: string, - buffer: AudioBuffer, - volume: number, - loop: boolean, - owner: string, - fx: FxVoice, - ): void { + private start({ id, file, files, volume, loop, owner, fx, duration }: Start): void { const context = this.audio(); // A context starts suspended where the page has not been interacted with; without this the // voice would run silently and end as if it had played. if (context.state === 'suspended') void context.resume(); const gain = context.createGain(); gain.gain.value = Math.max(0, Math.min(1, volume)); - const rings = fxRings(fx.plan); + const rings = fx.layers.some(fxRings); const wiring: Wiring = { context, + buffers: this.buffers, file, - buffer, loop, rings, sources: [], @@ -463,21 +557,24 @@ export class SoundLibrary implements AudioSink { knobs: [], stages: 0, }; - this.wire(wiring, fxChainOf(fx.plan, fx.params)).connect(gain).connect(context.destination); + for (const plan of fx.layers) this.wire(wiring, fxChainOf(plan, fx.params)).connect(gain); + gain.connect(context.destination); const { sources } = wiring; this.last = { file, volume: gain.gain.value, loop, + layers: fx.layers.length, pitches: wiring.pitches, stages: wiring.stages, - duration: buffer.duration, + duration: Math.max(0, ...files.map((name) => this.buffers.get(name)?.duration ?? 0)), }; const voice: Voice = { id, file, owner, sources, + finish: null, gain, fx, knobs: wiring.knobs, @@ -485,14 +582,15 @@ export class SoundLibrary implements AudioSink { ringing: false, quiet: null, }; - let playing = sources.length; - for (const source of sources) - source.ended(() => { - if (--playing === 0) this.ended(voice); - }); + const going = new Set(sources); + voice.finish = (source) => { + if (going.delete(source) && going.size === 0) this.ended(voice); + }; + for (const source of sources) source.ended(() => voice.finish?.(source)); this.voices.push(voice); this.started += 1; for (const source of sources) source.start(); + if (duration > 0) this.after(context.currentTime + duration, () => this.release(id)); } /** @@ -513,7 +611,7 @@ export class SoundLibrary implements AudioSink { const shareB = gainNode(context, chain.mix); wayB.connect(shareB).connect(tail); knobs.push({ param: shareB.gain, target: chain.mix, even: 'value' }); - } else tail = this.source(wiring, chain.pitch, chain.shifted); + } else tail = this.source(wiring, chain); for (const stage of chain.stages) { const made = stageNodes(context, stage); tail.connect(made.input); @@ -527,21 +625,29 @@ export class SoundLibrary implements AudioSink { /** * A way's sound: the pitch worklet where the way has a Pitch, which keeps the length whatever the * pitch does (#317), and otherwise the buffer as it is. Without the worklet a pitched way plays at - * its own pitch rather than not at all. + * its own pitch rather than not at all. A way with a source of its own (#331) plays that file, at + * its volume, and can fade out by itself when its note ends. */ - private source(wiring: Wiring, pitch: number, shifted: boolean): AudioNode { - const sound = this.sound(wiring, pitch, shifted); - if (!wiring.rings) return sound; - const fade = gainNode(wiring.context, 1); + private source(wiring: Wiring, chain: Way): AudioNode { + const sound = this.sound(wiring, chain); + const own = chain.source; + if (!wiring.rings && !own) return sound; + const fade = gainNode(wiring.context, own?.volume ?? 1); wiring.sources.at(-1)!.fade = fade; return sound.connect(fade); } - private sound(wiring: Wiring, pitch: number, shifted: boolean): AudioNode { - const { context, file, buffer, loop } = wiring; + private sound(wiring: Wiring, { pitch, shifted, source: own }: Way): AudioNode { + const { context } = wiring; + const file = own?.file ?? wiring.file; + const loop = own ? own.loop : wiring.loop; + const gate = own?.gate ?? false; + // A file that could not be loaded leaves its way silent rather than the whole note. + const buffer = wiring.buffers.get(file) ?? context.createBuffer(1, 1, context.sampleRate); wiring.pitches.push(pitch); if (shifted && this.canPitch) { const options: PitchOptions = { file, step: buffer.sampleRate / context.sampleRate, loop }; + if (own && loop) Object.assign(options, loopIn(buffer, own.loopStart, own.loopEnd)); if (!this.sent.has(file)) { options.channels = Array.from({ length: buffer.numberOfChannels }, (_, c) => buffer.getChannelData(c), @@ -566,6 +672,7 @@ export class SoundLibrary implements AudioSink { }; }, fade: null, + gate, }); wiring.knobs.push({ param: node.parameters.get('pitch')!, target: pitch, even: 'value' }); return node; @@ -573,6 +680,10 @@ export class SoundLibrary implements AudioSink { const source = context.createBufferSource(); source.buffer = buffer; source.loop = loop; + if (own && loop) { + source.loopStart = own.loopStart; + source.loopEnd = own.loopEnd; + } wiring.sources.push({ start: () => source.start(), stop: () => source.stop(), @@ -580,6 +691,7 @@ export class SoundLibrary implements AudioSink { source.onended = then; }, fade: null, + gate, }); if (shifted) wiring.knobs.push({ param: null, target: pitch, even: 'value' }); return source; @@ -590,7 +702,8 @@ export class SoundLibrary implements AudioSink { * clock along `ease`, or else in a quick glide. */ private glide(voice: Voice, time: number, ease: EaseShare): void { - const levels = levelsOf(fxChainOf(voice.fx.plan, voice.fx.params)); + const levels: number[] = []; + for (const plan of voice.fx.layers) levelsOf(fxChainOf(plan, voice.fx.params), levels); const now = this.audio().currentTime; voice.knobs.forEach((knob, i) => { const level = levels[i]; @@ -634,6 +747,8 @@ interface LastVoice { file: string; volume: number; loop: boolean; + /** How many layers of a note it played side by side (#331); 1 for a sound. */ + layers: number; /** Semitones of each way (#316), and how many gain and filter nodes it passed. */ pitches: number[]; stages: number; @@ -646,6 +761,24 @@ function pitches(plan: FxPlan): boolean { return 'mix' in plan ? pitches(plan.a) || pitches(plan.b) : plan.pitch.length > 0; } +/** The files the ways of a plan read with a source of their own (#331). */ +function filesOf(plan: FxPlan): string[] { + if ('mix' in plan) return [...filesOf(plan.a), ...filesOf(plan.b)]; + return plan.source ? [plan.source.file] : []; +} + +/** + * Where a loop runs in the samples of a sound, as the worklet takes it: between `start` and `end` + * seconds, 0 at the end being the end of the sound. One that does not fit is the whole sound, as + * Web Audio has it for a buffer source. + */ +function loopIn(buffer: AudioBuffer, start: number, end: number): { loopStart: number; loopEnd: number } { + const last = buffer.length; + const from = Math.round(Math.max(0, start) * buffer.sampleRate); + const to = end > 0 ? Math.min(last, Math.round(end * buffer.sampleRate)) : last; + return from < to ? { loopStart: from, loopEnd: to } : { loopStart: 0, loopEnd: last }; +} + function gainNode(context: BaseAudioContext, level: number): GainNode { const gain = context.createGain(); gain.gain.value = level; diff --git a/src/viewport/pitchWorklet.ts b/src/viewport/pitchWorklet.ts index 3499953e..1fcb0fcf 100644 --- a/src/viewport/pitchWorklet.ts +++ b/src/viewport/pitchWorklet.ts @@ -22,6 +22,9 @@ export interface PitchOptions { /** The sound's rate over the context's. */ step: number; loop: boolean; + /** Where a loop runs, in samples of the sound (#331); without them it is the whole sound. */ + loopStart?: number; + loopEnd?: number; } /** Every sound a voice has been played from, for the next voice of it – processors share this scope. */ @@ -31,10 +34,12 @@ class PitchProcessor extends AudioWorkletProcessor { private readonly player: PitchPlayer; private stopped = false; - constructor({ processorOptions: { file, channels, step, loop } }: { processorOptions: PitchOptions }) { + constructor({ processorOptions }: { processorOptions: PitchOptions }) { super(); + const { file, channels, step, loop, loopStart, loopEnd } = processorOptions; if (channels) sounds.set(file, channels); - this.player = new PitchPlayer(sounds.get(file) ?? [], step, sampleRate, loop); + const region = loop && loopEnd !== undefined ? { start: loopStart ?? 0, end: loopEnd } : loop; + this.player = new PitchPlayer(sounds.get(file) ?? [], step, sampleRate, region); this.port.onmessage = () => { this.stopped = true; }; diff --git a/tests/core/graph.test.ts b/tests/core/graph.test.ts index f6c296f4..5f9ed09a 100644 --- a/tests/core/graph.test.ts +++ b/tests/core/graph.test.ts @@ -79,10 +79,10 @@ describe('node registry', () => { if (def.category === 'event') expect(def.inputs.filter((p) => p.type === 'exec')).toHaveLength(0); // A state is a name in a machine; its ports only anchor transitions, so it never runs. else if (def.category === 'state') expect(def.run).toBeUndefined(); - // A shader node is compiled to GLSL, a particle node to a function and a sound FX node to a - // stage of a voice: none is ever run. + // A shader node is compiled to GLSL, a particle node to a function, a sound FX node to a + // stage of a voice and an instrument's own node to part of a note: none is ever run. else if (def.category === 'shader' || def.category === 'particles' || def.category === 'sound') - expect('emit' in def).toBe(true); + expect('emit' in def || 'instrument' in def).toBe(true); // The end of a block runs as part of the node that opens it (`Find` … `Match`). else if (defs.some((d) => d.opens?.type === def.type)) expect(def.run).toBeUndefined(); // Everything else does one of the two: a step (`run`), or an answer whenever asked (`read`). diff --git a/tests/core/instrument.test.ts b/tests/core/instrument.test.ts new file mode 100644 index 00000000..10114566 --- /dev/null +++ b/tests/core/instrument.test.ts @@ -0,0 +1,485 @@ +/** + * An instrument (#331) is a graph that makes a note a sound: Notes nodes bring the note in, Samples + * play sounds for it, the effects of a sound FX shape them and the Output sums them. It is compiled + * once per note; a trigger is on only for the notes it stands for. + */ +import { describe, expect, it } from 'vitest'; +import { CreateAssetCommand, CreateSoundCommand } from '../../src/core/commands/assetCommands'; +import { ConnectCommand } from '../../src/core/commands/graphCommands'; +import { + AddPortCommand, + RenamePortCommand, + SetPortDefaultCommand, +} from '../../src/core/commands/portCommands'; +import { callProblem, resolveDef, soundFxParamsMismatch } from '../../src/core/graph'; +import { createEmptyGraph, portsCompatible } from '../../src/core/graph/Graph'; +import { + compileInstrument, + instrumentBuildFor, + instrumentSourcesOf, +} from '../../src/core/instrument/compile'; +import { VELOCITY_PARAM } from '../../src/core/instrument/nodes'; +import { heldNote, noteFrequency, noteName } from '../../src/core/instrument/notes'; +import type { BehaviorAsset, InstrumentAsset, SoundFxAsset } from '../../src/core/model/Asset'; +import { isPlaceableType } from '../../src/core/model/AssetTypes'; +import { graphLookup } from '../../src/core/model/graphLookup'; +import { fxChainOf, type FxPlan } from '../../src/core/soundfx/nodes'; +import { paletteEntries } from '../../src/editor/graph/paletteEntries'; +import { keyboardLayout, noteAt, trackerNote } from '../../src/editor/instrument/keys'; +import type { AudioSink } from '../../src/runtime/systems/Audio'; +import { playNote } from '../../src/runtime/systems/behaviorContext'; +import { graphEditor, makeProject, type ProjectFixture, roundTrip } from '../helpers'; + +const KICK = `${'a1'.repeat(32)}.wav`; +const SNARE = `${'b2'.repeat(32)}.wav`; + +function makeInstrument(f: ProjectFixture = makeProject()) { + const command = new CreateAssetCommand('instrument', 'Piano'); + f.stack.execute(command); + const instrument = command.asset as InstrumentAsset; + const sound = (name: string, file: string) => { + const made = new CreateSoundCommand(name, file); + f.stack.execute(made); + return made.asset; + }; + return { ...f, instrument, sound, edit: graphEditor(f, instrument as unknown as BehaviorAsset) }; +} + +/** An instrument with nothing in it yet, for a test that builds its own graph. */ +function emptyInstrument() { + const f = makeInstrument(); + f.instrument.graph = createEmptyGraph(); + return f; +} + +const nodeOf = (f: { instrument: InstrumentAsset }, type: string) => + f.instrument.graph.nodes.find((n) => n.type === type)!; + +const compile = (f: ProjectFixture & { instrument: InstrumentAsset }, note: number) => + compileInstrument(f.instrument.graph, note, instrumentSourcesOf(f.project)); + +const way = (plan: FxPlan | undefined) => (plan && 'pitch' in plan ? plan : undefined); + +describe('notes', () => { + it('count as MIDI does: C4 is 60, A4 is 440 Hz, the lowest is C-1, sharps only', () => { + expect(noteName(60)).toBe('C4'); + expect(noteName(61)).toBe('C#4'); + expect(noteName(0)).toBe('C-1'); + expect(noteName(127)).toBe('G9'); + expect(noteFrequency(69)).toBe(440); + expect(noteFrequency(81)).toBeCloseTo(880, 9); + expect(heldNote(200)).toBe(127); + expect(heldNote(Number.NaN)).toBe(60); + }); +}); + +describe('the instrument asset', () => { + it('is not placeable and starts as Notes into a Sample into the Output', () => { + expect(isPlaceableType('instrument')).toBe(false); + const f = makeInstrument(); + expect(f.instrument.graph.nodes.map((n) => n.type)).toEqual([ + 'InstrumentNotes', + 'InstrumentSample', + 'InstrumentOutput', + ]); + expect(f.instrument.graph.edges.map((e) => `${e.from.port}>${e.to.port}`)).toEqual([ + 'any>trigger', + 'out>in', + ]); + // Its Sample has no sound yet, and says so. + expect(compile(f, 60)).toMatchObject({ layers: [], reason: 'No sound picked' }); + }); + + it('survives a round trip, keeping its own nodes and the effects it shares with a sound FX', () => { + const f = makeInstrument(); + f.edit.addNode('SoundFxGain', { gain: -6 }); + const back = roundTrip(f.project).assets.find((a) => a.id === f.instrument.id) as InstrumentAsset; + expect(back.graph.nodes.map((n) => n.type)).toEqual([ + 'InstrumentNotes', + 'InstrumentSample', + 'InstrumentOutput', + 'SoundFxGain', + ]); + expect(back.graph.edges).toHaveLength(2); + }); + + it('offers its own nodes and the effects, but no Input and no second Output', () => { + const f = makeInstrument(); + const offered = paletteEntries(f.project, f.instrument).map((e) => e.id); + expect(offered).toEqual( + expect.arrayContaining(['InstrumentNotes', 'InstrumentSample', 'InstrumentParameters', 'SoundFxDelay']), + ); + expect(offered).not.toContain('InstrumentOutput'); + expect(offered).not.toContain('SoundFxInput'); + expect(offered).not.toContain('SoundFxOutput'); + }); +}); + +describe('compiling a note', () => { + it('plays a Sample at its root as it is, and other notes shifted by the distance', () => { + const f = makeInstrument(); + const piano = f.sound('Piano C4', KICK); + nodeOf(f, 'InstrumentSample').params.sound = piano.id; + const root = compile(f, 60); + expect(root.reason).toBe(''); + expect(root.layers).toHaveLength(1); + expect(way(root.layers[0])).toMatchObject({ + pitch: [], + source: { file: KICK, volume: 1, loop: false, gate: true }, + }); + expect(way(compile(f, 64).layers[0])?.pitch).toEqual([{ value: 4 }]); + expect(way(compile(f, 67).layers[0])?.pitch).toEqual([{ value: 7 }]); + }); + + it('plays nothing for a note outside the range of its Notes node – silence, not a problem', () => { + const f = makeInstrument(); + nodeOf(f, 'InstrumentSample').params.sound = f.sound('Piano', KICK).id; + expect(compile(f, 72)).toEqual({ layers: [], problems: {}, reason: '' }); + expect(compile(f, 59).layers).toEqual([]); + expect(compile(f, 71).layers).toHaveLength(1); + }); + + it('plays a Sample that does not follow the note the same for every note', () => { + const f = makeInstrument(); + Object.assign(nodeOf(f, 'InstrumentSample').params, { + sound: f.sound('Kick', KICK).id, + followNote: false, + }); + expect(way(compile(f, 67).layers[0])?.pitch).toEqual([]); + }); + + it('a drum kit: one Notes node with single notes, a Sample on each, both into the Output', () => { + const f = emptyInstrument(); + const notes = f.edit.addNode('InstrumentNotes', { + from: '36', + to: '47', + singleNotes: true, + name36: 'Kick', + }); + const kick = f.edit.addNode('InstrumentSample', { sound: f.sound('Kick', KICK).id, followNote: false }); + const snare = f.edit.addNode('InstrumentSample', { sound: f.sound('Snare', SNARE).id, mode: 'oneshot' }); + const output = f.edit.addNode('InstrumentOutput'); + f.edit.connect(notes, 'n36', kick, 'trigger'); + f.edit.connect(notes, 'n38', snare, 'trigger'); + f.edit.connect(kick, 'out', output, 'in'); + f.edit.connect(snare, 'out', output, 'in'); + // The Output takes both: a second edge into it does not replace the first. + expect(f.instrument.graph.edges.filter((e) => e.to.node === output.id)).toHaveLength(2); + const files = (note: number) => compile(f, note).layers.map((plan) => way(plan)?.source?.file); + expect(files(36)).toEqual([KICK]); + expect(files(38)).toEqual([SNARE]); + expect(files(37)).toEqual([]); + expect(way(compile(f, 38).layers[0])?.source?.gate).toBe(false); + // The node names its output after the note's own name, the others after the note. + const def = resolveDef(notes, f.instrument.graph, graphLookup(f.project))!; + expect(def.outputs.map((p) => p.label).slice(0, 7)).toEqual([ + 'any', + 'note', + 'frequency', + 'velocity', + 'Kick', + 'C#2', + 'D2', + ]); + expect(def.outputs).toHaveLength(4 + 12); + }); + + it('two Notes nodes over one Sample: a trigger takes more than one edge, either one plays it', () => { + const f = emptyInstrument(); + const low = f.edit.addNode('InstrumentNotes', { from: '48', to: '52' }); + const high = f.edit.addNode('InstrumentNotes', { from: '70', to: '72' }); + const sample = f.edit.addNode('InstrumentSample', { sound: f.sound('Piano', KICK).id }); + const output = f.edit.addNode('InstrumentOutput'); + f.edit.wire([ + [low, 'any', sample, 'trigger'], + [high, 'any', sample, 'trigger'], + [sample, 'out', output, 'in'], + ]); + expect(compile(f, 50).layers).toHaveLength(1); + expect(compile(f, 71).layers).toHaveLength(1); + expect(compile(f, 60).layers).toHaveLength(0); + }); + + it('a Mix with one silent side is its other side', () => { + const f = emptyInstrument(); + const notes = f.edit.addNode('InstrumentNotes', { from: '60', to: '62', singleNotes: true }); + const a = f.edit.addNode('InstrumentSample', { sound: f.sound('A', KICK).id }); + const b = f.edit.addNode('InstrumentSample', { sound: f.sound('B', SNARE).id }); + const mix = f.edit.addNode('SoundFxMix', { mix: 0.5 }); + const output = f.edit.addNode('InstrumentOutput'); + f.edit.wire([ + [notes, 'n60', a, 'trigger'], + [notes, 'any', b, 'trigger'], + [a, 'out', mix, 'a'], + [b, 'out', mix, 'b'], + [mix, 'out', output, 'in'], + ]); + const both = compile(f, 60).layers[0]!; + expect('mix' in both).toBe(true); + const only = compile(f, 61).layers[0]; + expect(way(only)?.source?.file).toBe(SNARE); + }); + + it('leaves velocity and the parameters open for the voice, and fills in note and frequency', () => { + const f = makeInstrument(); + nodeOf(f, 'InstrumentSample').params.sound = f.sound('Piano', KICK).id; + const params = f.edit.addNode('InstrumentParameters'); + f.stack.execute(new AddPortCommand(f.instrument.id, params.id, { key: 'level', kind: 'number' })); + f.stack.execute(new SetPortDefaultCommand(f.instrument.id, params.id, 'level', -3)); + const sample = nodeOf(f, 'InstrumentSample'); + const output = nodeOf(f, 'InstrumentOutput'); + const notes = nodeOf(f, 'InstrumentNotes'); + f.instrument.graph.edges = f.instrument.graph.edges.filter((e) => e.to.node !== output.id); + const byVelocity = f.edit.addNode('SoundFxGain'); + const byParam = f.edit.addNode('SoundFxGain'); + const byNote = f.edit.addNode('SoundFxEq'); + f.stack.execute( + new ConnectCommand( + f.instrument.id, + { node: sample.id, port: 'out' }, + { node: byVelocity.id, port: 'in' }, + ), + ); + f.edit.wire([ + [notes, 'velocity', byVelocity, 'gain'], + [byVelocity, 'out', byParam, 'in'], + [params, 'level', byParam, 'gain'], + [byParam, 'out', byNote, 'in'], + [notes, 'note', byNote, 'high'], + [byNote, 'out', output, 'in'], + ]); + const plan = compile(f, 62).layers[0]!; + expect(plan.stages).toEqual([ + { kind: 'gain', db: { value: 1, param: VELOCITY_PARAM, min: -120, max: 24 } }, + { kind: 'gain', db: { value: -3, param: 'level', min: -120, max: 24 } }, + // The note is a number by now: 62, held to the band's row. + { kind: 'filter', filter: 'highshelf', frequency: 12000, gain: { value: 24 }, q: 1 }, + ]); + const chain = fxChainOf(plan, { [VELOCITY_PARAM]: -6, level: -9 }); + expect(chain.stages.map((s) => ('gain' in s ? s.gain : 0))).toEqual([ + 10 ** (-6 / 20), + 10 ** (-9 / 20), + 24, + ]); + }); + + it('sends a sound through a sound FX, its parameters bound to the inputs of the node', () => { + const f = makeInstrument(); + nodeOf(f, 'InstrumentSample').params.sound = f.sound('Piano', KICK).id; + // The FX: Input, a Gain fed by its parameter "depth" (default -6), a Pitch of 12, Output. + const made = new CreateAssetCommand('soundfx', 'Cave'); + f.stack.execute(made); + const fx = made.asset as SoundFxAsset; + const fxEdit = graphEditor(f, fx as unknown as BehaviorAsset); + const input = fxEdit.addNode('SoundFxInput'); + f.stack.execute(new AddPortCommand(fx.id, input.id, { key: 'depth', kind: 'number' })); + f.stack.execute(new SetPortDefaultCommand(fx.id, input.id, 'depth', -6)); + const gain = fxEdit.addNode('SoundFxGain'); + const pitch = fxEdit.addNode('SoundFxPitch', { semitones: 12 }); + const fxOut = fxEdit.addNode('SoundFxOutput'); + fxEdit.wire([ + [input, 'out', gain, 'in'], + [input, 'depth', gain, 'gain'], + [gain, 'out', pitch, 'in'], + [pitch, 'out', fxOut, 'in'], + ]); + // In the instrument: Sample → Sound FX → Output. + const sample = nodeOf(f, 'InstrumentSample'); + const output = nodeOf(f, 'InstrumentOutput'); + f.instrument.graph.edges = f.instrument.graph.edges.filter((e) => e.to.node !== output.id); + const through = f.edit.addNode('InstrumentSoundFx', { fx: fx.id }); + f.edit.wire([ + [sample, 'out', through, 'in'], + [through, 'out', output, 'in'], + ]); + const lookup = graphLookup(f.project); + const def = resolveDef(through, f.instrument.graph, lookup)!; + expect(def.label).toBe('Cave'); + expect(def.inputs.map((p) => [p.key, p.default])).toEqual([ + ['in', undefined], + ['depth', -6], + ]); + + // Left open, the parameter is the FX's default. + const open = way(compile(f, 64).layers[0])!; + expect(open.pitch).toEqual([{ value: 4 }, { value: 12 }]); + expect(open.stages).toEqual([{ kind: 'gain', db: { value: -6 } }]); + expect(open.source?.file).toBe(KICK); + + // Typed in at the node, it is a number – held to the FX's row. + through.params.depth = -500; + expect(way(compile(f, 64).layers[0])!.stages).toEqual([{ kind: 'gain', db: { value: -120 } }]); + + // Fed from the instrument's own parameter, it is that one. + const params = f.edit.addNode('InstrumentParameters'); + f.stack.execute(new AddPortCommand(f.instrument.id, params.id, { key: 'dark', kind: 'number' })); + f.edit.connect(params, 'dark', through, 'depth'); + expect(way(compile(f, 64).layers[0])!.stages).toEqual([ + { kind: 'gain', db: { value: 0, param: 'dark', min: -120, max: 24 } }, + ]); + + // Renaming the FX's parameter takes the edge into the node along. + f.stack.execute(new RenamePortCommand(fx.id, input.id, 'depth', 'floor')); + expect(f.instrument.graph.edges.some((e) => e.to.node === through.id && e.to.port === 'floor')).toBe( + true, + ); + expect(callProblem(through, lookup, f.instrument.id)).toBeNull(); + through.params.fx = ''; + expect(callProblem(through, lookup, f.instrument.id)).toBe('No sound FX picked'); + }); + + it('says what keeps it from compiling: a Sample nothing triggers, a sound that is gone', () => { + const f = makeInstrument(); + const sample = nodeOf(f, 'InstrumentSample'); + sample.params.sound = 'gone'; + expect(compile(f, 60).problems[sample.id]).toBe('That sound is gone'); + sample.params.sound = f.sound('Piano', KICK).id; + f.instrument.graph.edges = f.instrument.graph.edges.filter((e) => e.to.node !== sample.id); + expect(compile(f, 60).problems[sample.id]).toBe('Nothing triggers it: connect it to a Notes node'); + }); + + it('is compiled again when a sound it plays changes, not on every note', () => { + const f = makeInstrument(); + const piano = f.sound('Piano', KICK); + nodeOf(f, 'InstrumentSample').params.sound = piano.id; + const sources = instrumentSourcesOf(f.project); + const first = instrumentBuildFor(f.instrument, 60, sources); + expect(instrumentBuildFor(f.instrument, 60, sources)).toBe(first); + piano.file = SNARE; + const again = instrumentBuildFor(f.instrument, 60, instrumentSourcesOf(f.project)); + expect(way(again.layers[0])?.source?.file).toBe(SNARE); + }); +}); + +describe('ports of an instrument', () => { + it('a trigger fits only a trigger', () => { + expect(portsCompatible('trigger', 'trigger')).toBe(true); + expect(portsCompatible('trigger', 'boolean')).toBe(false); + expect(portsCompatible('trigger', 'any')).toBe(false); + expect(portsCompatible('audio', 'trigger')).toBe(false); + }); + + it('a Sample keeps every trigger connected to it, a Gain still only one sound', () => { + const f = makeInstrument(); + const notes = nodeOf(f, 'InstrumentNotes'); + notes.params.singleNotes = true; + const sample = nodeOf(f, 'InstrumentSample'); + f.edit.connect(notes, 'n61', sample, 'trigger'); + expect(f.instrument.graph.edges.filter((e) => e.to.node === sample.id)).toHaveLength(2); + const gain = f.edit.addNode('SoundFxGain'); + f.edit.connect(sample, 'out', gain, 'in'); + const other = f.edit.addNode('InstrumentSample'); + f.edit.connect(other, 'out', gain, 'in'); + expect(f.instrument.graph.edges.filter((e) => e.to.node === gain.id)).toHaveLength(1); + }); +}); + +describe('Play note', () => { + function sink() { + const notes: { + name: string; + layers: readonly FxPlan[]; + params: object; + volume: number; + duration: number; + }[] = []; + const audio = { + playNote: ( + name: string, + layers: readonly FxPlan[], + params: object, + volume: number, + _o: string, + duration: number, + ) => notes.push({ name, layers, params, volume, duration }), + } as unknown as AudioSink; + return { audio, notes }; + } + + it('hands the sink the layers of the note, its velocity as loudness and as a value, held to 0..1', () => { + const f = makeInstrument(); + nodeOf(f, 'InstrumentSample').params.sound = f.sound('Piano', KICK).id; + const { audio, notes } = sink(); + const struck = { note: 64, velocity: 0.5, volume: 0.8, duration: 0.25 }; + expect(playNote({ project: f.project, audio }, f.instrument.id, struck, '', { cut: 1 })).toBe(1); + expect(notes[0]).toMatchObject({ + name: `instrument:${f.instrument.id}`, + params: { cut: 1, [VELOCITY_PARAM]: 0.5 }, + volume: 0.4, + duration: 0.25, + }); + expect(way(notes[0]!.layers[0])?.pitch).toEqual([{ value: 4 }]); + playNote({ project: f.project, audio }, f.instrument.id, { ...struck, velocity: 3, volume: 2 }, '', {}); + expect(notes[1]).toMatchObject({ volume: 1, params: { [VELOCITY_PARAM]: 1 } }); + }); + + it('plays nothing for a note the instrument has nothing for, or an instrument that is gone', () => { + const f = makeInstrument(); + nodeOf(f, 'InstrumentSample').params.sound = f.sound('Piano', KICK).id; + const { audio, notes } = sink(); + const struck = { note: 30, velocity: 1, volume: 1, duration: 0 }; + expect(playNote({ project: f.project, audio }, f.instrument.id, struck, '', {})).toBe(0); + expect(playNote({ project: f.project, audio }, 'gone', { ...struck, note: 60 }, '', {})).toBe(0); + expect(notes).toEqual([]); + }); + + it('warns where its parameters come from a node naming another instrument', () => { + const f = makeInstrument(); + const behavior = new CreateAssetCommand('behavior', 'B'); + f.stack.execute(behavior); + const asset = behavior.asset as BehaviorAsset; + const edit = graphEditor(f, asset); + const play = edit.addNode('PlayNote', { instrument: f.instrument.id }); + const values = edit.addNode('InstrumentParameterValues', { soundFxAsset: 'other' }); + edit.connect(values, 'params', play, 'fxParams'); + expect(soundFxParamsMismatch(play, asset.graph)).toBe('The parameters are for another instrument'); + values.params.soundFxAsset = f.instrument.id; + expect(soundFxParamsMismatch(play, asset.graph)).toBeNull(); + expect(callProblem(values, graphLookup(f.project))).toBeNull(); + }); + + it('an Instrument parameters node offers what the Parameters node declares, and follows a rename', () => { + const f = makeInstrument(); + const params = f.edit.addNode('InstrumentParameters'); + f.stack.execute(new AddPortCommand(f.instrument.id, params.id, { key: 'cut', kind: 'number' })); + f.stack.execute(new SetPortDefaultCommand(f.instrument.id, params.id, 'cut', 0.7)); + expect(graphLookup(f.project).soundFx(f.instrument.id)?.params).toEqual([ + { key: 'cut', kind: 'number', default: 0.7 }, + ]); + const behavior = new CreateAssetCommand('behavior', 'B'); + f.stack.execute(behavior); + const asset = behavior.asset as BehaviorAsset; + const values = graphEditor(f, asset).addNode('InstrumentParameterValues', { + soundFxAsset: f.instrument.id, + }); + f.stack.execute(new AddPortCommand(asset.id, values.id, { key: 'cut', kind: 'number' })); + const def = resolveDef(values, asset.graph, graphLookup(f.project))!; + expect(def.inputs.map((p) => [p.key, p.default])).toEqual([['cut', 0.7]]); + f.stack.execute(new RenamePortCommand(f.instrument.id, params.id, 'cut', 'bright')); + expect(values.ports?.map((p) => p.key)).toEqual(['bright']); + }); +}); + +describe('the keys of the tab', () => { + it('lays out all 128 notes, black over white, and finds the note under a point', () => { + const keys = keyboardLayout(750, 60); + expect(keys).toHaveLength(128); + expect(keys.filter((k) => k.black)).toHaveLength(53); + const c4 = keys.find((k) => k.note === 60)!; + const cSharp4 = keys.find((k) => k.note === 61)!; + expect(noteAt(keys, c4.x + 1, 55)).toBe(60); + // Up top, the black key lies over the edge of the white one. + expect(noteAt(keys, cSharp4.x + cSharp4.w / 2, 5)).toBe(61); + expect(noteAt(keys, 800, 10)).toBeNull(); + }); + + it('plays the computer keyboard by where a key is, as a tracker does', () => { + expect(trackerNote('KeyZ', 3)).toBe(48); + expect(trackerNote('KeyS', 3)).toBe(49); + expect(trackerNote('KeyQ', 3)).toBe(60); + expect(trackerNote('Digit2', 3)).toBe(61); + expect(trackerNote('KeyP', 3)).toBe(76); + expect(trackerNote('KeyA', 3)).toBeNull(); + expect(trackerNote('KeyP', 8)).toBeNull(); + }); +}); diff --git a/tests/core/keep-on-scene-load.test.ts b/tests/core/keep-on-scene-load.test.ts index a8c39794..cf85025f 100644 --- a/tests/core/keep-on-scene-load.test.ts +++ b/tests/core/keep-on-scene-load.test.ts @@ -55,6 +55,8 @@ function start(f: Fixture, audio?: AudioSink) { const stoppedOwners: string[] = []; const sink: AudioSink = audio ?? { play: () => 0, + playNote: () => 0, + release: () => {}, tune: () => {}, length: () => 0, stop: () => {}, diff --git a/tests/core/node-catalog.test.ts b/tests/core/node-catalog.test.ts index 1fc15d0c..a2414003 100644 --- a/tests/core/node-catalog.test.ts +++ b/tests/core/node-catalog.test.ts @@ -2028,6 +2028,49 @@ const CASES: Record = { after: (ctx) => expect(ctx.played[0]?.fxParams).toEqual({ depth: -12 }), }, ], + PlayNote: [ + { + name: 'no instrument is a no-op that still continues the flow', + inputs: { instrument: '', note: 60 }, + next: 'exec', + outputs: { voice: 0 }, + after: (ctx) => expect(ctx.notes).toEqual([]), + }, + { + name: 'hands the instrument on with its note, velocity, volume and duration, and gives out the voice (#331)', + inputs: { + instrument: 'ins-1', + note: 64, + velocity: 0.5, + volume: 0.8, + duration: 0.25, + fxParams: { cut: 2 }, + }, + next: 'exec', + outputs: { voice: 1 }, + after: (ctx) => + expect(ctx.notes).toEqual([ + { + instrumentId: 'ins-1', + note: 64, + velocity: 0.5, + volume: 0.8, + duration: 0.25, + detached: false, + instrumentParams: { cut: 2 }, + }, + ]), + }, + ], + InstrumentParameterValues: [ + { + name: 'bundles its picked parameters as numbers, as for a sound FX', + ports: [{ key: 'cut', kind: 'number' }], + inputs: { cut: '0.5' }, + outputs: { params: { cut: 0.5 } }, + }, + { name: 'with none picked the bundle is empty', outputs: { params: {} } }, + ], SoundFxParameters: [ { name: 'bundles its picked parameters as numbers', diff --git a/tests/core/pitch-player.test.ts b/tests/core/pitch-player.test.ts index ee0c1afe..ec667444 100644 --- a/tests/core/pitch-player.test.ts +++ b/tests/core/pitch-player.test.ts @@ -21,7 +21,11 @@ function sine(frequency: number, seconds: number, rate = RATE): Float32Array { function render( sound: Float32Array, semitones: number | ((at: number) => number), - { step = 1, loop = false, blocks = Infinity } = {}, + { + step = 1, + loop = false, + blocks = Infinity, + }: { step?: number; loop?: boolean | { start: number; end: number }; blocks?: number } = {}, ): Float32Array { const player = new PitchPlayer([sound], step, RATE, loop); const out: number[] = []; @@ -138,3 +142,22 @@ describe('the pitch player', () => { expect(off).toBeLessThan(1e-5); }); }); + +describe('a loop between two points (#331)', () => { + it('plays up to the loop end, then on from the loop start, never ending', () => { + // A ramp, so every sample says where in the sound it was read. + const ramp = Float32Array.from({ length: 48000 }, (_, i) => i / 48000); + const out = render(ramp, 0, { loop: { start: 12000, end: 24000 }, blocks: 400 }); + expect(out.length).toBe(400 * BLOCK); + expect(out[20000]).toBeCloseTo(20000 / 48000, 3); + // 30000 is 6000 past the end: 6000 past the start. + expect(out[30000]).toBeCloseTo(18000 / 48000, 3); + expect(out[40000]).toBeCloseTo(16000 / 48000, 3); + }); + + it('takes a loop that does not fit the sound as the whole sound', () => { + const ramp = Float32Array.from({ length: 4800 }, (_, i) => i / 4800); + const out = render(ramp, 0, { loop: { start: 3000, end: 9000 }, blocks: 60 }); + expect(out[6000]).toBeCloseTo(1200 / 4800, 3); + }); +}); diff --git a/tests/core/soundfx.test.ts b/tests/core/soundfx.test.ts index cba93584..ebcc77d5 100644 --- a/tests/core/soundfx.test.ts +++ b/tests/core/soundfx.test.ts @@ -372,6 +372,8 @@ function fakeAudio(): AudioSink & { sink.tuned.push({ voice, params }); sink.times.push(time); }, + playNote: () => 0, + release: () => {}, length: () => 0.25, stop: () => {}, tail: () => 0, diff --git a/tests/core/sounds.test.ts b/tests/core/sounds.test.ts index d3a39011..9633c865 100644 --- a/tests/core/sounds.test.ts +++ b/tests/core/sounds.test.ts @@ -42,6 +42,8 @@ function fakeAudio(): AudioSink & { stoppedOwners: [] as string[], play: (file: string, volume: number, loop: boolean, owner: string) => sink.played.push({ file, volume, loop, owner }), + playNote: () => 0, + release: () => {}, tune: () => {}, length: () => 0, tail: () => 0, diff --git a/tests/helpers.ts b/tests/helpers.ts index 38a55d50..280ef673 100644 --- a/tests/helpers.ts +++ b/tests/helpers.ts @@ -214,6 +214,16 @@ export type FakeContext = NodeContext & { fx: string; fxParams: Readonly>; }[]; + /** The notes the context was asked to play (#331); a voice's number is how many have been played. */ + notes: { + instrumentId: string; + note: number; + velocity: number; + volume: number; + duration: number; + detached: boolean; + instrumentParams: Readonly>; + }[]; silenced: string[]; tuned: { voice: number; fxParams: Readonly>; time: number; ease: EaseShare }[]; /** The scenes the context was asked to load, in order. */ @@ -270,6 +280,9 @@ export function fakeContext(overrides: Partial = {}): FakeContext { // The number of a voice is how many have been played: unique, and never 0. playSound: (assetId, volume, loop, detached, fx, fxParams) => ctx.played.push({ assetId, volume, loop, detached, fx, fxParams }), + notes: [], + playNote: (instrumentId, note, velocity, volume, duration, detached, instrumentParams) => + ctx.notes.push({ instrumentId, note, velocity, volume, duration, detached, instrumentParams }), // Every sound lasts a quarter of a second, as if it had been loaded. soundLength: () => 0.25, // The tail of a voice is its number in tenths of a second, so a test can tell one from another. diff --git a/tests/viewport/sound-library.test.ts b/tests/viewport/sound-library.test.ts index 96dac53f..8eef7cc7 100644 --- a/tests/viewport/sound-library.test.ts +++ b/tests/viewport/sound-library.test.ts @@ -383,6 +383,7 @@ describe('the sound library', () => { file: FILE, volume: 0.25, loop: true, + layers: 1, pitches: [0], stages: 0, duration: 1.5, @@ -890,3 +891,108 @@ describe('the sound library', () => { }); }); }); + +describe('notes of an instrument (#331)', () => { + /** A way that plays its own file, as a Sample makes it. */ + const sample = ( + file: string, + extra: Partial['source']>> = {}, + ) => + ({ + pitch: [], + stages: [], + source: { file, volume: 0.5, loop: false, loopStart: 0, loopEnd: 0, gate: true, ...extra }, + }) as FxPlan; + + async function note(library: SoundLibrary, layers: FxPlan[], duration = 0): Promise { + const voice = library.playNote('instrument:i', layers, {}, 1, '', duration); + await settle(); + await arrive(); + return voice; + } + + it('plays its layers side by side as one voice, each way its own file behind its own volume', async () => { + const library = new SoundLibrary(); + const voice = await note(library, [sample(FILE), sample(OTHER, { gate: false })]); + expect(voice).toBe(1); + expect(library.stats()).toMatchObject({ started: 1, playing: 1, decoded: 2 }); + expect(library.stats().last).toMatchObject({ file: 'instrument:i', layers: 2, duration: 1.5 }); + expect(sources.map((source) => source.started)).toEqual([true, true]); + // Voice gain first, then a fade per way at the sound's volume. + expect(gains.map((gain) => gain.value)).toEqual([1, 0.5, 0.5]); + expect(wires).toEqual([ + 'source > gain', + 'gain > gain', + 'source > gain', + 'gain > gain', + 'gain > destination', + ]); + }); + + it('a release fades out the ways that end with the note and lets a one-shot play on', async () => { + const library = new SoundLibrary(); + const voice = await note(library, [sample(FILE), sample(OTHER, { gate: false })]); + library.release(voice); + expect(gains[1]!.ramps).toEqual([['linear', 0, 0.01]]); + expect(gains[2]!.ramps).toEqual([]); + clockTo(); + expect(sources.map((source) => source.stopped)).toEqual([true, false]); + expect(library.stats().playing).toBe(1); + // The one-shot ends by itself: now the voice is gone. + sources[1]!.onended?.(); + expect(library.stats().playing).toBe(0); + }); + + it('a note with a duration is released that long after it starts, on the audio clock', async () => { + const library = new SoundLibrary(); + await note(library, [sample(FILE)], 0.5); + expect(clocks.map((clock) => clock.stopAt)).toEqual([0.5]); + clockTo(0.5); + expect(gains[1]!.ramps).toEqual([['linear', 0, 0.01]]); + clockTo(); + expect(sources[0]!.stopped).toBe(true); + expect(library.stats().playing).toBe(0); + }); + + it('a note released before its bytes arrived still starts, and is released at once', async () => { + const library = new SoundLibrary(); + const voice = library.playNote( + 'instrument:i', + [sample(FILE), sample(OTHER, { gate: false })], + {}, + 1, + '', + 0, + ); + library.release(voice); + await settle(); + await arrive(); + expect(library.stats().started).toBe(1); + clockTo(); + expect(sources.map((source) => source.stopped)).toEqual([true, false]); + }); + + it('loops a way between its loop start and end', async () => { + const library = new SoundLibrary(); + await note(library, [sample(FILE, { loop: true, loopStart: 0.25, loopEnd: 1 })]); + expect(sources[0]).toMatchObject({ loop: true, loopStart: 0.25, loopEnd: 1 }); + }); + + it('hands a pitched way its loop in samples of the sound', async () => { + const library = new SoundLibrary(); + const pitched = { ...sample(FILE, { loop: true, loopStart: 0.25, loopEnd: 0 }), pitch: [{ value: 3 }] }; + await note(library, [pitched]); + // 800 samples a second, 1200 of them: from 200 to the end. + expect(worklets[0]!.options.processorOptions).toMatchObject({ + loop: true, + loopStart: 200, + loopEnd: 1200, + }); + }); + + it('plays nothing where no way has a file', () => { + const library = new SoundLibrary(); + expect(library.playNote('instrument:i', [], {}, 1, '', 0)).toBe(0); + expect(library.playNote('instrument:i', [{ pitch: [], stages: [] }], {}, 1, '', 0)).toBe(0); + }); +}); diff --git a/tools/e2e/04-assets.mjs b/tools/e2e/04-assets.mjs index 74287d80..888db557 100644 --- a/tools/e2e/04-assets.mjs +++ b/tools/e2e/04-assets.mjs @@ -2,7 +2,7 @@ import { colourDistance } from '../lib/harness.mjs'; import { inView } from './lib/scaffold.mjs'; /** One per entry of `ASSET_TYPES` – a new asset type counts here too. */ -const ASSET_TYPE_COUNT = 27; +const ASSET_TYPE_COUNT = 28; /** The asset panel: placing by dragging into the view, creating through "+", and resetting an asset value. */ export async function assets(t) { diff --git a/tools/e2e/34-sound.mjs b/tools/e2e/34-sound.mjs index 22bf7ea8..e460fe11 100644 --- a/tools/e2e/34-sound.mjs +++ b/tools/e2e/34-sound.mjs @@ -404,12 +404,13 @@ export async function sound(t) { const beforeEcho = await stats(); await click(fxPlayAt.x, fxPlayAt.y); const echoStarted = await waitFor(stats, (s) => s.started > beforeEcho.started); - const startedAt = Date.now(); + // Timed on the audio clock: the page's can fall behind it (task 523 saw a 2.95 s tail end 1 s in). + const audioNow = () => json('({ playing: __gm.sound.stats().playing, clock: __gm.sound.clock() })'); + const startedAt = (await audioNow()).clock; const echoLevels = await json(`__gm.sound.levels(${JSON.stringify(blip.file)})`); - await sleep(1000); - const ringing = await stats(); - const quiet = await waitFor(stats, (s) => s.playing === 0, 40); - const rangFor = (Date.now() - startedAt) / 1000; + const ringing = await waitFor(audioNow, (s) => s.clock - startedAt >= 1); + const quiet = await waitFor(audioNow, (s) => s.playing === 0, 40); + const rangFor = quiet.clock - startedAt; const echoKnobs = echoLevels?.[0]?.slice(2) ?? []; check( 'a Delay keeps the voice after its sound has ended, until its echoes have faded', @@ -421,10 +422,11 @@ export async function sound(t) { Math.abs(echoKnobs[2] / (20000 * 0.025 ** 0.3) - 1) < 1e-3 && Math.abs(echoKnobs[3] - 0.5) < 1e-6 && ringing.playing === 1 && + ringing.clock - startedAt < 2.4 && quiet.playing === 0 && rangFor > 2.4 && rangFor < 4.5, - JSON.stringify({ delayInfo, last: echoStarted.last, echoLevels, ringing: ringing.playing, rangFor }), + JSON.stringify({ delayInfo, last: echoStarted.last, echoLevels, ringing, startedAt, rangFor }), ); await click(fxPlayAt.x, fxPlayAt.y); await waitFor(stats, (s) => s.playing === 1); @@ -557,20 +559,23 @@ export async function sound(t) { ); await stop(); // The easings of an animation key: Quad in hands the clock a curve, which is still near the start - // 0.7 s in and only then falls to the end (-5). Read a little later than that for the round - // trips, it is above 3.1 until 1.1 s; linear is below 2.8 from 0.7 s on. + // after 0.8 s and only then falls to the end (-5): 7 - 12 * (t / 2)^2, where linear is 7 - 6 * t. + // Timed on the audio clock from the moment the voice is first seen, which is no earlier than its + // start: the curve is then further along than `into` says, and still well above linear. await mutate(` const tune = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph.nodes.find((n) => n.id === 'sn-tune'); Object.assign(tune.params, { ease: 'quad', direction: 'in' });`); await play(); - await waitFor(pitch, (s) => typeof s === 'number'); - await new Promise((resolve) => setTimeout(resolve, 700)); - const easedEarly = await pitch(); + const pitchNow = () => + json(`({ pitch: __gm.sound.levels(${JSON.stringify(blip.file)})?.[0]?.[0], clock: __gm.sound.clock() })`); + const sweepSeen = await waitFor(pitchNow, (s) => typeof s.pitch === 'number'); + const eased = await waitFor(pitchNow, (s) => s.clock - sweepSeen.clock >= 0.8); + const into = eased.clock - sweepSeen.clock; const easedEnd = await waitFor(pitch, (s) => typeof s === 'number' && s < -4.3); check( 'Set sound FX parameters with Quad in ramps slowly first, then on to the end', - easedEarly > 3.1 && easedEarly < 6.95 && easedEnd < -4.3, - JSON.stringify({ easedEarly, easedEnd }), + eased.pitch < 6.95 && eased.pitch > 7 - 6 * into + 1 && easedEnd < -4.3, + JSON.stringify({ into, eased: eased.pitch, linearThere: 7 - 6 * into, easedEnd }), ); await stop(); // Step holds the pitch for the whole time and then jumps there. @@ -738,6 +743,138 @@ export async function sound(t) { const project = __gm.state.project; project.assets = project.assets.filter((a) => a.id !== ${JSON.stringify(fxId)});`); + // 34g-5. An instrument (#331): "+ → Instrument" opens its tab on Notes → Sample → Output, which + // plays nothing until the Sample has a sound. With the blip, looped, every key of C4–B4 + // sounds; a key held by the pointer plays until it is let go, the computer keyboard plays + // as a tracker does, and a game's `Play note` releases its note after its duration. + await showTab('scene'); + await clickText('+'); + await pickAssetType(t, 'Instrument'); + const insCreated = await json(`{ + tab: __gm.tab(), + current: __gm.instrument.current(), + info: __gm.instrument.info(), + keys: __gm.instrument.keys(), + assets: __gm.state.project.assets.filter((a) => a.type === 'instrument') + .map((a) => ({ id: a.id, nodes: a.graph.nodes.map((n) => n.type) })), + }`); + check( + 'the + menu creates an instrument, Notes → Sample → Output, and opens its tab; no key sounds yet', + insCreated.tab === 'instrument' && + insCreated.assets.length === 1 && + insCreated.current === insCreated.assets[0].id && + insCreated.assets[0].nodes.join() === 'InstrumentNotes,InstrumentSample,InstrumentOutput' && + insCreated.info === 'No sound picked' && + insCreated.keys.sounding.length === 0, + JSON.stringify(insCreated), + ); + const insId = insCreated.assets[0]?.id; + // Bottom left: the fitted graph is one row of three nodes, its links among them. + const insCanvas = await json('__gm.instrument.canvasRect()'); + await click(insCanvas.x + 30, insCanvas.y + insCanvas.h - 30, 'right'); + const insPalette = await json('__gm.overlayTexts()'); + check( + 'its palette offers its own nodes and the effects, but no Input and no behavior node', + ['Notes', 'Sample', 'Parameters', 'Sound FX', 'Gain', 'Delay'].every((label) => + insPalette.includes(label), + ) && + !insPalette.includes('Input') && + !insPalette.includes('Branch'), + JSON.stringify(insPalette.slice(0, 14)), + ); + await press('Escape'); + await mutate(` + const ins = __gm.state.project.assets.find((a) => a.id === ${JSON.stringify(insId)}); + Object.assign(ins.graph.nodes.find((n) => n.type === 'InstrumentSample').params, { + sound: ${JSON.stringify(soundId)}, + loop: true, + });`); + const insReady = await json('{ info: __gm.instrument.info(), keys: __gm.instrument.keys() }'); + check( + 'with a sound picked the twelve keys from C4 to B4 sound, the others stay grey', + insReady.info === '12 notes' && + insReady.keys.sounding.join() === [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71].join(), + JSON.stringify(insReady), + ); + await shot('34-instrument'); + + await evaluate('__gm.sound.stopAll()'); + const beforeKey = await stats(); + const e4 = await json('__gm.instrument.keyAt(64)'); + await t.mouse('mouseMoved', e4.x, e4.y); + await t.mouse('mousePressed', e4.x, e4.y); + const heldKey = await waitFor(stats, (s) => s.started > beforeKey.started); + // Longer than the blip: a looped note that ended with its sound would be gone by now. + await sleep(600); + const stillHeld = await json('{ stats: __gm.sound.stats(), keys: __gm.instrument.keys() }'); + check( + 'holding E4 plays the blip four semitones up, looped, as one layer of the instrument, and lights the key', + heldKey.last?.file === `instrument:${insId}` && + heldKey.last.pitches.join() === '4' && + heldKey.last.layers === 1 && + stillHeld.stats.playing === 1 && + stillHeld.keys.held.join() === '64', + JSON.stringify({ heldKey, stillHeld }), + ); + await t.mouse('mouseReleased', e4.x, e4.y); + const letGo = await waitFor(stats, (s) => s.playing === 0); + check( + 'letting go of the key ends the note: its sample fades out and the voice goes', + letGo.playing === 0 && (await json('__gm.instrument.keys()')).held.length === 0, + JSON.stringify(letGo), + ); + + // The click left the keys focused. Z is C3 there – no sound in this instrument –, one octave up C4. + const beforeZ = await stats(); + await press('z'); + await sleep(200); + const lowZ = await stats(); + await press('ArrowRight'); + await t.keyDown('z'); + const typed = await waitFor(stats, (s) => s.started > lowZ.started); + const typedKeys = await json('__gm.instrument.keys()'); + await t.keyUp('z'); + const typedOff = await waitFor(stats, (s) => s.playing === 0); + check( + 'the computer keyboard plays as a tracker: Z is C3 and silent here, one octave up it is C4 until let go', + lowZ.started === beforeZ.started && + typedKeys.octave === 4 && + typedKeys.held.join() === '60' && + typed.last?.pitches.join() === '0' && + typedOff.playing === 0, + JSON.stringify({ beforeZ: beforeZ.started, lowZ: lowZ.started, typed, typedKeys, typedOff }), + ); + await press('ArrowLeft'); + + await mutate(` + const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph; + graph.nodes.push(${JSON.stringify( + node('sn-note', 'PlayNote', 260, 220, { instrument: insId, note: 67, duration: 0.3 }), + )}); + graph.edges.find((e) => e.id === 'sn-e1').to = { node: 'sn-note', port: 'exec' };`); + await showTab('scene'); + await evaluate('__gm.sound.stopAll()'); + const beforeNote = await stats(); + await play(); + const noted = await waitFor(stats, (s) => s.started > beforeNote.started); + await sleep(900); + const released = await stats(); + check( + 'a game’s Play note plays G4 – the blip seven semitones up – and lets it go after its duration', + noted.last?.file === `instrument:${insId}` && + noted.last.pitches.join() === '7' && + noted.playing === 1 && + released.playing === 0, + JSON.stringify({ noted, released }), + ); + await stop(); + await mutate(` + const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph; + graph.nodes = graph.nodes.filter((n) => n.id !== 'sn-note'); + graph.edges.find((e) => e.id === 'sn-e1').to = { node: 'sn-play', port: 'exec' }; + const project = __gm.state.project; + project.assets = project.assets.filter((a) => a.id !== ${JSON.stringify(insId)});`); + // 34h. Dropping a sound file onto the editor uploads it and makes an asset named after the file. const dropped = await evaluate(` (async () => {