From 23bdc3f775e737ce50a8737e06530fb0ed2e897f Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Tue, 6 Oct 2026 12:29:31 +0200 Subject: [PATCH] Play notes in time on the audio clock with a scheduler #332 Patterns and songs (#333, #334) have to play sample-accurate in time, whatever the frame rate does. What is counted in game time runs away from the sound. This lays the ground: musical time, a scheduler that plans notes ahead on the audio clock, and a player that starts and ends a note at a given second. Nothing in the editor plays a sequence yet, the check scripts drive it through `__gm.music`. - Ticks, 480 to the quarter as in MIDI files (decided in #332). Triplets and quintuplets land on whole ticks. Any meter, a beat being the note value under the line. The tempo always counts quarters. - `core/music/sequence.ts` is a pure rule: the notes starting in a window of playing time. Across the end of a loop it looks into the next round, so the seam has no gap. A note past the loop end ends there. Windows are whole ticks, and the seam is stepped exactly, so no rounding can drop the first note of a round. - `core/music/Scheduler.ts` plans the next 100 ms every 25 ms through a host, so it is tested with a clock of its own. The first note sounds 50 ms after the start. Changes to the sequence are heard from the next window on. Stop and seek fade what it started. A new tempo or loop counts from what is planned, and the playhead keeps its old pace until the clock gets there. - The position is the audio clock less `outputLatency`, as bar, beat and tick. - The beat comes from a Web Worker (`viewport/musicTimer.ts`), which keeps its pace in a hidden tab. - The sound library starts a note at `timing.at`: a buffer source by `start(when)`, the pitch worklet by counting down inside the render block it falls in. A note's end is set on the clock at once, its fade to the sample. Leaving a loop still waits for the page. - Delay synced to beats (decided in #332, noted from #327): a setting `Sync` (Seconds / Beats), `time` then in quarter notes, held to the 2 s of the line. Play sound and Play note have an input `tempo`, 120 BPM where nothing is connected. A scheduled note carries the tempo of its sequence. - Swing is left for #333/#334, as decided. Also fixes the red suite of 6f1f315 (task 526). Three checks in e2e 34 waited on the page's clock for things that run on the audio clock. In CI the voice was already 0.7 s old when first seen, and the 0.3 s note of Play note was over. `stats().last` now says when a voice started on the audio clock, and the Quad, Step and Play note checks hold every reading against it. Tests: - music.test.ts (new): 17 tests, time, windows, loop seam, scheduler with a jittery timer, stop, seek, tempo change - sound-library +4, soundfx +1, node-catalog +2 (tempo inputs) - Counter-checks: an unclamped note length, a window that never moves on, a buffer source ignoring its start time and a Delay ignoring Beats each turn 1, 6, 1 and 2 tests red - e2e 34 on melektaus-atx: ALL PASSED, 47 checks (+2). Nine notes of a looped bar at 150 BPM exactly 0.4 s apart, each planned 89 to 91 ms ahead. The position matched the clock less 32 ms latency to the tick. Offline the worklet started on samples 590, 12000 and 15200. - Gate on melektaus-atx: CHECK EXIT 0, 244 test files, 3786 tests Co-Authored-By: Claude Opus 5.5 --- src/core/graph/NodeContext.ts | 6 +- src/core/graph/nodes/action/PlayNote.ts | 7 +- src/core/graph/nodes/action/PlaySound.ts | 25 +- src/core/music/Scheduler.ts | 180 ++++++++++++++ src/core/music/sequence.ts | 84 +++++++ src/core/music/time.ts | 71 ++++++ src/core/soundfx/nodes.ts | 45 +++- src/editor/debugHook.ts | 18 ++ src/runtime/systems/Audio.ts | 10 + src/runtime/systems/behaviorContext.ts | 19 +- src/viewport/SoundLibrary.ts | 106 ++++++-- src/viewport/music.ts | 109 +++++++++ src/viewport/musicTimer.ts | 11 + src/viewport/pitchWorklet.ts | 19 +- tests/core/music.test.ts | 299 +++++++++++++++++++++++ tests/core/node-catalog.test.ts | 15 +- tests/core/soundfx.test.ts | 19 +- tests/helpers.ts | 10 +- tests/viewport/sound-library.test.ts | 80 +++++- tools/e2e/34-sound.mjs | 154 ++++++++++-- 20 files changed, 1201 insertions(+), 86 deletions(-) create mode 100644 src/core/music/Scheduler.ts create mode 100644 src/core/music/sequence.ts create mode 100644 src/core/music/time.ts create mode 100644 src/viewport/music.ts create mode 100644 src/viewport/musicTimer.ts create mode 100644 tests/core/music.test.ts diff --git a/src/core/graph/NodeContext.ts b/src/core/graph/NodeContext.ts index 0c68f407..b82e9479 100644 --- a/src/core/graph/NodeContext.ts +++ b/src/core/graph/NodeContext.ts @@ -240,6 +240,7 @@ export interface NodeContext { * the one case where the rule is in the way: the death sound of an object that destroys itself. * `fx` names a sound FX asset the voice is played through (#311); `''` plays it as it is. * `fxParams` are values for its parameters; one not given is the default the FX declares. + * `tempo` is what a Delay synced to beats counts by (#332). * Returns the number of the voice for `tuneSound`, or 0 where nothing is played. */ playSound( @@ -249,12 +250,14 @@ export interface NodeContext { detached: boolean, fx: string, fxParams: FxParams, + tempo: number, ): number; /** * Plays one note of an instrument asset (#331): `note` as MIDI counts it (60 is C4), struck with * `velocity` (0 to 1, how loud it is next to `volume`) and released after `duration` seconds, or * held until it is stopped where that is 0. Owned like a `playSound` voice, `instrumentParams` are - * values for the instrument's parameters. Returns the number of the voice, 0 where nothing plays. + * values for the instrument's parameters, `tempo` as for `playSound`. Returns the number of the + * voice, 0 where nothing plays. */ playNote( instrumentId: string, @@ -264,6 +267,7 @@ export interface NodeContext { duration: number, detached: boolean, instrumentParams: FxParams, + tempo: number, ): number; /** * How many seconds one run of that sound asset lasts, so a change can span it; 0 where it does not diff --git a/src/core/graph/nodes/action/PlayNote.ts b/src/core/graph/nodes/action/PlayNote.ts index 208049b8..8aa0ddb2 100644 --- a/src/core/graph/nodes/action/PlayNote.ts +++ b/src/core/graph/nodes/action/PlayNote.ts @@ -1,15 +1,16 @@ import { MIDDLE_C } from '../../../instrument/notes'; +import { heldTempo } from '../../../music/time'; import type { NodeDef } from '../../NodeRegistry'; import { asBoolean, asNumber, asString } from '../../values'; import { exec, num, withDefault } from '../shared'; -import { fxParamsOf } from './PlaySound'; +import { fxParamsOf, tempoPort } from './PlaySound'; export const PlayNote: NodeDef = { type: 'PlayNote', label: 'Play note', category: 'action', description: - 'Plays one note of an instrument asset. "note" is its number as MIDI counts it: 60 is C4, 61 C#4, 72 C5. "velocity" is how hard it is struck, 0 to 1 – it makes the note quieter and is handed to what the instrument does with it. "duration" is how many seconds until the note is released; at 0 it is held until a Stop sound ends it. A note belongs to this object like a sound and stops when the object is destroyed, unless "detached" is on. "params" takes values for the parameters of the instrument from an Instrument parameters node. "voice" is the number of this note, for Set sound FX parameters and Stop sound; 0 where the instrument plays nothing for it.', + 'Plays one note of an instrument asset. "note" is its number as MIDI counts it: 60 is C4, 61 C#4, 72 C5. "velocity" is how hard it is struck, 0 to 1 – it makes the note quieter and is handed to what the instrument does with it. "duration" is how many seconds until the note is released; at 0 it is held until a Stop sound ends it. A note belongs to this object like a sound and stops when the object is destroyed, unless "detached" is on. "params" takes values for the parameters of the instrument from an Instrument parameters node. "voice" is the number of this note, for Set sound FX parameters and Stop sound; 0 where the instrument plays nothing for it. "tempo" is what a Delay in the instrument synced to beats counts by, in quarter notes a minute.', inputs: [ exec(), { @@ -42,6 +43,7 @@ export const PlayNote: NodeDef = { field: { key: 'detached', label: 'Detached (outlives this object)', kind: 'boolean', default: false }, }, { key: 'fxParams', label: 'params', type: 'fxparams' }, + tempoPort(), ], outputs: [exec(), num('voice')], params: [], @@ -56,6 +58,7 @@ export const PlayNote: NodeDef = { asNumber(inputs.duration), asBoolean(inputs.detached), fxParamsOf(inputs.fxParams), + heldTempo(asNumber(inputs.tempo)), ) : 0; return { next: 'exec', outputs: { voice } }; diff --git a/src/core/graph/nodes/action/PlaySound.ts b/src/core/graph/nodes/action/PlaySound.ts index 37a7a7a8..36cc1c75 100644 --- a/src/core/graph/nodes/action/PlaySound.ts +++ b/src/core/graph/nodes/action/PlaySound.ts @@ -1,4 +1,5 @@ import { isStruct } from '../../../model/Properties'; +import { DEFAULT_TEMPO, heldTempo, MAX_TEMPO, MIN_TEMPO } from '../../../music/time'; import type { FxParams } from '../../../soundfx/nodes'; import type { NodeDef, PortDef } from '../../NodeRegistry'; import { asBoolean, asNumber, asString, type Value } from '../../values'; @@ -64,7 +65,8 @@ export const PlaySound: NodeDef = { 'nothing was played. "length" is how many seconds one run of the sound lasts, to hand to the ' + '"time" of Set sound FX parameters so a change spans the sound; 0 where nothing was played or ' + 'the sound has not been loaded yet. A pitch never changes it. "tail" is that length and the ' + - 'echoes of its FX after it, as they are when it starts – how long until the voice is quiet.', + 'echoes of its FX after it, as they are when it starts – how long until the voice is quiet. ' + + '"tempo" is what a Delay of its FX synced to beats counts by, in quarter notes a minute.', inputs: [ exec(), soundPort(), @@ -73,6 +75,7 @@ export const PlaySound: NodeDef = { detachedPort(), soundFxPort(), { key: 'fxParams', label: 'fx params', type: 'fxparams' }, + tempoPort(), ], outputs: [exec(), num('voice'), num('length'), num('tail')], params: [], @@ -86,6 +89,7 @@ export const PlaySound: NodeDef = { asBoolean(inputs.detached), asString(inputs.fx), fxParamsOf(inputs.fxParams), + heldTempo(asNumber(inputs.tempo)), ) : 0; const length = voice ? ctx.soundLength(sound) : 0; @@ -94,6 +98,25 @@ export const PlaySound: NodeDef = { }, }; +/** The tempo of a voice outside a song (#332), for a Delay synced to beats. */ +export function tempoPort(): PortDef { + return { + key: 'tempo', + label: 'tempo', + type: 'number', + default: DEFAULT_TEMPO, + field: { + key: 'tempo', + label: 'Tempo (BPM)', + kind: 'number', + default: DEFAULT_TEMPO, + min: MIN_TEMPO, + max: MAX_TEMPO, + step: 1, + }, + }; +} + /** The numbers of a parameter bundle; anything else hands in none. */ export function fxParamsOf(value: Value | undefined): FxParams { if (!isStruct(value)) return {}; diff --git a/src/core/music/Scheduler.ts b/src/core/music/Scheduler.ts new file mode 100644 index 00000000..8a6a3464 --- /dev/null +++ b/src/core/music/Scheduler.ts @@ -0,0 +1,180 @@ +import { heldLoop, type LoopRange, notesIn, playheadAt, type Sequence } from './sequence'; +import { + COMMON_TIME, + DEFAULT_TEMPO, + heldMeter, + heldTempo, + type Meter, + type Position, + positionOf, + secondsOfTicks, + ticksOfSeconds, +} from './time'; + +/** Seconds ahead of the audio clock the notes are planned, and how often that is done. */ +export const LOOKAHEAD = 0.1; +export const INTERVAL = 0.025; +/** Seconds between a start and its first note: the page has to get the note to the audio thread. */ +export const LEAD = 0.05; + +/** One note handed to the player: when it starts on the audio clock and how long it is held. */ +export interface ScheduledNote { + at: number; + duration: number; + note: number; + velocity: number; + instrument: string; + /** What a Delay synced to beats counts by. */ + tempo: number; +} + +/** What the scheduler needs of the audio side; a test hands in a clock of its own. */ +export interface SchedulerHost { + /** Seconds on the audio clock. */ + now(): number; + /** Seconds between the audio clock and what is heard. */ + latency(): number; + /** Starts the note, gives back the number of its voice; 0 where nothing plays. */ + play(note: ScheduledNote): number; + playing(voice: number): boolean; + stop(voices: readonly number[]): void; + /** Calls `pump` every `INTERVAL` seconds, also in a hidden tab, until the returned function is called. */ + wake(pump: () => void): () => void; +} + +export interface SchedulerOptions { + tempo?: number; + meter?: Meter; + loop?: LoopRange | null; +} + +/** + * Plays a sequence on the audio clock (#332): every `INTERVAL` it plans the notes of the next + * `LOOKAHEAD` seconds with their exact start, so a slow frame or a busy page never moves a note. + * Playing time is counted in whole ticks from an anchor on the audio clock. The sequence is read + * afresh for every window, so a change is heard from the next one on; what is planned stays. + */ +export class Scheduler { + private tempo: number; + private meter: Meter; + private loop: LoopRange | null; + /** Where the playhead started, in ticks of the sequence. */ + private from = 0; + /** Where on the audio clock playing time 0 is; null while stopped. */ + private anchor: number | null = null; + /** Playing time planned so far, in whole ticks. */ + private until = 0; + /** How it played before the last change of tempo or loop, for the time until that is reached. */ + private before: { anchor: number; from: number; tempo: number; loop: LoopRange | null } | null = null; + private voices: number[] = []; + private sleep: (() => void) | null = null; + + constructor( + private readonly host: SchedulerHost, + private readonly sequence: () => Sequence, + options: SchedulerOptions = {}, + ) { + this.tempo = heldTempo(options.tempo ?? DEFAULT_TEMPO); + this.meter = heldMeter(options.meter ?? COMMON_TIME); + this.loop = heldLoop(options.loop ?? null); + } + + get playing(): boolean { + return this.anchor !== null; + } + + /** Plays from `from` ticks of the sequence, or from where it was last started or sent. */ + start(from = this.from): void { + this.stop(); + this.from = Math.max(0, Math.round(from)); + this.anchor = this.host.now() + LEAD; + this.until = 0; + this.before = null; + this.sleep = this.host.wake(() => this.pump()); + this.pump(); + } + + /** Every note it started fades out, those still to come with them. */ + stop(): void { + this.sleep?.(); + this.sleep = null; + if (this.anchor !== null) this.from = Math.floor(this.headAt(this.host.now())); + this.anchor = null; + this.before = null; + if (this.voices.length > 0) this.host.stop(this.voices); + this.voices = []; + } + + /** A jump: playing on from there, cut off where it was, or only where the next start begins. */ + seek(tick: number): void { + if (this.playing) this.start(tick); + else this.from = Math.max(0, Math.round(tick)); + } + + /** Counted from what is planned already: a note in the next window keeps the tempo it had. */ + setTempo(tempo: number): void { + this.rebase(); + this.tempo = heldTempo(tempo); + } + + setLoop(loop: LoopRange | null): void { + this.rebase(); + this.loop = heldLoop(loop); + } + + setMeter(meter: Meter): void { + this.meter = heldMeter(meter); + } + + /** + * Where what is heard right now stands: the audio clock less the output latency, so a playhead + * drawn from it is not ahead of the sound. Before the first note it is where it starts. + */ + position(): Position & { ticks: number } { + const ticks = this.anchor === null ? this.from : this.headAt(this.host.now() - this.host.latency()); + return { ...positionOf(ticks, this.meter), ticks }; + } + + /** Plans the notes of the window ahead that are not planned yet. */ + pump(): void { + if (this.anchor === null) return; + const to = Math.floor(ticksOfSeconds(this.host.now() + LOOKAHEAD - this.anchor, this.tempo)); + if (to <= this.until) return; + this.voices = this.voices.filter((voice) => this.host.playing(voice)); + for (const due of notesIn(this.sequence(), this.from, this.loop, this.until, to)) { + const voice = this.host.play({ + at: this.anchor + secondsOfTicks(due.at, this.tempo), + duration: secondsOfTicks(due.length, this.tempo), + note: due.note, + velocity: due.velocity, + instrument: due.instrument, + tempo: this.tempo, + }); + if (voice !== 0) this.voices.push(voice); + } + this.until = to; + } + + /** The playhead at a second of the audio clock; before the anchor, where it starts. */ + private headAt(time: number): number { + if (this.anchor === null) return this.from; + const span = + this.before && time < this.anchor + ? this.before + : { anchor: this.anchor, from: this.from, tempo: this.tempo, loop: this.loop }; + const elapsed = ticksOfSeconds(time - span.anchor, span.tempo); + return playheadAt(Math.max(0, elapsed), span.from, span.loop); + } + + /** + * Moves the anchor to the end of what is planned, so a new tempo or loop starts there. Until the + * clock gets there, the playhead goes on as it did. + */ + private rebase(): void { + if (this.anchor === null) return; + this.before = { anchor: this.anchor, from: this.from, tempo: this.tempo, loop: this.loop }; + this.anchor += secondsOfTicks(this.until, this.tempo); + this.from = playheadAt(this.until, this.from, this.loop); + this.until = 0; + } +} diff --git a/src/core/music/sequence.ts b/src/core/music/sequence.ts new file mode 100644 index 00000000..12f0b8bd --- /dev/null +++ b/src/core/music/sequence.ts @@ -0,0 +1,84 @@ +/** + * What the scheduler plays (#332), before there are patterns and songs to make it of (#333, #334): + * notes of instruments at ticks, and a loop the playhead runs round. Everything here is counted in + * ticks of playing time – `elapsed` since the start, which only ever grows – and of the sequence – + * where the playhead stands, which jumps back at the end of a loop. + */ + +export interface SequenceNote { + /** Where it starts in the sequence, and how long it is held, in ticks. */ + tick: number; + length: number; + /** As MIDI counts it: 60 is C4. */ + note: number; + /** 0 to 1. */ + velocity: number; + /** The instrument asset that plays it. */ + instrument: string; +} + +export interface Sequence { + notes: readonly SequenceNote[]; +} + +/** From `start` up to `end`, in ticks of the sequence; `end` is where the playhead jumps back. */ +export interface LoopRange { + start: number; + end: number; +} + +/** A loop with nothing in it is none. */ +export function heldLoop(loop: LoopRange | null): LoopRange | null { + if (!loop || !(loop.end > loop.start) || !(loop.start >= 0)) return null; + return loop; +} + +/** + * Where the playhead stands after `elapsed` ticks of playing from `from`. Started before the end of + * the loop, it runs round it once it gets there; started behind it, it plays on. + */ +export function playheadAt(elapsed: number, from: number, loop: LoopRange | null): number { + if (!loop || from >= loop.end) return from + elapsed; + const toEnd = loop.end - from; + if (elapsed < toEnd) return from + elapsed; + return loop.start + ((elapsed - toEnd) % (loop.end - loop.start)); +} + +/** A note of the sequence where it falls in playing time, held no longer than the loop lets it. */ +export interface DueNote extends SequenceNote { + /** Ticks of playing time from the start. */ + at: number; +} + +/** + * The notes starting in the playing time from `start` up to `end`, both whole ticks. Across the end + * of a loop the window goes on at its start, so the next round is planned ahead like any other + * stretch and the seam has no gap. A note reaching past the end of a loop ends there: the playhead + * has jumped back. + */ +export function notesIn( + sequence: Sequence, + from: number, + loop: LoopRange | null, + start: number, + end: number, +): DueNote[] { + const due: DueNote[] = []; + const stretchEnd = loop !== null && from < loop.end ? loop.end : Infinity; + let elapsed = start; + let head = playheadAt(start, from, loop); + while (elapsed < end) { + const span = Math.min(end - elapsed, stretchEnd - head); + if (!(span > 0)) break; + for (const note of sequence.notes) + if (note.tick >= head && note.tick < head + span) + due.push({ + ...note, + at: elapsed + (note.tick - head), + length: Math.min(note.length, stretchEnd - note.tick), + }); + elapsed += span; + head = head + span >= stretchEnd ? loop!.start : head + span; + } + return due.sort((a, b) => a.at - b.at || a.note - b.note); +} diff --git a/src/core/music/time.ts b/src/core/music/time.ts new file mode 100644 index 00000000..6cbb77d0 --- /dev/null +++ b/src/core/music/time.ts @@ -0,0 +1,71 @@ +/** + * Musical time (#332): ticks, tempo and meter, and the seconds they come to. Ticks are whole numbers + * of a quarter note's share, so a grid, a triplet and a quintuplet all land on one exactly. + */ + +/** As MIDI files and most DAWs count: a 1/32 triplet is 40, a 1/16 quintuplet 96. */ +export const TICKS_PER_QUARTER = 480; + +/** A tempo always counts quarter notes per minute, in 6/8 as in 4/4. */ +export const DEFAULT_TEMPO = 120; +export const MIN_TEMPO = 20; +export const MAX_TEMPO = 400; + +/** A time signature: `beats` notes of the value `unit` to the bar – 3/4, 6/8, 7/8. */ +export interface Meter { + beats: number; + unit: number; +} + +export const COMMON_TIME: Meter = { beats: 4, unit: 4 }; +const UNITS = [1, 2, 4, 8, 16, 32]; +const MAX_BEATS = 32; + +/** A meter as written, or 4/4 where it is not one: a beat is a power of two, a bar 1 to 32 of them. */ +export function heldMeter(meter: Meter): Meter { + const { beats, unit } = meter; + if (!Number.isInteger(beats) || beats < 1 || beats > MAX_BEATS || !UNITS.includes(unit)) return COMMON_TIME; + return { beats, unit }; +} + +/** A tempo between `MIN_TEMPO` and `MAX_TEMPO`; 0 and anything that is not a number is the default. */ +export function heldTempo(tempo: number): number { + if (!Number.isFinite(tempo) || tempo <= 0) return DEFAULT_TEMPO; + return Math.max(MIN_TEMPO, Math.min(MAX_TEMPO, tempo)); +} + +export function ticksPerBeat(meter: Meter): number { + return (TICKS_PER_QUARTER * 4) / meter.unit; +} + +export function ticksPerBar(meter: Meter): number { + return meter.beats * ticksPerBeat(meter); +} + +export function secondsOfTicks(ticks: number, tempo: number): number { + return (ticks * 60) / (tempo * TICKS_PER_QUARTER); +} + +export function ticksOfSeconds(seconds: number, tempo: number): number { + return (seconds * tempo * TICKS_PER_QUARTER) / 60; +} + +/** Seconds of a number of quarter notes: 1 is a quarter, 0.5 an eighth, 0.75 a dotted eighth. */ +export function secondsOfQuarters(quarters: number, tempo: number): number { + return (quarters * 60) / tempo; +} + +/** Where a tick stands as a musician counts it: bar and beat from 1, the tick inside the beat from 0. */ +export interface Position { + bar: number; + beat: number; + tick: number; +} + +export function positionOf(ticks: number, meter: Meter): Position { + const at = Math.max(0, Math.floor(ticks)); + const bar = ticksPerBar(meter); + const beat = ticksPerBeat(meter); + const inBar = at % bar; + return { bar: Math.floor(at / bar) + 1, beat: Math.floor(inBar / beat) + 1, tick: inBar % beat }; +} diff --git a/src/core/soundfx/nodes.ts b/src/core/soundfx/nodes.ts index f0ffd446..f8a75240 100644 --- a/src/core/soundfx/nodes.ts +++ b/src/core/soundfx/nodes.ts @@ -2,6 +2,7 @@ import type { DeclaredPort } from '../graph/Graph'; import { SOUND_FX_INPUT, SOUND_FX_OUTPUT } from '../graph/nested'; import type { NodeDef, PortDef } from '../graph/NodeRegistry'; import type { PropertyBag } from '../model/Properties'; +import { DEFAULT_TEMPO, heldTempo, secondsOfQuarters } from '../music/time'; import type { FlatDef } from '../properties/PropertySchema'; export { SOUND_FX_INPUT, SOUND_FX_OUTPUT }; @@ -72,7 +73,9 @@ export type FxStagePlan = | { kind: 'filter'; filter: FxFilter; frequency: number; gain: FxAmount; q: number } | { kind: 'delay'; + /** Seconds, or quarter notes where `beats` (#332). */ time: FxAmount; + beats?: boolean; feedback: FxAmount; damping: FxAmount; mix: FxAmount; @@ -89,10 +92,14 @@ export const EMPTY_PLAN: FxPlan = { pitch: [], stages: [] }; /** Values for the parameters, by name; a name the plan does not have is left out. */ export type FxParams = Readonly>; -/** How a voice goes through its sound FX: the plan and the values it starts with, kept so they can change while it plays (#315). */ +/** + * How a voice goes through its sound FX: the plan and the values it starts with, kept so they can + * change while it plays (#315), and the tempo a Delay synced to beats counts by (#332). + */ export interface FxVoice { plan: FxPlan; params: FxParams; + tempo?: number; } export const DRY_VOICE: FxVoice = { plan: EMPTY_PLAN, params: {} }; @@ -276,7 +283,18 @@ export const SoundFxMix: SoundFxNodeDef = { /** As long as a delay line is ever made: every voice holds one this long, whatever its time. */ export const DELAY_MAX = 2; -const DELAY_TIME = { ...numberRow('time', 'Time (s)', 0.01, DELAY_MAX, 0.01), default: 0.25 }; +/** Up to two bars of 4/4 in beats; whatever it comes to is held to `DELAY_MAX` seconds. */ +const DELAY_TIME = { ...numberRow('time', 'Time', 0.01, 8, 0.01), default: 0.25 }; +const DELAY_SYNC = { + key: 'sync', + label: 'Sync', + kind: 'enum' as const, + default: 'seconds', + options: [ + { value: 'seconds', label: 'Seconds' }, + { value: 'beats', label: 'Beats' }, + ], +}; /** Never 1: the echoes would never fade, and above it they would grow. */ const DELAY_FEEDBACK = { ...numberRow('feedback', 'Feedback', 0, 0.95, 0.01), default: 0.4 }; const DELAY_DAMPING = { ...numberRow('damping', 'Damping', 0, 1, 0.01), default: 0.3 }; @@ -288,10 +306,10 @@ export const SoundFxDelay: SoundFxNodeDef = { label: 'Delay', category: 'sound', description: - 'Echoes the sound: it comes back after "time" seconds, again and again, each echo "feedback" times as loud as the one before. "damping" makes every echo duller than the last, like tape. "mix" is how loud the echoes are next to the sound, which itself stays as it is. With "ping-pong" the echoes take turns left and right. The echoes ring on after the sound ends; a voice lasts until they have faded.', + 'Echoes the sound: it comes back after "time", again and again, each echo "feedback" times as loud as the one before. "time" is in seconds, or with "sync" on beats in quarter notes at the tempo of the voice: 1 is a quarter, 0.5 an eighth, 0.75 a dotted eighth, 0.333 an eighth triplet. A Play sound or Play note says that tempo, a song its own. Either way an echo comes back after 2 s at most. "damping" makes every echo duller than the last, like tape. "mix" is how loud the echoes are next to the sound, which itself stays as it is. With "ping-pong" the echoes take turns left and right. The echoes ring on after the sound ends; a voice lasts until they have faded.', inputs: [audioIn, ...[DELAY_TIME, DELAY_FEEDBACK, DELAY_DAMPING, DELAY_MIX].map(numberIn)], outputs: [audioOut], - params: [PING_PONG], + params: [DELAY_SYNC, PING_PONG], emit: (sounds, amounts, settings) => { const mix = held(amounts.mix, DELAY_MIX); if (isNone(mix)) return arriving(sounds); @@ -299,6 +317,7 @@ export const SoundFxDelay: SoundFxNodeDef = { { kind: 'delay', time: held(amounts.time, DELAY_TIME), + beats: settings[DELAY_SYNC.key] === 'beats', feedback: held(amounts.feedback, DELAY_FEEDBACK), damping: held(amounts.damping, DELAY_DAMPING), mix, @@ -333,8 +352,13 @@ function amountOf(amount: FxAmount, params: FxParams): number { return Math.max(amount.min ?? -Infinity, Math.min(amount.max ?? Infinity, given)); } +/** Seconds of a delay's time, from beats at the voice's tempo where it is synced, never past the line. */ +function delaySeconds(time: number, beats: boolean, tempo: number): number { + return Math.min(DELAY_MAX, beats ? secondsOfQuarters(time, heldTempo(tempo)) : time); +} + /** A stage fed by a parameter stays at 0 as well: a value handed in later has to find it there. */ -function stagesOf(plan: FxPlan, params: FxParams): FxStage[] { +function stagesOf(plan: FxPlan, params: FxParams, tempo: number): FxStage[] { const stages: FxStage[] = []; for (const stage of plan.stages) { if (stage.kind === 'gain') { @@ -345,7 +369,7 @@ function stagesOf(plan: FxPlan, params: FxParams): FxStage[] { if (stage.kind === 'delay') { stages.push({ kind: 'delay', - time: amountOf(stage.time, params), + time: delaySeconds(amountOf(stage.time, params), stage.beats === true, tempo), feedback: amountOf(stage.feedback, params), damping: dampingCutoff(amountOf(stage.damping, params)), mix: amountOf(stage.mix, params), @@ -362,9 +386,10 @@ function stagesOf(plan: FxPlan, params: FxParams): FxStage[] { /** * The chain one voice is sent along: the plan with every parameter filled in. Its shape depends on * the plan alone, never on the values, so the values of a playing voice can change in place (#315). + * `tempo` is what a Delay synced to beats counts by (#332). */ -export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain { - const stages = stagesOf(plan, params); +export function fxChainOf(plan: FxPlan, params: FxParams = {}, tempo = DEFAULT_TEMPO): FxChain { + const stages = stagesOf(plan, params, tempo); if (!('mix' in plan)) { const pitch = plan.pitch.reduce((sum, amount) => sum + amountOf(amount, params), 0); const chain = { pitch, shifted: plan.pitch.length > 0, stages }; @@ -372,8 +397,8 @@ export function fxChainOf(plan: FxPlan, params: FxParams = {}): FxChain { } return { mix: amountOf(plan.mix, params), - a: fxChainOf(plan.a, params), - b: fxChainOf(plan.b, params), + a: fxChainOf(plan.a, params, tempo), + b: fxChainOf(plan.b, params, tempo), stages, }; } diff --git a/src/editor/debugHook.ts b/src/editor/debugHook.ts index dcbea97a..00f1add2 100644 --- a/src/editor/debugHook.ts +++ b/src/editor/debugHook.ts @@ -1,6 +1,9 @@ import type { Application } from 'pixi.js'; import { physicsStatus } from '../runtime/physics/engine'; +import type { SchedulerOptions } from '../core/music/Scheduler'; +import type { Sequence } from '../core/music/sequence'; import type { FxStage } from '../core/soundfx/nodes'; +import { Music } from '../viewport/music'; import { soundLibrary, stageNodes } from '../viewport/SoundLibrary'; import { curveHooks } from './debug/curveHooks'; import { documentHooks } from './debug/documentHooks'; @@ -22,6 +25,7 @@ import { rectOf } from './textProbe'; */ export function installDebugHook(app: Application, state: EditorState, editor: Editor): void { const context: HookContext = { app, state, editor }; + const music = new Music(() => state.project); (window as unknown as { __gm: unknown }).__gm = { state, /** The Pixi application, for probes that need the renderer or the stage. */ @@ -55,6 +59,20 @@ export function installDebugHook(app: Application, state: EditorState, editor: E /** The nodes a stage of a voice becomes, in any context – an offline one renders what it does. */ stageNodes: (context: BaseAudioContext, stage: FxStage) => stageNodes(context, stage), }, + /** + * The scheduler (#332), which nothing in the editor plays yet: a check hands it a sequence of + * notes and reads back what was planned for which second of the audio clock. + */ + music: { + play: (sequence: Sequence, options?: SchedulerOptions & { from?: number }) => + music.play(sequence, options), + edit: (sequence: Sequence) => music.edit(sequence), + stop: () => music.stop(), + position: () => music.current?.position() ?? null, + playing: () => music.current?.playing ?? false, + planned: () => [...music.planned], + latency: () => soundLibrary.latency(), + }, world: () => editor.worldSnapshot(), playInfo: () => editor.playInfo(), /** Which scene Play would start, and under which of the two rules (#183: set in the settings). */ diff --git a/src/runtime/systems/Audio.ts b/src/runtime/systems/Audio.ts index 52f89df7..2f335d04 100644 --- a/src/runtime/systems/Audio.ts +++ b/src/runtime/systems/Audio.ts @@ -32,6 +32,7 @@ export interface AudioSink { volume: number, owner: string, duration: number, + timing?: NoteTiming, ): number; /** * The end of a note (#331): its ways that end with the note fade out, those that play through @@ -68,6 +69,15 @@ export interface AudioSink { stopOwner(owner: string): void; } +/** + * When a note is heard (#332): `at` is the second of the audio clock it starts at, now where it is + * left out or already past; `tempo` is what a Delay synced to beats counts by, 120 where it is left out. + */ +export interface NoteTiming { + at?: number; + tempo?: number; +} + export const SILENT_AUDIO: AudioSink = { play: () => 0, playNote: () => 0, diff --git a/src/runtime/systems/behaviorContext.ts b/src/runtime/systems/behaviorContext.ts index 4dc99caa..4ff40bd8 100644 --- a/src/runtime/systems/behaviorContext.ts +++ b/src/runtime/systems/behaviorContext.ts @@ -23,7 +23,7 @@ import { DRY_VOICE, type FxParams } from '../../core/soundfx/nodes'; import type { Entity } from '../Entity'; import { PhysicsSystem } from '../physics/PhysicsSystem'; import type { World } from '../World'; -import type { AudioSink } from './Audio'; +import type { AudioSink, NoteTiming } from './Audio'; import type { Instance } from './behaviorPulses'; import type { Input } from './Input'; import { bodyOf, type CollisionSource, layerHolderOf, NOT_BLOCKED } from './solids'; @@ -167,10 +167,12 @@ export function behaviorContext( // eslint-disable-next-line no-console console.log(`[behavior ${entity.name}]`, message); }, - playSound: (assetId, volume, loop, detached, fx, fxParams) => - playSound(run, assetId, volume, loop, detached ? '' : entity.id, fx, fxParams), - playNote: (instrumentId, note, velocity, volume, duration, detached, params) => - playNote(run, instrumentId, { note, velocity, volume, duration }, detached ? '' : entity.id, params), + playSound: (assetId, volume, loop, detached, fx, fxParams, tempo) => + playSound(run, assetId, volume, loop, detached ? '' : entity.id, fx, fxParams, tempo), + playNote: (instrumentId, note, velocity, volume, duration, detached, params, tempo) => + playNote(run, instrumentId, { note, velocity, volume, duration }, detached ? '' : entity.id, params, { + tempo, + }), soundLength: (assetId) => { const file = soundFileOf(project, assetId); return file ? run.audio.length(file) : 0; @@ -240,6 +242,7 @@ function playSound( owner: string, fxId: string, fxParams: FxParams, + tempo: number, ): number { const asset = findAssetOf(project, assetId, 'sound'); if (!asset?.file) return 0; @@ -252,7 +255,7 @@ function playSound( level, loop ?? loopsOf(props), owner, - plan ? { plan, params: fxParams } : DRY_VOICE, + plan ? { plan, params: fxParams, tempo } : DRY_VOICE, ); } @@ -260,6 +263,7 @@ function playSound( * One note of an instrument (#331), as the sink understands it: the layers the instrument makes of * it, the velocity as loudness and as the value of its `velocity`, and the voice's volume held to * 0..1 like a sound's. An instrument that plays nothing for that note plays nothing at all. + * `timing` says when it starts and by which tempo (#332). */ export function playNote( { project, audio }: Pick, @@ -267,6 +271,7 @@ export function playNote( struck: { note: number; velocity: number; volume: number; duration: number }, owner: string, params: FxParams, + timing: NoteTiming = {}, ): number { const asset = findAssetOf(project, instrumentId, 'instrument'); if (!asset) return 0; @@ -276,7 +281,7 @@ export function playNote( const level = Math.max(0, Math.min(1, struck.volume * velocity)); const duration = Number.isFinite(struck.duration) ? Math.max(0, struck.duration) : 0; const values = { ...params, [VELOCITY_PARAM]: velocity }; - return audio.playNote(instrumentVoiceName(instrumentId), layers, values, level, owner, duration); + return audio.playNote(instrumentVoiceName(instrumentId), layers, values, level, owner, duration, timing); } /** What a note's voice is named by, where a sound's is named by its file. */ diff --git a/src/viewport/SoundLibrary.ts b/src/viewport/SoundLibrary.ts index 821b0c51..e1178a0f 100644 --- a/src/viewport/SoundLibrary.ts +++ b/src/viewport/SoundLibrary.ts @@ -12,7 +12,8 @@ import { type FxVoice, } from '../core/soundfx/nodes'; import type { EaseShare } from '../core/model/animation'; -import type { AudioSink } from '../runtime/systems/Audio'; +import { DEFAULT_TEMPO } from '../core/music/time'; +import type { AudioSink, NoteTiming } from '../runtime/systems/Audio'; import type { PitchOptions } from './pitchWorklet'; import pitchWorklet from './pitchWorklet.ts?worker&url'; @@ -44,7 +45,8 @@ interface Knob { * fall silent without a click while the echoes carry on (#324). */ interface Source { - start(): void; + /** At that second of the audio clock (#332); a worklet was told when it was made. */ + start(at: number): void; stop(): void; ended(then: () => void): void; fade: GainNode | null; @@ -82,11 +84,12 @@ interface Voice { /** * The plans of a voice and the values of their parameters: one plan for a sound played through its - * FX, one per layer for a note (#331). + * FX, one per layer for a note (#331), and the tempo a Delay synced to beats counts by (#332). */ interface Layered { layers: readonly FxPlan[]; params: FxParams; + tempo: number; } /** One way of a chain: what a source plays. */ @@ -98,6 +101,8 @@ type Way = Exclude; */ interface Wiring { context: AudioContext; + /** When its sources start on the audio clock. */ + at: number; buffers: ReadonlyMap; file: string; loop: boolean; @@ -120,6 +125,8 @@ interface Start { fx: Layered; /** Seconds until a note is released (#331); 0 holds it until it is released or stopped. */ duration: number; + /** When it starts on the audio clock (#332); one already past starts now. */ + at: number; } /** @@ -198,14 +205,15 @@ export class SoundLibrary implements AudioSink { */ play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE): number { if (file === '' || this.failed.has(file)) return 0; - const layered = { layers: [fx.plan], params: fx.params }; - return this.begin({ id: 0, file, files: [file], volume, loop, owner, fx: layered, duration: 0 }); + const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO }; + return this.begin({ id: 0, file, files: [file], volume, loop, owner, fx: layered, duration: 0, at: 0 }); } /** * One note of an instrument (#331): its layers played side by side as one voice, each way with its * own file. `name` stands for the instrument where a file would stand – what `levels` finds it by. - * A `duration` above 0 releases it that many seconds after it starts. + * A `duration` above 0 releases it that many seconds after it starts. `timing.at` starts it at that + * second of the audio clock, as the scheduler plans a note (#332). */ playNote( name: string, @@ -214,31 +222,42 @@ export class SoundLibrary implements AudioSink { volume: number, owner = '', duration = 0, + timing: NoteTiming = {}, ): number { const files = [...new Set(layers.flatMap(filesOf))].filter( (file) => file !== '' && !this.failed.has(file), ); if (files.length === 0) return 0; - const fx = { layers, params }; - return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration }); + const fx = { layers, params, tempo: timing.tempo ?? DEFAULT_TEMPO }; + const at = timing.at ?? 0; + return this.begin({ id: 0, file: name, files, volume, loop: false, owner, fx, duration, at }); } /** * The end of a note (#331): the ways that end with it fade out, a looping one that does not leaves * its loop, the others play on, and the voice goes once they have all ended – or rings on first - * where it has echoes. + * where it has echoes. `at` is the second of the audio clock it ends at (#332): the fade is set on + * the clock to the sample, leaving a loop waits for the page. */ - release(id: number): void { + release(id: number, at = 0): void { const waiting = this.pending.find((other) => other.id === id); if (waiting) waiting.released = true; const voice = this.voices.find((other) => other.id === id); if (!voice) return; - for (const source of voice.sources) if (!source.gate) source.leave?.(); const context = this.audio(); + const when = Math.max(at, context.currentTime); + const leaving = voice.sources.filter((source) => !source.gate && source.leave); + const leave = () => { + for (const source of leaving) source.leave!(); + }; + if (leaving.length > 0 && when > context.currentTime) + this.after(when, () => this.voices.includes(voice) && leave()); + else leave(); const gated = voice.sources.filter((source) => source.gate); - for (const source of gated) if (source.fade) fadeOut(source.fade.gain, context.currentTime); + for (const source of gated) if (source.fade) fadeOut(source.fade.gain, when); if (gated.length === 0) return; - this.after(context.currentTime + FADE, () => { + this.after(when + FADE, () => { + if (!this.voices.includes(voice)) return; for (const source of gated) { source.stop(); voice.finish?.(source); @@ -258,7 +277,7 @@ export class SoundLibrary implements AudioSink { const askedAt = this.cancelsFor(start.file, start.owner); const waiting: Pending = { id, - fx: { layers: start.fx.layers, params: { ...start.fx.params } }, + fx: { ...start.fx, params: { ...start.fx.params } }, timed: [], stopped: false, released: false, @@ -286,12 +305,12 @@ export class SoundLibrary implements AudioSink { // A quick change is where the voice starts; a timed one – and anything after it, so the order // holds – runs from the moment it is heard. if (time > 0 || waiting.timed.length > 0) waiting.timed.push({ params, time, ease }); - else waiting.fx = { layers: waiting.fx.layers, params: { ...waiting.fx.params, ...params } }; + else waiting.fx = { ...waiting.fx, params: { ...waiting.fx.params, ...params } }; return true; } const voice = this.voices.find((other) => other.id === id); if (!voice) return false; - voice.fx = { layers: voice.fx.layers, params: { ...voice.fx.params, ...params } }; + voice.fx = { ...voice.fx, params: { ...voice.fx.params, ...params } }; this.glide(voice, time, ease); // A longer feedback makes for a longer tail: it is counted again from where the change ends. if (voice.ringing) this.ringFor(voice, Math.max(0, time) + this.tailOf(voice.fx)); @@ -396,6 +415,34 @@ export class SoundLibrary implements AudioSink { return this.context?.currentTime ?? 0; } + /** Seconds on the audio clock, opening the context where there is none yet: what the scheduler plans by (#332). */ + time(): number { + const context = this.audio(); + if (context.state === 'suspended') void context.resume(); + return context.currentTime; + } + + /** Seconds between the audio clock and what is heard; 0 where the browser does not say. */ + latency(): number { + const context = this.context; + if (!context) return 0; + // Older Safari and Firefox have no `outputLatency`, some no `baseLatency` either. + const out = context.outputLatency; + if (Number.isFinite(out) && out > 0) return out; + return Number.isFinite(context.baseLatency) ? context.baseLatency : 0; + } + + /** + * Loads every file and the worklet these notes need, so the first note the scheduler plans is + * not late waiting on its bytes (#332). + */ + async prepare(layers: readonly FxPlan[]): Promise { + await Promise.all([ + ...[...new Set(layers.flatMap(filesOf))].filter((file) => file !== '').map((file) => this.load(file)), + layers.some(pitches) ? this.pitchWorklet() : null, + ]); + } + /** Where the pitch worklet is served from, for a check that runs it in an offline context (test hook). */ pitchWorkletUrl(): string { return pitchWorklet; @@ -520,7 +567,8 @@ export class SoundLibrary implements AudioSink { /** The longest tail of its layers: they are heard side by side. */ private tailOf(fx: Layered): number { return fx.layers.reduce( - (longest, plan) => (fxRings(plan) ? Math.max(longest, fxTailOf(fxChainOf(plan, fx.params))) : longest), + (longest, plan) => + fxRings(plan) ? Math.max(longest, fxTailOf(fxChainOf(plan, fx.params, fx.tempo))) : longest, 0, ); } @@ -547,16 +595,18 @@ export class SoundLibrary implements AudioSink { }; } - private start({ id, file, files, volume, loop, owner, fx, duration }: Start): void { + private start({ id, file, files, volume, loop, owner, fx, duration, at }: Start): void { const context = this.audio(); // A context starts suspended where the page has not been interacted with; without this the // voice would run silently and end as if it had played. if (context.state === 'suspended') void context.resume(); + const when = Math.max(at, context.currentTime); const gain = context.createGain(); gain.gain.value = Math.max(0, Math.min(1, volume)); const rings = fx.layers.some(fxRings); const wiring: Wiring = { context, + at: when, buffers: this.buffers, file, loop, @@ -566,7 +616,7 @@ export class SoundLibrary implements AudioSink { knobs: [], stages: 0, }; - for (const plan of fx.layers) this.wire(wiring, fxChainOf(plan, fx.params)).connect(gain); + for (const plan of fx.layers) this.wire(wiring, fxChainOf(plan, fx.params, fx.tempo)).connect(gain); gain.connect(context.destination); const { sources } = wiring; this.last = { @@ -577,6 +627,7 @@ export class SoundLibrary implements AudioSink { pitches: wiring.pitches, stages: wiring.stages, duration: Math.max(0, ...files.map((name) => this.buffers.get(name)?.duration ?? 0)), + at: when, }; const voice: Voice = { id, @@ -598,8 +649,8 @@ export class SoundLibrary implements AudioSink { for (const source of sources) source.ended(() => voice.finish?.(source)); this.voices.push(voice); this.started += 1; - for (const source of sources) source.start(); - if (duration > 0) this.after(context.currentTime + duration, () => this.release(id)); + for (const source of sources) source.start(when); + if (duration > 0) this.release(id, when + duration); } /** @@ -655,7 +706,12 @@ export class SoundLibrary implements AudioSink { const buffer = wiring.buffers.get(file) ?? context.createBuffer(1, 1, context.sampleRate); wiring.pitches.push(pitch); if (shifted && this.canPitch) { - const options: PitchOptions = { file, step: buffer.sampleRate / context.sampleRate, loop }; + const options: PitchOptions = { + file, + step: buffer.sampleRate / context.sampleRate, + loop, + startAt: wiring.at, + }; if (own && loop) Object.assign(options, loopIn(buffer, own.loopStart, own.loopEnd)); if (!this.sent.has(file)) { options.channels = Array.from({ length: buffer.numberOfChannels }, (_, c) => @@ -695,7 +751,7 @@ export class SoundLibrary implements AudioSink { source.loopEnd = own.loopEnd; } wiring.sources.push({ - start: () => source.start(), + start: (at) => source.start(at), stop: () => source.stop(), ended: (then) => { source.onended = then; @@ -720,7 +776,7 @@ export class SoundLibrary implements AudioSink { */ private glide(voice: Voice, time: number, ease: EaseShare): void { const levels: number[] = []; - for (const plan of voice.fx.layers) levelsOf(fxChainOf(plan, voice.fx.params), levels); + for (const plan of voice.fx.layers) levelsOf(fxChainOf(plan, voice.fx.params, voice.fx.tempo), levels); const now = this.audio().currentTime; voice.knobs.forEach((knob, i) => { const level = levels[i]; @@ -771,6 +827,8 @@ interface LastVoice { stages: number; /** Seconds of what was played: a pitched voice is as long as the sound itself (#312, #317). */ duration: number; + /** When it started on the audio clock, what a check times it from. */ + at: number; } /** Whether any way of a plan has a Pitch, and so needs the worklet. */ diff --git a/src/viewport/music.ts b/src/viewport/music.ts new file mode 100644 index 00000000..b86c1481 --- /dev/null +++ b/src/viewport/music.ts @@ -0,0 +1,109 @@ +import { INTERVAL, type ScheduledNote, Scheduler, type SchedulerOptions } from '../core/music/Scheduler'; +import type { Sequence } from '../core/music/sequence'; +import { instrumentBuildFor, instrumentSourcesOf } from '../core/instrument/compile'; +import { findAssetOf, type Project } from '../core/model/Project'; +import { playNote } from '../runtime/systems/behaviorContext'; +import TimerWorker from './musicTimer.ts?worker'; +import { soundLibrary } from './SoundLibrary'; + +/** Calls `pump` every `INTERVAL` from a worker; where there are no workers, from the page. */ +function wake(pump: () => void): () => void { + const ms = INTERVAL * 1000; + if (typeof Worker === 'undefined') { + const timer = setInterval(pump, ms); + return () => clearInterval(timer); + } + const worker = new TimerWorker(); + worker.onmessage = pump; + worker.postMessage(ms); + return () => worker.terminate(); +} + +/** One planned note as the player was given it, and where the audio clock stood then (test hook). */ +export interface PlannedNote extends ScheduledNote { + voice: number; + planned: number; +} + +/** + * The scheduler on the sound library's clock (#332), its notes played by the instruments of the + * project. Nothing in the editor plays a sequence yet – the piano roll (#333) will; until then the + * check scripts drive it through `__gm.music`. + */ +export class Music { + private scheduler: Scheduler | null = null; + private sequence: Sequence = { notes: [] }; + /** The last notes planned, newest last, for a check to hold against the clock. */ + readonly planned: PlannedNote[] = []; + + constructor(private readonly project: () => Project) {} + + /** Loads what the notes need, then plays from `from` ticks. */ + async play(sequence: Sequence, options: SchedulerOptions & { from?: number } = {}): Promise { + this.stop(); + this.sequence = sequence; + this.planned.length = 0; + await soundLibrary.prepare(this.layersOf(sequence)); + this.scheduler = new Scheduler( + { + now: () => soundLibrary.time(), + latency: () => soundLibrary.latency(), + play: (note) => this.playOne(note), + playing: (voice) => soundLibrary.playing(voice), + stop: (voices) => { + for (const voice of voices) soundLibrary.stopVoice(voice); + }, + wake, + }, + () => this.sequence, + options, + ); + this.scheduler.start(options.from ?? 0); + } + + /** A change heard from the next window on. */ + edit(sequence: Sequence): void { + this.sequence = sequence; + } + + stop(): void { + this.scheduler?.stop(); + this.scheduler = null; + } + + get current(): Scheduler | null { + return this.scheduler; + } + + private playOne(note: ScheduledNote): number { + const struck = { note: note.note, velocity: note.velocity, volume: 1, duration: note.duration }; + const voice = playNote( + { project: this.project(), audio: soundLibrary }, + note.instrument, + struck, + '', + {}, + { + at: note.at, + tempo: note.tempo, + }, + ); + this.planned.push({ ...note, voice, planned: soundLibrary.clock() }); + if (this.planned.length > 256) this.planned.shift(); + return voice; + } + + /** The layers of every note the sequence has, so their files are there before the first is due. */ + private layersOf(sequence: Sequence) { + const project = this.project(); + const sources = instrumentSourcesOf(project); + const seen = new Set(); + return sequence.notes.flatMap(({ instrument, note }) => { + const key = `${instrument}\u0000${note}`; + const asset = findAssetOf(project, instrument, 'instrument'); + if (seen.has(key) || !asset) return []; + seen.add(key); + return instrumentBuildFor(asset, note, sources).layers; + }); + } +} diff --git a/src/viewport/musicTimer.ts b/src/viewport/musicTimer.ts new file mode 100644 index 00000000..2320a4a3 --- /dev/null +++ b/src/viewport/musicTimer.ts @@ -0,0 +1,11 @@ +/** + * The scheduler's beat (#332): a worker's interval keeps its pace in a hidden tab, where the page's + * own timers are held to once a second and `requestAnimationFrame` stops. A message of so many + * milliseconds starts it, 0 stops it; each tick is an empty message back. + */ +let timer: ReturnType | null = null; + +self.onmessage = (event: MessageEvent) => { + if (timer !== null) clearInterval(timer); + timer = event.data > 0 ? setInterval(() => self.postMessage(0), event.data) : null; +}; diff --git a/src/viewport/pitchWorklet.ts b/src/viewport/pitchWorklet.ts index 4617bc4d..9de32ade 100644 --- a/src/viewport/pitchWorklet.ts +++ b/src/viewport/pitchWorklet.ts @@ -7,6 +7,7 @@ import { PitchPlayer } from '../core/soundfx/pitchPlayer'; /** What the audio thread has and the page's types do not. */ declare const sampleRate: number; +declare const currentTime: number; declare class AudioWorkletProcessor { readonly port: MessagePort; } @@ -25,6 +26,8 @@ export interface PitchOptions { /** Where a loop runs, in samples of the sound (#331); without them it is the whole sound. */ loopStart?: number; loopEnd?: number; + /** The second of the audio clock its first sample is heard at (#332); before it, silence. */ + startAt?: number; } /** Every sound a voice has been played from, for the next voice of it – processors share this scope. */ @@ -33,10 +36,12 @@ const sounds = new Map(); class PitchProcessor extends AudioWorkletProcessor { private readonly player: PitchPlayer; private stopped = false; + private startAt: number; constructor({ processorOptions }: { processorOptions: PitchOptions }) { super(); - const { file, channels, step, loop, loopStart, loopEnd } = processorOptions; + const { file, channels, step, loop, loopStart, loopEnd, startAt } = processorOptions; + this.startAt = startAt ?? 0; if (channels) sounds.set(file, channels); const region = loop && loopEnd !== undefined ? { start: loopStart ?? 0, end: loopEnd } : loop; this.player = new PitchPlayer(sounds.get(file) ?? [], step, sampleRate, region); @@ -52,7 +57,17 @@ class PitchProcessor extends AudioWorkletProcessor { parameters: Record, ): boolean { if (this.stopped) return false; - if (this.player.render(outputs[0]!, parameters.pitch!)) return true; + let out = outputs[0]!; + let pitch = parameters.pitch!; + // A block is silence until then, the one it falls in starts on that sample. + if (this.startAt > currentTime) { + const wait = Math.round((this.startAt - currentTime) * sampleRate); + if (wait >= (out[0]?.length ?? 0)) return true; + out = out.map((channel) => channel.subarray(wait)); + if (pitch.length > 1) pitch = pitch.subarray(wait); + this.startAt = 0; + } + if (this.player.render(out, pitch)) return true; this.port.postMessage('ended'); return false; } diff --git a/tests/core/music.test.ts b/tests/core/music.test.ts new file mode 100644 index 00000000..13fdaf96 --- /dev/null +++ b/tests/core/music.test.ts @@ -0,0 +1,299 @@ +import { describe, expect, it } from 'vitest'; +import { + INTERVAL, + LEAD, + LOOKAHEAD, + type ScheduledNote, + Scheduler, + type SchedulerHost, +} from '../../src/core/music/Scheduler'; +import { + heldLoop, + notesIn, + playheadAt, + type Sequence, + type SequenceNote, +} from '../../src/core/music/sequence'; +import { + heldMeter, + heldTempo, + positionOf, + secondsOfQuarters, + secondsOfTicks, + TICKS_PER_QUARTER, + ticksOfSeconds, + ticksPerBar, + ticksPerBeat, +} from '../../src/core/music/time'; + +const Q = TICKS_PER_QUARTER; + +function note(tick: number, length = Q / 2, pitch = 60): SequenceNote { + return { tick, length, note: pitch, velocity: 1, instrument: 'piano' }; +} + +describe('musical time (#332)', () => { + it('counts 480 ticks to the quarter, so triplets and quintuplets land on whole ticks', () => { + expect([Q / 3, Q / 5, Q / 6, Q / 8, Q / 12].every(Number.isInteger)).toBe(true); + expect(secondsOfTicks(Q, 120)).toBe(0.5); + expect(secondsOfTicks(Q / 3, 120)).toBeCloseTo(1 / 6); + expect(ticksOfSeconds(0.5, 120)).toBe(Q); + expect(secondsOfQuarters(0.75, 120)).toBe(0.375); + }); + + it('takes any meter: a beat is the note value under the line, a bar so many of them', () => { + expect([ticksPerBeat({ beats: 4, unit: 4 }), ticksPerBar({ beats: 4, unit: 4 })]).toEqual([Q, 4 * Q]); + expect([ticksPerBeat({ beats: 6, unit: 8 }), ticksPerBar({ beats: 6, unit: 8 })]).toEqual([Q / 2, 3 * Q]); + expect(ticksPerBar({ beats: 7, unit: 8 })).toBe(3.5 * Q); + expect(heldMeter({ beats: 3, unit: 4 })).toEqual({ beats: 3, unit: 4 }); + for (const odd of [ + { beats: 0, unit: 4 }, + { beats: 3.5, unit: 4 }, + { beats: 4, unit: 3 }, + { beats: 33, unit: 4 }, + ]) + expect(heldMeter(odd)).toEqual({ beats: 4, unit: 4 }); + }); + + it('holds a tempo to 20 to 400 BPM, 120 where it is none', () => { + expect([heldTempo(90), heldTempo(5), heldTempo(1000), heldTempo(0), heldTempo(Number.NaN)]).toEqual([ + 90, 20, 400, 120, 120, + ]); + }); + + it('names a tick as a musician counts it: bar and beat from 1, the tick in the beat from 0', () => { + expect(positionOf(0, { beats: 4, unit: 4 })).toEqual({ bar: 1, beat: 1, tick: 0 }); + expect(positionOf(5 * Q + 120, { beats: 4, unit: 4 })).toEqual({ bar: 2, beat: 2, tick: 120 }); + // 7/8: seven eighths of 240 ticks to the bar. + expect(positionOf(7 * 240 + 6 * 240 + 10, { beats: 7, unit: 8 })).toEqual({ bar: 2, beat: 7, tick: 10 }); + expect(positionOf(-30, { beats: 4, unit: 4 })).toEqual({ bar: 1, beat: 1, tick: 0 }); + }); +}); + +describe('the notes of a window of playing time (#332)', () => { + const sequence: Sequence = { notes: [note(0), note(Q), note(2 * Q), note(3 * Q)] }; + + it('runs the playhead round a loop it started in front of, and on past one it started behind', () => { + const loop = { start: Q, end: 3 * Q }; + expect([0, Q, 3 * Q - 1, 3 * Q, 4 * Q, 5 * Q].map((e) => playheadAt(e, 0, loop))).toEqual([ + 0, + Q, + 3 * Q - 1, + Q, + 2 * Q, + Q, + ]); + expect(playheadAt(Q, 4 * Q, loop)).toBe(5 * Q); + expect(playheadAt(Q, 0, null)).toBe(Q); + expect([heldLoop({ start: Q, end: Q }), heldLoop({ start: -1, end: Q })]).toEqual([null, null]); + }); + + it('gives the notes starting in it, by their start in playing time, and none twice across windows', () => { + expect(notesIn(sequence, 0, null, 0, Q + 1).map((n) => n.at)).toEqual([0, Q]); + const pieces = [0, 100, Q, Q + 1, 2 * Q + 37, 4 * Q] + .slice(1) + .map((end, i, ends) => [ends[i - 1] ?? 0, end]); + const joined = pieces.flatMap(([start, end]) => + notesIn(sequence, 0, null, start!, end!).map((n) => n.at), + ); + expect(joined).toEqual([0, Q, 2 * Q, 3 * Q]); + }); + + it('looks across the end of a loop into the next round, so the seam has no gap', () => { + const loop = { start: Q, end: 3 * Q }; + // Playing time 2.5 to 4.5 quarters: the end of the first round and the start of the second. + const due = notesIn(sequence, 0, loop, 2.5 * Q, 4.5 * Q); + expect(due.map((n) => [n.at, n.tick])).toEqual([ + [3 * Q, Q], + [4 * Q, 2 * Q], + ]); + // Round after round, every note once, two quarters apart. + const rounds = notesIn(sequence, 0, loop, 0, 11 * Q).map((n) => n.at); + expect(rounds).toEqual([0, Q, 2 * Q, 3 * Q, 4 * Q, 5 * Q, 6 * Q, 7 * Q, 8 * Q, 9 * Q, 10 * Q]); + const split = [0, 3 * Q - 7, 3 * Q, 3 * Q + 1, 7 * Q, 11 * Q]; + const pieces = split + .slice(1) + .flatMap((end, i) => notesIn(sequence, 0, loop, split[i]!, end).map((n) => n.at)); + expect(pieces).toEqual(rounds); + }); + + it('ends a note at the end of the loop, where the playhead jumps back', () => { + const long: Sequence = { notes: [note(2 * Q, 4 * Q)] }; + expect(notesIn(long, 0, { start: 0, end: 3 * Q }, 0, 3 * Q)[0]!.length).toBe(Q); + expect(notesIn(long, 0, null, 0, 3 * Q)[0]!.length).toBe(4 * Q); + }); + + it('keeps notes struck together in the order of their pitch', () => { + const chord: Sequence = { notes: [note(0, Q, 67), note(0, Q, 60), note(0, Q, 64)] }; + expect(notesIn(chord, 0, null, 0, 1).map((n) => n.note)).toEqual([60, 64, 67]); + }); +}); + +/** An audio clock the test moves by hand, and a player that only writes down what it is given. */ +function fakeHost() { + const host = { + time: 10, + latency: 0, + played: [] as ScheduledNote[], + stopped: [] as number[][], + sounding: new Set(), + pump: null as (() => void) | null, + asleep: 0, + }; + const sink: SchedulerHost = { + now: () => host.time, + latency: () => host.latency, + play: (scheduled) => { + host.played.push(scheduled); + host.sounding.add(host.played.length); + return host.played.length; + }, + playing: (voice) => host.sounding.has(voice), + stop: (voices) => void host.stopped.push([...voices]), + wake: (pump) => { + host.pump = pump; + return () => { + host.pump = null; + host.asleep += 1; + }; + }, + }; + /** Lets the clock run, the timer firing every `INTERVAL` on the way, as late as `jitter` says. */ + const run = (seconds: number, jitter = () => 0) => { + const end = host.time + seconds; + while (host.time + INTERVAL <= end + 1e-9) { + host.time += INTERVAL + jitter(); + host.pump?.(); + } + }; + return { host, sink, run }; +} + +describe('the scheduler on the audio clock (#332)', () => { + const quarters: Sequence = { notes: [0, 1, 2, 3, 4, 5, 6, 7].map((i) => note(i * Q)) }; + + it('plans each note at its exact second, a lookahead before it, the first a moment after the start', () => { + const { host, sink, run } = fakeHost(); + const scheduler = new Scheduler(sink, () => quarters, { tempo: 120 }); + scheduler.start(); + expect(host.played.map((n) => n.at)).toEqual([10 + LEAD]); + run(2); + // The fifth is due at 2.05 s, inside the 100 ms ahead of 2 s. + const offsets = host.played.map((n) => n.at - 10 - LEAD); + expect(offsets).toHaveLength(5); + offsets.forEach((offset, i) => expect(offset).toBeCloseTo(i * 0.5, 9)); + expect(host.played.every((n) => n.duration === 0.25 && n.tempo === 120 && n.instrument === 'piano')).toBe( + true, + ); + }); + + it('plans every note before it is due and never twice, however late the timer fires', () => { + const { host, sink, run } = fakeHost(); + const scheduler = new Scheduler(sink, () => quarters, { tempo: 150 }); + let planned = 0; + const late: number[] = []; + const watch = () => { + for (const n of host.played.slice(planned)) late.push(n.at - host.time); + planned = host.played.length; + }; + scheduler.start(); + watch(); + // Up to 60 ms late every time – a busy page – which is still inside the 100 ms ahead. + let k = 0; + const jitter = () => [0.06, 0, 0.03, 0.045][k++ % 4]!; + for (let i = 0; i < 160; i++) { + run(INTERVAL, jitter); + watch(); + } + expect(host.played).toHaveLength(8); + expect(Math.min(...late)).toBeGreaterThan(0); + host.played.forEach((n, i) => expect(n.at - 10 - LEAD).toBeCloseTo(i * 0.4, 9)); + }); + + it('runs a loop round and round without a gap at the seam', () => { + const { host, sink, run } = fakeHost(); + const twoNotes: Sequence = { notes: [note(0), note(Q)] }; + const scheduler = new Scheduler(sink, () => twoNotes, { tempo: 120, loop: { start: 0, end: 2 * Q } }); + scheduler.start(); + run(3); + const gaps = host.played.slice(1).map((n, i) => n.at - host.played[i]!.at); + expect(host.played.length).toBeGreaterThanOrEqual(6); + expect(gaps.every((gap) => Math.abs(gap - 0.5) < 1e-9)).toBe(true); + }); + + it('plays a change from the next window on and leaves what is planned', () => { + const { host, sink, run } = fakeHost(); + let sequence: Sequence = quarters; + const scheduler = new Scheduler(sink, () => sequence, { tempo: 120 }); + scheduler.start(); + run(0.6); + const before = host.played.length; + sequence = { notes: quarters.notes.map((n) => ({ ...n, note: 72 })) }; + run(2); + expect(host.played.slice(0, before).every((n) => n.note === 60)).toBe(true); + expect(host.played.slice(before).every((n) => n.note === 72)).toBe(true); + expect(host.played.length).toBeGreaterThan(before); + }); + + it('says where what is heard stands: the clock less the latency, in bars, beats and ticks', () => { + const { host, sink, run } = fakeHost(); + const scheduler = new Scheduler(sink, () => quarters, { tempo: 120, meter: { beats: 3, unit: 4 } }); + expect(scheduler.position()).toEqual({ bar: 1, beat: 1, tick: 0, ticks: 0 }); + scheduler.start(); + host.latency = 0.05; + run(1.75); + // 1.75 s on, less the lead and the latency: 1.65 s, 3.3 quarters – the second bar's first beat. + const heard = scheduler.position(); + expect([heard.bar, heard.beat]).toEqual([2, 1]); + expect(Math.abs(heard.tick - 0.3 * Q)).toBeLessThanOrEqual(1); + expect(heard.ticks).toBeCloseTo(3.3 * Q, 6); + }); + + it('stops every note it started, those still to come too, and plays on from there next time', () => { + const { host, sink, run } = fakeHost(); + const scheduler = new Scheduler(sink, () => quarters, { tempo: 120 }); + scheduler.start(); + run(0.6); + // The first note has ended by itself: only the second is left to stop. + host.sounding.delete(1); + run(0.025); + scheduler.stop(); + expect(host.stopped).toEqual([[2]]); + expect([scheduler.playing, host.pump, host.asleep]).toEqual([false, null, 1]); + expect(scheduler.position().ticks).toBe(Math.floor(ticksOfSeconds(0.625 - LEAD, 120))); + host.played = []; + scheduler.start(); + expect(host.played[0]?.note).toBeUndefined(); + run(0.5); + expect(host.played[0]!.at - host.time).toBeLessThan(LOOKAHEAD + 1e-9); + }); + + it('jumps: cuts what plays and goes on from the new place at once', () => { + const { host, sink, run } = fakeHost(); + const scheduler = new Scheduler(sink, () => quarters, { tempo: 120 }); + scheduler.start(); + run(0.3); + const started = host.time; + scheduler.seek(6 * Q); + expect(host.stopped).toEqual([[1]]); + expect(host.played.at(-1)).toMatchObject({ at: started + LEAD }); + expect(scheduler.position().ticks).toBe(6 * Q); + }); + + it('a new tempo counts from what is planned, and the playhead does not jump meanwhile', () => { + const { host, sink, run } = fakeHost(); + const scheduler = new Scheduler(sink, () => quarters, { tempo: 120 }); + scheduler.start(); + run(1.2); + const heard = scheduler.position().ticks; + const planned = host.played.length; + scheduler.setTempo(60); + expect(scheduler.position().ticks).toBeCloseTo(heard, 6); + run(3); + const after = host.played.slice(planned - 1).map((n) => n.at); + const gaps = after.slice(1).map((at, i) => at - after[i]!); + // The last note at the old tempo, then a second to the quarter. + expect(gaps.slice(1).every((gap) => Math.abs(gap - 1) < 1e-9)).toBe(true); + expect(host.played.at(-1)!.tempo).toBe(60); + }); +}); diff --git a/tests/core/node-catalog.test.ts b/tests/core/node-catalog.test.ts index a2414003..1d70ce24 100644 --- a/tests/core/node-catalog.test.ts +++ b/tests/core/node-catalog.test.ts @@ -1994,9 +1994,15 @@ const CASES: Record = { // `sounds.test.ts`). after: (ctx) => expect(ctx.played).toEqual([ - { assetId: 'snd-1', volume: 0.5, loop: true, detached: false, fx: '', fxParams: {} }, + { assetId: 'snd-1', volume: 0.5, loop: true, detached: false, fx: '', fxParams: {}, tempo: 120 }, ]), }, + { + name: 'hands on its tempo for a Delay synced to beats, held to 20 to 400 BPM (#332)', + inputs: { sound: 'snd-1', volume: 1, loop: '', tempo: 1000 }, + next: 'exec', + after: (ctx) => expect(ctx.played[0]?.tempo).toBe(400), + }, { name: 'detached says the voice is nobody’s, so destroying this object leaves it playing', inputs: { sound: 'snd-1', volume: 1, loop: '', detached: true }, @@ -2058,9 +2064,16 @@ const CASES: Record = { duration: 0.25, detached: false, instrumentParams: { cut: 2 }, + tempo: 120, }, ]), }, + { + name: 'hands on its tempo for a Delay synced to beats (#332)', + inputs: { instrument: 'ins-1', note: 60, tempo: 90 }, + next: 'exec', + after: (ctx) => expect(ctx.notes[0]?.tempo).toBe(90), + }, ], InstrumentParameterValues: [ { diff --git a/tests/core/soundfx.test.ts b/tests/core/soundfx.test.ts index ebcc77d5..3e3d60bb 100644 --- a/tests/core/soundfx.test.ts +++ b/tests/core/soundfx.test.ts @@ -18,6 +18,7 @@ import { EMPTY_PLAN, type FxChain, fxChainOf, + type FxDelay, type FxPlan, fxRings, type FxStage, @@ -54,7 +55,7 @@ function makeFx(f: ProjectFixture = makeProject()) { } /** Input, the given effects in a row, Output. */ -function line(effects: { type: string; params?: Record }[]) { +function line(effects: { type: string; params?: Record }[]) { const f = makeFx(); const input = f.edit.addNode('SoundFxInput'); let tail = input; @@ -306,6 +307,22 @@ describe('a Delay in a sound FX graph (#327)', () => { expect(fxChainOf(plan, { echo: 1 }).stages).toMatchObject([{ time: 1 }]); expect(fxChainOf(plan, { echo: 5 }).stages).toMatchObject([{ time: 2 }]); }); + + it('synced to beats counts its time in quarter notes at the tempo of the voice (#332)', () => { + const synced = (time: number) => + compileSoundFx(line([{ type: 'SoundFxDelay', params: { sync: 'beats', time } }]).fx.graph).compiled!; + expect(synced(0.75)).toMatchObject({ stages: [{ beats: true, time: { value: 0.75 } }] }); + // A dotted eighth at 120 BPM, an eighth triplet at 100, a quarter where no tempo is said. + expect(fxChainOf(synced(0.75), {}, 120).stages).toMatchObject([{ time: 0.375 }]); + expect((fxChainOf(synced(1 / 3), {}, 100).stages[0] as FxDelay).time).toBeCloseTo(0.2); + expect(fxChainOf(synced(1)).stages).toMatchObject([{ time: 0.5 }]); + // A whole note at 60 BPM is 4 s: held to the 2 s of the line, as is a tempo out of its range. + expect(fxChainOf(synced(4), {}, 60).stages).toMatchObject([{ time: 2 }]); + expect(fxChainOf(synced(1), {}, 1000).stages).toMatchObject([{ time: 0.15 }]); + // In seconds the tempo changes nothing. + const seconds = compileSoundFx(line([{ type: 'SoundFxDelay', params: { time: 0.75 } }]).fx.graph); + expect(fxChainOf(seconds.compiled!, {}, 60).stages).toMatchObject([{ time: 0.75 }]); + }); }); describe('the tail of a voice (#324)', () => { diff --git a/tests/helpers.ts b/tests/helpers.ts index 280ef673..54c1dc1d 100644 --- a/tests/helpers.ts +++ b/tests/helpers.ts @@ -213,6 +213,7 @@ export type FakeContext = NodeContext & { detached: boolean; fx: string; fxParams: Readonly>; + tempo: number; }[]; /** The notes the context was asked to play (#331); a voice's number is how many have been played. */ notes: { @@ -223,6 +224,7 @@ export type FakeContext = NodeContext & { duration: number; detached: boolean; instrumentParams: Readonly>; + tempo: number; }[]; silenced: string[]; tuned: { voice: number; fxParams: Readonly>; time: number; ease: EaseShare }[]; @@ -278,11 +280,11 @@ export function fakeContext(overrides: Partial = {}): FakeContext { silenced: [], loaded: [], // The number of a voice is how many have been played: unique, and never 0. - playSound: (assetId, volume, loop, detached, fx, fxParams) => - ctx.played.push({ assetId, volume, loop, detached, fx, fxParams }), + playSound: (assetId, volume, loop, detached, fx, fxParams, tempo) => + ctx.played.push({ assetId, volume, loop, detached, fx, fxParams, tempo }), notes: [], - playNote: (instrumentId, note, velocity, volume, duration, detached, instrumentParams) => - ctx.notes.push({ instrumentId, note, velocity, volume, duration, detached, instrumentParams }), + playNote: (instrumentId, note, velocity, volume, duration, detached, instrumentParams, tempo) => + ctx.notes.push({ instrumentId, note, velocity, volume, duration, detached, instrumentParams, tempo }), // Every sound lasts a quarter of a second, as if it had been loaded. soundLength: () => 0.25, // The tail of a voice is its number in tenths of a second, so a test can tell one from another. diff --git a/tests/viewport/sound-library.test.ts b/tests/viewport/sound-library.test.ts index 86190028..4ae41d58 100644 --- a/tests/viewport/sound-library.test.ts +++ b/tests/viewport/sound-library.test.ts @@ -60,9 +60,11 @@ interface FakeSource { playbackRate: FakeParam; onended: (() => void) | null; started: boolean; + /** The second of the audio clock it was started at. */ + startedAt: number; stopped: boolean; connect: (target: unknown) => unknown; - start: () => void; + start: (at?: number) => void; stop: () => void; } @@ -171,9 +173,11 @@ function fakeContext() { playbackRate: param(1), onended: null, started: false, + startedAt: -1, stopped: false, - start: () => { + start: (at?: number) => { source.started = true; + source.startedAt = at ?? 0; }, stop: () => { source.stopped = true; @@ -387,6 +391,7 @@ describe('the sound library', () => { pitches: [0], stages: 0, duration: 1.5, + at: 0, }); }); @@ -817,6 +822,30 @@ describe('the sound library', () => { expect(library.stats().playing).toBe(0); }); + it('counts an echo synced to beats by the tempo of the voice, never past the line (#332)', async () => { + const library = new SoundLibrary(); + const synced = (quarters: number): FxPlan => ({ + pitch: [], + stages: [ + { + kind: 'delay', + time: { value: quarters }, + beats: true, + feedback: { value: 0 }, + damping: { value: 0 }, + mix: { value: 1 }, + pingPong: false, + }, + ], + }); + const voice = await play(library, FILE, {}, { plan: synced(1), params: {}, tempo: 90 }); + expect(delays[0]!.delayTime.value).toBeCloseTo(60 / 90); + expect(library.tail(voice)).toBeCloseTo(60 / 90); + // Four quarters at 120 BPM are 2 s, which is as long as the line is; eight would be 4 s. + await play(library, FILE, {}, { plan: synced(8), params: {} }); + expect(delays[1]!.delayTime.value).toBe(2); + }); + it('lets the echoes ring when asked to: only the sound fades, the voice stays until they are gone', async () => { const library = new SoundLibrary(); await play(library, FILE, {}, { pitch: 0, stages: [echo()] }); @@ -943,17 +972,56 @@ describe('notes of an instrument (#331)', () => { expect(library.stats().playing).toBe(0); }); - it('a note with a duration is released that long after it starts, on the audio clock', async () => { + it('a note with a duration is released that long after it starts, its fade set on the audio clock at once (#332)', async () => { const library = new SoundLibrary(); await note(library, [sample(FILE)], 0.5); - expect(clocks.map((clock) => clock.stopAt)).toEqual([0.5]); - clockTo(0.5); - expect(gains[1]!.ramps).toEqual([['linear', 0, 0.01]]); + expect(gains[1]!.ramps).toEqual([ + ['set', 0.5, 0.5], + ['linear', 0, 0.51], + ]); + expect(clocks.map((clock) => clock.stopAt)).toEqual([0.51]); + expect(sources[0]!.stopped).toBe(false); clockTo(); expect(sources[0]!.stopped).toBe(true); expect(library.stats().playing).toBe(0); }); + it('a note planned ahead starts on that second of the audio clock and is released from there (#332)', async () => { + const library = new SoundLibrary(); + library.playNote('instrument:i', [sample(FILE)], {}, 1, '', 0.5, { at: 2 }); + await settle(); + await arrive(); + expect(sources[0]!.startedAt).toBe(2); + expect(library.stats().last?.at).toBe(2); + expect(gains[1]!.ramps).toEqual([ + ['set', 0.5, 2.5], + ['linear', 0, 2.51], + ]); + expect(clocks.map((clock) => clock.stopAt)).toEqual([2.51]); + // One whose second has passed by the time its bytes are there starts now. + library.playNote('instrument:i', [sample(FILE)], {}, 1, '', 0, { at: -1 }); + expect(sources[1]!.startedAt).toBe(0); + }); + + it('tells a pitched note planned ahead when to start, the worklet having no start of its own (#332)', async () => { + const library = new SoundLibrary(); + const pitched = { ...sample(FILE), pitch: [{ value: 7 }] } as FxPlan; + library.playNote('instrument:i', [pitched], {}, 1, '', 0, { at: 1.25 }); + await settle(); + await arrive(); + expect(worklets[0]!.options.processorOptions.startAt).toBe(1.25); + }); + + it('a note planned to end later leaves its loop then, not now (#332)', async () => { + const library = new SoundLibrary(); + await note(library, [sample(FILE, { loop: true, gate: false })], 1); + expect(sources[0]!.loop).toBe(true); + clockTo(0.99); + expect(sources[0]!.loop).toBe(true); + clockTo(1); + expect(sources[0]!.loop).toBe(false); + }); + it('a note released before its bytes arrived still starts, and is released at once', async () => { const library = new SoundLibrary(); const voice = library.playNote( diff --git a/tools/e2e/34-sound.mjs b/tools/e2e/34-sound.mjs index 9e8c33f9..58c60d72 100644 --- a/tools/e2e/34-sound.mjs +++ b/tools/e2e/34-sound.mjs @@ -34,6 +34,16 @@ export async function sound(t) { } return value; }; + // Every reading up to the one `done` takes: a check holds each against its own time on the audio + // clock, however late the first one comes in. + const trace = async (read, done, tries = 40) => { + const seen = [await read()]; + for (let i = 0; i < tries && !done(seen.at(-1)); i++) { + await sleep(150); + seen.push(await read()); + } + return seen; + }; await press('Escape'); await evaluate('document.activeElement instanceof HTMLElement && document.activeElement.blur()'); @@ -558,39 +568,50 @@ export async function sound(t) { JSON.stringify({ early, later, afterHalf }), ); await stop(); - // The easings of an animation key: Quad in hands the clock a curve, which is still near the start - // after 0.8 s and only then falls to the end (-5): 7 - 12 * (t / 2)^2, where linear is 7 - 6 * t. - // Timed on the audio clock from the moment the voice is first seen, which is no earlier than its - // start: the curve is then further along than `into` says, and still well above linear. + // The easings of an animation key: Quad in hands the clock a curve, 7 - 3 * t^2 towards -5 in 2 s, + // where linear is 7 - 6 * t. Each reading is timed on the audio clock from the voice's start, so a + // first one that comes in late is held against where the curve is then. Between 0.3 and 1.7 s the + // curve is more than a semitone above linear, and it never falls below it. await mutate(` const tune = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph.nodes.find((n) => n.id === 'sn-tune'); Object.assign(tune.params, { ease: 'quad', direction: 'in' });`); await play(); const pitchNow = () => json(`({ pitch: __gm.sound.levels(${JSON.stringify(blip.file)})?.[0]?.[0], clock: __gm.sound.clock() })`); - const sweepSeen = await waitFor(pitchNow, (s) => typeof s.pitch === 'number'); - const eased = await waitFor(pitchNow, (s) => s.clock - sweepSeen.clock >= 0.8); - const into = eased.clock - sweepSeen.clock; - const easedEnd = await waitFor(pitch, (s) => typeof s === 'number' && s < -4.3); + const sinceStart = async (seen) => { + const from = (await stats()).last.at; + return seen + .filter((s) => typeof s.pitch === 'number') + .map((s) => ({ t: +(s.clock - from).toFixed(3), pitch: +s.pitch.toFixed(3) })); + }; + const sweep = await sinceStart(await trace(pitchNow, (s) => typeof s.pitch === 'number' && s.pitch < -4.3)); + const midway = sweep.filter(({ t }) => t > 0.3 && t < 1.7); check( 'Set sound FX parameters with Quad in ramps slowly first, then on to the end', - eased.pitch < 6.95 && eased.pitch > 7 - 6 * into + 1 && easedEnd < -4.3, - JSON.stringify({ into, eased: eased.pitch, linearThere: 7 - 6 * into, easedEnd }), + midway.length > 0 && + midway.every(({ t, pitch }) => pitch > 7 - 6 * t + 1) && + sweep.every(({ t, pitch }) => pitch > 7 - 6 * t - 0.1) && + sweep.at(-1).pitch < -4.3, + JSON.stringify(sweep), ); await stop(); - // Step holds the pitch for the whole time and then jumps there. + // Step holds the pitch for the whole time and then jumps there: every reading before 2 s on the + // audio clock is still 7, the first one past it is the end. await mutate(` const tune = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph.nodes.find((n) => n.id === 'sn-tune'); - Object.assign(tune.params, { ease: 'step', time: 1 });`); + Object.assign(tune.params, { ease: 'step', time: 2 });`); await play(); - await waitFor(pitch, (s) => typeof s === 'number'); - await new Promise((resolve) => setTimeout(resolve, 500)); - const stepHeld = await pitch(); - const stepEnd = await waitFor(pitch, (s) => typeof s === 'number' && s < 6); + const steps = await sinceStart(await trace(pitchNow, (s) => typeof s.pitch === 'number' && s.pitch < 6)); + const held = steps.filter(({ pitch }) => pitch === 7); + const jumped = steps.find(({ pitch }) => pitch < 6); check( 'Set sound FX parameters with Step holds the pitch for the time, then jumps to the end', - stepHeld === 7 && Math.abs(stepEnd + 5) < 0.001, - JSON.stringify({ stepHeld, stepEnd }), + held.length > 0 && + held.every(({ t }) => t < 2.05) && + jumped !== undefined && + jumped.t > 1.95 && + Math.abs(jumped.pitch + 5) < 0.001, + JSON.stringify(steps), ); await stop(); // #317: the worklet itself, rendered in an offline context of this browser – what a muted check @@ -883,25 +904,104 @@ export async function sound(t) { await mutate(` const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph; graph.nodes.push(${JSON.stringify( - node('sn-note', 'PlayNote', 260, 220, { instrument: insId, note: 67, duration: 0.3 }), + node('sn-note', 'PlayNote', 260, 220, { instrument: insId, note: 67, duration: 1.5 }), )}); graph.edges.find((e) => e.id === 'sn-e1').to = { node: 'sn-note', port: 'exec' };`); await showTab('scene'); await evaluate('__gm.sound.stopAll()'); const beforeNote = await stats(); await play(); - const noted = await waitFor(stats, (s) => s.started > beforeNote.started); - await sleep(900); - const released = await stats(); + // The sample loops, so only the release after 1.5 s on the audio clock ends the note: it is heard + // well past the 0.25 s of the blip, never past its duration, and is gone soon after. + const noteNow = () => json('({ stats: __gm.sound.stats(), clock: __gm.sound.clock() })'); + const noted = await waitFor(noteNow, (s) => s.stats.started > beforeNote.started); + const heard = await trace(noteNow, (s) => s.stats.playing === 0); + const noteAt = noted.stats.last?.at ?? 0; + const heardAt = heard.filter((s) => s.stats.playing === 1).map((s) => +(s.clock - noteAt).toFixed(3)); + const goneAt = +(heard.at(-1).clock - noteAt).toFixed(3); check( 'a game’s Play note plays G4 – the blip seven semitones up – and lets it go after its duration', - noted.last?.file === `instrument:${insId}` && - noted.last.pitches.join() === '7' && - noted.playing === 1 && - released.playing === 0, - JSON.stringify({ noted, released }), + noted.stats.last?.file === `instrument:${insId}` && + noted.stats.last.pitches.join() === '7' && + heardAt.some((t) => t > 0.5) && + heardAt.every((t) => t < 1.6) && + heard.at(-1).stats.playing === 0 && + goneAt < 3, + JSON.stringify({ last: noted.stats.last, heardAt, goneAt }), ); await stop(); + + // #332: the scheduler, driven by the hook – nothing in the editor plays a sequence yet. A bar of + // quarters at 150 BPM in a loop: every note is planned before its second comes, each 0.4 s + // after the one before, also across the seam into the next round. Where the music stands + // is the audio clock less the output latency, counted from the first note. + const bar = [60, 62, 64, 65].map((pitch, i) => ({ + tick: i * 480, + length: 240, + note: pitch, + velocity: 1, + instrument: insId, + })); + await evaluate( + `__gm.music.play(${JSON.stringify({ notes: bar })}, { tempo: 150, loop: { start: 0, end: 1920 } })`, + ); + const planned = await waitFor( + () => json('__gm.music.planned()'), + (p) => p.length >= 9, + 40, + ); + const where = await json( + '({ position: __gm.music.position(), clock: __gm.sound.clock(), latency: __gm.music.latency() })', + ); + await evaluate('__gm.music.stop()'); + const musicOff = await waitFor(stats, (s) => s.playing === 0); + const apart = planned.slice(1, 9).map((p, i) => +(p.at - planned[i].at).toFixed(6)); + const ahead = planned.map((p) => +(p.at - p.planned).toFixed(3)); + const heardTicks = (((where.clock - where.latency - planned[0].at) * 150 * 480) / 60) % 1920; + check( + 'the scheduler plans a looped bar on the audio clock: every note ahead of its second, 0.4 s apart across the seam', + planned.length >= 9 && + planned + .slice(0, 9) + .map((p) => p.note) + .join() === '60,62,64,65,60,62,64,65,60' && + planned.every((p) => p.voice > 0 && Math.abs(p.duration - 0.2) < 1e-9 && p.tempo === 150) && + apart.every((gap) => Math.abs(gap - 0.4) < 1e-6) && + ahead.every((s) => s > 0 && s <= 0.16) && + Math.abs(where.position.ticks - heardTicks) < 1 && + where.position.bar === 1 && + musicOff.playing === 0 && + (await json('__gm.music.playing()')) === false, + JSON.stringify({ apart, ahead, where, heardTicks, playing: musicOff.playing }), + ); + + // A note planned ahead through the pitch worklet starts on its sample, also inside a render block: + // rendered offline at pitch 0, which plays the sound as it is, so the first sample heard is it. + const onsets = await evaluate(` + (async () => { + const RATE = 48000; + const onset = async (startAt) => { + const context = new OfflineAudioContext(1, RATE / 2, RATE); + await context.audioWorklet.addModule(__gm.sound.pitchWorkletUrl()); + const node = new AudioWorkletNode(context, 'pitch-player', { + numberOfInputs: 0, + outputChannelCount: [1], + parameterData: { pitch: 0 }, + processorOptions: { file: 'onset', channels: [new Float32Array(RATE).fill(0.5)], step: 1, loop: false, startAt }, + }); + node.connect(context.destination); + const out = (await context.startRendering()).getChannelData(0); + return out.findIndex((v) => Math.abs(v) > 1e-6); + }; + const at = [0.0123, 0.25, 0.05 + 0.4 * 2 / 3]; + return { expected: at.map((s) => Math.round(s * RATE)), heard: await Promise.all(at.map(onset)) }; + })()`); + check( + 'a pitched note planned ahead starts on its very sample, inside a render block too', + onsets.heard.join() === onsets.expected.join() && onsets.expected.some((i) => i % 128 !== 0), + JSON.stringify(onsets), + ); + await mutate(` const graph = __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph; graph.nodes = graph.nodes.filter((n) => n.id !== 'sn-note');