Stream long sounds from an Opus copy instead of decoding them (#349)

A sound was decoded whole as soon as the project opened, in the editor
and in a released game: five minutes of stereo held some 115 MB for as
long as the page was open. A sound with Stream on now plays from a copy
as Opus in Ogg, fetched a piece at a time and decoded a few seconds
ahead by WebCodecs, so a game holds a few megabytes for it.

- The Stream switch is a row of the sound, off by default. The Sound
  tab suggests it from a minute on and says how far the copy has got.
- StreamKeeper makes the copies in the browser, a moment after the
  last change, one at a time: the sound's own and one per clip of its
  cut, fades and Reverse baked in. A copy that is no longer wanted is
  taken off and deleted once the project is saved without it.
- streamTable says which played names stream from which copy. A clip
  without a fitting copy, a Sample and a Reverb impulse stay decoded.
  A sound with Stream on is never decoded ahead.
- The page fetches by Range and takes the whole file from a server
  that ignores it, as the CI's PHP server does. It keeps the bytes
  and the first two seconds decoded, the rest goes four seconds ahead.
- A jump decodes 32 packets before the place first. Measured in
  Chrome, what follows is then the same to the bit as decoding from
  the start. With 4 packets it was 0.06 off.
- The stream-player on the FX worklet reads the samples at the rate
  of the context along a Catmull-Rom curve. What comes late is silent
  and skipped once it is there, so a clip keeps its beat.
- The opening stretch goes with the player as it is made. Sent by
  message it came a few milliseconds late, and 557 samples of the
  attack were lost.
- Pitch on a streamed sound is a shifter after the player, as on a
  bus: the length stays and it is heard 42.7 ms late.
- Bakes of patterns and songs (#340) always stream now.
- The song timeline draws a streamed sound from its copy decoded at
  8 kHz, keeping only the peaks.

Tests:
- Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3,
  stream-queue 6, streams 11, streaming 6, sound-library +5).
- e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono,
  4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by
  5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and
  a reversed clip are the copy decoded whole to the sample (0).
- Counter-checks red as they should be: no pre-roll, no catch-up of
  late samples at the end, no opening stretch, a streamed sound
  decoded ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-09 20:27:32 +02:00
1 parent 84cd36f902
commit 2764919bb8
30 files changed
+2443 -146

No files matched your search

+386 -88
View File
@@ -31,11 +31,13 @@ import { limiterLatency } from '../core/soundfx/dynamics';
import type { ScheduledClip, ScheduledNote } from '../core/music/Scheduler';
import { dbToGain, SOUND_ITEM_MAX_DB } from '../core/music/song';
import { DEFAULT_TEMPO } from '../core/music/time';
import { playedPlan, type StreamedFile, type StreamPlan } from '../core/music/streams';
import { STREAM_RATE } from '../core/music/streamQueue';
import { type CutRange, cutChannels, cutRange, parseCutName } from '../core/model/sound';
import { shifterLatency } from '../core/soundfx/pitchPlayer';
import { REVERB_MAX_DECAY } from '../core/soundfx/reverb';
import type { AudioSink, ClipTiming, NoteTiming } from '../runtime/systems/Audio';
import type { ChipOptions, PitchOptions } from './fxWorklet';
import type { ChipOptions, PitchOptions, StreamOptions } from './fxWorklet';
import fxWorkletUrl from './fxWorklet.ts?worker&url';
import {
bothAtCenter,
@@ -57,6 +59,7 @@ import {
stopProcessor,
} from './fxNodes';
import { type Spectrum, SpectrumTaps } from './spectrum';
import { decodeSpan, StreamFeed, StreamFile } from './streaming';
import { wake } from './wake';
/** A voice is no louder than a sound item of a song turned all the way up. */
@@ -65,6 +68,8 @@ const MAX_VOLUME = dbToGain(SOUND_ITEM_MAX_DB);
const DEFAULT_BASE = '/game-maker/data/sounds';
/** The rate a sound loaded ahead of its first play is decoded at; a source resamples it on the way out. */
const DECODE_RATE = 48000;
/** The rate the picture of a streamed sound is decoded at (#349): enough for its peaks, a sixth of the size. */
const PICTURE_RATE = 8000;
/** The longest a send of a track is held back to meet the latest way of it (#350). */
const MAX_SEND_DELAY = 1;
@@ -199,6 +204,8 @@ interface Wiring {
/** Seconds into the sound its ways start, and whether it is a clip of a song, which fades out at its end (#334). */
offset: number;
clip: boolean;
/** The copy a way without a source of its own streams (#349), instead of a buffer of `file`. */
stream: StreamFile | null;
sources: Source[];
drivers: Driver[];
envelopes: Map<string, EnvelopeRun>;
@@ -229,6 +236,8 @@ interface Start {
clip?: { offset: number; end: number };
/** The track of a song it is heard on (#334), `''` straight out. */
bus?: string;
/** The copy `file` streams from (#349); its bytes are not loaded then, its head is. */
stream?: StreamFile;
}
/**
@@ -262,6 +271,9 @@ interface Pending {
*
* A way with a Pitch is played by a worklet (#317), loaded into the context with the first play that
* needs it; that play waits for it as it would for its bytes.
*
* A sound that streams (#349) is the exception to "decoded whole": a `stream-player` on that worklet
* plays it from its copy, fed a few seconds ahead (`streaming.ts`), and its file is never decoded.
*/
export class SoundLibrary implements AudioSink {
private base = DEFAULT_BASE;
@@ -331,6 +343,7 @@ export class SoundLibrary implements AudioSink {
/** The taps of the voices and buses playing now, and the node listened to (#374). */
private readonly spectra = new SpectrumTaps();
private readonly peaked = new Map<string, Float32Array>();
private readonly pictures = new Map<string, Promise<void>>();
/**
* The patterns and songs a game plays, each on a scheduler of its own. What it asks after are its
* own voices – asking `playing`, which counts the playings too, would ask the list it is in.
@@ -363,12 +376,40 @@ export class SoundLibrary implements AudioSink {
private mix: readonly MixBus[] = [];
/** The levels a running game set its buses to, over those of their assets; until `resetMix`. */
private readonly mixVolumes = new Map<string, number>();
/** The names streamed (#349) and the copies they stream from, by name and by copy. */
private streams: ReadonlyMap<string, StreamedFile> = new Map();
private readonly streamFiles = new Map<string, StreamFile>();
private readonly feeds = new Set<StreamFeed>();
/** Every sample streamed so far that came too late to be heard, of the feeds gone (test hook). */
private skippedBefore = 0;
/** Where the files are served from. Set once at start-up, before anything plays. */
configure(base: string): void {
this.base = base.replace(/\/+$/, '');
}
/**
* The names a game streams rather than decodes (#349), each with its copy (`streamTable`): the first
* seconds of every copy are fetched and decoded now, so it starts at once. A library that bakes,
* and a browser without Opus in WebCodecs, decode everything as before.
*/
setStreams(table: ReadonlyMap<string, StreamedFile>): void {
this.streams = table;
for (const name of table.keys()) void this.streamed(name)?.ready();
}
/** The copy a name streams from, while it streams and the copy has not failed. */
private streamed(name: string): StreamFile | null {
const entry = this.streams.get(name);
if (!entry || this.render || typeof AudioDecoder === 'undefined') return null;
let file = this.streamFiles.get(entry.file);
if (!file) {
file = new StreamFile(`${this.base}/${entry.file}`, entry.frames);
this.streamFiles.set(entry.file, file);
}
return file.failed ? null : file;
}
/**
* The number is handed out at once, before the bytes are there: a graph can tune a voice in the
* same frame it asked for it, and the voice then starts with those values.
@@ -376,6 +417,7 @@ export class SoundLibrary implements AudioSink {
play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE, bus = ''): number {
if (file === '' || this.hasFailed(file)) return 0;
const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO };
const stream = this.streamed(file);
return this.begin({
id: 0,
file,
@@ -387,6 +429,7 @@ export class SoundLibrary implements AudioSink {
duration: 0,
at: 0,
bus,
...(stream && { stream }),
});
}
@@ -424,6 +467,7 @@ export class SoundLibrary implements AudioSink {
if (file === '' || this.hasFailed(file)) return 0;
const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO };
const { at, offset, duration, volume, bus } = clip;
const stream = this.streamed(file);
return this.begin({
id: 0,
file,
@@ -436,6 +480,7 @@ export class SoundLibrary implements AudioSink {
at,
clip: { offset, end: at + Math.max(0, duration) },
bus,
...(stream && { stream }),
});
}
@@ -1050,19 +1095,102 @@ export class SoundLibrary implements AudioSink {
/**
* The bake of a playing (#340) as one voice on its master, heard from `heard`: its file once to its
* end, faded out there, or up to the loop end and then round its second round – in one source where
* the round follows at once, else in a second that takes over to the sample. 0 without its bytes.
* end, faded out there, or up to the loop end and then round its second round. Streamed where it can
* be (#349), else decoded – in one source where the round follows at once, else in a second that
* takes over to the sample. 0 without its bytes.
*/
private playBaked(id: number, bake: BakePlay, heard: number): number {
const buffer = this.buffer(bake.file);
if (!buffer) return 0;
const stream = this.hasWorklet ? this.streamed(bake.file) : null;
const streaming = stream?.head ? stream : null;
const buffer = streaming ? null : this.buffer(bake.file);
if (!streaming && !buffer) return 0;
const context = this.audio();
const bus = groupBus(id, '');
const at = Math.max(context.currentTime, heard - this.busLatency(bus));
const gain = context.createGain();
gain.gain.value = Math.max(0, Math.min(MAX_VOLUME, bake.gain));
gain.connect(this.busInput(bus));
const { end, until, round } = bake.playback;
const { end } = bake.playback;
const played = streaming
? this.bakeStreamed(streaming, bake.playback, gain, at)
: this.bakeDecoded(buffer!, bake.playback, gain, at);
const voice: Voice = {
id: this.nextId++,
file: bake.file,
owner: '',
bus,
at,
sources: played.sources,
ending: new Set(),
drivers: [],
envelopes: [],
finish: null,
gain,
fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO },
knobs: [],
rings: false,
ringing: false,
quiet: null,
vocoders: 0,
released: false,
ringFade: BAKE_LET_RING,
};
const going = new Set(voice.sources);
voice.finish = (source) => {
if (going.delete(source) && going.size === 0) this.ended(voice);
};
for (const source of played.sources) source.ended(() => voice.finish?.(source));
this.voices.push(voice);
this.started += 1;
this.last = {
file: bake.file,
volume: gain.gain.value,
loop: bake.playback.round !== null,
layers: 0,
pitches: [],
stages: 0,
duration: played.duration,
at,
envelopes: 0,
bus,
};
if (end !== null) {
gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE));
gain.gain.linearRampToValueAtTime(0, at + end);
}
played.start();
return voice.id;
}
/** A bake streamed (#349): one `stream-player` along the same plan, the round looped to the sample. */
private bakeStreamed(
stream: StreamFile,
{ end, until, round }: BakePlay['playback'],
gain: GainNode,
at: number,
): { sources: Source[]; start: () => void; duration: number } {
const frames = (seconds: number) => Math.round(seconds * STREAM_RATE);
const plan: StreamPlan =
end !== null
? { from: 0, until: frames(end), loop: null }
: {
from: 0,
until: frames(until),
loop: round && { start: frames(round.start), end: frames(round.end) },
};
const { node, source } = this.streamNode(this.audio(), stream, plan, at);
node.connect(gain);
return { sources: [source], start: () => {}, duration: stream.frames / STREAM_RATE };
}
/** A bake decoded, in one buffer source or two. */
private bakeDecoded(
buffer: AudioBuffer,
{ end, until, round }: BakePlay['playback'],
gain: GainNode,
at: number,
): { sources: Source[]; start: () => void; duration: number } {
const context = this.audio();
const made = (from: number, loop: { start: number; end: number } | null) => {
const node = context.createBufferSource();
node.buffer = buffer;
@@ -1090,56 +1218,15 @@ export class SoundLibrary implements AudioSink {
: round
? [made(0, null), made(round.start, round)]
: [made(0, null)];
const voice: Voice = {
id: this.nextId++,
file: bake.file,
owner: '',
bus,
at,
sources: parts.map((part) => part.source),
ending: new Set(),
drivers: [],
envelopes: [],
finish: null,
gain,
fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO },
knobs: [],
rings: false,
ringing: false,
quiet: null,
vocoders: 0,
released: false,
ringFade: BAKE_LET_RING,
};
const going = new Set(voice.sources);
voice.finish = (source) => {
if (going.delete(source) && going.size === 0) this.ended(voice);
};
for (const { source } of parts) source.ended(() => voice.finish?.(source));
this.voices.push(voice);
this.started += 1;
this.last = {
file: bake.file,
volume: gain.gain.value,
loop: round !== null,
layers: 0,
pitches: [],
stages: 0,
duration: buffer.duration,
at,
envelopes: 0,
bus,
};
const [head, next] = parts;
if (end !== null) {
gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE));
gain.gain.linearRampToValueAtTime(0, at + end);
head!.node.start(at, 0, end);
} else if (next) {
head!.node.start(at, 0, until);
next.node.start(at + until, next.from);
} else head!.node.start(at, 0);
return voice.id;
const start = () => {
if (end !== null) head!.node.start(at, 0, end);
else if (next) {
head!.node.start(at, 0, until);
next.node.start(at + until, next.from);
} else head!.node.start(at, 0);
};
return { sources: parts.map((part) => part.source), start, duration: buffer.duration };
}
/**
@@ -1304,8 +1391,12 @@ export class SoundLibrary implements AudioSink {
const asked: Start = { ...start, id };
// A Reverb's sound (#328) is waited for as well, but a voice is not silent for lack of it.
const impulseFiles = start.fx.layers.flatMap(impulsesOf);
const loaded = [...start.files, ...impulseFiles].every((file) => this.buffer(file) !== null);
const worklet = start.fx.layers.some(needsWorklet) ? this.fxWorklet() : null;
// A streamed sound (#349) waits for the head of its copy and the worklet that plays it instead.
const { stream } = start;
const loaded =
[...(stream ? [] : start.files), ...impulseFiles].every((file) => this.buffer(file) !== null) &&
(!stream || stream.head !== null);
const worklet = stream || start.fx.layers.some(needsWorklet) ? this.fxWorklet() : null;
if (loaded && (!worklet || this.hasWorklet !== null)) {
this.start(asked);
return id;
@@ -1321,19 +1412,23 @@ export class SoundLibrary implements AudioSink {
};
this.pending.push(waiting);
const impulsesLoaded = Promise.all(impulseFiles.map((file) => this.load(file)));
void Promise.all([Promise.all(start.files.map((file) => this.load(file))), worklet, impulsesLoaded]).then(
([buffers]) => {
this.pending = this.pending.filter((other) => other !== waiting);
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
const silent = buffers.every((buffer) => !buffer) && !start.fx.layers.some(hasChip);
if (silent || waiting.stopped) return;
if (this.cancelsFor(start.file, start.owner) !== askedAt) return;
this.start({ ...asked, fx: waiting.fx });
for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease);
if (waiting.released) this.release(id);
},
);
// A copy that cannot be streamed leaves the sound decoded, as it would be without one.
const sounds = stream
? stream.ready().then(async (ready) => [ready || (await this.load(start.file)) !== null])
: Promise.all(start.files.map((file) => this.load(file)));
void Promise.all([sounds, worklet, impulsesLoaded]).then(([buffers]) => {
this.pending = this.pending.filter((other) => other !== waiting);
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
const silent = buffers.every((buffer) => !buffer) && !start.fx.layers.some(hasChip);
if (silent || waiting.stopped) return;
if (this.cancelsFor(start.file, start.owner) !== askedAt) return;
const started: Start = { ...asked, fx: waiting.fx };
if (stream?.failed) delete started.stream;
this.start(started);
for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease);
if (waiting.released) this.release(id);
});
return id;
}
@@ -1463,7 +1558,13 @@ export class SoundLibrary implements AudioSink {
/** Ways playing through the pitch worklet now, at most `MAX_VOCODERS`. */
vocoders: number;
last: LastVoice | null;
/**
* Streaming (#349): sources playing from a copy now, samples that came too late to be heard, the
* bytes fetched of every copy, and the copies whose head is decoded.
*/
streaming: { playing: number; skipped: number; fetched: number; heads: number };
} {
const files = [...this.streamFiles.values()];
return {
playing: this.voices.length,
started: this.started,
@@ -1475,6 +1576,12 @@ export class SoundLibrary implements AudioSink {
worklet: this.hasWorklet,
vocoders: this.voices.reduce((sum, voice) => sum + vocodersOf(voice), 0),
last: this.last,
streaming: {
playing: this.feeds.size,
skipped: [...this.feeds].reduce((sum, feed) => sum + feed.skipped, this.skippedBefore),
fetched: files.reduce((sum, file) => sum + file.bytes, 0),
heads: files.filter((file) => file.head !== null).length,
},
};
}
@@ -1538,11 +1645,21 @@ export class SoundLibrary implements AudioSink {
* not late waiting on its bytes (#332).
*/
async prepare(layers: readonly FxPlan[], files: readonly string[] = []): Promise<void> {
// A Sample or a Reverb's sound is always decoded; a sound or a bake may stream (#349).
const decoded = new Set([...layers.flatMap(filesOf), ...layers.flatMap(impulsesOf)]);
const streamed = files.flatMap((file) => {
const stream = decoded.has(file) ? null : this.streamed(file);
return stream ? [{ file, stream }] : [];
});
const loaded = [...new Set([...decoded, ...files])].filter(
(file) => file !== '' && !streamed.some((entry) => entry.file === file),
);
await Promise.all([
...[...new Set([...layers.flatMap(filesOf), ...layers.flatMap(impulsesOf), ...files])]
.filter((file) => file !== '')
.map((file) => this.load(file)),
layers.some(needsWorklet) ? this.fxWorklet() : null,
...loaded.map((file) => this.load(file)),
...streamed.map(async ({ file, stream }) => {
if (!(await stream.ready())) await this.load(file);
}),
layers.some(needsWorklet) || streamed.length > 0 ? this.fxWorklet() : null,
]);
}
@@ -1562,7 +1679,10 @@ export class SoundLibrary implements AudioSink {
*/
duration(file: string): number | null {
const range = this.rangeOf(file);
return range ? (range.to - range.from) / range.whole.sampleRate : null;
if (range) return (range.to - range.from) / range.whole.sampleRate;
// A streamed one (#349) is as long as its copy, known without decoding a sample of it.
const stream = this.streams.get(file);
return stream ? stream.frames / STREAM_RATE : null;
}
/** The decoded samples of a file as it is, without a cut; null while it has not been loaded. */
@@ -1588,27 +1708,49 @@ export class SoundLibrary implements AudioSink {
if (known) return known;
const range = this.rangeOf(file);
if (!range) return null;
const { whole, from, to, reverse } = range;
const peaks = new Float32Array(PEAK_BUCKETS);
for (let c = 0; c < whole.numberOfChannels; c++) {
const data = whole.getChannelData(c);
for (let i = from; i < to; i++) {
const heard = reverse ? to - 1 - i : i - from;
const bucket = Math.floor((heard / (to - from)) * PEAK_BUCKETS);
peaks[bucket] = Math.max(peaks[bucket]!, Math.abs(data[i]!));
}
}
for (let b = 0; b < PEAK_BUCKETS; b++) peaks[b] = Math.min(1, peaks[b]!);
const peaks = peaksOf(range.whole, range);
this.peaked.set(file, peaks);
return peaks;
}
/**
* Loads what `peaks` and `duration` read for a name. A streamed one (#349) is not loaded whole for
* it: its copy is decoded at `PICTURE_RATE`, its peaks kept and the samples let go.
*/
loadPicture(name: string): Promise<void> {
const stream = this.streamed(name);
if (!stream) return this.load(name).then(() => {});
if (this.peaked.has(name)) return Promise.resolve();
let picture = this.pictures.get(name);
if (!picture) {
picture = this.draw(name, stream);
this.pictures.set(name, picture);
}
return picture;
}
private async draw(name: string, stream: StreamFile): Promise<void> {
try {
const bytes = await stream.whole();
const picture = await new OfflineAudioContext(1, 1, PICTURE_RATE).decodeAudioData(bytes);
this.peaked.set(name, peaksOf(picture, { from: 0, to: picture.length, reverse: false }));
} catch (error: unknown) {
console.error(`[sounds] cannot draw ${name}`, error);
}
}
/**
* Starts loading every file not held yet. Waiting for the first play left that play late: its
* start was not heard, and a graph tuning it in the meantime had moved on by the time it started.
* One that streams (#349) has the head of its copy loaded instead.
*/
preload(files: Iterable<string>): void {
for (const file of files) if (file) void this.load(file);
for (const file of files) {
if (!file) continue;
const stream = this.streamed(file);
if (stream) void stream.ready();
else void this.load(file);
}
}
/**
@@ -1643,6 +1785,13 @@ export class SoundLibrary implements AudioSink {
return work;
}
/** A file decoded at `rate` and not kept: what a stream copy is made from (#349). */
async decodeApart(file: string, rate: number): Promise<AudioBuffer> {
const response = await fetch(`${this.base}/${file}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
return new OfflineAudioContext(1, 1, rate).decodeAudioData(await response.arrayBuffer());
}
private async fetched(file: string): Promise<AudioBuffer> {
const response = await fetch(`${this.base}/${file}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
@@ -1806,6 +1955,7 @@ export class SoundLibrary implements AudioSink {
at: heard,
clip: asked,
bus = '',
stream,
}: Start): void {
const context = this.audio();
// A context starts suspended where the page has not been interacted with; without this the
@@ -1840,6 +1990,7 @@ export class SoundLibrary implements AudioSink {
rings,
offset: clip ? clip.offset + (when - at) : 0,
clip: clip !== undefined,
stream: stream ?? null,
sources: [],
drivers: [],
envelopes: new Map(),
@@ -1990,7 +2141,15 @@ export class SoundLibrary implements AudioSink {
return sound.connect(fade);
}
private sound(wiring: Wiring, { pitch, shifted, source: own, pitchMods }: Way): AudioNode {
private sound(wiring: Wiring, way: Way): AudioNode {
const { pitch, shifted, source: own, pitchMods } = way;
if (!own && wiring.stream && this.hasWorklet)
return this.streamSource(
wiring,
wiring.stream,
playedPlan(wiring.stream.frames, wiring.offset, wiring.loop),
way,
);
const { context } = wiring;
const file = own?.file ?? wiring.file;
const loop = own ? own.loop : wiring.loop;
@@ -2063,6 +2222,127 @@ export class SoundLibrary implements AudioSink {
return source;
}
/**
* A way streamed from its copy (#349): a `stream-player` fed along its plan. Its Pitch is a shifter
* of what it plays, as on a bus (#346): its length stays, and it is heard a frame late.
*/
private streamSource(
wiring: Wiring,
file: StreamFile,
plan: StreamPlan,
{ pitch, shifted, pitchMods }: Way,
): AudioNode {
const { context } = wiring;
wiring.pitches.push(pitch);
const { node, source } = this.streamNode(context, file, plan, wiring.at);
wiring.sources.push(source);
if (!shifted) return node;
const shifter = new AudioWorkletNode(context, 'pitch-shifter', {
numberOfInputs: 1,
numberOfOutputs: 1,
outputChannelCount: [2],
channelCount: 2,
channelCountMode: 'explicit',
parameterData: { pitch },
});
node.connect(shifter);
wiring.vocoders += 1;
wiring.drivers.push({ start: () => {}, stop: () => stopProcessor(shifter) });
const knob = shifter.parameters.get('pitch')!;
wiring.knobs.push({ param: knob, target: pitch, even: 'value' });
const follow = this.follower(wiring);
for (const mod of pitchMods ?? []) follow(mod, knob, false);
return shifter;
}
/** A `stream-player` fed along `plan`, heard from `at` on, and the source a voice holds it by. */
private streamNode(
context: BaseAudioContext,
file: StreamFile,
plan: StreamPlan,
at: number,
): { node: AudioWorkletNode; source: Source; feed: StreamFeed } {
let then: (() => void) | null = null;
const feed: StreamFeed = new StreamFeed(file, plan, () => {
this.dropFeed(feed);
then?.();
});
const options: StreamOptions = { channels: file.channels, startAt: at, first: feed.opening() };
const node = new AudioWorkletNode(context, 'stream-player', {
numberOfInputs: 0,
outputChannelCount: [file.channels],
processorOptions: options,
});
feed.attach(node);
this.feeds.add(feed);
const source: Source = {
start: () => {},
stop: () => {
this.dropFeed(feed);
feed.stop();
},
ended: (call) => {
then = call;
},
fade: null,
gate: false,
leave: null,
};
return { node, source, feed };
}
/**
* Streams a name along `plan` into an offline context at `STREAM_RATE`, heard from `at`, `frames`
* long, with the player and the feed a game has (test hook). The clock is held every half second
* until the feed has caught up: offline it runs faster than anything is decoded. Null where the
* name does not stream.
*/
async renderStream(
name: string,
plan: StreamPlan,
at: number,
frames: number,
): Promise<{ channels: Float32Array[]; skipped: number } | null> {
const file = this.streamed(name);
if (!file || !(await file.ready())) return null;
const context = new OfflineAudioContext(file.channels, frames, STREAM_RATE);
await context.audioWorklet.addModule(fxWorkletUrl);
const { node, source, feed } = this.streamNode(context, file, plan, at);
node.connect(context.destination);
// Where the plan starts past the head, the first stretch is decoded before the clock runs.
while (feed.busy) await new Promise((resolve) => setTimeout(resolve, 5));
const step = STREAM_RATE / 2;
const hold = (frame: number) => {
if (frame >= frames) return;
void context.suspend(frame / STREAM_RATE).then(async () => {
do await new Promise((resolve) => setTimeout(resolve, 5));
while (feed.busy);
hold(frame + step);
void context.resume();
});
};
hold(step);
const rendered = await context.startRendering();
const skipped = feed.skipped;
source.stop();
return {
channels: Array.from({ length: rendered.numberOfChannels }, (_, c) => rendered.getChannelData(c)),
skipped,
};
}
/** All of the copy a name streams from, decoded as the feed decodes it (test hook). */
async streamedWhole(name: string): Promise<Float32Array[] | null> {
const file = this.streamed(name);
if (!file || !(await file.ready())) return null;
return decodeSpan(file, 0, file.frames);
}
private dropFeed(feed: StreamFeed): void {
if (!this.feeds.delete(feed)) return;
this.skippedBefore += feed.skipped;
}
/**
* A Chip (#329): a worklet of its own that ends with the note. A Pitch on its way moves its
* frequency itself. Without the worklet the way is silent and ends at once.
@@ -2384,6 +2664,24 @@ function loopIn(buffer: AudioBuffer, start: number, end: number): { loopStart: n
return from < to ? { loopStart: from, loopEnd: to } : { loopStart: 0, loopEnd: last };
}
/** The loudest sample of each of `PEAK_BUCKETS` equal stretches of what is heard of `whole`, at most 1. */
function peaksOf(
whole: AudioBuffer,
{ from, to, reverse }: Pick<CutRange, 'from' | 'to' | 'reverse'>,
): Float32Array {
const peaks = new Float32Array(PEAK_BUCKETS);
for (let c = 0; c < whole.numberOfChannels; c++) {
const data = whole.getChannelData(c);
for (let i = from; i < to; i++) {
const heard = reverse ? to - 1 - i : i - from;
const bucket = Math.floor((heard / (to - from)) * PEAK_BUCKETS);
peaks[bucket] = Math.max(peaks[bucket]!, Math.abs(data[i]!));
}
}
for (let b = 0; b < PEAK_BUCKETS; b++) peaks[b] = Math.min(1, peaks[b]!);
return peaks;
}
/** The loudest sample of each side over the stretch a meter looks back over, 1 at full scale. */
function readMeter(meter: BusMeter): [number, number] {
const [left, right] = meter.sides.map((side) => {