Stream long sounds from an Opus copy instead of decoding them (#349)
A sound was decoded whole as soon as the project opened, in the editor and in a released game: five minutes of stereo held some 115 MB for as long as the page was open. A sound with Stream on now plays from a copy as Opus in Ogg, fetched a piece at a time and decoded a few seconds ahead by WebCodecs, so a game holds a few megabytes for it. - The Stream switch is a row of the sound, off by default. The Sound tab suggests it from a minute on and says how far the copy has got. - StreamKeeper makes the copies in the browser, a moment after the last change, one at a time: the sound's own and one per clip of its cut, fades and Reverse baked in. A copy that is no longer wanted is taken off and deleted once the project is saved without it. - streamTable says which played names stream from which copy. A clip without a fitting copy, a Sample and a Reverb impulse stay decoded. A sound with Stream on is never decoded ahead. - The page fetches by Range and takes the whole file from a server that ignores it, as the CI's PHP server does. It keeps the bytes and the first two seconds decoded, the rest goes four seconds ahead. - A jump decodes 32 packets before the place first. Measured in Chrome, what follows is then the same to the bit as decoding from the start. With 4 packets it was 0.06 off. - The stream-player on the FX worklet reads the samples at the rate of the context along a Catmull-Rom curve. What comes late is silent and skipped once it is there, so a clip keeps its beat. - The opening stretch goes with the player as it is made. Sent by message it came a few milliseconds late, and 557 samples of the attack were lost. - Pitch on a streamed sound is a shifter after the player, as on a bus: the length stays and it is heard 42.7 ms late. - Bakes of patterns and songs (#340) always stream now. - The song timeline draws a streamed sound from its copy decoded at 8 kHz, keeping only the peaks. Tests: - Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3, stream-queue 6, streams 11, streaming 6, sound-library +5). - e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono, 4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by 5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and a reversed clip are the copy decoded whole to the sample (0). - Counter-checks red as they should be: no pre-roll, no catch-up of late samples at the end, no opening stretch, a streamed sound decoded ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
84cd36f902
commit
2764919bb8
30 files changed
+2443
-146
No files matched your search
+386
-88
@@ -31,11 +31,13 @@ import { limiterLatency } from '../core/soundfx/dynamics';
|
||||
import type { ScheduledClip, ScheduledNote } from '../core/music/Scheduler';
|
||||
import { dbToGain, SOUND_ITEM_MAX_DB } from '../core/music/song';
|
||||
import { DEFAULT_TEMPO } from '../core/music/time';
|
||||
import { playedPlan, type StreamedFile, type StreamPlan } from '../core/music/streams';
|
||||
import { STREAM_RATE } from '../core/music/streamQueue';
|
||||
import { type CutRange, cutChannels, cutRange, parseCutName } from '../core/model/sound';
|
||||
import { shifterLatency } from '../core/soundfx/pitchPlayer';
|
||||
import { REVERB_MAX_DECAY } from '../core/soundfx/reverb';
|
||||
import type { AudioSink, ClipTiming, NoteTiming } from '../runtime/systems/Audio';
|
||||
import type { ChipOptions, PitchOptions } from './fxWorklet';
|
||||
import type { ChipOptions, PitchOptions, StreamOptions } from './fxWorklet';
|
||||
import fxWorkletUrl from './fxWorklet.ts?worker&url';
|
||||
import {
|
||||
bothAtCenter,
|
||||
@@ -57,6 +59,7 @@ import {
|
||||
stopProcessor,
|
||||
} from './fxNodes';
|
||||
import { type Spectrum, SpectrumTaps } from './spectrum';
|
||||
import { decodeSpan, StreamFeed, StreamFile } from './streaming';
|
||||
import { wake } from './wake';
|
||||
|
||||
/** A voice is no louder than a sound item of a song turned all the way up. */
|
||||
@@ -65,6 +68,8 @@ const MAX_VOLUME = dbToGain(SOUND_ITEM_MAX_DB);
|
||||
const DEFAULT_BASE = '/game-maker/data/sounds';
|
||||
/** The rate a sound loaded ahead of its first play is decoded at; a source resamples it on the way out. */
|
||||
const DECODE_RATE = 48000;
|
||||
/** The rate the picture of a streamed sound is decoded at (#349): enough for its peaks, a sixth of the size. */
|
||||
const PICTURE_RATE = 8000;
|
||||
|
||||
/** The longest a send of a track is held back to meet the latest way of it (#350). */
|
||||
const MAX_SEND_DELAY = 1;
|
||||
@@ -199,6 +204,8 @@ interface Wiring {
|
||||
/** Seconds into the sound its ways start, and whether it is a clip of a song, which fades out at its end (#334). */
|
||||
offset: number;
|
||||
clip: boolean;
|
||||
/** The copy a way without a source of its own streams (#349), instead of a buffer of `file`. */
|
||||
stream: StreamFile | null;
|
||||
sources: Source[];
|
||||
drivers: Driver[];
|
||||
envelopes: Map<string, EnvelopeRun>;
|
||||
@@ -229,6 +236,8 @@ interface Start {
|
||||
clip?: { offset: number; end: number };
|
||||
/** The track of a song it is heard on (#334), `''` straight out. */
|
||||
bus?: string;
|
||||
/** The copy `file` streams from (#349); its bytes are not loaded then, its head is. */
|
||||
stream?: StreamFile;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -262,6 +271,9 @@ interface Pending {
|
||||
*
|
||||
* A way with a Pitch is played by a worklet (#317), loaded into the context with the first play that
|
||||
* needs it; that play waits for it as it would for its bytes.
|
||||
*
|
||||
* A sound that streams (#349) is the exception to "decoded whole": a `stream-player` on that worklet
|
||||
* plays it from its copy, fed a few seconds ahead (`streaming.ts`), and its file is never decoded.
|
||||
*/
|
||||
export class SoundLibrary implements AudioSink {
|
||||
private base = DEFAULT_BASE;
|
||||
@@ -331,6 +343,7 @@ export class SoundLibrary implements AudioSink {
|
||||
/** The taps of the voices and buses playing now, and the node listened to (#374). */
|
||||
private readonly spectra = new SpectrumTaps();
|
||||
private readonly peaked = new Map<string, Float32Array>();
|
||||
private readonly pictures = new Map<string, Promise<void>>();
|
||||
/**
|
||||
* The patterns and songs a game plays, each on a scheduler of its own. What it asks after are its
|
||||
* own voices – asking `playing`, which counts the playings too, would ask the list it is in.
|
||||
@@ -363,12 +376,40 @@ export class SoundLibrary implements AudioSink {
|
||||
private mix: readonly MixBus[] = [];
|
||||
/** The levels a running game set its buses to, over those of their assets; until `resetMix`. */
|
||||
private readonly mixVolumes = new Map<string, number>();
|
||||
/** The names streamed (#349) and the copies they stream from, by name and by copy. */
|
||||
private streams: ReadonlyMap<string, StreamedFile> = new Map();
|
||||
private readonly streamFiles = new Map<string, StreamFile>();
|
||||
private readonly feeds = new Set<StreamFeed>();
|
||||
/** Every sample streamed so far that came too late to be heard, of the feeds gone (test hook). */
|
||||
private skippedBefore = 0;
|
||||
|
||||
/** Where the files are served from. Set once at start-up, before anything plays. */
|
||||
configure(base: string): void {
|
||||
this.base = base.replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* The names a game streams rather than decodes (#349), each with its copy (`streamTable`): the first
|
||||
* seconds of every copy are fetched and decoded now, so it starts at once. A library that bakes,
|
||||
* and a browser without Opus in WebCodecs, decode everything as before.
|
||||
*/
|
||||
setStreams(table: ReadonlyMap<string, StreamedFile>): void {
|
||||
this.streams = table;
|
||||
for (const name of table.keys()) void this.streamed(name)?.ready();
|
||||
}
|
||||
|
||||
/** The copy a name streams from, while it streams and the copy has not failed. */
|
||||
private streamed(name: string): StreamFile | null {
|
||||
const entry = this.streams.get(name);
|
||||
if (!entry || this.render || typeof AudioDecoder === 'undefined') return null;
|
||||
let file = this.streamFiles.get(entry.file);
|
||||
if (!file) {
|
||||
file = new StreamFile(`${this.base}/${entry.file}`, entry.frames);
|
||||
this.streamFiles.set(entry.file, file);
|
||||
}
|
||||
return file.failed ? null : file;
|
||||
}
|
||||
|
||||
/**
|
||||
* The number is handed out at once, before the bytes are there: a graph can tune a voice in the
|
||||
* same frame it asked for it, and the voice then starts with those values.
|
||||
@@ -376,6 +417,7 @@ export class SoundLibrary implements AudioSink {
|
||||
play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE, bus = ''): number {
|
||||
if (file === '' || this.hasFailed(file)) return 0;
|
||||
const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO };
|
||||
const stream = this.streamed(file);
|
||||
return this.begin({
|
||||
id: 0,
|
||||
file,
|
||||
@@ -387,6 +429,7 @@ export class SoundLibrary implements AudioSink {
|
||||
duration: 0,
|
||||
at: 0,
|
||||
bus,
|
||||
...(stream && { stream }),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -424,6 +467,7 @@ export class SoundLibrary implements AudioSink {
|
||||
if (file === '' || this.hasFailed(file)) return 0;
|
||||
const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO };
|
||||
const { at, offset, duration, volume, bus } = clip;
|
||||
const stream = this.streamed(file);
|
||||
return this.begin({
|
||||
id: 0,
|
||||
file,
|
||||
@@ -436,6 +480,7 @@ export class SoundLibrary implements AudioSink {
|
||||
at,
|
||||
clip: { offset, end: at + Math.max(0, duration) },
|
||||
bus,
|
||||
...(stream && { stream }),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1050,19 +1095,102 @@ export class SoundLibrary implements AudioSink {
|
||||
|
||||
/**
|
||||
* The bake of a playing (#340) as one voice on its master, heard from `heard`: its file once to its
|
||||
* end, faded out there, or up to the loop end and then round its second round – in one source where
|
||||
* the round follows at once, else in a second that takes over to the sample. 0 without its bytes.
|
||||
* end, faded out there, or up to the loop end and then round its second round. Streamed where it can
|
||||
* be (#349), else decoded – in one source where the round follows at once, else in a second that
|
||||
* takes over to the sample. 0 without its bytes.
|
||||
*/
|
||||
private playBaked(id: number, bake: BakePlay, heard: number): number {
|
||||
const buffer = this.buffer(bake.file);
|
||||
if (!buffer) return 0;
|
||||
const stream = this.hasWorklet ? this.streamed(bake.file) : null;
|
||||
const streaming = stream?.head ? stream : null;
|
||||
const buffer = streaming ? null : this.buffer(bake.file);
|
||||
if (!streaming && !buffer) return 0;
|
||||
const context = this.audio();
|
||||
const bus = groupBus(id, '');
|
||||
const at = Math.max(context.currentTime, heard - this.busLatency(bus));
|
||||
const gain = context.createGain();
|
||||
gain.gain.value = Math.max(0, Math.min(MAX_VOLUME, bake.gain));
|
||||
gain.connect(this.busInput(bus));
|
||||
const { end, until, round } = bake.playback;
|
||||
const { end } = bake.playback;
|
||||
const played = streaming
|
||||
? this.bakeStreamed(streaming, bake.playback, gain, at)
|
||||
: this.bakeDecoded(buffer!, bake.playback, gain, at);
|
||||
const voice: Voice = {
|
||||
id: this.nextId++,
|
||||
file: bake.file,
|
||||
owner: '',
|
||||
bus,
|
||||
at,
|
||||
sources: played.sources,
|
||||
ending: new Set(),
|
||||
drivers: [],
|
||||
envelopes: [],
|
||||
finish: null,
|
||||
gain,
|
||||
fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO },
|
||||
knobs: [],
|
||||
rings: false,
|
||||
ringing: false,
|
||||
quiet: null,
|
||||
vocoders: 0,
|
||||
released: false,
|
||||
ringFade: BAKE_LET_RING,
|
||||
};
|
||||
const going = new Set(voice.sources);
|
||||
voice.finish = (source) => {
|
||||
if (going.delete(source) && going.size === 0) this.ended(voice);
|
||||
};
|
||||
for (const source of played.sources) source.ended(() => voice.finish?.(source));
|
||||
this.voices.push(voice);
|
||||
this.started += 1;
|
||||
this.last = {
|
||||
file: bake.file,
|
||||
volume: gain.gain.value,
|
||||
loop: bake.playback.round !== null,
|
||||
layers: 0,
|
||||
pitches: [],
|
||||
stages: 0,
|
||||
duration: played.duration,
|
||||
at,
|
||||
envelopes: 0,
|
||||
bus,
|
||||
};
|
||||
if (end !== null) {
|
||||
gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE));
|
||||
gain.gain.linearRampToValueAtTime(0, at + end);
|
||||
}
|
||||
played.start();
|
||||
return voice.id;
|
||||
}
|
||||
|
||||
/** A bake streamed (#349): one `stream-player` along the same plan, the round looped to the sample. */
|
||||
private bakeStreamed(
|
||||
stream: StreamFile,
|
||||
{ end, until, round }: BakePlay['playback'],
|
||||
gain: GainNode,
|
||||
at: number,
|
||||
): { sources: Source[]; start: () => void; duration: number } {
|
||||
const frames = (seconds: number) => Math.round(seconds * STREAM_RATE);
|
||||
const plan: StreamPlan =
|
||||
end !== null
|
||||
? { from: 0, until: frames(end), loop: null }
|
||||
: {
|
||||
from: 0,
|
||||
until: frames(until),
|
||||
loop: round && { start: frames(round.start), end: frames(round.end) },
|
||||
};
|
||||
const { node, source } = this.streamNode(this.audio(), stream, plan, at);
|
||||
node.connect(gain);
|
||||
return { sources: [source], start: () => {}, duration: stream.frames / STREAM_RATE };
|
||||
}
|
||||
|
||||
/** A bake decoded, in one buffer source or two. */
|
||||
private bakeDecoded(
|
||||
buffer: AudioBuffer,
|
||||
{ end, until, round }: BakePlay['playback'],
|
||||
gain: GainNode,
|
||||
at: number,
|
||||
): { sources: Source[]; start: () => void; duration: number } {
|
||||
const context = this.audio();
|
||||
const made = (from: number, loop: { start: number; end: number } | null) => {
|
||||
const node = context.createBufferSource();
|
||||
node.buffer = buffer;
|
||||
@@ -1090,56 +1218,15 @@ export class SoundLibrary implements AudioSink {
|
||||
: round
|
||||
? [made(0, null), made(round.start, round)]
|
||||
: [made(0, null)];
|
||||
const voice: Voice = {
|
||||
id: this.nextId++,
|
||||
file: bake.file,
|
||||
owner: '',
|
||||
bus,
|
||||
at,
|
||||
sources: parts.map((part) => part.source),
|
||||
ending: new Set(),
|
||||
drivers: [],
|
||||
envelopes: [],
|
||||
finish: null,
|
||||
gain,
|
||||
fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO },
|
||||
knobs: [],
|
||||
rings: false,
|
||||
ringing: false,
|
||||
quiet: null,
|
||||
vocoders: 0,
|
||||
released: false,
|
||||
ringFade: BAKE_LET_RING,
|
||||
};
|
||||
const going = new Set(voice.sources);
|
||||
voice.finish = (source) => {
|
||||
if (going.delete(source) && going.size === 0) this.ended(voice);
|
||||
};
|
||||
for (const { source } of parts) source.ended(() => voice.finish?.(source));
|
||||
this.voices.push(voice);
|
||||
this.started += 1;
|
||||
this.last = {
|
||||
file: bake.file,
|
||||
volume: gain.gain.value,
|
||||
loop: round !== null,
|
||||
layers: 0,
|
||||
pitches: [],
|
||||
stages: 0,
|
||||
duration: buffer.duration,
|
||||
at,
|
||||
envelopes: 0,
|
||||
bus,
|
||||
};
|
||||
const [head, next] = parts;
|
||||
if (end !== null) {
|
||||
gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE));
|
||||
gain.gain.linearRampToValueAtTime(0, at + end);
|
||||
head!.node.start(at, 0, end);
|
||||
} else if (next) {
|
||||
head!.node.start(at, 0, until);
|
||||
next.node.start(at + until, next.from);
|
||||
} else head!.node.start(at, 0);
|
||||
return voice.id;
|
||||
const start = () => {
|
||||
if (end !== null) head!.node.start(at, 0, end);
|
||||
else if (next) {
|
||||
head!.node.start(at, 0, until);
|
||||
next.node.start(at + until, next.from);
|
||||
} else head!.node.start(at, 0);
|
||||
};
|
||||
return { sources: parts.map((part) => part.source), start, duration: buffer.duration };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1304,8 +1391,12 @@ export class SoundLibrary implements AudioSink {
|
||||
const asked: Start = { ...start, id };
|
||||
// A Reverb's sound (#328) is waited for as well, but a voice is not silent for lack of it.
|
||||
const impulseFiles = start.fx.layers.flatMap(impulsesOf);
|
||||
const loaded = [...start.files, ...impulseFiles].every((file) => this.buffer(file) !== null);
|
||||
const worklet = start.fx.layers.some(needsWorklet) ? this.fxWorklet() : null;
|
||||
// A streamed sound (#349) waits for the head of its copy and the worklet that plays it instead.
|
||||
const { stream } = start;
|
||||
const loaded =
|
||||
[...(stream ? [] : start.files), ...impulseFiles].every((file) => this.buffer(file) !== null) &&
|
||||
(!stream || stream.head !== null);
|
||||
const worklet = stream || start.fx.layers.some(needsWorklet) ? this.fxWorklet() : null;
|
||||
if (loaded && (!worklet || this.hasWorklet !== null)) {
|
||||
this.start(asked);
|
||||
return id;
|
||||
@@ -1321,19 +1412,23 @@ export class SoundLibrary implements AudioSink {
|
||||
};
|
||||
this.pending.push(waiting);
|
||||
const impulsesLoaded = Promise.all(impulseFiles.map((file) => this.load(file)));
|
||||
void Promise.all([Promise.all(start.files.map((file) => this.load(file))), worklet, impulsesLoaded]).then(
|
||||
([buffers]) => {
|
||||
this.pending = this.pending.filter((other) => other !== waiting);
|
||||
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
|
||||
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
|
||||
const silent = buffers.every((buffer) => !buffer) && !start.fx.layers.some(hasChip);
|
||||
if (silent || waiting.stopped) return;
|
||||
if (this.cancelsFor(start.file, start.owner) !== askedAt) return;
|
||||
this.start({ ...asked, fx: waiting.fx });
|
||||
for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease);
|
||||
if (waiting.released) this.release(id);
|
||||
},
|
||||
);
|
||||
// A copy that cannot be streamed leaves the sound decoded, as it would be without one.
|
||||
const sounds = stream
|
||||
? stream.ready().then(async (ready) => [ready || (await this.load(start.file)) !== null])
|
||||
: Promise.all(start.files.map((file) => this.load(file)));
|
||||
void Promise.all([sounds, worklet, impulsesLoaded]).then(([buffers]) => {
|
||||
this.pending = this.pending.filter((other) => other !== waiting);
|
||||
// A `Stop sound` – or the destruction of the behavior that asked – that happened while the
|
||||
// bytes were in flight has to win, or a sound the run already ended would start afterwards.
|
||||
const silent = buffers.every((buffer) => !buffer) && !start.fx.layers.some(hasChip);
|
||||
if (silent || waiting.stopped) return;
|
||||
if (this.cancelsFor(start.file, start.owner) !== askedAt) return;
|
||||
const started: Start = { ...asked, fx: waiting.fx };
|
||||
if (stream?.failed) delete started.stream;
|
||||
this.start(started);
|
||||
for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease);
|
||||
if (waiting.released) this.release(id);
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
@@ -1463,7 +1558,13 @@ export class SoundLibrary implements AudioSink {
|
||||
/** Ways playing through the pitch worklet now, at most `MAX_VOCODERS`. */
|
||||
vocoders: number;
|
||||
last: LastVoice | null;
|
||||
/**
|
||||
* Streaming (#349): sources playing from a copy now, samples that came too late to be heard, the
|
||||
* bytes fetched of every copy, and the copies whose head is decoded.
|
||||
*/
|
||||
streaming: { playing: number; skipped: number; fetched: number; heads: number };
|
||||
} {
|
||||
const files = [...this.streamFiles.values()];
|
||||
return {
|
||||
playing: this.voices.length,
|
||||
started: this.started,
|
||||
@@ -1475,6 +1576,12 @@ export class SoundLibrary implements AudioSink {
|
||||
worklet: this.hasWorklet,
|
||||
vocoders: this.voices.reduce((sum, voice) => sum + vocodersOf(voice), 0),
|
||||
last: this.last,
|
||||
streaming: {
|
||||
playing: this.feeds.size,
|
||||
skipped: [...this.feeds].reduce((sum, feed) => sum + feed.skipped, this.skippedBefore),
|
||||
fetched: files.reduce((sum, file) => sum + file.bytes, 0),
|
||||
heads: files.filter((file) => file.head !== null).length,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1538,11 +1645,21 @@ export class SoundLibrary implements AudioSink {
|
||||
* not late waiting on its bytes (#332).
|
||||
*/
|
||||
async prepare(layers: readonly FxPlan[], files: readonly string[] = []): Promise<void> {
|
||||
// A Sample or a Reverb's sound is always decoded; a sound or a bake may stream (#349).
|
||||
const decoded = new Set([...layers.flatMap(filesOf), ...layers.flatMap(impulsesOf)]);
|
||||
const streamed = files.flatMap((file) => {
|
||||
const stream = decoded.has(file) ? null : this.streamed(file);
|
||||
return stream ? [{ file, stream }] : [];
|
||||
});
|
||||
const loaded = [...new Set([...decoded, ...files])].filter(
|
||||
(file) => file !== '' && !streamed.some((entry) => entry.file === file),
|
||||
);
|
||||
await Promise.all([
|
||||
...[...new Set([...layers.flatMap(filesOf), ...layers.flatMap(impulsesOf), ...files])]
|
||||
.filter((file) => file !== '')
|
||||
.map((file) => this.load(file)),
|
||||
layers.some(needsWorklet) ? this.fxWorklet() : null,
|
||||
...loaded.map((file) => this.load(file)),
|
||||
...streamed.map(async ({ file, stream }) => {
|
||||
if (!(await stream.ready())) await this.load(file);
|
||||
}),
|
||||
layers.some(needsWorklet) || streamed.length > 0 ? this.fxWorklet() : null,
|
||||
]);
|
||||
}
|
||||
|
||||
@@ -1562,7 +1679,10 @@ export class SoundLibrary implements AudioSink {
|
||||
*/
|
||||
duration(file: string): number | null {
|
||||
const range = this.rangeOf(file);
|
||||
return range ? (range.to - range.from) / range.whole.sampleRate : null;
|
||||
if (range) return (range.to - range.from) / range.whole.sampleRate;
|
||||
// A streamed one (#349) is as long as its copy, known without decoding a sample of it.
|
||||
const stream = this.streams.get(file);
|
||||
return stream ? stream.frames / STREAM_RATE : null;
|
||||
}
|
||||
|
||||
/** The decoded samples of a file as it is, without a cut; null while it has not been loaded. */
|
||||
@@ -1588,27 +1708,49 @@ export class SoundLibrary implements AudioSink {
|
||||
if (known) return known;
|
||||
const range = this.rangeOf(file);
|
||||
if (!range) return null;
|
||||
const { whole, from, to, reverse } = range;
|
||||
const peaks = new Float32Array(PEAK_BUCKETS);
|
||||
for (let c = 0; c < whole.numberOfChannels; c++) {
|
||||
const data = whole.getChannelData(c);
|
||||
for (let i = from; i < to; i++) {
|
||||
const heard = reverse ? to - 1 - i : i - from;
|
||||
const bucket = Math.floor((heard / (to - from)) * PEAK_BUCKETS);
|
||||
peaks[bucket] = Math.max(peaks[bucket]!, Math.abs(data[i]!));
|
||||
}
|
||||
}
|
||||
for (let b = 0; b < PEAK_BUCKETS; b++) peaks[b] = Math.min(1, peaks[b]!);
|
||||
const peaks = peaksOf(range.whole, range);
|
||||
this.peaked.set(file, peaks);
|
||||
return peaks;
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads what `peaks` and `duration` read for a name. A streamed one (#349) is not loaded whole for
|
||||
* it: its copy is decoded at `PICTURE_RATE`, its peaks kept and the samples let go.
|
||||
*/
|
||||
loadPicture(name: string): Promise<void> {
|
||||
const stream = this.streamed(name);
|
||||
if (!stream) return this.load(name).then(() => {});
|
||||
if (this.peaked.has(name)) return Promise.resolve();
|
||||
let picture = this.pictures.get(name);
|
||||
if (!picture) {
|
||||
picture = this.draw(name, stream);
|
||||
this.pictures.set(name, picture);
|
||||
}
|
||||
return picture;
|
||||
}
|
||||
|
||||
private async draw(name: string, stream: StreamFile): Promise<void> {
|
||||
try {
|
||||
const bytes = await stream.whole();
|
||||
const picture = await new OfflineAudioContext(1, 1, PICTURE_RATE).decodeAudioData(bytes);
|
||||
this.peaked.set(name, peaksOf(picture, { from: 0, to: picture.length, reverse: false }));
|
||||
} catch (error: unknown) {
|
||||
console.error(`[sounds] cannot draw ${name}`, error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Starts loading every file not held yet. Waiting for the first play left that play late: its
|
||||
* start was not heard, and a graph tuning it in the meantime had moved on by the time it started.
|
||||
* One that streams (#349) has the head of its copy loaded instead.
|
||||
*/
|
||||
preload(files: Iterable<string>): void {
|
||||
for (const file of files) if (file) void this.load(file);
|
||||
for (const file of files) {
|
||||
if (!file) continue;
|
||||
const stream = this.streamed(file);
|
||||
if (stream) void stream.ready();
|
||||
else void this.load(file);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1643,6 +1785,13 @@ export class SoundLibrary implements AudioSink {
|
||||
return work;
|
||||
}
|
||||
|
||||
/** A file decoded at `rate` and not kept: what a stream copy is made from (#349). */
|
||||
async decodeApart(file: string, rate: number): Promise<AudioBuffer> {
|
||||
const response = await fetch(`${this.base}/${file}`);
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
return new OfflineAudioContext(1, 1, rate).decodeAudioData(await response.arrayBuffer());
|
||||
}
|
||||
|
||||
private async fetched(file: string): Promise<AudioBuffer> {
|
||||
const response = await fetch(`${this.base}/${file}`);
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
@@ -1806,6 +1955,7 @@ export class SoundLibrary implements AudioSink {
|
||||
at: heard,
|
||||
clip: asked,
|
||||
bus = '',
|
||||
stream,
|
||||
}: Start): void {
|
||||
const context = this.audio();
|
||||
// A context starts suspended where the page has not been interacted with; without this the
|
||||
@@ -1840,6 +1990,7 @@ export class SoundLibrary implements AudioSink {
|
||||
rings,
|
||||
offset: clip ? clip.offset + (when - at) : 0,
|
||||
clip: clip !== undefined,
|
||||
stream: stream ?? null,
|
||||
sources: [],
|
||||
drivers: [],
|
||||
envelopes: new Map(),
|
||||
@@ -1990,7 +2141,15 @@ export class SoundLibrary implements AudioSink {
|
||||
return sound.connect(fade);
|
||||
}
|
||||
|
||||
private sound(wiring: Wiring, { pitch, shifted, source: own, pitchMods }: Way): AudioNode {
|
||||
private sound(wiring: Wiring, way: Way): AudioNode {
|
||||
const { pitch, shifted, source: own, pitchMods } = way;
|
||||
if (!own && wiring.stream && this.hasWorklet)
|
||||
return this.streamSource(
|
||||
wiring,
|
||||
wiring.stream,
|
||||
playedPlan(wiring.stream.frames, wiring.offset, wiring.loop),
|
||||
way,
|
||||
);
|
||||
const { context } = wiring;
|
||||
const file = own?.file ?? wiring.file;
|
||||
const loop = own ? own.loop : wiring.loop;
|
||||
@@ -2063,6 +2222,127 @@ export class SoundLibrary implements AudioSink {
|
||||
return source;
|
||||
}
|
||||
|
||||
/**
|
||||
* A way streamed from its copy (#349): a `stream-player` fed along its plan. Its Pitch is a shifter
|
||||
* of what it plays, as on a bus (#346): its length stays, and it is heard a frame late.
|
||||
*/
|
||||
private streamSource(
|
||||
wiring: Wiring,
|
||||
file: StreamFile,
|
||||
plan: StreamPlan,
|
||||
{ pitch, shifted, pitchMods }: Way,
|
||||
): AudioNode {
|
||||
const { context } = wiring;
|
||||
wiring.pitches.push(pitch);
|
||||
const { node, source } = this.streamNode(context, file, plan, wiring.at);
|
||||
wiring.sources.push(source);
|
||||
if (!shifted) return node;
|
||||
const shifter = new AudioWorkletNode(context, 'pitch-shifter', {
|
||||
numberOfInputs: 1,
|
||||
numberOfOutputs: 1,
|
||||
outputChannelCount: [2],
|
||||
channelCount: 2,
|
||||
channelCountMode: 'explicit',
|
||||
parameterData: { pitch },
|
||||
});
|
||||
node.connect(shifter);
|
||||
wiring.vocoders += 1;
|
||||
wiring.drivers.push({ start: () => {}, stop: () => stopProcessor(shifter) });
|
||||
const knob = shifter.parameters.get('pitch')!;
|
||||
wiring.knobs.push({ param: knob, target: pitch, even: 'value' });
|
||||
const follow = this.follower(wiring);
|
||||
for (const mod of pitchMods ?? []) follow(mod, knob, false);
|
||||
return shifter;
|
||||
}
|
||||
|
||||
/** A `stream-player` fed along `plan`, heard from `at` on, and the source a voice holds it by. */
|
||||
private streamNode(
|
||||
context: BaseAudioContext,
|
||||
file: StreamFile,
|
||||
plan: StreamPlan,
|
||||
at: number,
|
||||
): { node: AudioWorkletNode; source: Source; feed: StreamFeed } {
|
||||
let then: (() => void) | null = null;
|
||||
const feed: StreamFeed = new StreamFeed(file, plan, () => {
|
||||
this.dropFeed(feed);
|
||||
then?.();
|
||||
});
|
||||
const options: StreamOptions = { channels: file.channels, startAt: at, first: feed.opening() };
|
||||
const node = new AudioWorkletNode(context, 'stream-player', {
|
||||
numberOfInputs: 0,
|
||||
outputChannelCount: [file.channels],
|
||||
processorOptions: options,
|
||||
});
|
||||
feed.attach(node);
|
||||
this.feeds.add(feed);
|
||||
const source: Source = {
|
||||
start: () => {},
|
||||
stop: () => {
|
||||
this.dropFeed(feed);
|
||||
feed.stop();
|
||||
},
|
||||
ended: (call) => {
|
||||
then = call;
|
||||
},
|
||||
fade: null,
|
||||
gate: false,
|
||||
leave: null,
|
||||
};
|
||||
return { node, source, feed };
|
||||
}
|
||||
|
||||
/**
|
||||
* Streams a name along `plan` into an offline context at `STREAM_RATE`, heard from `at`, `frames`
|
||||
* long, with the player and the feed a game has (test hook). The clock is held every half second
|
||||
* until the feed has caught up: offline it runs faster than anything is decoded. Null where the
|
||||
* name does not stream.
|
||||
*/
|
||||
async renderStream(
|
||||
name: string,
|
||||
plan: StreamPlan,
|
||||
at: number,
|
||||
frames: number,
|
||||
): Promise<{ channels: Float32Array[]; skipped: number } | null> {
|
||||
const file = this.streamed(name);
|
||||
if (!file || !(await file.ready())) return null;
|
||||
const context = new OfflineAudioContext(file.channels, frames, STREAM_RATE);
|
||||
await context.audioWorklet.addModule(fxWorkletUrl);
|
||||
const { node, source, feed } = this.streamNode(context, file, plan, at);
|
||||
node.connect(context.destination);
|
||||
// Where the plan starts past the head, the first stretch is decoded before the clock runs.
|
||||
while (feed.busy) await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
const step = STREAM_RATE / 2;
|
||||
const hold = (frame: number) => {
|
||||
if (frame >= frames) return;
|
||||
void context.suspend(frame / STREAM_RATE).then(async () => {
|
||||
do await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
while (feed.busy);
|
||||
hold(frame + step);
|
||||
void context.resume();
|
||||
});
|
||||
};
|
||||
hold(step);
|
||||
const rendered = await context.startRendering();
|
||||
const skipped = feed.skipped;
|
||||
source.stop();
|
||||
return {
|
||||
channels: Array.from({ length: rendered.numberOfChannels }, (_, c) => rendered.getChannelData(c)),
|
||||
skipped,
|
||||
};
|
||||
}
|
||||
|
||||
/** All of the copy a name streams from, decoded as the feed decodes it (test hook). */
|
||||
async streamedWhole(name: string): Promise<Float32Array[] | null> {
|
||||
const file = this.streamed(name);
|
||||
if (!file || !(await file.ready())) return null;
|
||||
return decodeSpan(file, 0, file.frames);
|
||||
}
|
||||
|
||||
private dropFeed(feed: StreamFeed): void {
|
||||
if (!this.feeds.delete(feed)) return;
|
||||
this.skippedBefore += feed.skipped;
|
||||
}
|
||||
|
||||
/**
|
||||
* A Chip (#329): a worklet of its own that ends with the note. A Pitch on its way moves its
|
||||
* frequency itself. Without the worklet the way is silent and ends at once.
|
||||
@@ -2384,6 +2664,24 @@ function loopIn(buffer: AudioBuffer, start: number, end: number): { loopStart: n
|
||||
return from < to ? { loopStart: from, loopEnd: to } : { loopStart: 0, loopEnd: last };
|
||||
}
|
||||
|
||||
/** The loudest sample of each of `PEAK_BUCKETS` equal stretches of what is heard of `whole`, at most 1. */
|
||||
function peaksOf(
|
||||
whole: AudioBuffer,
|
||||
{ from, to, reverse }: Pick<CutRange, 'from' | 'to' | 'reverse'>,
|
||||
): Float32Array {
|
||||
const peaks = new Float32Array(PEAK_BUCKETS);
|
||||
for (let c = 0; c < whole.numberOfChannels; c++) {
|
||||
const data = whole.getChannelData(c);
|
||||
for (let i = from; i < to; i++) {
|
||||
const heard = reverse ? to - 1 - i : i - from;
|
||||
const bucket = Math.floor((heard / (to - from)) * PEAK_BUCKETS);
|
||||
peaks[bucket] = Math.max(peaks[bucket]!, Math.abs(data[i]!));
|
||||
}
|
||||
}
|
||||
for (let b = 0; b < PEAK_BUCKETS; b++) peaks[b] = Math.min(1, peaks[b]!);
|
||||
return peaks;
|
||||
}
|
||||
|
||||
/** The loudest sample of each side over the stretch a meter looks back over, 1 at full scale. */
|
||||
function readMeter(meter: BusMeter): [number, number] {
|
||||
const [left, right] = meter.sides.map((side) => {
|
||||
|
||||
Reference in new issue
Block a user