From 2764919bb8a4591ebac1eb6dad860750bef69e69 Mon Sep 17 00:00:00 2001 From: MelekTaus Date: Fri, 9 Oct 2026 20:27:32 +0200 Subject: [PATCH] Stream long sounds from an Opus copy instead of decoding them (#349) A sound was decoded whole as soon as the project opened, in the editor and in a released game: five minutes of stereo held some 115 MB for as long as the page was open. A sound with Stream on now plays from a copy as Opus in Ogg, fetched a piece at a time and decoded a few seconds ahead by WebCodecs, so a game holds a few megabytes for it. - The Stream switch is a row of the sound, off by default. The Sound tab suggests it from a minute on and says how far the copy has got. - StreamKeeper makes the copies in the browser, a moment after the last change, one at a time: the sound's own and one per clip of its cut, fades and Reverse baked in. A copy that is no longer wanted is taken off and deleted once the project is saved without it. - streamTable says which played names stream from which copy. A clip without a fitting copy, a Sample and a Reverb impulse stay decoded. A sound with Stream on is never decoded ahead. - The page fetches by Range and takes the whole file from a server that ignores it, as the CI's PHP server does. It keeps the bytes and the first two seconds decoded, the rest goes four seconds ahead. - A jump decodes 32 packets before the place first. Measured in Chrome, what follows is then the same to the bit as decoding from the start. With 4 packets it was 0.06 off. - The stream-player on the FX worklet reads the samples at the rate of the context along a Catmull-Rom curve. What comes late is silent and skipped once it is there, so a clip keeps its beat. - The opening stretch goes with the player as it is made. Sent by message it came a few milliseconds late, and 557 samples of the attack were lost. - Pitch on a streamed sound is a shifter after the player, as on a bus: the length stays and it is heard 42.7 ms late. - Bakes of patterns and songs (#340) always stream now. - The song timeline draws a streamed sound from its copy decoded at 8 kHz, keeping only the peaks. Tests: - Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3, stream-queue 6, streams 11, streaming 6, sound-library +5). - e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono, 4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by 5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and a reversed clip are the copy decoded whole to the sample (0). - Counter-checks red as they should be: no pre-roll, no catch-up of late samples at the end, no opening stretch, a streamed sound decoded ahead. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- src/core/model/Asset.ts | 17 + src/core/model/projectFiles.ts | 7 +- src/core/music/ogg.ts | 123 +++++-- src/core/music/streamQueue.ts | 144 ++++++++ src/core/music/streams.ts | 154 +++++++++ src/core/properties/PropertySchema.ts | 8 +- src/core/properties/sounds.ts | 6 + src/core/store/serialize.ts | 14 +- src/editor/Editor.ts | 6 +- src/editor/debug/soundHooks.ts | 59 ++++ src/editor/music/BakeButton.ts | 18 +- src/editor/music/dropWhenSaved.ts | 16 + src/editor/song/SongTimeline.ts | 4 +- src/editor/sound/SoundEditor.ts | 25 +- src/editor/sound/SoundPicker.ts | 5 +- src/editor/sound/StreamKeeper.ts | 164 +++++++++ src/editor/state.ts | 19 +- src/viewport/SoundLibrary.ts | 474 +++++++++++++++++++++----- src/viewport/fxWorklet.ts | 47 ++- src/viewport/projectFiles.ts | 3 + src/viewport/streaming.ts | 376 ++++++++++++++++++++ tests/core/ogg.test.ts | 56 ++- tests/core/sounds.test.ts | 6 +- tests/core/stream-queue.test.ts | 83 +++++ tests/core/streams.test.ts | 180 ++++++++++ tests/viewport/fakeStream.ts | 105 ++++++ tests/viewport/sound-library.test.ts | 118 ++++++- tests/viewport/streaming.test.ts | 152 +++++++++ tools/e2e/34-sound.mjs | 198 +++++++++++ 30 files changed, 2443 insertions(+), 146 deletions(-) create mode 100644 src/core/music/streamQueue.ts create mode 100644 src/core/music/streams.ts create mode 100644 src/editor/music/dropWhenSaved.ts create mode 100644 src/editor/sound/StreamKeeper.ts create mode 100644 src/viewport/streaming.ts create mode 100644 tests/core/stream-queue.test.ts create mode 100644 tests/core/streams.test.ts create mode 100644 tests/viewport/fakeStream.ts create mode 100644 tests/viewport/streaming.test.ts diff --git a/README.md b/README.md index b9d19b3e..a328c947 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ npm run build # writes dist/ and regenerates index.html; the folder is wh The browser suite and the three example games need a machine with a GPU: ```bash -node tools/seed-e2e.mjs && node tools/e2e.mjs # the editor, 1928 checks (GM_STEPS=20 for one step) +node tools/seed-e2e.mjs && node tools/e2e.mjs # the editor, 1935 checks (GM_STEPS=20 for one step) # every tool signs in first – see the Accounts page node tools/seed-jumprun.mjs && node tools/smoke-jumprun.mjs node tools/seed-tetris.mjs && node tools/smoke-tetris.mjs diff --git a/src/core/model/Asset.ts b/src/core/model/Asset.ts index 40a23b0b..302c071f 100644 --- a/src/core/model/Asset.ts +++ b/src/core/model/Asset.ts @@ -317,6 +317,21 @@ export interface SoundAsset extends AssetBase { file: string; /** Absent for a sound without clips. */ clips?: SoundClip[]; + /** The copy it is streamed from while its `stream` is on (#349). */ + stream?: SoundStream; +} + +/** + * A sound or a clip of it as Opus in Ogg (#349), what it is streamed from rather than decoded whole. + * Made in the browser from what it plays; `from` says what that was – the file, or the name of the + * cut a clip is – and once it is something else the copy is stale and made again. + */ +export interface SoundStream { + file: string; + from: string; + /** Samples per channel at 48 kHz, and the size of the file, to show. */ + frames: number; + bytes: number; } /** @@ -329,6 +344,8 @@ export interface SoundClip { id: string; name: string; props: PropertyBag; + /** Its cut baked to a copy of its own, while its sound streams (#349). */ + stream?: SoundStream; } /** diff --git a/src/core/model/projectFiles.ts b/src/core/model/projectFiles.ts index f73a5e51..353c1115 100644 --- a/src/core/model/projectFiles.ts +++ b/src/core/model/projectFiles.ts @@ -1,4 +1,5 @@ import { validBake } from '../music/bake'; +import { streamsOf } from '../properties/sounds'; import { isFontFile } from './font'; import { celFilesOf } from './pixelart'; import type { Project } from './Project'; @@ -17,7 +18,8 @@ export interface ProjectFiles { * Every file an asset names, used or not: which frame an animation or a tile reaches next is only * known while it plays, and a file fetched then shows as a gap. A name that is no stored file – * an upload that failed – is left out, as the libraries would never get it anyway. So is a bake that - * no longer holds (#340): it would never be played, and decoded it weighs as much as the music. + * no longer holds (#340): it would never be played. So is a sound that streams (#349): the sound + * library fetches the start of its copy instead, and decoded ahead it would weigh what streaming saves. */ export function projectFiles(project: Project): ProjectFiles { const files: ProjectFiles = { textures: [], fonts: [], cels: [], sounds: [] }; @@ -25,7 +27,8 @@ export function projectFiles(project: Project): ProjectFiles { if (asset.type === 'texture' && isTextureFile(asset.file)) files.textures.push(asset.file); else if (asset.type === 'font' && isFontFile(asset.file)) files.fonts.push(asset.file); else if (asset.type === 'pixelart') files.cels.push(...celFilesOf(asset)); - else if (asset.type === 'sound' && isSoundFile(asset.file)) files.sounds.push(asset.file); + else if (asset.type === 'sound' && isSoundFile(asset.file) && !streamsOf(asset.props)) + files.sounds.push(asset.file); else if ((asset.type === 'pattern' || asset.type === 'song') && validBake(project, asset)) files.sounds.push(asset.bake!.file); } diff --git a/src/core/music/ogg.ts b/src/core/music/ogg.ts index e69a4063..baf1ba1f 100644 --- a/src/core/music/ogg.ts +++ b/src/core/music/ogg.ts @@ -140,38 +140,111 @@ export interface ReadOpus { pages: number; } -/** Reads a file `oggOpus` wrote, checking every page; throws at the first that is not right. */ +/** Reads a whole file `oggOpus` wrote, checking every page; throws at the first that is not right. */ export function readOggOpus(bytes: Uint8Array): ReadOpus { - const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - const all: Uint8Array[] = []; - let open: Uint8Array[] = []; - let granule = 0; - let pages = 0; - for (let at = 0; at < bytes.length; pages++) { - if (text(bytes.subarray(at, at + 4)) !== 'OggS') throw new Error(`no page at byte ${at}`); - const count = bytes[at + 26]!; - const lacing = bytes.subarray(at + 27, at + 27 + count); - const size = 27 + count + lacing.reduce((sum, value) => sum + value, 0); - const page = bytes.slice(at, at + size); - const crc = view.getUint32(at + 22, true); - page.fill(0, 22, 26); - if (oggCrc(page) !== crc) throw new Error(`page ${pages} has a wrong checksum`); - granule = view.getUint32(at + 6, true) + view.getUint32(at + 10, true) * 2 ** 32; - let from = at + 27 + count; + const reader = new OggOpusReader(); + reader.push(bytes); + if (reader.channels === null || reader.preSkip === null) throw new Error('no OpusHead'); + return { + channels: reader.channels, + preSkip: reader.preSkip, + granule: reader.granule, + packets: reader.packets.map((packet) => packet.data), + pages: reader.pages, + }; +} + +/** A packet of a stream (#349) and where it lies, in samples counted from the first, pre-skip and all. */ +export interface PlacedPacket { + data: Uint8Array; + at: number; + samples: number; +} + +/** + * Reads Ogg Opus as its bytes come in (#349), a page once all of it is there, checking each; throws + * at the first that is not right. The packets after the tags are placed by the samples each stands for. + */ +export class OggOpusReader { + channels: number | null = null; + preSkip: number | null = null; + /** The granule position of the last page read, and whether it was the last of the stream. */ + granule = 0; + ended = false; + pages = 0; + readonly packets: PlacedPacket[] = []; + private held: Uint8Array = new Uint8Array(0); + private open: Uint8Array[] = []; + private headers = 0; + private next = 0; + + push(bytes: Uint8Array): void { + let data = this.held.length > 0 ? concat([this.held, bytes]) : bytes; + let at = 0; + while (data.length - at >= 27) { + if (text(data.subarray(at, at + 4)) !== 'OggS') throw new Error(`no page at byte ${at}`); + const count = data[at + 26]!; + if (data.length - at < 27 + count) break; + const lacing = data.subarray(at + 27, at + 27 + count); + const size = 27 + count + lacing.reduce((sum, value) => sum + value, 0); + if (data.length - at < size) break; + this.page(data.subarray(at, at + size), lacing); + at += size; + } + data = data.subarray(at); + this.held = data.length > 0 ? data.slice() : new Uint8Array(0); + } + + private page(page: Uint8Array, lacing: Uint8Array): void { + const view = new DataView(page.buffer, page.byteOffset, page.byteLength); + const crc = view.getUint32(22, true); + const blank = page.slice(); + blank.fill(0, 22, 26); + if (oggCrc(blank) !== crc) throw new Error(`page ${this.pages} has a wrong checksum`); + this.granule = view.getUint32(6, true) + view.getUint32(10, true) * 2 ** 32; + this.ended = (page[5]! & 0x04) !== 0; + this.pages += 1; + let from = 27 + lacing.length; for (const value of lacing) { - open.push(bytes.subarray(from, from + value)); + this.open.push(page.subarray(from, from + value)); from += value; if (value < 255) { - all.push(concat(open)); - open = []; + this.packet(concat(this.open)); + this.open = []; } } - at += size; } - const head = all[0]; - const preSkip = head ? preSkipOf(head) : null; - if (!head || preSkip === null) throw new Error('no OpusHead'); - return { channels: head[9]!, preSkip, granule, packets: all.slice(2), pages }; + + private packet(data: Uint8Array): void { + if (this.headers === 0) { + this.preSkip = preSkipOf(data); + if (this.preSkip === null) throw new Error('no OpusHead'); + this.channels = data[9]!; + } + if (this.headers < 2) { + this.headers += 1; + return; + } + const samples = opusPacketSamples(data); + this.packets.push({ data, at: this.next, samples }); + this.next += samples; + } +} + +/** The samples per channel at 48 kHz an Opus packet stands for, from its TOC byte (RFC 6716 3.1). */ +export function opusPacketSamples(packet: Uint8Array): number { + if (packet.length === 0) return 0; + const toc = packet[0]!; + const config = toc >> 3; + const size = + config < 12 + ? [480, 960, 1920, 2880][config & 3]! + : config < 16 + ? [480, 960][config & 1]! + : [120, 240, 480, 960][config & 3]!; + const code = toc & 3; + const frames = code === 0 ? 1 : code === 3 ? (packet[1] ?? 0) & 63 : 2; + return size * frames; } function ascii(value: string): Uint8Array { diff --git a/src/core/music/streamQueue.ts b/src/core/music/streamQueue.ts new file mode 100644 index 00000000..06720e8f --- /dev/null +++ b/src/core/music/streamQueue.ts @@ -0,0 +1,144 @@ +/** + * What a streamed sound (#349) is played from on the audio thread: the samples the page decoded, in + * the order they are heard, at `STREAM_RATE`. It reads them out at the rate of the context, `step` + * of its samples per sample heard, between two of them along a cubic curve. Time runs on whatever + * has arrived: a stretch that comes late is silent where it would have been heard and skipped once it + * is there, so a sound in a song stays on its beat. + */ +export const STREAM_RATE = 48000; + +export class StreamQueue { + private readonly chunks: Float32Array[][] = []; + /** Where in the first chunk the next sample is, and how far between it and the one after. */ + private index = 0; + private fraction = 0; + /** The sample of each channel read last, for the curve. */ + private readonly last: number[]; + private finished = false; + /** Samples of the stream read so far, heard or skipped, and those that were due but not there. */ + used = 0; + owed = 0; + /** Samples that came too late to be heard (test hook). */ + skipped = 0; + + constructor( + readonly channels: number, + private readonly step: number, + ) { + this.last = Array.from({ length: channels }, () => 0); + } + + push(chunk: Float32Array[]): void { + if (chunk.length === 0 || (chunk[0]?.length ?? 0) === 0) return; + this.chunks.push(chunk); + this.payOwed(); + } + + /** Nothing more comes: it ends once what is held has been read. */ + end(): void { + this.finished = true; + this.payOwed(); + } + + /** Samples held and not yet read. */ + get held(): number { + let count = -this.index; + for (const chunk of this.chunks) count += chunk[0]!.length; + return Math.max(0, count); + } + + /** Whether all of it has been read. */ + get done(): boolean { + return this.finished && this.held === 0; + } + + /** + * Fills `out` from what is held; what is not there yet is silence. Says whether it goes on: false + * once it has ended and been read to the end. + */ + render(out: Float32Array[]): boolean { + const length = out[0]?.length ?? 0; + for (let i = 0; i < length; i++) { + if (!this.ahead(2)) { + if (this.finished && this.held === 0) { + for (const channel of out) channel.fill(0, i); + return false; + } + if (!this.finished) { + for (const channel of out) channel[i] = 0; + this.owe(); + continue; + } + } + const t = this.fraction; + for (let c = 0; c < out.length; c++) { + const side = Math.min(c, this.channels - 1); + out[c]![i] = cubic(this.last[side]!, this.at(side, 0), this.at(side, 1), this.at(side, 2), t); + } + this.advance(this.step); + } + return !this.done; + } + + /** Whether `count` samples past the next one are held, or the stream ends before them. */ + private ahead(count: number): boolean { + if (this.finished) return this.held > 0; + return this.held > count; + } + + /** The sample `k` places after the next one, 0 past the end. */ + private at(side: number, k: number): number { + let offset = this.index + k; + for (const chunk of this.chunks) { + const data = chunk[Math.min(side, chunk.length - 1)]!; + if (offset < data.length) return data[offset]!; + offset -= data.length; + } + return 0; + } + + private advance(by: number): void { + this.fraction += by; + while (this.fraction >= 1) { + this.fraction -= 1; + this.take(); + } + } + + /** Moves past the next sample, keeping it for the curve. */ + private take(): void { + for (let side = 0; side < this.channels; side++) this.last[side] = this.at(side, 0); + this.index += 1; + this.used += 1; + while (this.chunks.length > 0 && this.index >= this.chunks[0]![0]!.length) { + this.index -= this.chunks[0]![0]!.length; + this.chunks.shift(); + } + } + + /** A sample was due and not there: skipped once it is. */ + private owe(): void { + this.fraction += this.step; + while (this.fraction >= 1) { + this.fraction -= 1; + this.owed += 1; + } + } + + private payOwed(): void { + while (this.owed > 0 && this.held > 0) { + this.take(); + this.owed -= 1; + this.skipped += 1; + } + } +} + +/** Catmull-Rom between `p1` and `p2`, `t` of the way; at 0 it is `p1` to the bit. */ +function cubic(p0: number, p1: number, p2: number, p3: number, t: number): number { + if (t === 0) return p1; + const a = -0.5 * p0 + 1.5 * p1 - 1.5 * p2 + 0.5 * p3; + const b = p0 - 2.5 * p1 + 2 * p2 - 0.5 * p3; + const c = -0.5 * p0 + 0.5 * p2; + return ((a * t + b) * t + c) * t + p1; +} diff --git a/src/core/music/streams.ts b/src/core/music/streams.ts new file mode 100644 index 00000000..d9a8f97a --- /dev/null +++ b/src/core/music/streams.ts @@ -0,0 +1,154 @@ +/** + * Sounds streamed rather than decoded whole (#349). A sound with `stream` on is played from a copy as + * Opus in Ogg, made in the browser, and each of its clips from a copy of its own cut, so no clip needs + * the whole file either. A bake (#340) is Opus already and always streams. What a game plays is still + * named as before – the file, or the cut a clip is – and the table here says which of those names to + * stream from which copy. + */ +import type { SoundAsset, SoundClip, SoundStream } from '../model/Asset'; +import type { Project } from '../model/Project'; +import { clipsOf, cutName, isSoundFile } from '../model/sound'; +import { cutOf, streamsOf } from '../properties/sounds'; +import { STREAM_RATE } from './streamQueue'; + +/** Seconds from which the Sound tab suggests streaming. */ +export const STREAM_SUGGESTED = 60; + +/** What a name is streamed from: the copy, and how many samples per channel it holds at `STREAM_RATE`. */ +export interface StreamedFile { + file: string; + frames: number; +} + +/** What a sound or a clip played streamed is read as (#349), in samples of the copy at `STREAM_RATE`. */ +export interface StreamPlan { + /** Where it starts. */ + from: number; + /** Where its first run ends: the end of the copy, or of the head of a bake. */ + until: number; + /** What it runs round after that; null to end there. */ + loop: { start: number; end: number } | null; +} + +/** A sound or a clip played from `offset` seconds in, round and round where it loops. */ +export function playedPlan(frames: number, offset: number, loop: boolean): StreamPlan { + const at = Math.max(0, Math.round(offset * STREAM_RATE)); + const from = loop && frames > 0 ? at % frames : Math.min(at, frames); + return { from, until: frames, loop: loop && frames > 0 ? { start: 0, end: frames } : null }; +} + +/** Walks a plan in stretches of the copy, each read straight through; a loop starts a new one. */ +export class StreamCursor { + private at: number; + private end: number; + + constructor(private readonly plan: StreamPlan) { + this.at = plan.from; + this.end = plan.until; + } + + /** Where the next stretch starts; null once all of it has been read. */ + peek(): number | null { + if (this.at < this.end) return this.at; + const loop = this.plan.loop; + return loop && loop.end > loop.start ? loop.start : null; + } + + /** The next stretch, at most `most` samples; null once all of it has been read. */ + next(most: number): { from: number; to: number } | null { + if (this.at >= this.end) { + const loop = this.plan.loop; + if (!loop || loop.end <= loop.start) return null; + this.at = loop.start; + this.end = loop.end; + } + const from = this.at; + this.at = Math.min(this.end, from + Math.max(1, most)); + return { from, to: this.at }; + } +} + +/** + * Every name a game streams, with its copy: each sound with `stream` on whose copy is of its file, + * each clip of one whose copy is of its cut – a clip without one plays decoded until it is there – and + * every bake, which streams from itself. + */ +export function streamTable(project: Project): Map { + const table = new Map(); + for (const asset of project.assets) { + if (asset.type === 'sound') { + if (!streamsOf(asset.props) || !asset.file) continue; + const own = heldCopy(asset.stream, asset.file); + if (own) table.set(asset.file, own); + for (const clip of clipsOf(asset)) { + const played = cutName(asset.file, cutOf(clip.props)); + const copy = played === asset.file ? own : heldCopy(clip.stream, played); + if (copy) table.set(played, copy); + } + } else if ((asset.type === 'pattern' || asset.type === 'song') && asset.bake) { + const bake = asset.bake; + table.set(bake.file, { file: bake.file, frames: bake.head + bake.round }); + } + } + return table; +} + +function heldCopy(stream: SoundStream | undefined, played: string): StreamedFile | null { + return stream && stream.from === played ? { file: stream.file, frames: stream.frames } : null; +} + +/** A copy to make: of a sound, or of one of its clips, from what it plays now. */ +export interface StreamJob { + assetId: string; + clipId: string | null; + from: string; +} + +/** A copy that is to go: its sound no longer streams, or a clip of it has become the whole file. */ +export interface StreamDrop { + assetId: string; + clipId: string | null; +} + +/** What is missing and what is left over, sounds in their order, each sound before its clips. */ +export function streamWork(project: Project): { jobs: StreamJob[]; drops: StreamDrop[] } { + const jobs: StreamJob[] = []; + const drops: StreamDrop[] = []; + for (const asset of project.assets) { + if (asset.type !== 'sound') continue; + const on = streamsOf(asset.props) && asset.file !== ''; + const keep = (owner: SoundAsset | SoundClip, clipId: string | null, from: string | null) => { + if (from === null) { + if (owner.stream) drops.push({ assetId: asset.id, clipId }); + } else if (owner.stream?.from !== from) jobs.push({ assetId: asset.id, clipId, from }); + }; + keep(asset, null, on ? asset.file : null); + for (const clip of clipsOf(asset)) { + const played = cutName(asset.file, cutOf(clip.props)); + keep(clip, clip.id, on && played !== asset.file ? played : null); + } + } + return { jobs, drops }; +} + +/** Held to what a copy may be, else none. */ +export function heldStream(raw: unknown): SoundStream | null { + if (typeof raw !== 'object' || raw === null) return null; + const s = raw as Record; + const whole = (value: unknown) => typeof value === 'number' && Number.isInteger(value) && value >= 0; + if (typeof s.file !== 'string' || !isSoundFile(s.file) || typeof s.from !== 'string' || s.from === '') + return null; + if (!whole(s.frames) || !whole(s.bytes)) return null; + return { file: s.file, from: s.from, frames: s.frames as number, bytes: s.bytes as number }; +} + +/** What a copy is uploaded as: like a bake's, behind a mark the sound picker leaves out by. */ +const STREAM_MARK = '[stream] '; + +export function streamUploadName(name: string): string { + return `${STREAM_MARK}${name.replace(/[/\\]/g, '-')}.ogg`; +} + +export function isStreamUpload(name: string): boolean { + return name.startsWith(STREAM_MARK); +} diff --git a/src/core/properties/PropertySchema.ts b/src/core/properties/PropertySchema.ts index e67bde58..87f94de9 100644 --- a/src/core/properties/PropertySchema.ts +++ b/src/core/properties/PropertySchema.ts @@ -103,7 +103,11 @@ import { } from './particles'; import { SHADER_KEY, shaderOf, shaderParamDefs } from './shaders'; import { SHADER_RESOLUTION_KEY, SHADER_RESOLUTION_OPTIONS } from './cameraShader'; -import { LOOP_KEY as SOUND_LOOP_KEY, VOLUME_KEY as SOUND_VOLUME_KEY } from './sounds'; +import { + LOOP_KEY as SOUND_LOOP_KEY, + STREAM_KEY as SOUND_STREAM_KEY, + VOLUME_KEY as SOUND_VOLUME_KEY, +} from './sounds'; import { DEFAULT_BARS, DEFAULT_GRID, @@ -692,6 +696,8 @@ const ASSET_SCHEMAS: Record = { sound: [ { key: SOUND_VOLUME_KEY, label: 'Volume', kind: 'number', default: 1, min: 0, max: 1, step: 0.05 }, { key: SOUND_LOOP_KEY, label: 'Loop', kind: 'boolean', default: false }, + // A long one is streamed from a copy as Opus rather than decoded whole (#349). + { key: SOUND_STREAM_KEY, label: 'Stream', kind: 'boolean', default: false }, FX_ROW, ], // The effects are the asset's graph (see SoundFxAsset), not a property. diff --git a/src/core/properties/sounds.ts b/src/core/properties/sounds.ts index dc0968ec..e8ea86f0 100644 --- a/src/core/properties/sounds.ts +++ b/src/core/properties/sounds.ts @@ -7,6 +7,8 @@ import type { PropertyDef } from './PropertySchema'; export const VOLUME_KEY = 'volume'; /** Whether this sound is music: it repeats until something stops it. */ export const LOOP_KEY = 'loop'; +/** Whether a game streams it from a copy as Opus rather than decoding all of it first (#349). */ +export const STREAM_KEY = 'stream'; /** Where in its file a clip starts and ends, in seconds; an end of 0 is the end of the file (#343). */ export const START_KEY = 'start'; export const END_KEY = 'end'; @@ -47,6 +49,10 @@ export function loopsOf(bag: PropertyBag): boolean { return booleanOf(bag[LOOP_KEY]); } +export function streamsOf(bag: PropertyBag): boolean { + return booleanOf(bag[STREAM_KEY]); +} + /** The stretch of its file a clip is, its fades in seconds. */ export function cutOf(bag: PropertyBag): SoundCut { const seconds = (key: string, fallback: number) => Math.max(0, coerceAs(bag[key], 'number', fallback)); diff --git a/src/core/store/serialize.ts b/src/core/store/serialize.ts index c8508779..73183ae2 100644 --- a/src/core/store/serialize.ts +++ b/src/core/store/serialize.ts @@ -22,6 +22,7 @@ import { isShaderNodeDef } from '../shader/nodes'; import { isSoundFxNodeDef } from '../soundfx/nodes'; import { INSTRUMENT_SAMPLE, isInstrumentNodeDef, withSamplePlay } from '../instrument/nodes'; import { heldBake } from '../music/bake'; +import { heldStream } from '../music/streams'; import { heldPatternNotes } from '../music/pattern'; import { isEmbeddable } from '../music/embedded'; import { heldSongItems, heldSongTracks } from '../music/song'; @@ -263,7 +264,16 @@ export function deserializeProject(raw: unknown, fallbackId = ''): Project { } else if (type === 'sound') { const file = typeof a.file === 'string' && isSoundFile(a.file) ? a.file : ''; const clips = normalizeClips(a.clips, takenIds); - assets.push({ id, type, name, props, file, ...(clips.length > 0 && { clips }) }); + const stream = heldStream(a.stream); + assets.push({ + id, + type, + name, + props, + file, + ...(clips.length > 0 && { clips }), + ...(stream && { stream }), + }); } else if (type === 'tile') { // A picture naming nothing is kept, like an animation's dangling root: it reads as missing // in the tab and comes back when the texture it named is restored by an undo. @@ -763,10 +773,12 @@ function normalizeClips(raw: unknown, takenIds: Set): SoundClip[] { const id = typeof c.id === 'string' ? c.id : ''; if (!id || takenIds.has(id)) continue; takenIds.add(id); + const stream = heldStream(c.stream); clips.push({ id, name: typeof c.name === 'string' ? c.name : '', props: normalizeBag(c.props, CLIP_ROWS, true), + ...(stream && { stream }), }); } return clips; diff --git a/src/editor/Editor.ts b/src/editor/Editor.ts index 538ce83e..17068d69 100644 --- a/src/editor/Editor.ts +++ b/src/editor/Editor.ts @@ -9,6 +9,7 @@ import { pluginsOf } from '../core/store/projectPlugins'; import type { FontStore } from '../core/store/FontStore'; import type { SnapshotStore } from '../core/store/SnapshotStore'; import type { SoundStore } from '../core/store/SoundStore'; +import { StreamKeeper } from './sound/StreamKeeper'; import type { TextureStore } from '../core/store/TextureStore'; import type { UsersStore } from '../core/store/UsersStore'; import { gameClock, timeStep } from '../core/time'; @@ -249,6 +250,8 @@ export class Editor implements PlayHost { readonly shaderEditor: ShaderEditor; readonly particlesEditor: ParticlesEditor; readonly soundEditor: SoundEditor; + /** Makes the copies streamed sounds play from (#349). */ + readonly streamKeeper: StreamKeeper; readonly soundFxEditor: SoundFxEditor; readonly instrumentEditor: InstrumentEditor; readonly patternEditor: PatternEditor; @@ -349,7 +352,8 @@ export class Editor implements PlayHost { this.stateMachineEditor = new StateMachineEditor(state); this.shaderEditor = new ShaderEditor(state); this.particlesEditor = new ParticlesEditor(state); - this.soundEditor = new SoundEditor(state); + this.streamKeeper = new StreamKeeper(state, sounds); + this.soundEditor = new SoundEditor(state, this.streamKeeper); this.soundFxEditor = new SoundFxEditor(state); this.instrumentEditor = new InstrumentEditor(state); this.patternEditor = new PatternEditor(state, sounds); diff --git a/src/editor/debug/soundHooks.ts b/src/editor/debug/soundHooks.ts index 33399ca1..4eabc44c 100644 --- a/src/editor/debug/soundHooks.ts +++ b/src/editor/debug/soundHooks.ts @@ -1,5 +1,7 @@ import { findAssetOf } from '../../core/model/Project'; import { clipsOf, findSound } from '../../core/model/sound'; +import { STREAM_RATE } from '../../core/music/streamQueue'; +import { playedPlan, streamTable } from '../../core/music/streams'; import { playedSoundOf } from '../../core/properties/playedSound'; import { soundLibrary } from '../../viewport/SoundLibrary'; import { centreOf, type HookContext } from './hookHelpers'; @@ -72,5 +74,62 @@ export function soundHooks({ state, editor }: HookContext) { }; }, }, + /** Streamed sounds (#349). */ + streams: { + /** Done once every copy wanted has been made. */ + idle: () => editor.streamKeeper.idle(), + /** Every name a game streams, with its copy. */ + table: () => [...streamTable(state.project)].map(([name, copy]) => ({ name, ...copy })), + /** + * Streams `seconds` of a name offline from `offset` seconds in, heard from `at`, looped or not, + * and holds it against its copy decoded whole: the largest difference of any sample from what + * should be heard there – silence before `at` – against the feed's own decoder and against the + * browser's `decodeAudioData`, and how many samples came too late. + */ + exact: async (name: string, offset: number, loop: boolean, at: number, seconds: number) => { + const entry = streamTable(state.project).get(name); + if (!entry) return null; + const plan = playedPlan(entry.frames, offset, loop); + const frames = Math.round((at + seconds) * STREAM_RATE); + const [played, whole, browser] = await Promise.all([ + soundLibrary.renderStream(name, plan, at, frames), + soundLibrary.streamedWhole(name), + soundLibrary.decodeApart(entry.file, STREAM_RATE), + ]); + if (!played || !whole) return null; + const first = Math.round(at * STREAM_RATE); + const placeOf = (i: number) => + loop ? (plan.from + i - first) % entry.frames : plan.from + i - first; + const worst = (reference: (c: number, place: number) => number) => { + let most = 0; + played.channels.forEach((side, c) => { + for (let i = 0; i < frames; i++) { + const place = placeOf(i); + const expected = i < first || place >= entry.frames ? 0 : reference(c, place); + most = Math.max(most, Math.abs(side[i]! - expected)); + } + }); + return most; + }; + return { + frames: entry.frames, + decoded: browser.length, + own: worst((c, place) => whole[c]![place]!), + browser: worst( + (c, place) => browser.getChannelData(Math.min(c, browser.numberOfChannels - 1))[place]!, + ), + skipped: played.skipped, + }; + }, + /** What a sound and each of its clips keep of their copies. */ + copies: (id: string) => { + const asset = findAssetOf(state.project, id, 'sound'); + if (!asset) return null; + return { + sound: asset.stream ?? null, + clips: clipsOf(asset).map((clip) => ({ id: clip.id, stream: clip.stream ?? null })), + }; + }, + }, }; } diff --git a/src/editor/music/BakeButton.ts b/src/editor/music/BakeButton.ts index 7c88f32a..d70015b9 100644 --- a/src/editor/music/BakeButton.ts +++ b/src/editor/music/BakeButton.ts @@ -12,6 +12,7 @@ import { Button, formatBytes, Menu, Theme } from '../../ui'; import { BakeCancelled, makeBake } from '../../viewport/bakeRender'; import { HEADER_ICON_SVG } from '../panels/headerIcons'; import type { EditorState } from '../state'; +import { dropWhenSaved } from './dropWhenSaved'; import { clockText } from './timeView'; /** Where a bake stands for the asset shown. */ @@ -128,7 +129,7 @@ export class BakeButton extends Button { } const old = this.current(asset.id)?.bake?.file; this.state.setBake(asset.id, { file: uploaded.file, bytes: uploaded.bytes, ...rest }); - if (old && old !== uploaded.file) this.dropWhenSaved(old); + if (old && old !== uploaded.file) dropWhenSaved(this.state, this.sounds, old); this.state.setNotice( `Baked ${asset.name}: ${clockText(rest.length / BAKE_RATE)}, ${formatBytes(uploaded.bytes)}`, ); @@ -145,7 +146,7 @@ export class BakeButton extends Button { private remove(asset: BakedAsset): void { const old = asset.bake?.file; this.state.setBake(asset.id, null); - if (old) this.dropWhenSaved(old); + if (old) dropWhenSaved(this.state, this.sounds, old); } private current(id: string): BakedAsset | null { @@ -162,17 +163,4 @@ export class BakeButton extends Button { private changed(asset: BakedAsset): void { this.state.setNotice(`${asset.name} changed while it was baked – bake it again`, 'error'); } - - /** - * Deletes the file of a bake that has gone, once the project no longer names it on the server: the - * server keeps a file any saved project or snapshot names (409), and so does this. - */ - private dropWhenSaved(file: string): void { - const off = this.state.on('save', () => { - const status = this.state.saveStatus; - if (status === 'dirty' || status === 'saving') return; - off(); - if (status === 'saved') void this.sounds.delete(file).catch(() => {}); - }); - } } diff --git a/src/editor/music/dropWhenSaved.ts b/src/editor/music/dropWhenSaved.ts new file mode 100644 index 00000000..3378d151 --- /dev/null +++ b/src/editor/music/dropWhenSaved.ts @@ -0,0 +1,16 @@ +import type { SoundStore } from '../../core/store/SoundStore'; +import type { EditorState } from '../state'; + +/** + * Deletes a file of the sounds a bake (#340) or a stream copy (#349) left behind, once the project no + * longer names it on the server: the server keeps a file any saved project or snapshot names (409), + * and so does this. + */ +export function dropWhenSaved(state: EditorState, sounds: SoundStore, file: string): void { + const off = state.on('save', () => { + const status = state.saveStatus; + if (status === 'dirty' || status === 'saving') return; + off(); + if (status === 'saved') void sounds.delete(file).catch(() => {}); + }); +} diff --git a/src/editor/song/SongTimeline.ts b/src/editor/song/SongTimeline.ts index 310ae97d..9ed7f55a 100644 --- a/src/editor/song/SongTimeline.ts +++ b/src/editor/song/SongTimeline.ts @@ -1368,9 +1368,9 @@ export class SongTimeline extends Widget { } private ensureLoaded(file: string): void { - if (!file || this.loading.has(file) || soundLibrary.duration(file) !== null) return; + if (!file || this.loading.has(file) || soundLibrary.peaks(file) !== null) return; this.loading.add(file); - void soundLibrary.load(file).then(() => this.invalidate()); + void soundLibrary.loadPicture(file).then(() => this.invalidate()); } override layout(): void { diff --git a/src/editor/sound/SoundEditor.ts b/src/editor/sound/SoundEditor.ts index cd5c3251..369aa325 100644 --- a/src/editor/sound/SoundEditor.ts +++ b/src/editor/sound/SoundEditor.ts @@ -12,8 +12,10 @@ import { FADE_OUT_CURVE_KEY, FADE_OUT_KEY, REVERSE_KEY, + streamsOf, volumeOf, } from '../../core/properties/sounds'; +import { STREAM_SUGGESTED } from '../../core/music/streams'; import { Button, Dropdown, Label, Menu, NumberField } from '../../ui'; import { soundLibrary } from '../../viewport/SoundLibrary'; import { MASTER_MIX } from '../../core/music/mixer'; @@ -26,6 +28,7 @@ import { claimTransport, releaseTransport } from '../music/transport'; import { HEADER_ICON_SVG } from '../panels/headerIcons'; import { TOOLBAR_ICON_SVG } from '../panels/toolbarIcons'; import type { EditorState } from '../state'; +import type { StreamKeeper } from './StreamKeeper'; import { WaveCanvas } from './WaveCanvas'; const INFO_WIDTH = 140; @@ -71,7 +74,10 @@ export class SoundEditor extends AssetTabEditor { private playingId = ''; private readonly halt = () => this.stop(); - constructor(state: EditorState) { + constructor( + state: EditorState, + private readonly streams: StreamKeeper, + ) { super(state, { infoWidth: INFO_WIDTH }); this.playButton = new Button('Play', { icon: TOOLBAR_ICON_SVG.play, @@ -125,6 +131,8 @@ export class SoundEditor extends AssetTabEditor { state.on('load', () => this.stop()); // Picking a clip changes what the header plays and edits. state.on('selection', () => this.refresh()); + // How far a stream copy has got (#349) comes as a notice, not as a change of the project. + state.on('notice', () => this.refresh()); } override selectedId(): string { @@ -251,7 +259,20 @@ export class SoundEditor extends AssetTabEditor { if (!asset.file) return 'Not uploaded'; if (!span || whole === null) return ''; const kept = span.end - span.start; - return kept >= whole ? `${whole.toFixed(2)} s` : `${kept.toFixed(2)} s of ${whole.toFixed(2)} s`; + const length = kept >= whole ? `${whole.toFixed(2)} s` : `${kept.toFixed(2)} s of ${whole.toFixed(2)} s`; + return `${length}${this.streamText(asset, whole)}`; + } + + /** + * Whether it streams (#349): its copy is made, being made or yet to be; a long one that does not + * is pointed at the switch. + */ + private streamText(asset: SoundAsset, whole: number): string { + const making = this.streams.making(asset.id); + if (making !== null) return ` · copy ${Math.round(making * 100)}%`; + if (streamsOf(resolveAssetProps(this.state.project, asset))) + return asset.stream?.from === asset.file ? ' · streamed' : ' · copy waiting'; + return whole >= STREAM_SUGGESTED ? ' · long, try Stream' : ''; } protected override syncExtras(asset: SoundAsset | null): void { diff --git a/src/editor/sound/SoundPicker.ts b/src/editor/sound/SoundPicker.ts index d193d750..244fe05f 100644 --- a/src/editor/sound/SoundPicker.ts +++ b/src/editor/sound/SoundPicker.ts @@ -1,5 +1,6 @@ import { Container, type FederatedPointerEvent, Graphics, type Text } from 'pixi.js'; import { isBakeUpload } from '../../core/music/bake'; +import { isStreamUpload } from '../../core/music/streams'; import type { StoredSound } from '../../core/store/SoundStore'; import { drawSvg, formatBytes, textStyle, Theme, truncate, crispText } from '../../ui'; import { soundLibrary } from '../../viewport/SoundLibrary'; @@ -121,7 +122,9 @@ class SoundPicker extends FilePicker { /** A bake (#340) is in the sounds, but no sound to pick. */ protected async fetchItems(): Promise { - const sounds = (await this.deps.sounds.list()).filter((sound) => !isBakeUpload(sound.name)); + const sounds = (await this.deps.sounds.list()).filter( + (sound) => !isBakeUpload(sound.name) && !isStreamUpload(sound.name), + ); return withAssetNames(sounds, this.deps.state.project, 'sound'); } diff --git a/src/editor/sound/StreamKeeper.ts b/src/editor/sound/StreamKeeper.ts new file mode 100644 index 00000000..dc0076ce --- /dev/null +++ b/src/editor/sound/StreamKeeper.ts @@ -0,0 +1,164 @@ +import type { SoundAsset } from '../../core/model/Asset'; +import { findAsset } from '../../core/model/Project'; +import { clipsOf, cutName, cutChannels, cutRange, parseCutName, soundRefName } from '../../core/model/sound'; +import { STREAM_RATE } from '../../core/music/streamQueue'; +import { type StreamJob, streamUploadName, streamWork } from '../../core/music/streams'; +import { cutOf, streamsOf } from '../../core/properties/sounds'; +import type { SoundStore } from '../../core/store/SoundStore'; +import { BakeCancelled, encodeOpus } from '../../viewport/bakeRender'; +import { soundLibrary } from '../../viewport/SoundLibrary'; +import { dropWhenSaved } from '../music/dropWhenSaved'; +import type { EditorState } from '../state'; + +/** Milliseconds the project has to stay as it is before a copy is made: a clip dragged about is one. */ +const SETTLE_MS = 600; + +/** + * Makes the copies streamed sounds are played from (#349). A moment after the project last changed, + * each sound with `stream` on gets a copy of its file as Opus, and each of its clips one of its cut, + * one at a time, from the file decoded for it and let go again. A copy no longer wanted is taken off + * its sound and deleted once the project has been saved without it. Not while the game runs. + */ +export class StreamKeeper { + private job: { job: StreamJob; share: number } | null = null; + private timer: ReturnType | null = null; + /** Copies that could not be made, so one broken file is not tried over and over. */ + private readonly failed = new Set(); + private waiting: (() => void)[] = []; + + constructor( + private readonly state: EditorState, + private readonly sounds: SoundStore, + ) { + for (const event of ['project', 'load', 'mode'] as const) state.on(event, () => this.later()); + this.later(); + } + + /** How far the copy of that sound, or of one of its clips, has got, 0 to 1; null while none is made. */ + making(assetId: string): number | null { + return this.job?.job.assetId === assetId ? this.job.share : null; + } + + /** Done once there is nothing left to make (test hook). */ + idle(): Promise { + if (!this.job && this.timer === null && this.pending().length === 0) return Promise.resolve(); + return new Promise((resolve) => this.waiting.push(resolve)); + } + + private later(wait = SETTLE_MS): void { + if (this.timer !== null) clearTimeout(this.timer); + this.timer = setTimeout(() => { + this.timer = null; + void this.work(); + }, wait); + } + + private pending(): StreamJob[] { + return streamWork(this.state.project).jobs.filter((job) => !this.failed.has(keyOf(job))); + } + + private async work(): Promise { + if (this.job || this.state.mode === 'play') return; + for (const drop of streamWork(this.state.project).drops) { + const old = this.ownerOf(drop.assetId, drop.clipId)?.stream?.file; + this.state.setStream(drop.assetId, drop.clipId, null); + if (old && !this.named(old)) dropWhenSaved(this.state, this.sounds, old); + } + const next = this.pending()[0]; + if (!next) { + for (const resolve of this.waiting.splice(0)) resolve(); + return; + } + this.job = { job: next, share: 0 }; + try { + await this.make(this.job); + } catch (error: unknown) { + // Called off because what it was made of changed: the next round makes the right one. + if (error instanceof BakeCancelled) return this.later(0); + this.failed.add(keyOf(next)); + this.state.setNotice( + `No stream copy of ${this.label(next)}: ${error instanceof Error ? error.message : String(error)}`, + 'error', + ); + } finally { + this.job = null; + } + this.later(0); + } + + private async make(running: { job: StreamJob; share: number }): Promise { + const { job } = running; + const label = this.label(job); + const { file, cut } = parseCutName(job.from); + const whole = await soundLibrary.decodeApart(file, STREAM_RATE); + let channels = Array.from({ length: Math.min(2, whole.numberOfChannels) }, (_, c) => + whole.getChannelData(c), + ); + if (cut) channels = cutChannels(channels, cutRange(whole.length, whole.sampleRate, cut)); + const data = await encodeOpus(channels, { + progress: (share) => { + const before = Math.round(running.share * 100); + running.share = share; + const now = Math.round(share * 100); + if (now !== before) this.state.setNotice(`Making a stream copy of ${label}: ${now}%`, 'quiet'); + }, + cancelled: () => !this.stillWanted(job), + }); + if (!this.stillWanted(job)) return; + const uploaded = await this.sounds.upload(data.slice().buffer, 'audio/ogg', streamUploadName(label)); + // Nothing names the file yet, so it can go at once. + if (!this.stillWanted(job)) return void this.sounds.delete(uploaded.file).catch(() => {}); + const old = this.ownerOf(job.assetId, job.clipId)?.stream?.file; + const frames = channels[0]?.length ?? 0; + // Done: what the Sound tab reads when the copy lands is that it streams, not how far it got. + this.job = null; + this.state.setStream(job.assetId, job.clipId, { + file: uploaded.file, + from: job.from, + frames, + bytes: uploaded.bytes, + }); + if (old && old !== uploaded.file && !this.named(old)) dropWhenSaved(this.state, this.sounds, old); + this.state.setNotice(`Made a stream copy of ${label}`, 'quiet'); + } + + /** Whether the sound still streams and still plays what the copy is being made of. */ + private stillWanted(job: StreamJob): boolean { + const asset = this.sound(job.assetId); + if (!asset || !streamsOf(asset.props)) return false; + if (job.clipId === null) return asset.file === job.from; + const clip = clipsOf(asset).find((c) => c.id === job.clipId); + return clip !== undefined && cutName(asset.file, cutOf(clip.props)) === job.from; + } + + /** Whether a copy is still named by another sound or clip – a sound and its duplicate share one. */ + private named(file: string): boolean { + return this.state.project.assets.some( + (asset) => + asset.type === 'sound' && + (asset.stream?.file === file || clipsOf(asset).some((clip) => clip.stream?.file === file)), + ); + } + + private sound(id: string): SoundAsset | null { + const asset = findAsset(this.state.project, id); + return asset?.type === 'sound' ? asset : null; + } + + private ownerOf(assetId: string, clipId: string | null): { stream?: SoundAsset['stream'] } | null { + const asset = this.sound(assetId); + if (!asset) return null; + return clipId === null ? asset : (clipsOf(asset).find((clip) => clip.id === clipId) ?? null); + } + + private label(job: StreamJob): string { + const asset = this.sound(job.assetId); + if (!asset) return 'a sound'; + const clip = clipsOf(asset).find((c) => c.id === job.clipId) ?? null; + return soundRefName({ asset, clip }); + } +} + +function keyOf(job: StreamJob): string { + return `${job.assetId}:${job.clipId ?? ''}:${job.from}`; +} diff --git a/src/editor/state.ts b/src/editor/state.ts index ff1d4741..73631df1 100644 --- a/src/editor/state.ts +++ b/src/editor/state.ts @@ -6,7 +6,7 @@ import type { ObjectPick, PickEnd } from './objectPick'; import { findKey, findTrack, hasKey, type KeyRef, sameKey } from '../core/model/animation'; import { type Command, CommandError } from '../core/commands/Command'; import { CommandStack } from '../core/commands/CommandStack'; -import type { Asset, GraphAsset, SceneAsset } from '../core/model/Asset'; +import type { Asset, GraphAsset, SceneAsset, SoundStream } from '../core/model/Asset'; import { findAnnotation, type Graph } from '../core/graph'; import { findObject } from '../core/model/hierarchy'; import { allShapes } from '../core/model/vectorTree'; @@ -443,6 +443,23 @@ export class EditorState { this.emit('project'); } + /** + * The copy a sound or one of its clips streams from (#349), or none: saved, and like a bake no undo + * step – it is made again whenever it no longer fits. + */ + setStream(assetId: string, clipId: string | null, stream: SoundStream | null): void { + for (const project of [this.project, this.edit?.project]) { + const asset = project && findAsset(project, assetId); + if (asset?.type !== 'sound') continue; + const owner = clipId === null ? asset : clipsOf(asset).find((clip) => clip.id === clipId); + if (!owner) continue; + if (stream) owner.stream = stream; + else delete owner.stream; + } + this.setSaveStatus('dirty'); + this.emit('project'); + } + /** * Runs a command through the undo stack. Returns false (and sets a notice) when invalid. * In play mode the command changes the runtime copy only (see `enterPlay`). diff --git a/src/viewport/SoundLibrary.ts b/src/viewport/SoundLibrary.ts index 19da8461..c8b7ad6b 100644 --- a/src/viewport/SoundLibrary.ts +++ b/src/viewport/SoundLibrary.ts @@ -31,11 +31,13 @@ import { limiterLatency } from '../core/soundfx/dynamics'; import type { ScheduledClip, ScheduledNote } from '../core/music/Scheduler'; import { dbToGain, SOUND_ITEM_MAX_DB } from '../core/music/song'; import { DEFAULT_TEMPO } from '../core/music/time'; +import { playedPlan, type StreamedFile, type StreamPlan } from '../core/music/streams'; +import { STREAM_RATE } from '../core/music/streamQueue'; import { type CutRange, cutChannels, cutRange, parseCutName } from '../core/model/sound'; import { shifterLatency } from '../core/soundfx/pitchPlayer'; import { REVERB_MAX_DECAY } from '../core/soundfx/reverb'; import type { AudioSink, ClipTiming, NoteTiming } from '../runtime/systems/Audio'; -import type { ChipOptions, PitchOptions } from './fxWorklet'; +import type { ChipOptions, PitchOptions, StreamOptions } from './fxWorklet'; import fxWorkletUrl from './fxWorklet.ts?worker&url'; import { bothAtCenter, @@ -57,6 +59,7 @@ import { stopProcessor, } from './fxNodes'; import { type Spectrum, SpectrumTaps } from './spectrum'; +import { decodeSpan, StreamFeed, StreamFile } from './streaming'; import { wake } from './wake'; /** A voice is no louder than a sound item of a song turned all the way up. */ @@ -65,6 +68,8 @@ const MAX_VOLUME = dbToGain(SOUND_ITEM_MAX_DB); const DEFAULT_BASE = '/game-maker/data/sounds'; /** The rate a sound loaded ahead of its first play is decoded at; a source resamples it on the way out. */ const DECODE_RATE = 48000; +/** The rate the picture of a streamed sound is decoded at (#349): enough for its peaks, a sixth of the size. */ +const PICTURE_RATE = 8000; /** The longest a send of a track is held back to meet the latest way of it (#350). */ const MAX_SEND_DELAY = 1; @@ -199,6 +204,8 @@ interface Wiring { /** Seconds into the sound its ways start, and whether it is a clip of a song, which fades out at its end (#334). */ offset: number; clip: boolean; + /** The copy a way without a source of its own streams (#349), instead of a buffer of `file`. */ + stream: StreamFile | null; sources: Source[]; drivers: Driver[]; envelopes: Map; @@ -229,6 +236,8 @@ interface Start { clip?: { offset: number; end: number }; /** The track of a song it is heard on (#334), `''` straight out. */ bus?: string; + /** The copy `file` streams from (#349); its bytes are not loaded then, its head is. */ + stream?: StreamFile; } /** @@ -262,6 +271,9 @@ interface Pending { * * A way with a Pitch is played by a worklet (#317), loaded into the context with the first play that * needs it; that play waits for it as it would for its bytes. + * + * A sound that streams (#349) is the exception to "decoded whole": a `stream-player` on that worklet + * plays it from its copy, fed a few seconds ahead (`streaming.ts`), and its file is never decoded. */ export class SoundLibrary implements AudioSink { private base = DEFAULT_BASE; @@ -331,6 +343,7 @@ export class SoundLibrary implements AudioSink { /** The taps of the voices and buses playing now, and the node listened to (#374). */ private readonly spectra = new SpectrumTaps(); private readonly peaked = new Map(); + private readonly pictures = new Map>(); /** * The patterns and songs a game plays, each on a scheduler of its own. What it asks after are its * own voices – asking `playing`, which counts the playings too, would ask the list it is in. @@ -363,12 +376,40 @@ export class SoundLibrary implements AudioSink { private mix: readonly MixBus[] = []; /** The levels a running game set its buses to, over those of their assets; until `resetMix`. */ private readonly mixVolumes = new Map(); + /** The names streamed (#349) and the copies they stream from, by name and by copy. */ + private streams: ReadonlyMap = new Map(); + private readonly streamFiles = new Map(); + private readonly feeds = new Set(); + /** Every sample streamed so far that came too late to be heard, of the feeds gone (test hook). */ + private skippedBefore = 0; /** Where the files are served from. Set once at start-up, before anything plays. */ configure(base: string): void { this.base = base.replace(/\/+$/, ''); } + /** + * The names a game streams rather than decodes (#349), each with its copy (`streamTable`): the first + * seconds of every copy are fetched and decoded now, so it starts at once. A library that bakes, + * and a browser without Opus in WebCodecs, decode everything as before. + */ + setStreams(table: ReadonlyMap): void { + this.streams = table; + for (const name of table.keys()) void this.streamed(name)?.ready(); + } + + /** The copy a name streams from, while it streams and the copy has not failed. */ + private streamed(name: string): StreamFile | null { + const entry = this.streams.get(name); + if (!entry || this.render || typeof AudioDecoder === 'undefined') return null; + let file = this.streamFiles.get(entry.file); + if (!file) { + file = new StreamFile(`${this.base}/${entry.file}`, entry.frames); + this.streamFiles.set(entry.file, file); + } + return file.failed ? null : file; + } + /** * The number is handed out at once, before the bytes are there: a graph can tune a voice in the * same frame it asked for it, and the voice then starts with those values. @@ -376,6 +417,7 @@ export class SoundLibrary implements AudioSink { play(file: string, volume: number, loop: boolean, owner = '', fx: FxVoice = DRY_VOICE, bus = ''): number { if (file === '' || this.hasFailed(file)) return 0; const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO }; + const stream = this.streamed(file); return this.begin({ id: 0, file, @@ -387,6 +429,7 @@ export class SoundLibrary implements AudioSink { duration: 0, at: 0, bus, + ...(stream && { stream }), }); } @@ -424,6 +467,7 @@ export class SoundLibrary implements AudioSink { if (file === '' || this.hasFailed(file)) return 0; const layered = { layers: [fx.plan], params: fx.params, tempo: fx.tempo ?? DEFAULT_TEMPO }; const { at, offset, duration, volume, bus } = clip; + const stream = this.streamed(file); return this.begin({ id: 0, file, @@ -436,6 +480,7 @@ export class SoundLibrary implements AudioSink { at, clip: { offset, end: at + Math.max(0, duration) }, bus, + ...(stream && { stream }), }); } @@ -1050,19 +1095,102 @@ export class SoundLibrary implements AudioSink { /** * The bake of a playing (#340) as one voice on its master, heard from `heard`: its file once to its - * end, faded out there, or up to the loop end and then round its second round – in one source where - * the round follows at once, else in a second that takes over to the sample. 0 without its bytes. + * end, faded out there, or up to the loop end and then round its second round. Streamed where it can + * be (#349), else decoded – in one source where the round follows at once, else in a second that + * takes over to the sample. 0 without its bytes. */ private playBaked(id: number, bake: BakePlay, heard: number): number { - const buffer = this.buffer(bake.file); - if (!buffer) return 0; + const stream = this.hasWorklet ? this.streamed(bake.file) : null; + const streaming = stream?.head ? stream : null; + const buffer = streaming ? null : this.buffer(bake.file); + if (!streaming && !buffer) return 0; const context = this.audio(); const bus = groupBus(id, ''); const at = Math.max(context.currentTime, heard - this.busLatency(bus)); const gain = context.createGain(); gain.gain.value = Math.max(0, Math.min(MAX_VOLUME, bake.gain)); gain.connect(this.busInput(bus)); - const { end, until, round } = bake.playback; + const { end } = bake.playback; + const played = streaming + ? this.bakeStreamed(streaming, bake.playback, gain, at) + : this.bakeDecoded(buffer!, bake.playback, gain, at); + const voice: Voice = { + id: this.nextId++, + file: bake.file, + owner: '', + bus, + at, + sources: played.sources, + ending: new Set(), + drivers: [], + envelopes: [], + finish: null, + gain, + fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO }, + knobs: [], + rings: false, + ringing: false, + quiet: null, + vocoders: 0, + released: false, + ringFade: BAKE_LET_RING, + }; + const going = new Set(voice.sources); + voice.finish = (source) => { + if (going.delete(source) && going.size === 0) this.ended(voice); + }; + for (const source of played.sources) source.ended(() => voice.finish?.(source)); + this.voices.push(voice); + this.started += 1; + this.last = { + file: bake.file, + volume: gain.gain.value, + loop: bake.playback.round !== null, + layers: 0, + pitches: [], + stages: 0, + duration: played.duration, + at, + envelopes: 0, + bus, + }; + if (end !== null) { + gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE)); + gain.gain.linearRampToValueAtTime(0, at + end); + } + played.start(); + return voice.id; + } + + /** A bake streamed (#349): one `stream-player` along the same plan, the round looped to the sample. */ + private bakeStreamed( + stream: StreamFile, + { end, until, round }: BakePlay['playback'], + gain: GainNode, + at: number, + ): { sources: Source[]; start: () => void; duration: number } { + const frames = (seconds: number) => Math.round(seconds * STREAM_RATE); + const plan: StreamPlan = + end !== null + ? { from: 0, until: frames(end), loop: null } + : { + from: 0, + until: frames(until), + loop: round && { start: frames(round.start), end: frames(round.end) }, + }; + const { node, source } = this.streamNode(this.audio(), stream, plan, at); + node.connect(gain); + return { sources: [source], start: () => {}, duration: stream.frames / STREAM_RATE }; + } + + /** A bake decoded, in one buffer source or two. */ + private bakeDecoded( + buffer: AudioBuffer, + { end, until, round }: BakePlay['playback'], + gain: GainNode, + at: number, + ): { sources: Source[]; start: () => void; duration: number } { + const context = this.audio(); const made = (from: number, loop: { start: number; end: number } | null) => { const node = context.createBufferSource(); node.buffer = buffer; @@ -1090,56 +1218,15 @@ export class SoundLibrary implements AudioSink { : round ? [made(0, null), made(round.start, round)] : [made(0, null)]; - const voice: Voice = { - id: this.nextId++, - file: bake.file, - owner: '', - bus, - at, - sources: parts.map((part) => part.source), - ending: new Set(), - drivers: [], - envelopes: [], - finish: null, - gain, - fx: { layers: [], params: {}, tempo: DEFAULT_TEMPO }, - knobs: [], - rings: false, - ringing: false, - quiet: null, - vocoders: 0, - released: false, - ringFade: BAKE_LET_RING, - }; - const going = new Set(voice.sources); - voice.finish = (source) => { - if (going.delete(source) && going.size === 0) this.ended(voice); - }; - for (const { source } of parts) source.ended(() => voice.finish?.(source)); - this.voices.push(voice); - this.started += 1; - this.last = { - file: bake.file, - volume: gain.gain.value, - loop: round !== null, - layers: 0, - pitches: [], - stages: 0, - duration: buffer.duration, - at, - envelopes: 0, - bus, - }; const [head, next] = parts; - if (end !== null) { - gain.gain.setValueAtTime(gain.gain.value, at + Math.max(0, end - FADE)); - gain.gain.linearRampToValueAtTime(0, at + end); - head!.node.start(at, 0, end); - } else if (next) { - head!.node.start(at, 0, until); - next.node.start(at + until, next.from); - } else head!.node.start(at, 0); - return voice.id; + const start = () => { + if (end !== null) head!.node.start(at, 0, end); + else if (next) { + head!.node.start(at, 0, until); + next.node.start(at + until, next.from); + } else head!.node.start(at, 0); + }; + return { sources: parts.map((part) => part.source), start, duration: buffer.duration }; } /** @@ -1304,8 +1391,12 @@ export class SoundLibrary implements AudioSink { const asked: Start = { ...start, id }; // A Reverb's sound (#328) is waited for as well, but a voice is not silent for lack of it. const impulseFiles = start.fx.layers.flatMap(impulsesOf); - const loaded = [...start.files, ...impulseFiles].every((file) => this.buffer(file) !== null); - const worklet = start.fx.layers.some(needsWorklet) ? this.fxWorklet() : null; + // A streamed sound (#349) waits for the head of its copy and the worklet that plays it instead. + const { stream } = start; + const loaded = + [...(stream ? [] : start.files), ...impulseFiles].every((file) => this.buffer(file) !== null) && + (!stream || stream.head !== null); + const worklet = stream || start.fx.layers.some(needsWorklet) ? this.fxWorklet() : null; if (loaded && (!worklet || this.hasWorklet !== null)) { this.start(asked); return id; @@ -1321,19 +1412,23 @@ export class SoundLibrary implements AudioSink { }; this.pending.push(waiting); const impulsesLoaded = Promise.all(impulseFiles.map((file) => this.load(file))); - void Promise.all([Promise.all(start.files.map((file) => this.load(file))), worklet, impulsesLoaded]).then( - ([buffers]) => { - this.pending = this.pending.filter((other) => other !== waiting); - // A `Stop sound` – or the destruction of the behavior that asked – that happened while the - // bytes were in flight has to win, or a sound the run already ended would start afterwards. - const silent = buffers.every((buffer) => !buffer) && !start.fx.layers.some(hasChip); - if (silent || waiting.stopped) return; - if (this.cancelsFor(start.file, start.owner) !== askedAt) return; - this.start({ ...asked, fx: waiting.fx }); - for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease); - if (waiting.released) this.release(id); - }, - ); + // A copy that cannot be streamed leaves the sound decoded, as it would be without one. + const sounds = stream + ? stream.ready().then(async (ready) => [ready || (await this.load(start.file)) !== null]) + : Promise.all(start.files.map((file) => this.load(file))); + void Promise.all([sounds, worklet, impulsesLoaded]).then(([buffers]) => { + this.pending = this.pending.filter((other) => other !== waiting); + // A `Stop sound` – or the destruction of the behavior that asked – that happened while the + // bytes were in flight has to win, or a sound the run already ended would start afterwards. + const silent = buffers.every((buffer) => !buffer) && !start.fx.layers.some(hasChip); + if (silent || waiting.stopped) return; + if (this.cancelsFor(start.file, start.owner) !== askedAt) return; + const started: Start = { ...asked, fx: waiting.fx }; + if (stream?.failed) delete started.stream; + this.start(started); + for (const { params, time, ease } of waiting.timed) this.tune(id, params, time, ease); + if (waiting.released) this.release(id); + }); return id; } @@ -1463,7 +1558,13 @@ export class SoundLibrary implements AudioSink { /** Ways playing through the pitch worklet now, at most `MAX_VOCODERS`. */ vocoders: number; last: LastVoice | null; + /** + * Streaming (#349): sources playing from a copy now, samples that came too late to be heard, the + * bytes fetched of every copy, and the copies whose head is decoded. + */ + streaming: { playing: number; skipped: number; fetched: number; heads: number }; } { + const files = [...this.streamFiles.values()]; return { playing: this.voices.length, started: this.started, @@ -1475,6 +1576,12 @@ export class SoundLibrary implements AudioSink { worklet: this.hasWorklet, vocoders: this.voices.reduce((sum, voice) => sum + vocodersOf(voice), 0), last: this.last, + streaming: { + playing: this.feeds.size, + skipped: [...this.feeds].reduce((sum, feed) => sum + feed.skipped, this.skippedBefore), + fetched: files.reduce((sum, file) => sum + file.bytes, 0), + heads: files.filter((file) => file.head !== null).length, + }, }; } @@ -1538,11 +1645,21 @@ export class SoundLibrary implements AudioSink { * not late waiting on its bytes (#332). */ async prepare(layers: readonly FxPlan[], files: readonly string[] = []): Promise { + // A Sample or a Reverb's sound is always decoded; a sound or a bake may stream (#349). + const decoded = new Set([...layers.flatMap(filesOf), ...layers.flatMap(impulsesOf)]); + const streamed = files.flatMap((file) => { + const stream = decoded.has(file) ? null : this.streamed(file); + return stream ? [{ file, stream }] : []; + }); + const loaded = [...new Set([...decoded, ...files])].filter( + (file) => file !== '' && !streamed.some((entry) => entry.file === file), + ); await Promise.all([ - ...[...new Set([...layers.flatMap(filesOf), ...layers.flatMap(impulsesOf), ...files])] - .filter((file) => file !== '') - .map((file) => this.load(file)), - layers.some(needsWorklet) ? this.fxWorklet() : null, + ...loaded.map((file) => this.load(file)), + ...streamed.map(async ({ file, stream }) => { + if (!(await stream.ready())) await this.load(file); + }), + layers.some(needsWorklet) || streamed.length > 0 ? this.fxWorklet() : null, ]); } @@ -1562,7 +1679,10 @@ export class SoundLibrary implements AudioSink { */ duration(file: string): number | null { const range = this.rangeOf(file); - return range ? (range.to - range.from) / range.whole.sampleRate : null; + if (range) return (range.to - range.from) / range.whole.sampleRate; + // A streamed one (#349) is as long as its copy, known without decoding a sample of it. + const stream = this.streams.get(file); + return stream ? stream.frames / STREAM_RATE : null; } /** The decoded samples of a file as it is, without a cut; null while it has not been loaded. */ @@ -1588,27 +1708,49 @@ export class SoundLibrary implements AudioSink { if (known) return known; const range = this.rangeOf(file); if (!range) return null; - const { whole, from, to, reverse } = range; - const peaks = new Float32Array(PEAK_BUCKETS); - for (let c = 0; c < whole.numberOfChannels; c++) { - const data = whole.getChannelData(c); - for (let i = from; i < to; i++) { - const heard = reverse ? to - 1 - i : i - from; - const bucket = Math.floor((heard / (to - from)) * PEAK_BUCKETS); - peaks[bucket] = Math.max(peaks[bucket]!, Math.abs(data[i]!)); - } - } - for (let b = 0; b < PEAK_BUCKETS; b++) peaks[b] = Math.min(1, peaks[b]!); + const peaks = peaksOf(range.whole, range); this.peaked.set(file, peaks); return peaks; } + /** + * Loads what `peaks` and `duration` read for a name. A streamed one (#349) is not loaded whole for + * it: its copy is decoded at `PICTURE_RATE`, its peaks kept and the samples let go. + */ + loadPicture(name: string): Promise { + const stream = this.streamed(name); + if (!stream) return this.load(name).then(() => {}); + if (this.peaked.has(name)) return Promise.resolve(); + let picture = this.pictures.get(name); + if (!picture) { + picture = this.draw(name, stream); + this.pictures.set(name, picture); + } + return picture; + } + + private async draw(name: string, stream: StreamFile): Promise { + try { + const bytes = await stream.whole(); + const picture = await new OfflineAudioContext(1, 1, PICTURE_RATE).decodeAudioData(bytes); + this.peaked.set(name, peaksOf(picture, { from: 0, to: picture.length, reverse: false })); + } catch (error: unknown) { + console.error(`[sounds] cannot draw ${name}`, error); + } + } + /** * Starts loading every file not held yet. Waiting for the first play left that play late: its * start was not heard, and a graph tuning it in the meantime had moved on by the time it started. + * One that streams (#349) has the head of its copy loaded instead. */ preload(files: Iterable): void { - for (const file of files) if (file) void this.load(file); + for (const file of files) { + if (!file) continue; + const stream = this.streamed(file); + if (stream) void stream.ready(); + else void this.load(file); + } } /** @@ -1643,6 +1785,13 @@ export class SoundLibrary implements AudioSink { return work; } + /** A file decoded at `rate` and not kept: what a stream copy is made from (#349). */ + async decodeApart(file: string, rate: number): Promise { + const response = await fetch(`${this.base}/${file}`); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + return new OfflineAudioContext(1, 1, rate).decodeAudioData(await response.arrayBuffer()); + } + private async fetched(file: string): Promise { const response = await fetch(`${this.base}/${file}`); if (!response.ok) throw new Error(`HTTP ${response.status}`); @@ -1806,6 +1955,7 @@ export class SoundLibrary implements AudioSink { at: heard, clip: asked, bus = '', + stream, }: Start): void { const context = this.audio(); // A context starts suspended where the page has not been interacted with; without this the @@ -1840,6 +1990,7 @@ export class SoundLibrary implements AudioSink { rings, offset: clip ? clip.offset + (when - at) : 0, clip: clip !== undefined, + stream: stream ?? null, sources: [], drivers: [], envelopes: new Map(), @@ -1990,7 +2141,15 @@ export class SoundLibrary implements AudioSink { return sound.connect(fade); } - private sound(wiring: Wiring, { pitch, shifted, source: own, pitchMods }: Way): AudioNode { + private sound(wiring: Wiring, way: Way): AudioNode { + const { pitch, shifted, source: own, pitchMods } = way; + if (!own && wiring.stream && this.hasWorklet) + return this.streamSource( + wiring, + wiring.stream, + playedPlan(wiring.stream.frames, wiring.offset, wiring.loop), + way, + ); const { context } = wiring; const file = own?.file ?? wiring.file; const loop = own ? own.loop : wiring.loop; @@ -2063,6 +2222,127 @@ export class SoundLibrary implements AudioSink { return source; } + /** + * A way streamed from its copy (#349): a `stream-player` fed along its plan. Its Pitch is a shifter + * of what it plays, as on a bus (#346): its length stays, and it is heard a frame late. + */ + private streamSource( + wiring: Wiring, + file: StreamFile, + plan: StreamPlan, + { pitch, shifted, pitchMods }: Way, + ): AudioNode { + const { context } = wiring; + wiring.pitches.push(pitch); + const { node, source } = this.streamNode(context, file, plan, wiring.at); + wiring.sources.push(source); + if (!shifted) return node; + const shifter = new AudioWorkletNode(context, 'pitch-shifter', { + numberOfInputs: 1, + numberOfOutputs: 1, + outputChannelCount: [2], + channelCount: 2, + channelCountMode: 'explicit', + parameterData: { pitch }, + }); + node.connect(shifter); + wiring.vocoders += 1; + wiring.drivers.push({ start: () => {}, stop: () => stopProcessor(shifter) }); + const knob = shifter.parameters.get('pitch')!; + wiring.knobs.push({ param: knob, target: pitch, even: 'value' }); + const follow = this.follower(wiring); + for (const mod of pitchMods ?? []) follow(mod, knob, false); + return shifter; + } + + /** A `stream-player` fed along `plan`, heard from `at` on, and the source a voice holds it by. */ + private streamNode( + context: BaseAudioContext, + file: StreamFile, + plan: StreamPlan, + at: number, + ): { node: AudioWorkletNode; source: Source; feed: StreamFeed } { + let then: (() => void) | null = null; + const feed: StreamFeed = new StreamFeed(file, plan, () => { + this.dropFeed(feed); + then?.(); + }); + const options: StreamOptions = { channels: file.channels, startAt: at, first: feed.opening() }; + const node = new AudioWorkletNode(context, 'stream-player', { + numberOfInputs: 0, + outputChannelCount: [file.channels], + processorOptions: options, + }); + feed.attach(node); + this.feeds.add(feed); + const source: Source = { + start: () => {}, + stop: () => { + this.dropFeed(feed); + feed.stop(); + }, + ended: (call) => { + then = call; + }, + fade: null, + gate: false, + leave: null, + }; + return { node, source, feed }; + } + + /** + * Streams a name along `plan` into an offline context at `STREAM_RATE`, heard from `at`, `frames` + * long, with the player and the feed a game has (test hook). The clock is held every half second + * until the feed has caught up: offline it runs faster than anything is decoded. Null where the + * name does not stream. + */ + async renderStream( + name: string, + plan: StreamPlan, + at: number, + frames: number, + ): Promise<{ channels: Float32Array[]; skipped: number } | null> { + const file = this.streamed(name); + if (!file || !(await file.ready())) return null; + const context = new OfflineAudioContext(file.channels, frames, STREAM_RATE); + await context.audioWorklet.addModule(fxWorkletUrl); + const { node, source, feed } = this.streamNode(context, file, plan, at); + node.connect(context.destination); + // Where the plan starts past the head, the first stretch is decoded before the clock runs. + while (feed.busy) await new Promise((resolve) => setTimeout(resolve, 5)); + const step = STREAM_RATE / 2; + const hold = (frame: number) => { + if (frame >= frames) return; + void context.suspend(frame / STREAM_RATE).then(async () => { + do await new Promise((resolve) => setTimeout(resolve, 5)); + while (feed.busy); + hold(frame + step); + void context.resume(); + }); + }; + hold(step); + const rendered = await context.startRendering(); + const skipped = feed.skipped; + source.stop(); + return { + channels: Array.from({ length: rendered.numberOfChannels }, (_, c) => rendered.getChannelData(c)), + skipped, + }; + } + + /** All of the copy a name streams from, decoded as the feed decodes it (test hook). */ + async streamedWhole(name: string): Promise { + const file = this.streamed(name); + if (!file || !(await file.ready())) return null; + return decodeSpan(file, 0, file.frames); + } + + private dropFeed(feed: StreamFeed): void { + if (!this.feeds.delete(feed)) return; + this.skippedBefore += feed.skipped; + } + /** * A Chip (#329): a worklet of its own that ends with the note. A Pitch on its way moves its * frequency itself. Without the worklet the way is silent and ends at once. @@ -2384,6 +2664,24 @@ function loopIn(buffer: AudioBuffer, start: number, end: number): { loopStart: n return from < to ? { loopStart: from, loopEnd: to } : { loopStart: 0, loopEnd: last }; } +/** The loudest sample of each of `PEAK_BUCKETS` equal stretches of what is heard of `whole`, at most 1. */ +function peaksOf( + whole: AudioBuffer, + { from, to, reverse }: Pick, +): Float32Array { + const peaks = new Float32Array(PEAK_BUCKETS); + for (let c = 0; c < whole.numberOfChannels; c++) { + const data = whole.getChannelData(c); + for (let i = from; i < to; i++) { + const heard = reverse ? to - 1 - i : i - from; + const bucket = Math.floor((heard / (to - from)) * PEAK_BUCKETS); + peaks[bucket] = Math.max(peaks[bucket]!, Math.abs(data[i]!)); + } + } + for (let b = 0; b < PEAK_BUCKETS; b++) peaks[b] = Math.min(1, peaks[b]!); + return peaks; +} + /** The loudest sample of each side over the stretch a meter looks back over, 1 at full scale. */ function readMeter(meter: BusMeter): [number, number] { const [left, right] = meter.sides.map((side) => { diff --git a/src/viewport/fxWorklet.ts b/src/viewport/fxWorklet.ts index e3841bc8..659ed22c 100644 --- a/src/viewport/fxWorklet.ts +++ b/src/viewport/fxWorklet.ts @@ -3,7 +3,8 @@ * plays one sound through a `PitchPlayer`, its pitch the AudioParam `pitch` in semitones, so it * glides, ramps and eases like any other knob. A Bitcrusher (#322) runs a `Crusher`, a Chip * (#329) a `ChipPlayer`, a Phaser and a Flanger (#341) theirs, the pitch of a bus (#346) a - * `PitchShifter`, a Limiter (#326) a `Limiter`, a Tremolo and a Vibrato (#320) theirs. Loaded into + * `PitchShifter`, a Limiter (#326) a `Limiter`, a Tremolo and a Vibrato (#320) theirs, a streamed + * sound (#349) a `StreamQueue`. Loaded into * the context once by the sound library; nothing on the page imports it. */ import { type ChipShape, ChipPlayer } from '../core/soundfx/chip'; @@ -12,6 +13,7 @@ import { Limiter, type LimiterNumbers } from '../core/soundfx/dynamics'; import { LFO_SHAPES, type LfoNumbers, Tremolo, Vibrato } from '../core/soundfx/lfo'; import { Flanger, type FlangerNumbers, Phaser, type PhaserNumbers } from '../core/soundfx/phaser'; import { PitchPlayer, PitchShifter } from '../core/soundfx/pitchPlayer'; +import { STREAM_RATE, StreamQueue } from '../core/music/streamQueue'; /** What the audio thread has and the page's types do not. */ declare const sampleRate: number; @@ -167,6 +169,49 @@ class ShiftProcessor extends Effect { registerProcessor('pitch-shifter', ShiftProcessor); +export interface StreamOptions { + channels: number; + /** The second of the audio clock its first sample is heard at; before it, silence. */ + startAt: number; + /** What it starts with, so it is there for the first sample; the rest comes by message. */ + first: Float32Array[]; +} + +/** Samples of the stream read between two reports to the page. */ +const STREAM_REPORT = 4800; + +/** + * A streamed sound (#349): plays the samples the page sends it, in order, and tells it how far it has + * read so it can send more. 'end' says nothing more comes. + */ +class StreamProcessor extends Source { + private readonly queue: StreamQueue; + private reported = 0; + + constructor({ processorOptions }: { processorOptions: StreamOptions }) { + super(processorOptions.startAt); + this.queue = new StreamQueue(processorOptions.channels, STREAM_RATE / sampleRate); + this.queue.push(processorOptions.first); + this.port.onmessage = (event: MessageEvent) => { + const data: unknown = event.data; + if (Array.isArray(data)) this.queue.push(data as Float32Array[]); + else if (data === 'end') this.queue.end(); + else this.stop(); + }; + } + + protected render(out: Float32Array[]): boolean { + const going = this.queue.render(out); + if (this.queue.used - this.reported >= STREAM_REPORT) { + this.reported = this.queue.used; + this.port.postMessage({ used: this.queue.used, skipped: this.queue.skipped }); + } + return going; + } +} + +registerProcessor('stream-player', StreamProcessor); + /** A Bitcrusher (#322). */ class CrushProcessor extends Effect { private readonly crusher = new Crusher(sampleRate); diff --git a/src/viewport/projectFiles.ts b/src/viewport/projectFiles.ts index 12ce59a9..59983582 100644 --- a/src/viewport/projectFiles.ts +++ b/src/viewport/projectFiles.ts @@ -1,6 +1,7 @@ import type { Renderer } from 'pixi.js'; import type { Project } from '../core/model/Project'; import { projectFiles } from '../core/model/projectFiles'; +import { streamTable } from '../core/music/streams'; import { requestRender } from '../ui'; import { fontLibrary } from './FontLibrary'; import { installPixelArtPictures } from './pixelArtPicture'; @@ -46,5 +47,7 @@ export function preloadProjectFiles(project: Project): void { textureLibrary.preload(files.textures); fontLibrary.preload(files.fonts); pixelCels.preload(files.cels); + // Which of them stream (#349) first, so their copies are fetched rather than they decoded. + soundLibrary.setStreams(streamTable(project)); soundLibrary.preload(files.sounds); } diff --git a/src/viewport/streaming.ts b/src/viewport/streaming.ts new file mode 100644 index 00000000..20376a0c --- /dev/null +++ b/src/viewport/streaming.ts @@ -0,0 +1,376 @@ +/** + * Streaming a sound (#349) on the page's side: the copy is fetched a piece at a time, read page by + * page, and decoded by WebCodecs a few seconds ahead of what is heard; the samples go to a + * `stream-player` on the worklet and are let go there once played. Only the compressed bytes and the + * first seconds of each copy are kept, so a long sound weighs a few megabytes rather than a hundred. + */ +import { OggOpusReader, type PlacedPacket } from '../core/music/ogg'; +import { STREAM_RATE } from '../core/music/streamQueue'; +import { StreamCursor, type StreamPlan } from '../core/music/streams'; + +/** Bytes fetched at a time, some 13 s at 160 kbit/s. */ +const CHUNK_BYTES = 256 * 1024; +/** Samples of each copy decoded once it is named, so it starts at once (`StreamFile.ready`). */ +const HEAD = 2 * STREAM_RATE; +/** + * Samples decoded before a place jumped to and thrown away: measured in Chrome, 32 packets of 20 ms + * make what follows the same to the bit as decoding from the start, 4 come out 0.06 off. + */ +const PREROLL = 32 * 960; +/** Samples kept sent ahead of what the worklet has read, and the most decoded at once. */ +const LEAD = 4 * STREAM_RATE; +const BATCH = STREAM_RATE; + +/** One copy, shared by every voice streaming it: its bytes as they come, and its first seconds decoded. */ +export class StreamFile { + readonly reader = new OggOpusReader(); + /** Samples per channel without the pre-skip, as the asset says. */ + head: Float32Array[] | null = null; + failed = false; + private fetched = 0; + private complete = false; + private fetching: Promise | null = null; + private heading: Promise | null = null; + + constructor( + readonly url: string, + readonly frames: number, + ) {} + + get channels(): number { + return this.reader.channels ?? 2; + } + + get preSkip(): number { + return this.reader.preSkip ?? 0; + } + + /** Bytes fetched so far (test hook). */ + get bytes(): number { + return this.fetched; + } + + /** Its head fetched and decoded; false where it cannot be had. */ + ready(): Promise { + return (this.heading ??= (async () => { + try { + this.head = await decodeSpan(this, 0, Math.min(HEAD, this.frames)); + return true; + } catch (error: unknown) { + this.failed = true; + console.error(`[sounds] cannot stream ${this.url}`, error); + return false; + } + })()); + } + + /** Fetches until the packet holding stream sample `sample` is there, or the copy has ended. */ + async reach(sample: number): Promise { + while (!this.complete && this.lastEnd() <= sample) await this.more(); + } + + /** Every byte of the copy, fetched if it is not yet: what the picture of it is decoded from. */ + async whole(): Promise { + const response = await fetch(this.url); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + return response.arrayBuffer(); + } + + private lastEnd(): number { + const last = this.reader.packets.at(-1); + return last ? last.at + last.samples : 0; + } + + private more(): Promise { + return (this.fetching ??= this.fetchMore().finally(() => { + this.fetching = null; + })); + } + + /** The next piece by `Range`; a server that sends the whole file instead is taken at its word. */ + private async fetchMore(): Promise { + const response = await fetch(this.url, { + headers: { Range: `bytes=${this.fetched}-${this.fetched + CHUNK_BYTES - 1}` }, + }); + if (response.status === 416) { + this.complete = true; + return; + } + if (!response.ok) throw new Error(`HTTP ${response.status}`); + const bytes = new Uint8Array(await response.arrayBuffer()); + if (response.status === 200 && this.fetched > 0) throw new Error('the server sent all of it again'); + this.reader.push(bytes); + this.fetched += bytes.length; + const total = Number(response.headers.get('content-range')?.split('/')[1]); + if (response.status === 200 || bytes.length === 0 || (Number.isFinite(total) && this.fetched >= total)) + this.complete = true; + } +} + +/** + * Decodes stream samples on one decoder, reading on from where it stopped while it is asked for what + * follows, and settling first wherever it is asked to jump. + */ +class StreamDecoder { + private decoder: AudioDecoder | null = null; + /** The next packet to hand it, and the stream sample the next sample it gives back is at. */ + private next = 0; + private outAt = 0; + /** What it has given back and not yet been asked for, from `outAt` back. */ + private spare: Float32Array[] = []; + private outputs: Float32Array[][] = []; + private fed = 0; + private given = 0; + private wake: (() => void) | null = null; + private failure: Error | null = null; + + constructor(private readonly file: StreamFile) {} + + /** The samples of the copy from `from` to `to`, counted without its pre-skip. */ + async read(from: number, to: number): Promise { + const { file } = this; + const start = from + file.preSkip; + const end = to + file.preSkip; + const held = () => this.outAt - (this.spare[0]?.length ?? 0); + if (!this.decoder || start !== held()) await this.jump(start); + const spareFrom = held(); + const parts: Float32Array[][] = [this.spare]; + let have = this.outAt; + this.spare = []; + while (have < end) { + await file.reach(have); + const packet = file.reader.packets[this.next]; + if (!packet) break; + this.feed(packet); + const got = await this.take(); + parts.push(got); + have += got[0]?.length ?? 0; + } + this.outAt = have; + const joined = join(parts, file.channels, have - spareFrom); + const offset = start - spareFrom; + const kept = joined.map((side) => side.slice(offset, offset + (end - start))); + this.spare = joined.map((side) => side.slice(offset + (end - start))); + return kept; + } + + close(): void { + if (this.decoder && this.decoder.state !== 'closed') this.decoder.close(); + this.decoder = null; + } + + /** A new decoder, from far enough before `sample` that it has settled there; what came before goes. */ + private async jump(sample: number): Promise { + this.close(); + const { file } = this; + await file.reach(sample); + const packets = file.reader.packets; + let index = 0; + while (index + 1 < packets.length && packets[index + 1]!.at <= sample - PREROLL) index += 1; + this.next = index; + this.outAt = packets[index]?.at ?? 0; + this.spare = []; + this.outputs = []; + this.fed = this.given = 0; + this.failure = null; + const decoder = new AudioDecoder({ + output: (data) => { + const sides = Array.from({ length: file.channels }, (_, c) => { + const side = new Float32Array(data.numberOfFrames); + data.copyTo(side, { planeIndex: Math.min(c, data.numberOfChannels - 1), format: 'f32-planar' }); + return side; + }); + data.close(); + this.outputs.push(sides); + this.given += 1; + this.wake?.(); + }, + error: (error) => { + this.failure = error; + this.wake?.(); + }, + }); + // No description: Chrome would apply the pre-skip by itself, and only to the first packet it gets. + decoder.configure({ codec: 'opus', sampleRate: STREAM_RATE, numberOfChannels: file.channels }); + this.decoder = decoder; + // Every packet that ends before where it is asked for settles it and is thrown away; `read` cuts + // the rest off the one it starts in. + while (this.next < packets.length && packets[this.next]!.at + packets[this.next]!.samples <= sample) { + this.feed(packets[this.next]!); + await this.take(); + } + this.outAt = packets[this.next]?.at ?? this.outAt; + } + + private feed(packet: PlacedPacket): void { + const decoder = this.decoder!; + decoder.decode( + new EncodedAudioChunk({ + type: 'key', + timestamp: Math.round((packet.at * 1e6) / STREAM_RATE), + data: packet.data, + }), + ); + this.next += 1; + this.fed += 1; + } + + /** The samples of the packet handed over last, once the decoder has given them back. */ + private async take(): Promise { + while (this.given < this.fed && !this.failure) + await new Promise((resolve) => { + this.wake = resolve; + }); + this.wake = null; + if (this.failure) throw this.failure; + const out = this.outputs.splice(0); + return join( + out, + this.file.channels, + out.reduce((sum, part) => sum + (part[0]?.length ?? 0), 0), + ); + } +} + +/** The samples of a copy from `from` to `to` on a decoder of their own: what a head is. */ +export async function decodeSpan(file: StreamFile, from: number, to: number): Promise { + await file.reach(0); + if (file.reader.channels === null) throw new Error('no Opus in it'); + const decoder = new StreamDecoder(file); + try { + return await decoder.read(from, to); + } finally { + decoder.close(); + } +} + +function join(parts: readonly Float32Array[][], channels: number, length: number): Float32Array[] { + return Array.from({ length: channels }, (_, c) => { + const side = new Float32Array(Math.max(0, length)); + let at = 0; + for (const part of parts) { + const data = part[c] ?? part[0]; + if (!data) continue; + side.set(data.subarray(0, Math.max(0, length - at)), at); + at += data.length; + } + return side; + }); +} + +/** + * Keeps one `stream-player` fed along its plan: the head where the plan is in it, decoded samples + * after that, never more than `LEAD` ahead of what it has read. Tells `ended` once it has played out. + */ +export class StreamFeed { + private readonly cursor: StreamCursor; + private readonly decoder: StreamDecoder; + private node: AudioWorkletNode | null = null; + private sent = 0; + private used = 0; + private pumping = false; + private stopped = false; + private finished = false; + /** Samples that came too late to be heard, as the worklet last said (test hook). */ + skipped = 0; + + constructor( + private readonly file: StreamFile, + plan: StreamPlan, + private readonly ended: () => void, + ) { + this.cursor = new StreamCursor(plan); + this.decoder = new StreamDecoder(file); + } + + /** + * What the player starts with where its plan starts in the head: handed over as it is made, so it + * is there for the first sample heard. Sent by message, it came a few milliseconds late, and a + * sound started at once lost its attack. Empty where the plan starts past the head. + */ + opening(): Float32Array[] { + const at = this.cursor.peek(); + const head = this.file.head; + if (at === null || !head || at >= (head[0]?.length ?? 0)) return []; + const span = this.cursor.next(head[0]!.length - at)!; + this.sent += span.to - span.from; + return head.map((side) => side.slice(span.from, span.to)); + } + + /** Feeds that player from now on. */ + attach(node: AudioWorkletNode): void { + this.node = node; + node.port.onmessage = (event: MessageEvent) => { + const data: unknown = event.data; + if (data === 'ended') { + this.close(); + this.ended(); + } else if (typeof data === 'object' && data !== null && 'used' in data) { + const report = data as { used: number; skipped: number }; + this.used = report.used; + this.skipped = report.skipped; + void this.pump(); + } + }; + void this.pump(); + } + + stop(): void { + if (this.stopped) return; + this.close(); + if (!this.node) return; + this.node.port.postMessage('stop'); + this.node.port.close(); + this.node.disconnect(); + } + + /** Whether it is decoding or sending right now: what an offline clock waits for (test hook). */ + get busy(): boolean { + return this.pumping; + } + + /** Read anew each time: a stop comes in while a stretch is being decoded. */ + private halted(): boolean { + return this.stopped; + } + + private close(): void { + this.stopped = true; + if (this.node) this.node.port.onmessage = null; + this.decoder.close(); + } + + private async pump(): Promise { + const node = this.node; + if (!node || this.pumping || this.stopped || this.finished) return; + this.pumping = true; + try { + while (!this.halted() && this.sent - this.used < LEAD) { + const at = this.cursor.peek(); + if (at === null) { + this.finished = true; + node.port.postMessage('end'); + break; + } + const head = this.file.head; + const inHead = head !== null && at < (head[0]?.length ?? 0); + const span = this.cursor.next(inHead ? head[0]!.length - at : BATCH)!; + const samples = inHead + ? head.map((side) => side.slice(span.from, span.to)) + : await this.decoder.read(span.from, span.to); + if (this.halted()) break; + node.port.postMessage( + samples, + samples.map((side) => side.buffer), + ); + this.sent += span.to - span.from; + } + } catch (error: unknown) { + console.error(`[sounds] streaming ${this.file.url} broke off`, error); + if (!this.halted()) node.port.postMessage('end'); + this.finished = true; + } finally { + this.pumping = false; + } + } +} diff --git a/tests/core/ogg.test.ts b/tests/core/ogg.test.ts index 66910c81..d1ea7e25 100644 --- a/tests/core/ogg.test.ts +++ b/tests/core/ogg.test.ts @@ -3,7 +3,15 @@ * page says it has got to, and the pre-skip and length a decoder trims by. */ import { describe, expect, it } from 'vitest'; -import { oggCrc, oggOpus, opusHead, preSkipOf, readOggOpus } from '../../src/core/music/ogg'; +import { + OggOpusReader, + oggCrc, + oggOpus, + opusHead, + opusPacketSamples, + preSkipOf, + readOggOpus, +} from '../../src/core/music/ogg'; const packet = (size: number, fill: number) => ({ data: new Uint8Array(size).fill(fill), samples: 960 }); @@ -65,3 +73,49 @@ describe('Opus in Ogg', () => { expect(preSkipOf(new Uint8Array(19))).toBeNull(); }); }); + +describe('Opus in Ogg as it comes in (#349)', () => { + /** A CELT packet of 20 ms at full band: config 31, one frame. */ + const celt = (size: number, fill: number) => { + const data = new Uint8Array(size).fill(fill); + data[0] = 31 << 3; + return { data, samples: 960 }; + }; + const file = oggOpus({ + channels: 2, + preSkip: 312, + frames: 300 * 960 - 312, + packets: Array.from({ length: 300 }, (_, i) => celt(100 + (i % 300), i % 256)), + }); + + it('reads a file pushed in pieces as it reads it whole, each packet placed where it starts', () => { + const reader = new OggOpusReader(); + for (let at = 0, size = 1; at < file.length; at += size, size = (size * 7 + 3) % 4000) { + reader.push(file.subarray(at, at + size)); + } + const whole = readOggOpus(file); + expect(reader.packets.map((p) => p.data)).toEqual(whole.packets); + expect(reader.packets.slice(0, 3).map((p) => p.at)).toEqual([0, 960, 1920]); + expect(reader.packets.at(-1)!.at).toBe(299 * 960); + expect([reader.channels, reader.preSkip, reader.ended]).toEqual([2, 312, true]); + }); + + it('holds a page back until all of it is there', () => { + const reader = new OggOpusReader(); + reader.push(file.subarray(0, 4000)); + const seen = reader.packets.length; + expect(reader.ended).toBe(false); + reader.push(file.subarray(4000)); + expect(reader.packets.length).toBeGreaterThan(seen); + expect(reader.packets).toHaveLength(300); + }); + + it('counts the samples of a packet from its TOC byte', () => { + expect(opusPacketSamples(Uint8Array.of(31 << 3))).toBe(960); + expect(opusPacketSamples(Uint8Array.of((28 << 3) | 1))).toBe(240); + expect(opusPacketSamples(Uint8Array.of((3 << 3) | 2))).toBe(2 * 2880); + expect(opusPacketSamples(Uint8Array.of((13 << 3) | 3, 3))).toBe(3 * 960); + expect(opusPacketSamples(Uint8Array.of(12 << 3))).toBe(480); + expect(opusPacketSamples(new Uint8Array(0))).toBe(0); + }); +}); diff --git a/tests/core/sounds.test.ts b/tests/core/sounds.test.ts index dae05696..832313a1 100644 --- a/tests/core/sounds.test.ts +++ b/tests/core/sounds.test.ts @@ -37,6 +37,8 @@ import { loopsOf, REVERSE_KEY, START_KEY, + STREAM_KEY, + streamsOf, VOLUME_KEY, volumeOf, } from '../../src/core/properties/sounds'; @@ -129,8 +131,10 @@ describe('the sound asset', () => { expect(asset).toMatchObject({ type: 'sound', file: '' }); expect(volumeOf(asset.props)).toBe(1); expect(loopsOf(asset.props)).toBe(false); + // Decoded whole unless it is switched to stream (#349). + expect(streamsOf(asset.props)).toBe(false); // The cut belongs to a clip of the sound (#357), the sound is the whole file. - expect(schemaFor('sound').map((def) => def.key)).toEqual([VOLUME_KEY, LOOP_KEY, 'fx']); + expect(schemaFor('sound').map((def) => def.key)).toEqual([VOLUME_KEY, LOOP_KEY, STREAM_KEY, 'fx']); expect(CLIP_ROWS.map((def) => def.key)).toEqual([ START_KEY, END_KEY, diff --git a/tests/core/stream-queue.test.ts b/tests/core/stream-queue.test.ts new file mode 100644 index 00000000..a584e6ce --- /dev/null +++ b/tests/core/stream-queue.test.ts @@ -0,0 +1,83 @@ +/** + * What a streamed sound (#349) is played from on the audio thread: the samples in order, read at the + * rate of the context, silent where they come late and skipped once they are there. + */ +import { describe, expect, it } from 'vitest'; +import { STREAM_RATE, StreamQueue } from '../../src/core/music/streamQueue'; + +const ramp = (from: number, count: number) => Float32Array.from({ length: count }, (_, i) => from + i); +const block = (size = 128) => [new Float32Array(size), new Float32Array(size)]; + +describe('the queue of a streamed sound', () => { + it('plays what it is sent sample for sample at the rate of the stream, across its chunks', () => { + const queue = new StreamQueue(2, 1); + queue.push([ramp(0, 100), ramp(1000, 100)]); + queue.push([ramp(100, 100), ramp(1100, 100)]); + queue.end(); + const out = block(); + expect(queue.render(out)).toBe(true); + expect(Array.from(out[0]!.subarray(98, 102))).toEqual([98, 99, 100, 101]); + expect(Array.from(out[1]!.subarray(0, 2))).toEqual([1000, 1001]); + expect(queue.render(out)).toBe(false); + expect(Array.from(out[0]!.subarray(70, 74))).toEqual([198, 199, 0, 0]); + expect(queue.used).toBe(200); + }); + + it('plays a mono stream on both sides', () => { + const queue = new StreamQueue(1, 1); + queue.push([ramp(5, 300)]); + const out = block(); + queue.render(out); + expect(out[1]![3]).toBe(8); + }); + + it('reads it at the rate of a context running at 44.1 kHz along a smooth curve', () => { + const step = STREAM_RATE / 44100; + const queue = new StreamQueue(1, step); + const sine = (i: number) => Math.sin((2 * Math.PI * 440 * i) / STREAM_RATE); + for (let at = 0; at < 2 * STREAM_RATE; at += 4800) + queue.push([Float32Array.from({ length: 4800 }, (_, i) => sine(at + i))]); + const out = [new Float32Array(44100)]; + queue.render(out); + let worst = 0; + for (let i = 4; i < 44000; i++) worst = Math.max(worst, Math.abs(out[0]![i]! - sine(i * step))); + expect(worst).toBeLessThan(1e-4); + expect(queue.used).toBe(Math.floor(44100 * step)); + }); + + it('is silent while what is due has not come, and skips as much once it has, so it keeps time', () => { + const queue = new StreamQueue(1, 1); + queue.push([ramp(0, 64)]); + const out = [new Float32Array(128)]; + queue.render(out); + // The last two it had are held for the curve and come late with the rest. + expect(out[0]![61]).toBe(61); + expect(out[0]![100]).toBe(0); + queue.push([ramp(64, 1000)]); + expect(queue.used).toBe(128); + expect(queue.skipped).toBe(66); + queue.render(out); + expect(out[0]![0]).toBe(128); + }); + + it('says it has played out once it has ended and all of it has been read', () => { + const queue = new StreamQueue(2, 1); + queue.push([ramp(0, 300), ramp(0, 300)]); + queue.end(); + expect(queue.render(block())).toBe(true); + expect(queue.render(block())).toBe(true); + const out = block(); + expect(queue.render(out)).toBe(false); + expect(Array.from(out[0]!.subarray(42, 46))).toEqual([298, 299, 0, 0]); + expect(queue.done).toBe(true); + }); + + it('skips what came too late even when nothing more comes after it', () => { + const queue = new StreamQueue(1, 1); + queue.push([ramp(0, 10)]); + queue.render(block()); + queue.end(); + expect(queue.done).toBe(true); + expect(queue.render(block())).toBe(false); + }); +}); diff --git a/tests/core/streams.test.ts b/tests/core/streams.test.ts new file mode 100644 index 00000000..60c01502 --- /dev/null +++ b/tests/core/streams.test.ts @@ -0,0 +1,180 @@ +/** + * Sounds streamed rather than decoded whole (#349): which names a game streams from which copy, what + * copies are missing or left over, how a played sound is walked through its copy, and that a copy is + * kept with the project. + */ +import { describe, expect, it } from 'vitest'; +import { + CreateAssetCommand, + CreateSoundCommand, + SetAssetPropertyCommand, +} from '../../src/core/commands/assetCommands'; +import { AddClipCommand, SetClipPropsCommand } from '../../src/core/commands/clipCommands'; +import type { SongAsset, SoundAsset, SoundStream } from '../../src/core/model/Asset'; +import { projectFiles } from '../../src/core/model/projectFiles'; +import { cutName } from '../../src/core/model/sound'; +import { STREAM_RATE } from '../../src/core/music/streamQueue'; +import { + heldStream, + isStreamUpload, + playedPlan, + StreamCursor, + streamTable, + streamUploadName, + streamWork, +} from '../../src/core/music/streams'; +import { cutOf } from '../../src/core/properties/sounds'; +import { deserializeProject, serializeProject } from '../../src/core/store/serialize'; +import { makeSpriteScene } from '../helpers'; + +const FILE = `${'a1'.repeat(6)}/${'b2'.repeat(32)}.wav`; +const COPY = `${'a1'.repeat(6)}/${'c3'.repeat(32)}.ogg`; +const CLIP_COPY = `${'a1'.repeat(6)}/${'d4'.repeat(32)}.ogg`; + +function streamed() { + const f = makeSpriteScene('Box'); + const made = new CreateSoundCommand('Music', FILE); + f.stack.execute(made); + const sound = made.asset; + const clip = new AddClipCommand(sound.id, { start: 1, end: 3 }); + f.stack.execute(clip); + const whole = new AddClipCommand(sound.id, { start: 0, end: 0 }); + f.stack.execute(whole); + const turn = (on: boolean) => f.stack.execute(new SetAssetPropertyCommand(sound.id, 'stream', on)); + const copy = (from: string, file: string): SoundStream => ({ file, from, frames: 96000, bytes: 4000 }); + const cut = () => cutName(FILE, cutOf(sound.clips![0]!.props)); + return { ...f, sound, clipId: clip.clipId, wholeId: whole.clipId, turn, copy, cut }; +} + +describe('which sounds stream', () => { + it('streams nothing while the switch is off, whatever copies there are', () => { + const f = streamed(); + f.sound.stream = f.copy(FILE, COPY); + expect(streamTable(f.project).size).toBe(0); + }); + + it('streams a sound from a copy of its file, a clip from a copy of its cut, a whole-file clip as the sound', () => { + const f = streamed(); + f.turn(true); + f.sound.stream = f.copy(FILE, COPY); + f.sound.clips![0]!.stream = f.copy(f.cut(), CLIP_COPY); + expect([...streamTable(f.project)]).toEqual([ + [FILE, { file: COPY, frames: 96000 }], + [f.cut(), { file: CLIP_COPY, frames: 96000 }], + ]); + }); + + it('plays a sound or a clip decoded while its copy is of something else', () => { + const f = streamed(); + f.turn(true); + f.sound.stream = f.copy(FILE, COPY); + f.sound.clips![0]!.stream = f.copy(f.cut(), CLIP_COPY); + f.stack.execute(new SetClipPropsCommand(f.sound.id, f.clipId, { end: 2.5 })); + expect([...streamTable(f.project).keys()]).toEqual([FILE]); + }); + + it('asks for a copy of the sound and of each clip with a cut of its own, and lets them go once it is off', () => { + const f = streamed(); + expect(streamWork(f.project)).toEqual({ jobs: [], drops: [] }); + f.turn(true); + expect(streamWork(f.project).jobs).toEqual([ + { assetId: f.sound.id, clipId: null, from: FILE }, + { assetId: f.sound.id, clipId: f.clipId, from: f.cut() }, + ]); + f.sound.stream = f.copy(FILE, COPY); + f.sound.clips![0]!.stream = f.copy(f.cut(), CLIP_COPY); + expect(streamWork(f.project)).toEqual({ jobs: [], drops: [] }); + f.stack.execute(new SetClipPropsCommand(f.sound.id, f.clipId, { reverse: true })); + expect(streamWork(f.project).jobs).toEqual([{ assetId: f.sound.id, clipId: f.clipId, from: f.cut() }]); + f.turn(false); + expect(streamWork(f.project)).toEqual({ + jobs: [], + drops: [ + { assetId: f.sound.id, clipId: null }, + { assetId: f.sound.id, clipId: f.clipId }, + ], + }); + }); + + it('leaves a sound that streams out of what is decoded ahead, its copy made or not', () => { + const f = streamed(); + expect(projectFiles(f.project).sounds).toEqual([FILE]); + f.turn(true); + expect(projectFiles(f.project).sounds).toEqual([]); + }); + + it('streams every bake from itself', () => { + const f = streamed(); + const made = new CreateAssetCommand('song', 'Theme'); + f.stack.execute(made); + (made.asset as SongAsset).bake = { + file: COPY, + print: 'x', + peak: 1, + length: 100, + loopEnd: 80, + head: 100, + round: 40, + bytes: 9, + }; + expect(streamTable(f.project).get(COPY)).toEqual({ file: COPY, frames: 140 }); + }); +}); + +describe('walking a copy', () => { + it('plays a sound from where it is asked to, round its whole length where it loops', () => { + expect(playedPlan(1000, 0, false)).toEqual({ from: 0, until: 1000, loop: null }); + expect(playedPlan(STREAM_RATE * 2, 2.5, true)).toEqual({ + from: STREAM_RATE / 2, + until: STREAM_RATE * 2, + loop: { start: 0, end: STREAM_RATE * 2 }, + }); + expect(playedPlan(1000, 1, false).from).toBe(1000); + }); + + it('reads straight on to the end of a run, then round the loop, starting a new stretch there', () => { + const cursor = new StreamCursor({ from: 50, until: 120, loop: { start: 100, end: 130 } }); + const read: number[][] = []; + for (let i = 0; i < 5; i++) { + const span = cursor.next(40)!; + read.push([span.from, span.to]); + } + expect(read).toEqual([ + [50, 90], + [90, 120], + [100, 130], + [100, 130], + [100, 130], + ]); + expect(cursor.peek()).toBe(100); + }); + + it('ends a run without a loop', () => { + const cursor = new StreamCursor({ from: 0, until: 10, loop: null }); + expect(cursor.next(100)).toEqual({ from: 0, to: 10 }); + expect(cursor.peek()).toBeNull(); + expect(cursor.next(100)).toBeNull(); + }); +}); + +describe('a copy kept with the project', () => { + it('comes back from a save, and one that is not right is dropped', () => { + const f = streamed(); + f.sound.stream = f.copy(FILE, COPY); + f.sound.clips![0]!.stream = f.copy(f.cut(), CLIP_COPY); + const back = deserializeProject(serializeProject(f.project)); + const sound = back.assets.find((a) => a.id === f.sound.id) as SoundAsset; + expect(sound.stream).toEqual(f.copy(FILE, COPY)); + expect(sound.clips![0]!.stream).toEqual(f.copy(f.cut(), CLIP_COPY)); + expect(sound.clips![1]!.stream).toBeUndefined(); + expect(heldStream({ ...f.copy(FILE, COPY), frames: 1.5 })).toBeNull(); + expect(heldStream({ ...f.copy('', COPY) })).toBeNull(); + expect(heldStream({ ...f.copy(FILE, 'notes.txt') })).toBeNull(); + }); + + it('is uploaded under a mark the sound picker leaves out by', () => { + expect(streamUploadName('Music / Intro')).toBe('[stream] Music - Intro.ogg'); + expect(isStreamUpload(streamUploadName('Music'))).toBe(true); + expect(isStreamUpload('Music.ogg')).toBe(false); + }); +}); diff --git a/tests/viewport/fakeStream.ts b/tests/viewport/fakeStream.ts new file mode 100644 index 00000000..e356e9b9 --- /dev/null +++ b/tests/viewport/fakeStream.ts @@ -0,0 +1,105 @@ +/** + * What streaming (#349) needs of a browser, stubbed: a copy as Opus in Ogg served with or without + * `Range`, and an `AudioDecoder` whose every sample is where it lies in the stream – so a check sees + * to the sample whether what is played is what it should be. + */ +import { vi } from 'vitest'; +import { oggOpus } from '../../src/core/music/ogg'; + +export const PRE_SKIP = 312; +/** Bytes of each packet: big enough that a copy of a few seconds comes in several pieces. */ +const PACKET_BYTES = 4000; + +/** A copy of `seconds` seconds: CELT packets of 20 ms. */ +export function fakeCopy(seconds: number, channels = 2): { bytes: Uint8Array; frames: number } { + const count = Math.ceil((seconds * 48000 + PRE_SKIP) / 960); + const packets = Array.from({ length: count }, () => { + const data = new Uint8Array(PACKET_BYTES); + data[0] = 31 << 3; + return { data, samples: 960 }; + }); + const frames = Math.round(seconds * 48000); + return { bytes: oggOpus({ channels, preSkip: PRE_SKIP, frames, packets }), frames }; +} + +export interface FakeServer { + /** Every request, as the range it asked for ('' for the whole file). */ + asked: string[]; +} + +/** Serves these files by URL ending; `ranges` false answers every request with the whole file. */ +export function serveCopies(files: Record, ranges = true): FakeServer { + const server: FakeServer = { asked: [] }; + vi.stubGlobal('fetch', (url: string, init?: { headers?: Record }) => { + const name = Object.keys(files).find((file) => url.endsWith(file)); + if (!name) return Promise.resolve({ ok: false, status: 404 }); + const bytes = files[name]!; + const range = init?.headers?.Range ?? ''; + server.asked.push(range); + const [from, to] = range ? range.replace('bytes=', '').split('-').map(Number) : [0, bytes.length - 1]; + if (!ranges || !range) return Promise.resolve(answer(200, bytes, null)); + if (from! >= bytes.length) return Promise.resolve(answer(416, new Uint8Array(0), null)); + const part = bytes.slice(from, Math.min(bytes.length, to! + 1)); + return Promise.resolve(answer(206, part, `bytes ${from}-${from! + part.length - 1}/${bytes.length}`)); + }); + return server; +} + +function answer(status: number, bytes: Uint8Array, contentRange: string | null) { + return { + ok: status >= 200 && status < 300, + status, + headers: { get: (name: string) => (name.toLowerCase() === 'content-range' ? contentRange : null) }, + arrayBuffer: () => Promise.resolve(bytes.slice().buffer), + }; +} + +/** Every decoder made, with the timestamps of what it was given and whether it was closed. */ +export const decoders: { fed: number[]; closed: boolean; channels: number }[] = []; + +/** An `AudioDecoder` that gives back, for each packet, 960 samples counting up from where it starts. */ +export function stubDecoder(): void { + decoders.length = 0; + vi.stubGlobal( + 'EncodedAudioChunk', + class { + readonly timestamp: number; + constructor(init: { timestamp: number }) { + this.timestamp = init.timestamp; + } + }, + ); + vi.stubGlobal( + 'AudioDecoder', + class { + state = 'unconfigured'; + private readonly made = { fed: [] as number[], closed: false, channels: 0 }; + constructor(private readonly init: { output: (data: unknown) => void }) { + decoders.push(this.made); + } + configure(config: { numberOfChannels: number }) { + this.state = 'configured'; + this.made.channels = config.numberOfChannels; + } + decode(chunk: { timestamp: number }) { + const at = Math.round((chunk.timestamp * 48000) / 1e6); + this.made.fed.push(at); + queueMicrotask(() => { + if (this.made.closed) return; + this.init.output({ + numberOfFrames: 960, + numberOfChannels: this.made.channels, + copyTo: (into: Float32Array, { planeIndex }: { planeIndex: number }) => { + for (let i = 0; i < into.length; i++) into[i] = at + i + planeIndex * 1e6; + }, + close: () => {}, + }); + }); + } + close() { + this.state = 'closed'; + this.made.closed = true; + } + }, + ); +} diff --git a/tests/viewport/sound-library.test.ts b/tests/viewport/sound-library.test.ts index 7b63a13b..2b367449 100644 --- a/tests/viewport/sound-library.test.ts +++ b/tests/viewport/sound-library.test.ts @@ -18,10 +18,11 @@ import { import type { MixBus } from '../../src/core/music/mixer'; import type { BakePlay, SequencePlay } from '../../src/core/music/playbacks'; import type { ScheduledNote } from '../../src/core/music/Scheduler'; -import type { LfoOptions, PitchOptions } from '../../src/viewport/fxWorklet'; +import type { LfoOptions, PitchOptions, StreamOptions } from '../../src/viewport/fxWorklet'; import { saturationCurve } from '../../src/core/soundfx/saturation'; import { stageNodes } from '../../src/viewport/fxNodes'; import { SoundLibrary } from '../../src/viewport/SoundLibrary'; +import { fakeCopy, PRE_SKIP, serveCopies, stubDecoder } from './fakeStream'; /** What a test wants to hear: a chain whose pitch, where it is not 0, is a Pitch typed in. */ type Chain = @@ -368,7 +369,10 @@ beforeEach(() => { constructor(_context: unknown, name: string, options: FakeWorklet['options']) { if (!moduleLoaded) throw new DOMException('no such processor', 'InvalidStateError'); const parameters = new Map( - Object.entries(options.parameterData).map(([key, value]) => [key, param(value)] as const), + // A stream player (#349) is made without any. + Object.entries( + (options.parameterData as FakeWorklet['options']['parameterData'] | undefined) ?? {}, + ).map(([key, value]) => [key, param(value)] as const), ); const kind = name === 'bit-crusher' @@ -379,7 +383,9 @@ beforeEach(() => { ? name : name === 'pitch-shifter' ? 'shifter' - : 'pitch'; + : name === 'stream-player' + ? 'stream' + : 'pitch'; const node: FakeWorklet = wired(kind, { options, pitch: parameters.get('pitch') ?? param(0), @@ -1983,6 +1989,112 @@ describe('a bake (#340)', () => { }); }); +describe('a sound streamed (#349)', () => { + const COPY = 'copy.ogg'; + const BAKED = 'baked.ogg'; + /** Long enough that the waits below see the feed settle. */ + const drain = async () => { + for (let i = 0; i < 200; i++) await settle(); + }; + const streaming = async () => { + stubDecoder(); + const copy = fakeCopy(10); + serveCopies({ [COPY]: copy.bytes, [BAKED]: copy.bytes }); + const library = new SoundLibrary(false); + library.setStreams( + new Map([ + [FILE, { file: COPY, frames: copy.frames }], + [BAKED, { file: BAKED, frames: copy.frames }], + ]), + ); + await drain(); + return library; + }; + const player = () => worklets.find((w) => w.kind === 'stream')!; + /** What a player was handed as it was made and sent after, channel 0. */ + const heard = (node: FakeWorklet) => + [(node.options.processorOptions as unknown as StreamOptions).first, ...node.posted] + .filter((chunk): chunk is Float32Array[] => Array.isArray(chunk) && chunk.length > 0) + .flatMap((chunk) => Array.from(chunk[0]!)); + + it('plays from its copy on a stream player as long as the copy, none of the file decoded', async () => { + const library = await streaming(); + expect(library.duration(FILE)).toBe(10); + const voice = library.play(FILE, 1, true); + await drain(); + expect(player().options).toMatchObject({ + outputChannelCount: [2], + processorOptions: { channels: 2, startAt: 0 }, + }); + expect(heard(player())[0]).toBe(PRE_SKIP); + expect((player().options.processorOptions as unknown as StreamOptions).first[0]).toHaveLength(96000); + expect(wires).toEqual(['stream > gain', ...INTO_MASTER]); + expect(decodes).toHaveLength(0); + expect(library.stats()).toMatchObject({ decoded: 0, playing: 1, streaming: { playing: 1, heads: 2 } }); + library.stopVoice(voice); + clockTo(); + expect(player().posted.at(-1)).toBe('stop'); + expect(library.stats()).toMatchObject({ playing: 0, streaming: { playing: 0 } }); + }); + + it('moves its pitch with a shifter of what it plays, as a bus does', async () => { + const library = await streaming(); + await play(library, FILE, {}, { pitch: 7, stages: [] }); + await drain(); + expect(wires).toEqual(['stream > shifter', 'shifter > gain', ...INTO_MASTER]); + expect(worklets.find((w) => w.kind === 'shifter')!.pitch.value).toBe(7); + expect(library.stats()).toMatchObject({ vocoders: 1, last: { pitches: [7] } }); + }); + + it('starts a clip of a song as far into its copy as its offset is', async () => { + const library = await streaming(); + library.playClip(FILE, { at: 0, offset: 12.5, duration: 4, volume: 1, bus: '' }); + await drain(); + expect(heard(player())[0]).toBe(PRE_SKIP + 2.5 * 48000); + }); + + it('plays a sound whose copy cannot be had decoded, as without one', async () => { + stubDecoder(); + vi.stubGlobal('fetch', (url: string) => + Promise.resolve( + url.endsWith(COPY) + ? { ok: false, status: 404 } + : { ok: true, arrayBuffer: () => Promise.resolve(new ArrayBuffer(8)) }, + ), + ); + const library = new SoundLibrary(false); + library.setStreams(new Map([[FILE, { file: COPY, frames: 480000 }]])); + await drain(); + await play(library); + expect(worklets.filter((w) => w.kind === 'stream')).toHaveLength(0); + expect(sources.filter((s) => s.started)).toHaveLength(1); + }); + + it('prepares a bake by the head of its file and streams it round its round, to the sample', async () => { + const library = await streaming(); + library.playSequence( + 'pattern:p', + { + sequence: { notes: [{ tick: 0, length: 480, velocity: 1, instrument: 'i', note: 60 }] }, + tempo: 120, + length: 1920, + loop: { start: 0, end: 1920 }, + needs: { layers: [], files: [] }, + strike: () => 0, + bake: { file: BAKED, gain: 0.5, playback: { end: null, until: 2, round: { start: 3, end: 5 } } }, + }, + '', + ); + await drain(); + expect(decodes).toHaveLength(0); + expect(sources).toHaveLength(0); + const sent = heard(player()); + expect(sent[2 * 48000 - 1]).toBe(PRE_SKIP + 2 * 48000 - 1); + expect(sent[2 * 48000]).toBe(PRE_SKIP + 3 * 48000); + expect(library.playbacks()[0]?.baked).toBe(true); + }); +}); + describe('a sound FX on a bus (#346)', () => { /** One frame of the shifter at the fake context's 48 kHz. */ const LATE = 2048 / 48000; diff --git a/tests/viewport/streaming.test.ts b/tests/viewport/streaming.test.ts new file mode 100644 index 00000000..e0c0c40b --- /dev/null +++ b/tests/viewport/streaming.test.ts @@ -0,0 +1,152 @@ +/** + * Streaming a sound (#349) on the page's side: the copy fetched a piece at a time, its first seconds + * decoded ahead, and a `stream-player` kept fed along its plan – every sample where it belongs, at a + * jump as well as round a loop, and no further ahead than it needs. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { StreamPlan } from '../../src/core/music/streams'; +import { StreamFeed, StreamFile } from '../../src/viewport/streaming'; +import { decoders, fakeCopy, PRE_SKIP, serveCopies, stubDecoder } from './fakeStream'; + +const URL = '/sounds/copy.ogg'; +const settle = () => new Promise((resolve) => setTimeout(resolve, 0)); + +/** A worklet node as the feed sees it: what was posted to it, and a way to report what it read. */ +function fakeNode() { + const posted: unknown[] = []; + const node = { + port: { + onmessage: null as ((event: { data: unknown }) => void) | null, + postMessage: (data: unknown) => posted.push(data), + close: () => {}, + }, + disconnect: () => {}, + }; + const samples = () => + (posted.filter((data) => Array.isArray(data) && data.length > 0) as Float32Array[][]).flatMap((chunk) => + Array.from(chunk[0]!), + ); + const read = (used: number) => node.port.onmessage?.({ data: { used, skipped: 0 } }); + return { node: node as unknown as AudioWorkletNode, posted, samples, read }; +} + +/** A feed as the library makes it: its opening handed over first, then attached to the player. */ +function feedOf(file: StreamFile, plan: StreamPlan, fake: ReturnType, ended = () => {}) { + const feed = new StreamFeed(file, plan, ended); + fake.posted.push(feed.opening()); + feed.attach(fake.node); + return feed; +} + +/** Waits until the feed has nothing left to do for now. */ +async function drain(): Promise { + for (let i = 0; i < 400; i++) await settle(); +} + +beforeEach(() => stubDecoder()); +afterEach(() => vi.unstubAllGlobals()); + +describe('a copy streamed', () => { + it('fetches pieces until its first two seconds are there and decodes them, its pre-skip cut off', async () => { + const copy = fakeCopy(10); + const server = serveCopies({ [URL]: copy.bytes }); + const file = new StreamFile(URL, copy.frames); + expect(await file.ready()).toBe(true); + expect(file.head![0]!.length).toBe(96000); + expect([file.head![0]![0], file.head![0]![95999], file.head![1]![0]]).toEqual([ + PRE_SKIP, + PRE_SKIP + 95999, + PRE_SKIP + 1e6, + ]); + expect(server.asked[0]).toBe('bytes=0-262143'); + expect(file.bytes).toBeLessThan(copy.bytes.length); + }); + + it('takes the whole file from a server that leaves `Range` out', async () => { + const copy = fakeCopy(5); + const server = serveCopies({ [URL]: copy.bytes }, false); + const file = new StreamFile(URL, copy.frames); + expect(await file.ready()).toBe(true); + expect(server.asked).toHaveLength(1); + expect(file.bytes).toBe(copy.bytes.length); + }); + + it('sends its head at once, then decodes on from there to the sample, no more than four seconds ahead', async () => { + const copy = fakeCopy(10); + serveCopies({ [URL]: copy.bytes }); + const file = new StreamFile(URL, copy.frames); + await file.ready(); + const player = fakeNode(); + const { samples, read } = player; + const feed = new StreamFeed(file, { from: 0, until: copy.frames, loop: null }, () => {}); + // The head goes with the player as it is made; the rest comes by message. + expect(feed.opening()[0]).toHaveLength(96000); + feed.attach(player.node); + expect(samples()).toHaveLength(0); + await drain(); + const sent = samples(); + expect(sent).toHaveLength(2 * 48000); + expect(sent.every((value, i) => value === PRE_SKIP + 96000 + i)).toBe(true); + read(48000); + await drain(); + expect(samples()).toHaveLength(3 * 48000); + expect(samples()[3 * 48000 - 1]).toBe(PRE_SKIP + 5 * 48000 - 1); + }); + + it('jumps in mid-way after settling the decoder on what comes before', async () => { + const copy = fakeCopy(10); + serveCopies({ [URL]: copy.bytes }); + const file = new StreamFile(URL, copy.frames); + await file.ready(); + const player = fakeNode(); + const { samples } = player; + const from = 6 * 48000 + 123; + feedOf(file, { from, until: copy.frames, loop: null }, player); + await drain(); + expect(samples()[0]).toBe(PRE_SKIP + from); + expect(samples().every((value, i) => value === PRE_SKIP + from + i)).toBe(true); + const fed = decoders.at(-1)!.fed; + expect(fed[0]).toBeLessThanOrEqual(PRE_SKIP + from - 32 * 960); + expect(fed[0]).toBeGreaterThan(PRE_SKIP + from - 34 * 960); + }); + + it('goes round its loop to the sample and says when it has played out once', async () => { + const copy = fakeCopy(3); + serveCopies({ [URL]: copy.bytes }); + const file = new StreamFile(URL, copy.frames); + await file.ready(); + const looped = fakeNode(); + const start = 100000; + feedOf(file, { from: 0, until: copy.frames, loop: { start, end: copy.frames } }, looped); + await drain(); + const sent = looped.samples(); + expect(sent[copy.frames - 1]).toBe(PRE_SKIP + copy.frames - 1); + expect(sent[copy.frames]).toBe(PRE_SKIP + start); + const round = copy.frames - start; + expect(sent.length).toBeGreaterThan(copy.frames + round); + expect(sent.slice(copy.frames).every((value, i) => value === PRE_SKIP + start + (i % round))).toBe(true); + + const once = fakeNode(); + let ended = false; + feedOf(file, { from: 0, until: copy.frames, loop: null }, once, () => (ended = true)); + await drain(); + expect(once.samples()).toHaveLength(copy.frames); + expect(once.posted.at(-1)).toBe('end'); + once.node.port.onmessage?.({ data: 'ended' } as never); + expect(ended).toBe(true); + }); + + it('stops the player and its decoder', async () => { + const copy = fakeCopy(10); + serveCopies({ [URL]: copy.bytes }); + const file = new StreamFile(URL, copy.frames); + await file.ready(); + const player = fakeNode(); + const { posted } = player; + const feed = feedOf(file, { from: 5 * 48000, until: copy.frames, loop: null }, player); + await drain(); + feed.stop(); + expect(posted.at(-1)).toBe('stop'); + expect(decoders.at(-1)!.closed).toBe(true); + }); +}); diff --git a/tools/e2e/34-sound.mjs b/tools/e2e/34-sound.mjs index 299ca4ef..9f6e28bf 100644 --- a/tools/e2e/34-sound.mjs +++ b/tools/e2e/34-sound.mjs @@ -1,4 +1,7 @@ +import { execSync } from 'node:child_process'; +import { sineWav } from '../lib/audio.mjs'; import { edge, node, port, variable, written } from '../lib/builders.mjs'; +import { uploadSound } from '../lib/projects.mjs'; import { LANES, MAKE_OBJECT, pickAssetType } from './lib/scaffold.mjs'; /** @@ -2570,6 +2573,201 @@ export async function sound(t) { p.assets = p.assets.filter((a) => a.type !== 'bus'); p.routing = { nodes: [], edges: [] };`); + // 34h-4. A long sound streamed (#349): five minutes of mono, decoded whole some 55 MB. With Stream + // on, the editor makes a copy of it as Opus and a game plays from that, through its bus and + // the master; the renderer grows by a few megabytes while it plays, not by the decoded + // file. The counter-proof right after turns Stream off: decoded, it grows by about that much. + const long = await uploadSound(t.baseUrl, sineWav(300, 330), 'long.wav'); + const rendererMb = () => + execSync('ps -eo rss,args') + .toString() + .split('\n') + .filter((line) => line.includes('--type=renderer')) + .reduce((sum, line) => sum + Number(line.trim().split(/\s+/)[0]) / 1024, 0); + const restingMb = async () => { + await t.cdp.send('HeapProfiler.collectGarbage'); + await sleep(1500); + return rendererMb(); + }; + // What the game runs: On start plays the long sound, or a song. + const startWith = (played) => + mutate(` + __gm.state.project.assets.find((a) => a.id === 'sn-logic').graph = ${JSON.stringify({ + nodes: [node('sn-start', 'OnStart', 0, 0), played], + edges: [edge('sn-e1', 'sn-start', 'exec', played.id, 'exec')], + })};`); + const playsLong = node('sn-play', 'PlaySound', 260, 0, { sound: 'sn-long', volume: 1, loop: '' }); + await evaluate('__gm.sound.stopAll()'); + await mutate(` + const p = __gm.state.project; + p.assets.push( + { id: 'sn-long', type: 'sound', name: 'long', props: { volume: 1, loop: true, fx: '', stream: true }, file: ${JSON.stringify(long.file)} }, + { id: 'sn-bus', type: 'bus', name: 'Music', props: { volume: 1, fx: '' } }, + ); + p.routing = { + nodes: [ + { id: 'sn-long', type: 'RoutedAsset', x: 0, y: 0, params: { name: 'long', kind: 'sound' } }, + { id: 'sn-bus', type: 'RoutedAsset', x: 320, y: 0, params: { name: 'Music', kind: 'bus' } }, + { id: '#master', type: 'RoutedMaster', x: 640, y: -80, params: {} }, + ], + edges: [ + { id: 'sn-r1', from: { node: 'sn-long', port: 'out' }, to: { node: 'sn-bus', port: 'in' } }, + { id: 'sn-r2', from: { node: 'sn-bus', port: 'out' }, to: { node: '#master', port: 'in' } }, + ], + };`); + await startWith(playsLong); + await evaluate('__gm.streams.idle()'); + const copies = await json("__gm.streams.copies('sn-long')"); + const streamed = (await json('__gm.streams.table()')).find((entry) => entry.name === long.file); + check( + 'with Stream on the editor makes a copy of the long sound as Opus, five minutes to the sample, and a game streams it from there', + copies?.sound?.from === long.file && + copies.sound.frames === 300 * 48000 && + copies.sound.bytes > 0 && + streamed?.file === copies.sound.file && + streamed.frames === 300 * 48000, + JSON.stringify({ long, copies, streamed }), + ); + + const decodedBefore = (await stats()).decoded; + const restStreamed = await restingMb(); + await play(); + const tapsLong = JSON.stringify(['mix', 'mix:sn-bus']); + await json(`__gm.routing.peaks(${tapsLong})`); + const streaming = await waitFor(stats, (s) => s.streaming.playing > 0, 40); + const heardLong = await waitFor( + () => json(`__gm.routing.peaks(${tapsLong})`), + (peaks) => Math.min(Math.max(...(peaks.mix ?? [0])), Math.max(...(peaks['mix:sn-bus'] ?? [0]))) > 0.01, + ); + await sleep(2500); + const playingStreamed = rendererMb(); + await t.clickToolbar('pause'); + await sleep(500); + const paused = { clock: await json('__gm.sound.clock()'), state: await json('__gm.sound.clockState()') }; + await sleep(500); + const pausedLater = await json('__gm.sound.clock()'); + await t.clickToolbar('pause'); + await sleep(1000); + const resumedLong = await stats(); + await stop(); + const stoppedLong = await stats(); + check( + 'played, the streamed sound is heard on its bus and the master, the renderer growing by less than 20 MB, none of the file decoded', + streaming?.last?.file === long.file && + streaming.last.bus === 'mix:sn-bus' && + streaming.decoded === decodedBefore && + playingStreamed - restStreamed < 20 && + Math.max(...(heardLong?.mix ?? [0])) > 0.01, + JSON.stringify({ restStreamed, playingStreamed, streaming, heardLong }), + ); + check( + 'the pause holds the stream where it is, it plays on after it without a sample lost, and stopping ends it', + paused.state === 'suspended' && + pausedLater === paused.clock && + resumedLong.streaming.playing === 1 && + resumedLong.streaming.skipped === 0 && + stoppedLong.streaming.playing === 0 && + stoppedLong.playing === 0, + JSON.stringify({ paused, pausedLater, resumedLong: resumedLong.streaming, stoppedLong }), + ); + + // The counter-proof: Stream off, the game plays it decoded, and the renderer holds all of it. + // Measured from before it is switched off, as from then on the editor decodes it ahead, and before + // anything below has had large buffers made: Chrome does not hand all of that back at once. + await evaluate('__gm.sound.stopAll()'); + const restDecoded = await restingMb(); + await mutate(`__gm.state.project.assets.find((a) => a.id === 'sn-long').props.stream = false;`); + await play(); + const decodedNow = await waitFor(stats, (s) => s.playing > 0 && s.last?.file === long.file, 60); + await sleep(2500); + const playingDecoded = rendererMb(); + await stop(); + await evaluate('__gm.streams.idle()'); + check( + 'with Stream off the same sound is decoded whole, and the renderer grows by more than 40 MB', + decodedNow?.streaming.playing === 0 && + decodedNow.decoded > decodedBefore && + playingDecoded - restDecoded > 40 && + (await json("__gm.streams.copies('sn-long')"))?.sound === null, + JSON.stringify({ restDecoded, playingDecoded, decodedNow }), + ); + + // The Sound tab points a long sound at the switch, and says once it streams. + await evaluate("__gm.state.openAsset('sn-long')"); + await sleep(300); + const longInfo = await json('__gm.soundTab.info()'); + await mutate(`__gm.state.project.assets.find((a) => a.id === 'sn-long').props.stream = true;`); + await evaluate('__gm.streams.idle()'); + await sleep(200); + const streamedInfo = await json('__gm.soundTab.info()'); + check( + 'the Sound tab suggests Stream for a sound of a minute or more, and says when it is streamed', + longInfo === '300.00 s · long, try Stream' && streamedInfo === '300.00 s · streamed', + JSON.stringify({ longInfo, streamedInfo }), + ); + // On a track of a song it starts on its beat, a clip of its own streamed from a copy of its cut. + await mutate(` + const sound = __gm.state.project.assets.find((a) => a.id === 'sn-long'); + sound.clips = [{ id: 'sn-long-clip', name: 'Middle', props: { start: 10, end: 20, fadeIn: 5, fadeOut: 5, fadeInCurve: 'linear', fadeOutCurve: 'linear', reverse: true } }];`); + await clickText('+'); + await pickAssetType(t, 'Song'); + await sleep(200); + const [longSong] = await assetsOfType('song'); + await mutate(` + const song = __gm.state.project.assets.find((a) => a.id === ${JSON.stringify(longSong.id)}); + song.items = [{ id: 'sn-i1', track: song.tracks[0].id, kind: 'sound', asset: 'sn-long-clip', tick: 0, length: 3840, offset: 0, volume: 0, transpose: 0, instrument: '', fx: '' }];`); + await evaluate('__gm.streams.idle()'); + await startWith( + node('sn-song', 'PlaySong', 260, 0, { song: longSong.id, volume: 1, fadeIn: 0, loop: false }), + ); + await showTab('scene'); + await play(); + const songRun = await waitFor( + async () => ({ runs: await json('__gm.sound.playbacks()'), stats: await stats() }), + (got) => got.runs[0]?.clips.length > 0 && got.stats.streaming.playing > 0, + 40, + ); + await stop(); + const clipCopy = (await json("__gm.streams.copies('sn-long')"))?.clips[0]?.stream; + const clipName = await json("__gm.soundTab.played('sn-long-clip')"); + const songClip = songRun?.runs[0]?.clips[0]; + check( + 'a clip of the streamed sound gets a copy of its cut, and a song streams it on its track, started on its beat', + clipCopy?.from === clipName && + clipCopy.frames === 10 * 48000 && + songClip?.sound === 'sn-long-clip' && + songRun.stats.last?.file === clipName && + Math.abs(songRun.stats.last.at - songClip.at) < 1e-6, + JSON.stringify({ clipCopy, clipName, songClip, last: songRun?.stats.last }), + ); + + // Offline, to the sample: from 0.25 s on, two seconds into the copy; round the seam of its loop; + // and the reversed clip from its start – each against the copy decoded whole. + const exactly = async (...args) => + JSON.parse(await evaluate(`__gm.streams.exact(...${JSON.stringify(args)}).then(JSON.stringify)`)); + const exact = { + start: await exactly(long.file, 2, false, 0.25, 2), + seam: await exactly(long.file, 299, true, 0, 2.5), + clip: await exactly(clipName, 9, true, 0.1, 2), + }; + check( + 'streamed, it starts on its sample, runs on past where it jumped in and round its loop as the copy decoded whole', + Object.values(exact).every( + (run) => + run !== null && + run.own < 1e-6 && + run.browser < 0.01 && + run.skipped === 0 && + run.decoded === run.frames, + ), + JSON.stringify(exact), + ); + + await mutate(` + const p = __gm.state.project; + p.assets = p.assets.filter((a) => !['sn-long', 'sn-bus', ${JSON.stringify(longSong.id)}].includes(a.id)); + p.routing = { nodes: [], edges: [] };`); + // 34i. The step takes back everything it made, so the steps after it see the fixture again. await mutate(` const gone = new Set(['sn-logic', ...${JSON.stringify(both.map((a) => a.id))}]);