Stream long sounds from an Opus copy instead of decoding them (#349)

A sound was decoded whole as soon as the project opened, in the editor
and in a released game: five minutes of stereo held some 115 MB for as
long as the page was open. A sound with Stream on now plays from a copy
as Opus in Ogg, fetched a piece at a time and decoded a few seconds
ahead by WebCodecs, so a game holds a few megabytes for it.

- The Stream switch is a row of the sound, off by default. The Sound
  tab suggests it from a minute on and says how far the copy has got.
- StreamKeeper makes the copies in the browser, a moment after the
  last change, one at a time: the sound's own and one per clip of its
  cut, fades and Reverse baked in. A copy that is no longer wanted is
  taken off and deleted once the project is saved without it.
- streamTable says which played names stream from which copy. A clip
  without a fitting copy, a Sample and a Reverb impulse stay decoded.
  A sound with Stream on is never decoded ahead.
- The page fetches by Range and takes the whole file from a server
  that ignores it, as the CI's PHP server does. It keeps the bytes
  and the first two seconds decoded, the rest goes four seconds ahead.
- A jump decodes 32 packets before the place first. Measured in
  Chrome, what follows is then the same to the bit as decoding from
  the start. With 4 packets it was 0.06 off.
- The stream-player on the FX worklet reads the samples at the rate
  of the context along a Catmull-Rom curve. What comes late is silent
  and skipped once it is there, so a clip keeps its beat.
- The opening stretch goes with the player as it is made. Sent by
  message it came a few milliseconds late, and 557 samples of the
  attack were lost.
- Pitch on a streamed sound is a shifter after the player, as on a
  bus: the length stays and it is heard 42.7 ms late.
- Bakes of patterns and songs (#340) always stream now.
- The song timeline draws a streamed sound from its copy decoded at
  8 kHz, keeping only the peaks.

Tests:
- Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3,
  stream-queue 6, streams 11, streaming 6, sound-library +5).
- e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono,
  4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by
  5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and
  a reversed clip are the copy decoded whole to the sample (0).
- Counter-checks red as they should be: no pre-roll, no catch-up of
  late samples at the end, no opening stretch, a streamed sound
  decoded ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/editor/Editor.ts

@@ -9,6 +9,7 @@ import { pluginsOf } from '../core/store/projectPlugins';
 import type { FontStore } from '../core/store/FontStore';
 import type { SnapshotStore } from '../core/store/SnapshotStore';
 import type { SoundStore } from '../core/store/SoundStore';
+import { StreamKeeper } from './sound/StreamKeeper';
 import type { TextureStore } from '../core/store/TextureStore';
 import type { UsersStore } from '../core/store/UsersStore';
 import { gameClock, timeStep } from '../core/time';
@@ -249,6 +250,8 @@ export class Editor implements PlayHost {
   readonly shaderEditor: ShaderEditor;
   readonly particlesEditor: ParticlesEditor;
   readonly soundEditor: SoundEditor;
+  /** Makes the copies streamed sounds play from (#349). */
+  readonly streamKeeper: StreamKeeper;
   readonly soundFxEditor: SoundFxEditor;
   readonly instrumentEditor: InstrumentEditor;
   readonly patternEditor: PatternEditor;
@@ -349,7 +352,8 @@ export class Editor implements PlayHost {
     this.stateMachineEditor = new StateMachineEditor(state);
     this.shaderEditor = new ShaderEditor(state);
     this.particlesEditor = new ParticlesEditor(state);
-    this.soundEditor = new SoundEditor(state);
+    this.streamKeeper = new StreamKeeper(state, sounds);
+    this.soundEditor = new SoundEditor(state, this.streamKeeper);
     this.soundFxEditor = new SoundFxEditor(state);
     this.instrumentEditor = new InstrumentEditor(state);
     this.patternEditor = new PatternEditor(state, sounds);

src/editor/debug/soundHooks.ts

@@ -1,5 +1,7 @@
 import { findAssetOf } from '../../core/model/Project';
 import { clipsOf, findSound } from '../../core/model/sound';
+import { STREAM_RATE } from '../../core/music/streamQueue';
+import { playedPlan, streamTable } from '../../core/music/streams';
 import { playedSoundOf } from '../../core/properties/playedSound';
 import { soundLibrary } from '../../viewport/SoundLibrary';
 import { centreOf, type HookContext } from './hookHelpers';
@@ -72,5 +74,62 @@ export function soundHooks({ state, editor }: HookContext) {
         };
       },
     },
+    /** Streamed sounds (#349). */
+    streams: {
+      /** Done once every copy wanted has been made. */
+      idle: () => editor.streamKeeper.idle(),
+      /** Every name a game streams, with its copy. */
+      table: () => [...streamTable(state.project)].map(([name, copy]) => ({ name, ...copy })),
+      /**
+       * Streams `seconds` of a name offline from `offset` seconds in, heard from `at`, looped or not,
+       * and holds it against its copy decoded whole: the largest difference of any sample from what
+       * should be heard there – silence before `at` – against the feed's own decoder and against the
+       * browser's `decodeAudioData`, and how many samples came too late.
+       */
+      exact: async (name: string, offset: number, loop: boolean, at: number, seconds: number) => {
+        const entry = streamTable(state.project).get(name);
+        if (!entry) return null;
+        const plan = playedPlan(entry.frames, offset, loop);
+        const frames = Math.round((at + seconds) * STREAM_RATE);
+        const [played, whole, browser] = await Promise.all([
+          soundLibrary.renderStream(name, plan, at, frames),
+          soundLibrary.streamedWhole(name),
+          soundLibrary.decodeApart(entry.file, STREAM_RATE),
+        ]);
+        if (!played || !whole) return null;
+        const first = Math.round(at * STREAM_RATE);
+        const placeOf = (i: number) =>
+          loop ? (plan.from + i - first) % entry.frames : plan.from + i - first;
+        const worst = (reference: (c: number, place: number) => number) => {
+          let most = 0;
+          played.channels.forEach((side, c) => {
+            for (let i = 0; i < frames; i++) {
+              const place = placeOf(i);
+              const expected = i < first || place >= entry.frames ? 0 : reference(c, place);
+              most = Math.max(most, Math.abs(side[i]! - expected));
+            }
+          });
+          return most;
+        };
+        return {
+          frames: entry.frames,
+          decoded: browser.length,
+          own: worst((c, place) => whole[c]![place]!),
+          browser: worst(
+            (c, place) => browser.getChannelData(Math.min(c, browser.numberOfChannels - 1))[place]!,
+          ),
+          skipped: played.skipped,
+        };
+      },
+      /** What a sound and each of its clips keep of their copies. */
+      copies: (id: string) => {
+        const asset = findAssetOf(state.project, id, 'sound');
+        if (!asset) return null;
+        return {
+          sound: asset.stream ?? null,
+          clips: clipsOf(asset).map((clip) => ({ id: clip.id, stream: clip.stream ?? null })),
+        };
+      },
+    },
   };
 }

src/editor/music/BakeButton.ts

@@ -12,6 +12,7 @@ import { Button, formatBytes, Menu, Theme } from '../../ui';
 import { BakeCancelled, makeBake } from '../../viewport/bakeRender';
 import { HEADER_ICON_SVG } from '../panels/headerIcons';
 import type { EditorState } from '../state';
+import { dropWhenSaved } from './dropWhenSaved';
 import { clockText } from './timeView';
 
 /** Where a bake stands for the asset shown. */
@@ -128,7 +129,7 @@ export class BakeButton extends Button {
       }
       const old = this.current(asset.id)?.bake?.file;
       this.state.setBake(asset.id, { file: uploaded.file, bytes: uploaded.bytes, ...rest });
-      if (old && old !== uploaded.file) this.dropWhenSaved(old);
+      if (old && old !== uploaded.file) dropWhenSaved(this.state, this.sounds, old);
       this.state.setNotice(
         `Baked ${asset.name}: ${clockText(rest.length / BAKE_RATE)}, ${formatBytes(uploaded.bytes)}`,
       );
@@ -145,7 +146,7 @@ export class BakeButton extends Button {
   private remove(asset: BakedAsset): void {
     const old = asset.bake?.file;
     this.state.setBake(asset.id, null);
-    if (old) this.dropWhenSaved(old);
+    if (old) dropWhenSaved(this.state, this.sounds, old);
   }
 
   private current(id: string): BakedAsset | null {
@@ -162,17 +163,4 @@ export class BakeButton extends Button {
   private changed(asset: BakedAsset): void {
     this.state.setNotice(`${asset.name} changed while it was baked – bake it again`, 'error');
   }
-
-  /**
-   * Deletes the file of a bake that has gone, once the project no longer names it on the server: the
-   * server keeps a file any saved project or snapshot names (409), and so does this.
-   */
-  private dropWhenSaved(file: string): void {
-    const off = this.state.on('save', () => {
-      const status = this.state.saveStatus;
-      if (status === 'dirty' || status === 'saving') return;
-      off();
-      if (status === 'saved') void this.sounds.delete(file).catch(() => {});
-    });
-  }
 }

src/editor/music/dropWhenSaved.ts

@@ -0,0 +1,16 @@
+import type { SoundStore } from '../../core/store/SoundStore';
+import type { EditorState } from '../state';
+
+/**
+ * Deletes a file of the sounds a bake (#340) or a stream copy (#349) left behind, once the project no
+ * longer names it on the server: the server keeps a file any saved project or snapshot names (409),
+ * and so does this.
+ */
+export function dropWhenSaved(state: EditorState, sounds: SoundStore, file: string): void {
+  const off = state.on('save', () => {
+    const status = state.saveStatus;
+    if (status === 'dirty' || status === 'saving') return;
+    off();
+    if (status === 'saved') void sounds.delete(file).catch(() => {});
+  });
+}

src/editor/song/SongTimeline.ts

@@ -1368,9 +1368,9 @@ export class SongTimeline extends Widget {
   }
 
   private ensureLoaded(file: string): void {
-    if (!file || this.loading.has(file) || soundLibrary.duration(file) !== null) return;
+    if (!file || this.loading.has(file) || soundLibrary.peaks(file) !== null) return;
     this.loading.add(file);
-    void soundLibrary.load(file).then(() => this.invalidate());
+    void soundLibrary.loadPicture(file).then(() => this.invalidate());
   }
 
   override layout(): void {

src/editor/sound/SoundEditor.ts

@@ -12,8 +12,10 @@ import {
   FADE_OUT_CURVE_KEY,
   FADE_OUT_KEY,
   REVERSE_KEY,
+  streamsOf,
   volumeOf,
 } from '../../core/properties/sounds';
+import { STREAM_SUGGESTED } from '../../core/music/streams';
 import { Button, Dropdown, Label, Menu, NumberField } from '../../ui';
 import { soundLibrary } from '../../viewport/SoundLibrary';
 import { MASTER_MIX } from '../../core/music/mixer';
@@ -26,6 +28,7 @@ import { claimTransport, releaseTransport } from '../music/transport';
 import { HEADER_ICON_SVG } from '../panels/headerIcons';
 import { TOOLBAR_ICON_SVG } from '../panels/toolbarIcons';
 import type { EditorState } from '../state';
+import type { StreamKeeper } from './StreamKeeper';
 import { WaveCanvas } from './WaveCanvas';
 
 const INFO_WIDTH = 140;
@@ -71,7 +74,10 @@ export class SoundEditor extends AssetTabEditor<SoundAsset, WaveCanvas> {
   private playingId = '';
   private readonly halt = () => this.stop();
 
-  constructor(state: EditorState) {
+  constructor(
+    state: EditorState,
+    private readonly streams: StreamKeeper,
+  ) {
     super(state, { infoWidth: INFO_WIDTH });
     this.playButton = new Button('Play', {
       icon: TOOLBAR_ICON_SVG.play,
@@ -125,6 +131,8 @@ export class SoundEditor extends AssetTabEditor<SoundAsset, WaveCanvas> {
     state.on('load', () => this.stop());
     // Picking a clip changes what the header plays and edits.
     state.on('selection', () => this.refresh());
+    // How far a stream copy has got (#349) comes as a notice, not as a change of the project.
+    state.on('notice', () => this.refresh());
   }
 
   override selectedId(): string {
@@ -251,7 +259,20 @@ export class SoundEditor extends AssetTabEditor<SoundAsset, WaveCanvas> {
     if (!asset.file) return 'Not uploaded';
     if (!span || whole === null) return '';
     const kept = span.end - span.start;
-    return kept >= whole ? `${whole.toFixed(2)} s` : `${kept.toFixed(2)} s of ${whole.toFixed(2)} s`;
+    const length = kept >= whole ? `${whole.toFixed(2)} s` : `${kept.toFixed(2)} s of ${whole.toFixed(2)} s`;
+    return `${length}${this.streamText(asset, whole)}`;
+  }
+
+  /**
+   * Whether it streams (#349): its copy is made, being made or yet to be; a long one that does not
+   * is pointed at the switch.
+   */
+  private streamText(asset: SoundAsset, whole: number): string {
+    const making = this.streams.making(asset.id);
+    if (making !== null) return ` · copy ${Math.round(making * 100)}%`;
+    if (streamsOf(resolveAssetProps(this.state.project, asset)))
+      return asset.stream?.from === asset.file ? ' · streamed' : ' · copy waiting';
+    return whole >= STREAM_SUGGESTED ? ' · long, try Stream' : '';
   }
 
   protected override syncExtras(asset: SoundAsset | null): void {

src/editor/sound/SoundPicker.ts

@@ -1,5 +1,6 @@
 import { Container, type FederatedPointerEvent, Graphics, type Text } from 'pixi.js';
 import { isBakeUpload } from '../../core/music/bake';
+import { isStreamUpload } from '../../core/music/streams';
 import type { StoredSound } from '../../core/store/SoundStore';
 import { drawSvg, formatBytes, textStyle, Theme, truncate, crispText } from '../../ui';
 import { soundLibrary } from '../../viewport/SoundLibrary';
@@ -121,7 +122,9 @@ class SoundPicker extends FilePicker<SoundItem> {
 
   /** A bake (#340) is in the sounds, but no sound to pick. */
   protected async fetchItems(): Promise<SoundItem[]> {
-    const sounds = (await this.deps.sounds.list()).filter((sound) => !isBakeUpload(sound.name));
+    const sounds = (await this.deps.sounds.list()).filter(
+      (sound) => !isBakeUpload(sound.name) && !isStreamUpload(sound.name),
+    );
     return withAssetNames(sounds, this.deps.state.project, 'sound');
   }
 

src/editor/sound/StreamKeeper.ts

@@ -0,0 +1,164 @@
+import type { SoundAsset } from '../../core/model/Asset';
+import { findAsset } from '../../core/model/Project';
+import { clipsOf, cutName, cutChannels, cutRange, parseCutName, soundRefName } from '../../core/model/sound';
+import { STREAM_RATE } from '../../core/music/streamQueue';
+import { type StreamJob, streamUploadName, streamWork } from '../../core/music/streams';
+import { cutOf, streamsOf } from '../../core/properties/sounds';
+import type { SoundStore } from '../../core/store/SoundStore';
+import { BakeCancelled, encodeOpus } from '../../viewport/bakeRender';
+import { soundLibrary } from '../../viewport/SoundLibrary';
+import { dropWhenSaved } from '../music/dropWhenSaved';
+import type { EditorState } from '../state';
+
+/** Milliseconds the project has to stay as it is before a copy is made: a clip dragged about is one. */
+const SETTLE_MS = 600;
+
+/**
+ * Makes the copies streamed sounds are played from (#349). A moment after the project last changed,
+ * each sound with `stream` on gets a copy of its file as Opus, and each of its clips one of its cut,
+ * one at a time, from the file decoded for it and let go again. A copy no longer wanted is taken off
+ * its sound and deleted once the project has been saved without it. Not while the game runs.
+ */
+export class StreamKeeper {
+  private job: { job: StreamJob; share: number } | null = null;
+  private timer: ReturnType<typeof setTimeout> | null = null;
+  /** Copies that could not be made, so one broken file is not tried over and over. */
+  private readonly failed = new Set<string>();
+  private waiting: (() => void)[] = [];
+
+  constructor(
+    private readonly state: EditorState,
+    private readonly sounds: SoundStore,
+  ) {
+    for (const event of ['project', 'load', 'mode'] as const) state.on(event, () => this.later());
+    this.later();
+  }
+
+  /** How far the copy of that sound, or of one of its clips, has got, 0 to 1; null while none is made. */
+  making(assetId: string): number | null {
+    return this.job?.job.assetId === assetId ? this.job.share : null;
+  }
+
+  /** Done once there is nothing left to make (test hook). */
+  idle(): Promise<void> {
+    if (!this.job && this.timer === null && this.pending().length === 0) return Promise.resolve();
+    return new Promise((resolve) => this.waiting.push(resolve));
+  }
+
+  private later(wait = SETTLE_MS): void {
+    if (this.timer !== null) clearTimeout(this.timer);
+    this.timer = setTimeout(() => {
+      this.timer = null;
+      void this.work();
+    }, wait);
+  }
+
+  private pending(): StreamJob[] {
+    return streamWork(this.state.project).jobs.filter((job) => !this.failed.has(keyOf(job)));
+  }
+
+  private async work(): Promise<void> {
+    if (this.job || this.state.mode === 'play') return;
+    for (const drop of streamWork(this.state.project).drops) {
+      const old = this.ownerOf(drop.assetId, drop.clipId)?.stream?.file;
+      this.state.setStream(drop.assetId, drop.clipId, null);
+      if (old && !this.named(old)) dropWhenSaved(this.state, this.sounds, old);
+    }
+    const next = this.pending()[0];
+    if (!next) {
+      for (const resolve of this.waiting.splice(0)) resolve();
+      return;
+    }
+    this.job = { job: next, share: 0 };
+    try {
+      await this.make(this.job);
+    } catch (error: unknown) {
+      // Called off because what it was made of changed: the next round makes the right one.
+      if (error instanceof BakeCancelled) return this.later(0);
+      this.failed.add(keyOf(next));
+      this.state.setNotice(
+        `No stream copy of ${this.label(next)}: ${error instanceof Error ? error.message : String(error)}`,
+        'error',
+      );
+    } finally {
+      this.job = null;
+    }
+    this.later(0);
+  }
+
+  private async make(running: { job: StreamJob; share: number }): Promise<void> {
+    const { job } = running;
+    const label = this.label(job);
+    const { file, cut } = parseCutName(job.from);
+    const whole = await soundLibrary.decodeApart(file, STREAM_RATE);
+    let channels = Array.from({ length: Math.min(2, whole.numberOfChannels) }, (_, c) =>
+      whole.getChannelData(c),
+    );
+    if (cut) channels = cutChannels(channels, cutRange(whole.length, whole.sampleRate, cut));
+    const data = await encodeOpus(channels, {
+      progress: (share) => {
+        const before = Math.round(running.share * 100);
+        running.share = share;
+        const now = Math.round(share * 100);
+        if (now !== before) this.state.setNotice(`Making a stream copy of ${label}: ${now}%`, 'quiet');
+      },
+      cancelled: () => !this.stillWanted(job),
+    });
+    if (!this.stillWanted(job)) return;
+    const uploaded = await this.sounds.upload(data.slice().buffer, 'audio/ogg', streamUploadName(label));
+    // Nothing names the file yet, so it can go at once.
+    if (!this.stillWanted(job)) return void this.sounds.delete(uploaded.file).catch(() => {});
+    const old = this.ownerOf(job.assetId, job.clipId)?.stream?.file;
+    const frames = channels[0]?.length ?? 0;
+    // Done: what the Sound tab reads when the copy lands is that it streams, not how far it got.
+    this.job = null;
+    this.state.setStream(job.assetId, job.clipId, {
+      file: uploaded.file,
+      from: job.from,
+      frames,
+      bytes: uploaded.bytes,
+    });
+    if (old && old !== uploaded.file && !this.named(old)) dropWhenSaved(this.state, this.sounds, old);
+    this.state.setNotice(`Made a stream copy of ${label}`, 'quiet');
+  }
+
+  /** Whether the sound still streams and still plays what the copy is being made of. */
+  private stillWanted(job: StreamJob): boolean {
+    const asset = this.sound(job.assetId);
+    if (!asset || !streamsOf(asset.props)) return false;
+    if (job.clipId === null) return asset.file === job.from;
+    const clip = clipsOf(asset).find((c) => c.id === job.clipId);
+    return clip !== undefined && cutName(asset.file, cutOf(clip.props)) === job.from;
+  }
+
+  /** Whether a copy is still named by another sound or clip – a sound and its duplicate share one. */
+  private named(file: string): boolean {
+    return this.state.project.assets.some(
+      (asset) =>
+        asset.type === 'sound' &&
+        (asset.stream?.file === file || clipsOf(asset).some((clip) => clip.stream?.file === file)),
+    );
+  }
+
+  private sound(id: string): SoundAsset | null {
+    const asset = findAsset(this.state.project, id);
+    return asset?.type === 'sound' ? asset : null;
+  }
+
+  private ownerOf(assetId: string, clipId: string | null): { stream?: SoundAsset['stream'] } | null {
+    const asset = this.sound(assetId);
+    if (!asset) return null;
+    return clipId === null ? asset : (clipsOf(asset).find((clip) => clip.id === clipId) ?? null);
+  }
+
+  private label(job: StreamJob): string {
+    const asset = this.sound(job.assetId);
+    if (!asset) return 'a sound';
+    const clip = clipsOf(asset).find((c) => c.id === job.clipId) ?? null;
+    return soundRefName({ asset, clip });
+  }
+}
+
+function keyOf(job: StreamJob): string {
+  return `${job.assetId}:${job.clipId ?? ''}:${job.from}`;
+}

src/editor/state.ts

@@ -6,7 +6,7 @@ import type { ObjectPick, PickEnd } from './objectPick';
 import { findKey, findTrack, hasKey, type KeyRef, sameKey } from '../core/model/animation';
 import { type Command, CommandError } from '../core/commands/Command';
 import { CommandStack } from '../core/commands/CommandStack';
-import type { Asset, GraphAsset, SceneAsset } from '../core/model/Asset';
+import type { Asset, GraphAsset, SceneAsset, SoundStream } from '../core/model/Asset';
 import { findAnnotation, type Graph } from '../core/graph';
 import { findObject } from '../core/model/hierarchy';
 import { allShapes } from '../core/model/vectorTree';
@@ -443,6 +443,23 @@ export class EditorState {
     this.emit('project');
   }
 
+  /**
+   * The copy a sound or one of its clips streams from (#349), or none: saved, and like a bake no undo
+   * step – it is made again whenever it no longer fits.
+   */
+  setStream(assetId: string, clipId: string | null, stream: SoundStream | null): void {
+    for (const project of [this.project, this.edit?.project]) {
+      const asset = project && findAsset(project, assetId);
+      if (asset?.type !== 'sound') continue;
+      const owner = clipId === null ? asset : clipsOf(asset).find((clip) => clip.id === clipId);
+      if (!owner) continue;
+      if (stream) owner.stream = stream;
+      else delete owner.stream;
+    }
+    this.setSaveStatus('dirty');
+    this.emit('project');
+  }
+
   /**
    * Runs a command through the undo stack. Returns false (and sets a notice) when invalid.
    * In play mode the command changes the runtime copy only (see `enterPlay`).