Stream long sounds from an Opus copy instead of decoding them (#349)

A sound was decoded whole as soon as the project opened, in the editor
and in a released game: five minutes of stereo held some 115 MB for as
long as the page was open. A sound with Stream on now plays from a copy
as Opus in Ogg, fetched a piece at a time and decoded a few seconds
ahead by WebCodecs, so a game holds a few megabytes for it.

- The Stream switch is a row of the sound, off by default. The Sound
  tab suggests it from a minute on and says how far the copy has got.
- StreamKeeper makes the copies in the browser, a moment after the
  last change, one at a time: the sound's own and one per clip of its
  cut, fades and Reverse baked in. A copy that is no longer wanted is
  taken off and deleted once the project is saved without it.
- streamTable says which played names stream from which copy. A clip
  without a fitting copy, a Sample and a Reverb impulse stay decoded.
  A sound with Stream on is never decoded ahead.
- The page fetches by Range and takes the whole file from a server
  that ignores it, as the CI's PHP server does. It keeps the bytes
  and the first two seconds decoded, the rest goes four seconds ahead.
- A jump decodes 32 packets before the place first. Measured in
  Chrome, what follows is then the same to the bit as decoding from
  the start. With 4 packets it was 0.06 off.
- The stream-player on the FX worklet reads the samples at the rate
  of the context along a Catmull-Rom curve. What comes late is silent
  and skipped once it is there, so a clip keeps its beat.
- The opening stretch goes with the player as it is made. Sent by
  message it came a few milliseconds late, and 557 samples of the
  attack were lost.
- Pitch on a streamed sound is a shifter after the player, as on a
  bus: the length stays and it is heard 42.7 ms late.
- Bakes of patterns and songs (#340) always stream now.
- The song timeline draws a streamed sound from its copy decoded at
  8 kHz, keeping only the peaks.

Tests:
- Gate: CHECK EXIT 0, 280 files, 4312 tests (31 new: ogg +3,
  stream-queue 6, streams 11, streaming 6, sound-library +5).
- e2e 34: ALL PASSED, 106 checks (7 new). Five minutes of mono,
  4.8 MB as WAV, 3.6 MB as its copy: streamed the renderer grows by
  5.8 MB, decoded by 69.6 MB. Offline at 48 kHz, start, loop seam and
  a reversed clip are the copy decoded whole to the sample (0).
- Counter-checks red as they should be: no pre-roll, no catch-up of
  late samples at the end, no opening stretch, a streamed sound
  decoded ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/editor/sound/SoundEditor.ts

@@ -12,8 +12,10 @@ import {
   FADE_OUT_CURVE_KEY,
   FADE_OUT_KEY,
   REVERSE_KEY,
+  streamsOf,
   volumeOf,
 } from '../../core/properties/sounds';
+import { STREAM_SUGGESTED } from '../../core/music/streams';
 import { Button, Dropdown, Label, Menu, NumberField } from '../../ui';
 import { soundLibrary } from '../../viewport/SoundLibrary';
 import { MASTER_MIX } from '../../core/music/mixer';
@@ -26,6 +28,7 @@ import { claimTransport, releaseTransport } from '../music/transport';
 import { HEADER_ICON_SVG } from '../panels/headerIcons';
 import { TOOLBAR_ICON_SVG } from '../panels/toolbarIcons';
 import type { EditorState } from '../state';
+import type { StreamKeeper } from './StreamKeeper';
 import { WaveCanvas } from './WaveCanvas';
 
 const INFO_WIDTH = 140;
@@ -71,7 +74,10 @@ export class SoundEditor extends AssetTabEditor<SoundAsset, WaveCanvas> {
   private playingId = '';
   private readonly halt = () => this.stop();
 
-  constructor(state: EditorState) {
+  constructor(
+    state: EditorState,
+    private readonly streams: StreamKeeper,
+  ) {
     super(state, { infoWidth: INFO_WIDTH });
     this.playButton = new Button('Play', {
       icon: TOOLBAR_ICON_SVG.play,
@@ -125,6 +131,8 @@ export class SoundEditor extends AssetTabEditor<SoundAsset, WaveCanvas> {
     state.on('load', () => this.stop());
     // Picking a clip changes what the header plays and edits.
     state.on('selection', () => this.refresh());
+    // How far a stream copy has got (#349) comes as a notice, not as a change of the project.
+    state.on('notice', () => this.refresh());
   }
 
   override selectedId(): string {
@@ -251,7 +259,20 @@ export class SoundEditor extends AssetTabEditor<SoundAsset, WaveCanvas> {
     if (!asset.file) return 'Not uploaded';
     if (!span || whole === null) return '';
     const kept = span.end - span.start;
-    return kept >= whole ? `${whole.toFixed(2)} s` : `${kept.toFixed(2)} s of ${whole.toFixed(2)} s`;
+    const length = kept >= whole ? `${whole.toFixed(2)} s` : `${kept.toFixed(2)} s of ${whole.toFixed(2)} s`;
+    return `${length}${this.streamText(asset, whole)}`;
+  }
+
+  /**
+   * Whether it streams (#349): its copy is made, being made or yet to be; a long one that does not
+   * is pointed at the switch.
+   */
+  private streamText(asset: SoundAsset, whole: number): string {
+    const making = this.streams.making(asset.id);
+    if (making !== null) return ` · copy ${Math.round(making * 100)}%`;
+    if (streamsOf(resolveAssetProps(this.state.project, asset)))
+      return asset.stream?.from === asset.file ? ' · streamed' : ' · copy waiting';
+    return whole >= STREAM_SUGGESTED ? ' · long, try Stream' : '';
   }
 
   protected override syncExtras(asset: SoundAsset | null): void {

src/editor/sound/SoundPicker.ts

@@ -1,5 +1,6 @@
 import { Container, type FederatedPointerEvent, Graphics, type Text } from 'pixi.js';
 import { isBakeUpload } from '../../core/music/bake';
+import { isStreamUpload } from '../../core/music/streams';
 import type { StoredSound } from '../../core/store/SoundStore';
 import { drawSvg, formatBytes, textStyle, Theme, truncate, crispText } from '../../ui';
 import { soundLibrary } from '../../viewport/SoundLibrary';
@@ -121,7 +122,9 @@ class SoundPicker extends FilePicker<SoundItem> {
 
   /** A bake (#340) is in the sounds, but no sound to pick. */
   protected async fetchItems(): Promise<SoundItem[]> {
-    const sounds = (await this.deps.sounds.list()).filter((sound) => !isBakeUpload(sound.name));
+    const sounds = (await this.deps.sounds.list()).filter(
+      (sound) => !isBakeUpload(sound.name) && !isStreamUpload(sound.name),
+    );
     return withAssetNames(sounds, this.deps.state.project, 'sound');
   }
 

src/editor/sound/StreamKeeper.ts

@@ -0,0 +1,164 @@
+import type { SoundAsset } from '../../core/model/Asset';
+import { findAsset } from '../../core/model/Project';
+import { clipsOf, cutName, cutChannels, cutRange, parseCutName, soundRefName } from '../../core/model/sound';
+import { STREAM_RATE } from '../../core/music/streamQueue';
+import { type StreamJob, streamUploadName, streamWork } from '../../core/music/streams';
+import { cutOf, streamsOf } from '../../core/properties/sounds';
+import type { SoundStore } from '../../core/store/SoundStore';
+import { BakeCancelled, encodeOpus } from '../../viewport/bakeRender';
+import { soundLibrary } from '../../viewport/SoundLibrary';
+import { dropWhenSaved } from '../music/dropWhenSaved';
+import type { EditorState } from '../state';
+
+/** Milliseconds the project has to stay as it is before a copy is made: a clip dragged about is one. */
+const SETTLE_MS = 600;
+
+/**
+ * Makes the copies streamed sounds are played from (#349). A moment after the project last changed,
+ * each sound with `stream` on gets a copy of its file as Opus, and each of its clips one of its cut,
+ * one at a time, from the file decoded for it and let go again. A copy no longer wanted is taken off
+ * its sound and deleted once the project has been saved without it. Not while the game runs.
+ */
+export class StreamKeeper {
+  private job: { job: StreamJob; share: number } | null = null;
+  private timer: ReturnType<typeof setTimeout> | null = null;
+  /** Copies that could not be made, so one broken file is not tried over and over. */
+  private readonly failed = new Set<string>();
+  private waiting: (() => void)[] = [];
+
+  constructor(
+    private readonly state: EditorState,
+    private readonly sounds: SoundStore,
+  ) {
+    for (const event of ['project', 'load', 'mode'] as const) state.on(event, () => this.later());
+    this.later();
+  }
+
+  /** How far the copy of that sound, or of one of its clips, has got, 0 to 1; null while none is made. */
+  making(assetId: string): number | null {
+    return this.job?.job.assetId === assetId ? this.job.share : null;
+  }
+
+  /** Done once there is nothing left to make (test hook). */
+  idle(): Promise<void> {
+    if (!this.job && this.timer === null && this.pending().length === 0) return Promise.resolve();
+    return new Promise((resolve) => this.waiting.push(resolve));
+  }
+
+  private later(wait = SETTLE_MS): void {
+    if (this.timer !== null) clearTimeout(this.timer);
+    this.timer = setTimeout(() => {
+      this.timer = null;
+      void this.work();
+    }, wait);
+  }
+
+  private pending(): StreamJob[] {
+    return streamWork(this.state.project).jobs.filter((job) => !this.failed.has(keyOf(job)));
+  }
+
+  private async work(): Promise<void> {
+    if (this.job || this.state.mode === 'play') return;
+    for (const drop of streamWork(this.state.project).drops) {
+      const old = this.ownerOf(drop.assetId, drop.clipId)?.stream?.file;
+      this.state.setStream(drop.assetId, drop.clipId, null);
+      if (old && !this.named(old)) dropWhenSaved(this.state, this.sounds, old);
+    }
+    const next = this.pending()[0];
+    if (!next) {
+      for (const resolve of this.waiting.splice(0)) resolve();
+      return;
+    }
+    this.job = { job: next, share: 0 };
+    try {
+      await this.make(this.job);
+    } catch (error: unknown) {
+      // Called off because what it was made of changed: the next round makes the right one.
+      if (error instanceof BakeCancelled) return this.later(0);
+      this.failed.add(keyOf(next));
+      this.state.setNotice(
+        `No stream copy of ${this.label(next)}: ${error instanceof Error ? error.message : String(error)}`,
+        'error',
+      );
+    } finally {
+      this.job = null;
+    }
+    this.later(0);
+  }
+
+  private async make(running: { job: StreamJob; share: number }): Promise<void> {
+    const { job } = running;
+    const label = this.label(job);
+    const { file, cut } = parseCutName(job.from);
+    const whole = await soundLibrary.decodeApart(file, STREAM_RATE);
+    let channels = Array.from({ length: Math.min(2, whole.numberOfChannels) }, (_, c) =>
+      whole.getChannelData(c),
+    );
+    if (cut) channels = cutChannels(channels, cutRange(whole.length, whole.sampleRate, cut));
+    const data = await encodeOpus(channels, {
+      progress: (share) => {
+        const before = Math.round(running.share * 100);
+        running.share = share;
+        const now = Math.round(share * 100);
+        if (now !== before) this.state.setNotice(`Making a stream copy of ${label}: ${now}%`, 'quiet');
+      },
+      cancelled: () => !this.stillWanted(job),
+    });
+    if (!this.stillWanted(job)) return;
+    const uploaded = await this.sounds.upload(data.slice().buffer, 'audio/ogg', streamUploadName(label));
+    // Nothing names the file yet, so it can go at once.
+    if (!this.stillWanted(job)) return void this.sounds.delete(uploaded.file).catch(() => {});
+    const old = this.ownerOf(job.assetId, job.clipId)?.stream?.file;
+    const frames = channels[0]?.length ?? 0;
+    // Done: what the Sound tab reads when the copy lands is that it streams, not how far it got.
+    this.job = null;
+    this.state.setStream(job.assetId, job.clipId, {
+      file: uploaded.file,
+      from: job.from,
+      frames,
+      bytes: uploaded.bytes,
+    });
+    if (old && old !== uploaded.file && !this.named(old)) dropWhenSaved(this.state, this.sounds, old);
+    this.state.setNotice(`Made a stream copy of ${label}`, 'quiet');
+  }
+
+  /** Whether the sound still streams and still plays what the copy is being made of. */
+  private stillWanted(job: StreamJob): boolean {
+    const asset = this.sound(job.assetId);
+    if (!asset || !streamsOf(asset.props)) return false;
+    if (job.clipId === null) return asset.file === job.from;
+    const clip = clipsOf(asset).find((c) => c.id === job.clipId);
+    return clip !== undefined && cutName(asset.file, cutOf(clip.props)) === job.from;
+  }
+
+  /** Whether a copy is still named by another sound or clip – a sound and its duplicate share one. */
+  private named(file: string): boolean {
+    return this.state.project.assets.some(
+      (asset) =>
+        asset.type === 'sound' &&
+        (asset.stream?.file === file || clipsOf(asset).some((clip) => clip.stream?.file === file)),
+    );
+  }
+
+  private sound(id: string): SoundAsset | null {
+    const asset = findAsset(this.state.project, id);
+    return asset?.type === 'sound' ? asset : null;
+  }
+
+  private ownerOf(assetId: string, clipId: string | null): { stream?: SoundAsset['stream'] } | null {
+    const asset = this.sound(assetId);
+    if (!asset) return null;
+    return clipId === null ? asset : (clipsOf(asset).find((clip) => clip.id === clipId) ?? null);
+  }
+
+  private label(job: StreamJob): string {
+    const asset = this.sound(job.assetId);
+    if (!asset) return 'a sound';
+    const clip = clipsOf(asset).find((c) => c.id === job.clipId) ?? null;
+    return soundRefName({ asset, clip });
+  }
+}
+
+function keyOf(job: StreamJob): string {
+  return `${job.assetId}:${job.clipId ?? ''}:${job.from}`;
+}