Record every frame with Real time off on Start recording (#423)

At 4K and 60 fps the user's computer reaches about 39 fps in real time.
Reading each frame back from the GPU takes longer than a frame, and
neither reading back later nor the canvas route helped. With Real time
off the run moves on by one frame of the recording at a time, however
long the computer takes, so the file has every frame.

- Start recording has a new check Real time, on by default. Only the
  node records so, the editor recording stays in real time.
- SteadyPace in recording.ts is the pace as a pure class. A step is
  1/fps once the frame before is taken and the sound up to it is made,
  otherwise 0 and the run holds. A frame's time is its number.
- The sound is made apart, as decided in the issue. AudioSwitch routes
  what the run plays to the live library, or to the offline library of
  a SteadySound while such a recording lasts. That library has the same
  mix, buses, FX and limiter and is rendered up to each next frame.
- A driven video holds at every picture asked for, and the frame waits
  for it. Its sound is decoded from the MP4 and played in step through
  its bus.
- Two Chrome quirks measured and worked around. A held video hands
  over the picture of a seek before seeked, so the frame callback is
  asked for when the seek begins. A seek right onto where a picture
  begins shows that one or the one before, so such a seek lands half a
  millisecond further. Together 30 s became 10.7 s for 2 s of video.
- The notice tells how far the recording has got, and the line at the
  end how long it took.

Deviations and limits:

- Input is not ignored, it still reaches the game, which runs slower.
- At most ten minutes per recording. Chrome holds the whole buffer of
  the OfflineAudioContext while rendering, 230 MB for ten minutes.
- A free video plays on in real time and does not fit. A sound playing
  before the start stays at the speakers.

Found along the way:

- A check in e2e 60 was red since 77365f1. It expected one log line,
  the start line of the recording came on top. CI was off, so nobody
  saw it. The expectation is updated.

Tests:

- First gate run red: 1 test in node-catalog.test.ts still expected
  the settings of Start recording without realTime. Updated.
- Second gate run on melektaus-atx green: 310 files,
  4818 tests.
- e2e 66 + 68 all passed, 50 checks, 68j new with 3 checks. ffmpeg
  reads all 120 frames 1/60 s apart, the colour changes at frames
  30/60/90, the video's sound from the first frame, the blip 8 ms off.
- e2e 60 (20 checks) and 67 green.
- Counter-check without the pace: all three 68j checks red, 32 of 119
  frames, colour changes at 33/62/96, no sound. AudioSwitch's unit test
  goes red when the ids are mixed up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-14 12:57:15 +02:00
1 parent 77365f17d3
commit 87b2ee9402
22 files changed
+1228 -64

No files matched your search

+54 -5
View File
@@ -25,7 +25,16 @@ import {
import { videoDrivenOf, videoLevelOf, videoTimeOf } from '../core/properties/videos';
import { FileLibrary } from './FileLibrary';
import { soundLibrary } from './SoundLibrary';
import { followVideo, readSpeed, type SpeedReading } from './videoFollow';
import type { HeardVideo } from './SteadySound';
import {
followVideo,
PICTURE_NUDGE,
pictureShown,
readSpeed,
type ShownPicture,
showsPictureOf,
type SpeedReading,
} from './videoFollow';
import { chaseSeeks, seekVideo, type SeekChase } from './videoSeek';
const VIDEO_BASE = dataPath('videos');
@@ -74,6 +83,8 @@ interface Drive extends SeekChase {
/** How loud the wearer hears it, 0–1, on top of the video's own volume: its Video volume. */
level: number;
reading: SpeedReading | null;
/** The picture it was held at last, as Chrome handed it over; null while it plays. */
shown: ShownPicture | null;
}
interface Player {
@@ -99,6 +110,8 @@ export class VideoLibrary {
private readonly listeners = new Set<() => void>();
private heard = false;
private paused = false;
/** Driven players hold at every picture asked rather than play: a recording takes each (#423). */
private stepped = false;
private sweeper: ReturnType<typeof setInterval> | null = null;
private husher: ReturnType<typeof setInterval> | null = null;
/** When the page was last shown, by `performance.now()`; see `sweep`. */
@@ -207,21 +220,54 @@ export class VideoLibrary {
return [...this.players.values()].filter((player) => !player.drive);
}
/**
* While a recording not in real time goes (#423), every driven player holds at the picture its
* wearer asks for and seeks to the next: the frames come slower than it would play, and each has
* to show its own picture. Their sound is the recording's business then.
*/
setStepped(stepped: boolean): void {
this.stepped = stepped;
}
/** What a driven player would let be heard in a game now (#423); null for one that is not. */
heardOf(key: string): HeardVideo | null {
const player = this.players.get(key);
const drive = player?.drive;
if (!player || !drive || player.gone || drive.heard !== 'game') return null;
const { play, element } = player;
return {
key,
file: play.file,
time: drivenTimeOf(play, drive.time, element.duration),
level: play.volume * drive.level,
...(play.bus !== undefined && { bus: play.bus }),
};
}
private follow(player: Player, drive: Drive): void {
const { element, play } = player;
const step = followVideo({
target: drivenTimeOf(play, drive.time, element.duration),
speed: drive.reading?.speed ?? 0,
speed: this.stepped ? 0 : (drive.reading?.speed ?? 0),
current: element.currentTime,
// A video that ran to the end of its file starts over from its first picture on play().
end: element.ended ? element.currentTime : Math.min(play.end, element.duration),
});
if (step.seek !== null) seekVideo(element, drive, step.seek);
// Held frame by frame for a recording (#423), a seek within the picture shown brings no other.
const shown =
this.stepped &&
step.seek !== null &&
element.paused &&
!drive.landing &&
showsPictureOf(drive.shown, step.seek);
if (step.seek !== null && !shown)
seekVideo(element, drive, this.stepped ? step.seek + PICTURE_NUDGE : step.seek);
if (step.kind === 'hold') {
element.muted = true;
if (!element.paused) element.pause();
return;
}
drive.shown = null;
if (Math.abs(element.playbackRate - step.rate) > 0.001) element.playbackRate = step.rate;
if (step.heard && drive.heard !== 'none') this.hear(player, drive.heard === 'game');
else element.muted = true;
@@ -315,7 +361,7 @@ export class VideoLibrary {
const play = parseVideoKey(key);
if (!play) return null;
const drive: Drive | null = isDrivenVideoKey(key)
? { time, heard: 'none', level: 1, reading: null, wanted: null, landing: false }
? { time, heard: 'none', level: 1, reading: null, wanted: null, landing: false, shown: null }
: null;
const at = drive ? drivenTimeOf(play, time, NaN) : play.start;
const player: Player = {
@@ -344,7 +390,10 @@ export class VideoLibrary {
player.element,
drive,
() => player.gone,
() => player.texture?.source.update(),
(shown) => {
player.texture?.source.update();
if (shown !== null && player.element.paused) drive.shown = pictureShown(drive.shown, shown);
},
);
else {
this.watch(player);