Play videos on sprites, converted on the server with clips (#402)

A video becomes an asset like a sound: uploaded once, converted to one
format by a worker of its own, cut into clips in its tab, and worn by a
sprite through its Texture row. Only Chrome is a target, without alpha.

- Server: POST ?r=videos queues the upload under the hash of its bytes,
  so the MP4's name is known at once. The worker (VideoWorker over
  VideoFormat) writes H.264/AAC, yuv420p, constant frame rate, a
  keyframe every 15 frames, at most 1920x1080, with a poster and the
  details beside it. A failed job says why and is tried again on the
  next upload.
- An upload may be up to 2 GB. It is streamed from the request to disk
  and hashed there, so php-fpm holds none of it in memory. The other
  routes keep their 256 MB.
- The worker runs in the container game-maker-video on stein2, a
  compose project of its own (tools/video/), and in CI beside the PHP
  server. The CI image has ffmpeg for it.
- Model: asset type video with size, length, frame rate and sound, and
  clips like a sound's. The clip helpers and commands serve both.
- A sprite's Texture row offers videos and clips after the pictures.
  resolvePicture names a playing video by a key, and the texture library
  hands it to a VideoLibrary with one player per key.
- The editor shows a placed video as a quiet preview. Play starts it
  over at its clip's start and lets it be heard through the master bus,
  a pause holds it, Esc makes it quiet again.
- Video tab: the picture, a ruler, the clip strip and a track on the
  Sound tab's time axis. A drag spans a clip, a handle moves an edge,
  Play plays the picked clip.
- The picker shows a video on its way with its progress and adopts an
  upload from it once it is ready. Dropping a video file does the same.

Decided on my own:

- Two sprites wearing the same clip share one player and show the same
  picture. Control per instance belongs to the follow-up with nodes.
- A video plays in real time, the game's time scale does not slow it.
- The container may use 3 of the 4 cores of stein2.

Findings:

- The first real upload (279 MB) got 413: the API's own ceiling of
  256 MB for every request, not nginx or the Brandmauer.
- Chrome has no picture of a video that never played, and a Pixi
  VideoSource drawn before the first frame stays empty for good. The
  texture is made in the first frame callback (openVideo).
- A player nobody asks for in 2 s is let go. The inspector's PREVIEW
  asked only on an edit, so with the sprite not on screen its texture
  was destroyed under it and the batcher threw on every frame. The
  preview now redraws every frame while it shows a video, like a Time
  shader. A let-go texture keeps an empty source instead of none.
- VideoSource.load() reloads the element and cuts off an earlier
  play(). Playing starts after loadeddata.

Tests:

- PHP videos.test.php: 68 checks, with the real ffmpeg
- 300 MB upload through Brandmauer and nginx: 201 in 21 s, php-fpm at
  13.7 MB RSS at most
- Unit videos.test.ts: 20 tests
- e2e 66 (stein2, atx was off): ALL PASSED, 18 checks; e2e 04 with the
  new asset count green
- Gate (atx): 294 files, 4520 tests. serialize.test.ts used video as
  its example of an unknown type and now uses a made-up one.
- Counter-checks: a keyframe gap of 30 fails the PHP check, a key that
  keeps the video's end fails 2 unit tests, a clip without its end
  fails "plays only its stretch" in e2e 66

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/core/commands/assetCommands.ts

@@ -10,6 +10,7 @@ import type {
   TextureAsset,
   TilemapLayer,
   VariablesAsset,
+  VideoAsset,
 } from '../model/Asset';
 import type { AssetType } from '../model/AssetTypes';
 import { embeddedIn } from '../music/embedded';
@@ -33,6 +34,7 @@ import {
 } from '../model/Scope';
 import type { SceneObject } from '../model/SceneObject';
 import { isTextureFile } from '../model/texture';
+import { isVideoFile } from '../model/video';
 import { cellsWithout } from '../model/tilemap';
 import { ASSET_FX_KEY } from '../properties/assetFx';
 import { coerceValue, defaultsFor, initialOf, initialPropsFor } from '../properties/PropertySchema';
@@ -108,6 +110,8 @@ export function buildAsset(type: AssetType, name: string, id = newId()): Asset {
       return { ...base, type: 'font', file: '' };
     case 'sound':
       return { ...base, type: 'sound', file: '' };
+    case 'video':
+      return { ...base, type: 'video', file: '', width: 0, height: 0, duration: 0, fps: 0, audio: false };
     case 'soundfx':
       return { ...base, type: 'soundfx', graph: createEmptyGraph() };
     case 'instrument':
@@ -311,6 +315,25 @@ export class CreateSoundCommand extends ImportFileCommand<SoundAsset> {
   }
 }
 
+/** What a video says about itself once the worker has made it (#402), copied into the asset. */
+export interface VideoDetails {
+  width: number;
+  height: number;
+  duration: number;
+  fps: number;
+  audio: boolean;
+}
+
+export class CreateVideoCommand extends ImportFileCommand<VideoAsset> {
+  constructor(name: string, file: string, details: VideoDetails) {
+    super(
+      { id: newId(), name, props: defaultsFor('video'), type: 'video', file, ...details },
+      'video',
+      isVideoFile,
+    );
+  }
+}
+
 /**
  * Makes an asset a song kept to itself a project asset (#355): it shows in the lists, and a name a
  * listed asset has already gets a number.

src/core/commands/clipCommands.ts

@@ -1,29 +1,41 @@
-/** The clips of a sound (#357): stretches of its file with a name, as frames are of a texture. */
+/**
+ * The clips of a sound (#357) or a video (#402): stretches of its file with a name, as frames are
+ * of a texture.
+ */
 import { newId } from '../ids';
-import type { SoundAsset, SoundClip } from '../model/Asset';
+import type { SoundClip } from '../model/Asset';
 import type { PropertyBag, PropertyValue } from '../model/Properties';
 import type { Project } from '../model/Project';
-import { clipsOf, nextClipName, uniqueClipName } from '../model/sound';
+import { type ClipOwner, clipsOf, hasClips, nextClipName, uniqueClipName } from '../model/sound';
+import type { PropertyDef } from '../properties/PropertySchema';
 import { CLIP_ROWS } from '../properties/sounds';
+import { VIDEO_CLIP_ROWS } from '../properties/videos';
 import { type Removed, removeFirst, restoreAt } from '../util/list';
 import { type Command, CommandError } from './Command';
 import { requireAsset } from './require';
 
-function requireSound(project: Project, assetId: string): SoundAsset {
+/** A clip is the same shape on a sound and on a video; only its rows differ. */
+type Clip = SoundClip;
+
+function requireSound(project: Project, assetId: string): ClipOwner {
   const asset = requireAsset(project, assetId);
-  if (asset.type !== 'sound') throw new CommandError(`${asset.name} is not a sound`);
+  if (!hasClips(asset)) throw new CommandError(`${asset.name} has no clips`);
   return asset;
 }
 
-function requireClip(asset: SoundAsset, clipId: string): SoundClip {
-  const clip = clipsOf(asset).find((c) => c.id === clipId);
+function requireClip(asset: ClipOwner, clipId: string): Clip {
+  const clip = clipsOf<Clip>(asset).find((c) => c.id === clipId);
   if (!clip) throw new CommandError(`Clip ${clipId} not found in ${asset.name}`);
   return clip;
 }
 
-/** What a clip may hold: a row of `CLIP_ROWS`, a finite number at least 0, a curve it offers. */
-function heldClipValue(key: string, value: PropertyValue): PropertyValue {
-  const def = CLIP_ROWS.find((row) => row.key === key);
+function clipRowsOf(asset: ClipOwner): readonly PropertyDef[] {
+  return asset.type === 'video' ? VIDEO_CLIP_ROWS : CLIP_ROWS;
+}
+
+/** What a clip may hold: a row of its asset's clip rows, a finite number at least 0, a curve it offers. */
+function heldClipValue(asset: ClipOwner, key: string, value: PropertyValue): PropertyValue {
+  const def = clipRowsOf(asset).find((row) => row.key === key);
   if (!def) throw new CommandError(`A clip has no "${key}"`);
   if (def.kind === 'number' && (typeof value !== 'number' || !Number.isFinite(value) || value < 0))
     throw new CommandError(`${def.label} needs a number of at least 0`);
@@ -47,14 +59,14 @@ export class AddClipCommand implements Command {
   execute(project: Project): void {
     const asset = requireSound(project, this.assetId);
     const props: PropertyBag = {};
-    for (const [key, value] of Object.entries(this.props)) props[key] = heldClipValue(key, value);
+    for (const [key, value] of Object.entries(this.props)) props[key] = heldClipValue(asset, key, value);
     // Named per execution: a redo has to fit the names that exist when it runs.
-    asset.clips = [...clipsOf(asset), { id: this.clipId, name: nextClipName(asset), props }];
+    asset.clips = [...clipsOf<Clip>(asset), { id: this.clipId, name: nextClipName(asset), props }];
   }
 
   undo(project: Project): void {
     const asset = requireSound(project, this.assetId);
-    asset.clips = clipsOf(asset).filter((c) => c.id !== this.clipId);
+    asset.clips = clipsOf<Clip>(asset).filter((c) => c.id !== this.clipId);
   }
 }
 
@@ -73,8 +85,11 @@ export class SetClipPropsCommand implements Command {
   ) {}
 
   execute(project: Project): void {
-    const clip = requireClip(requireSound(project, this.assetId), this.clipId);
-    const held = Object.entries(this.values).map(([key, value]) => [key, heldClipValue(key, value)] as const);
+    const asset = requireSound(project, this.assetId);
+    const clip = requireClip(asset, this.clipId);
+    const held = Object.entries(this.values).map(
+      ([key, value]) => [key, heldClipValue(asset, key, value)] as const,
+    );
     this.previous = held.map(([key]) => [key, clip.props[key]]);
     for (const [key, value] of held) clip.props[key] = value;
   }
@@ -117,7 +132,7 @@ export class RenameClipCommand implements Command {
  */
 export class RemoveClipCommand implements Command {
   readonly label = 'Delete clip';
-  private removed?: Removed<SoundClip>;
+  private removed?: Removed<Clip>;
 
   constructor(
     private readonly assetId: string,
@@ -126,7 +141,7 @@ export class RemoveClipCommand implements Command {
 
   execute(project: Project): void {
     const asset = requireSound(project, this.assetId);
-    const clips = [...clipsOf(asset)];
+    const clips = [...clipsOf<Clip>(asset)];
     const removed = removeFirst(clips, (c) => c.id === this.clipId);
     if (!removed) throw new CommandError(`Clip ${this.clipId} not found in ${asset.name}`);
     this.removed = removed;
@@ -136,7 +151,7 @@ export class RemoveClipCommand implements Command {
   undo(project: Project): void {
     if (!this.removed) return;
     const asset = requireSound(project, this.assetId);
-    const clips = [...clipsOf(asset)];
+    const clips = [...clipsOf<Clip>(asset)];
     restoreAt(clips, [this.removed]);
     asset.clips = clips;
   }

src/core/model/Asset.ts

@@ -348,6 +348,31 @@ export interface SoundClip {
   stream?: SoundStream;
 }
 
+/**
+ * `file` names the video in `data/videos/` (#402): `{folder}/{sha256}.mp4`, the MP4 the worker made
+ * from the upload, with a poster `.jpg` and its details beside it. The size, length, frame rate and
+ * whether it has sound are copied in when it is picked, so a sprite can be measured before the file
+ * loads. A sprite wears it through `texture`; its clips are stretches of it, as a sound's are.
+ */
+export interface VideoAsset extends AssetBase {
+  type: 'video';
+  file: string;
+  width: number;
+  height: number;
+  duration: number;
+  fps: number;
+  audio: boolean;
+  /** Absent for a video without clips. */
+  clips?: VideoClip[];
+}
+
+/** A stretch of a video with a name, `props` its start and end (see `properties/videos.ts`). */
+export interface VideoClip {
+  id: string;
+  name: string;
+  props: PropertyBag;
+}
+
 /**
  * What a sound is played through (#311): the graph runs from its `Input` over effect nodes to its
  * `Output`, and a `Play sound` naming it sends the voice along that line (see `core/soundfx`).
@@ -658,7 +683,8 @@ export type Asset =
   | PatternAsset
   | SongAsset
   | BusAsset
-  | ParticlesAsset;
+  | ParticlesAsset
+  | VideoAsset;
 
 /** The assets whose `variables` are declared by the user – see `requireVariableOwner`. */
 export type VariableOwnerAsset = BehaviorAsset | VariablesAsset | StructAsset;

src/core/model/AssetTypes.ts

@@ -30,6 +30,7 @@ export const ASSET_TYPES = [
   'song',
   'bus',
   'particles',
+  'video',
 ] as const;
 
 export type AssetType = (typeof ASSET_TYPES)[number];
@@ -66,6 +67,7 @@ export const ASSET_TYPE_LABELS: Record<AssetType, string> = {
   song: 'Song',
   bus: 'Bus',
   particles: 'Particles',
+  video: 'Video',
 };
 
 /**
@@ -79,7 +81,8 @@ export const ASSET_TYPE_LABELS: Record<AssetType, string> = {
  * A shader function is part of other shaders (#303), worn by nothing itself. A sound FX is what a
  * sound is played through (#311), heard but never seen, an instrument what a note is played
  * with (#331), a pattern the notes it plays (#333) and a song what lays them out (#334). A bus is
- * where sounds are mixed on their way out (#325).
+ * where sounds are mixed on their way out (#325). A video is a moving picture a sprite wears through
+ * `texture` (#402), as it wears an image.
  */
 export type PlaceableAssetType = Exclude<
   AssetType,
@@ -103,6 +106,7 @@ export type PlaceableAssetType = Exclude<
   | 'bus'
   | 'variables'
   | 'struct'
+  | 'video'
 >;
 
 const NOT_PLACEABLE: ReadonlySet<AssetType> = new Set([
@@ -126,6 +130,7 @@ const NOT_PLACEABLE: ReadonlySet<AssetType> = new Set([
   'bus',
   'variables',
   'struct',
+  'video',
 ]);
 
 export function isPlaceableType(type: AssetType): type is PlaceableAssetType {

src/core/model/sound.ts

@@ -1,6 +1,6 @@
 import { nextFreeName, nextNumberedName } from '../commands/naming';
 import { newId } from '../ids';
-import type { Asset, SoundAsset, SoundClip } from './Asset';
+import type { Asset, SoundAsset, SoundClip, VideoAsset } from './Asset';
 import { findAsset, type Project } from './Project';
 import { storedFilePattern } from './storedFile';
 
@@ -13,10 +13,18 @@ export function isSoundFile(name: string): boolean {
 const CLIP_SEPARATOR = ' / ';
 const DEFAULT_CLIP_NAME = 'Clip';
 
-export function clipsOf(asset: SoundAsset): readonly SoundClip[] {
+/** The clips of a sound (#357) or a video (#402). */
+export function clipsOf<C>(asset: { clips?: C[] }): readonly C[] {
   return asset.clips ?? [];
 }
 
+/** What has clips: a sound and a video, each a file with named stretches of it. */
+export type ClipOwner = SoundAsset | VideoAsset;
+
+export function hasClips(asset: Asset): asset is ClipOwner {
+  return asset.type === 'sound' || asset.type === 'video';
+}
+
 /** What a sound field names (#357): a sound, the whole file, or one of its clips. */
 export interface SoundRef {
   asset: SoundAsset;
@@ -39,10 +47,12 @@ export function findSound(project: Project, id: string): SoundRef | null {
   return null;
 }
 
-/** The asset among `assets` an id names: the asset itself, or the sound one of whose clips it is. */
+/** The asset among `assets` an id names: the asset itself, or the one one of whose clips it is. */
 export function assetNamedBy(assets: readonly Asset[], id: string): Asset | undefined {
   if (id === '') return undefined;
-  return assets.find((a) => a.id === id || (a.type === 'sound' && clipsOf(a).some((c) => c.id === id)));
+  return assets.find(
+    (a) => a.id === id || (hasClips(a) && clipsOf<{ id: string }>(a).some((c) => c.id === id)),
+  );
 }
 
 /**
@@ -50,8 +60,8 @@ export function assetNamedBy(assets: readonly Asset[], id: string): Asset | unde
  * each old id with its new one in `ids` so the places that named a clip can follow.
  */
 export function renewClipIds(copy: Asset, ids: Map<string, string>): void {
-  if (copy.type !== 'sound') return;
-  for (const clip of clipsOf(copy)) {
+  if (!hasClips(copy)) return;
+  for (const clip of clipsOf<{ id: string }>(copy)) {
     const id = newId();
     ids.set(clip.id, id);
     clip.id = id;
@@ -63,18 +73,18 @@ export function soundRefName(ref: SoundRef, soundName = ref.asset.name): string
   return ref.clip ? `${soundName}${CLIP_SEPARATOR}${ref.clip.name}` : soundName;
 }
 
-/** "Clip 1", 2, 3 …: the first number none of the sound's clips has. */
-export function nextClipName(asset: SoundAsset): string {
+/** "Clip 1", 2, 3 …: the first number none of the asset's clips has. */
+export function nextClipName(asset: ClipOwner): string {
   return nextNumberedName(
-    clipsOf(asset).map((c) => c.name),
+    clipsOf<{ name: string }>(asset).map((c) => c.name),
     DEFAULT_CLIP_NAME,
   );
 }
 
-/** A name free among the sound's clips: `name`, else "name 2", "name 3" … */
-export function uniqueClipName(asset: SoundAsset, name: string, ignoreId = ''): string {
+/** A name free among the asset's clips: `name`, else "name 2", "name 3" … */
+export function uniqueClipName(asset: ClipOwner, name: string, ignoreId = ''): string {
   return nextFreeName(
-    clipsOf(asset)
+    clipsOf<{ id: string; name: string }>(asset)
       .filter((c) => c.id !== ignoreId)
       .map((c) => c.name),
     name,

src/core/model/texture.ts

@@ -4,6 +4,7 @@ import { findPixelFrame, pixelFrameRect } from './pixelart';
 import { type PropertyBag, textOf, type Vec2 } from './Properties';
 import { findAsset, type Project } from './Project';
 import { storedFilePattern } from './storedFile';
+import { findVideo, isVideoFile, videoKey, videoPlayOf, type VideoRef } from './video';
 
 const FILE_PATTERN = storedFilePattern('png|jpg|webp|gif');
 
@@ -95,13 +96,17 @@ export function resolveTexture(project: Project, props: PropertyBag): TextureRef
   return resolvePicture(project, textOf(props.texture ?? ''));
 }
 
-/** A texture, a frame or a generating asset; any other asset's id is no picture. */
+/** A texture, a frame, a video or a clip of one, or a generating asset; any other asset's id is no picture. */
 export function resolvePicture(project: Project, id: string): TextureRef | null {
   if (!id) return null;
   const asset = findAsset(project, id);
-  if (asset) return asset.type === 'texture' ? refFor(asset, null) : generatedRef(asset, project);
+  if (asset?.type === 'texture') return refFor(asset, null);
+  if (asset?.type === 'video') return videoRef({ asset, clip: null });
+  if (asset) return generatedRef(asset, project);
   const found = findFrame(project, id);
   if (found) return refFor(found.asset, found.frame);
+  const video = findVideo(project, id);
+  if (video) return videoRef(video);
   // A frame of pixel art (#130) is a cut of its generated sheet, as a texture frame is of its image.
   const pixel = findPixelFrame(project, id);
   return pixel ? generatedRef(pixel.asset, project, pixelFrameRect(pixel.asset, pixel.index)) : null;
@@ -128,6 +133,19 @@ function generatedRef(asset: Asset, project: Project, rect: FrameRect | null = n
   };
 }
 
+/** A playing video (#402) reads like a texture of its size; the key says what plays, see `videoKey`. */
+function videoRef(ref: VideoRef): TextureRef | null {
+  if (!isVideoFile(ref.asset.file)) return null;
+  return {
+    file: videoKey(videoPlayOf(ref)),
+    smooth: true,
+    rect: null,
+    sourceWidth: ref.asset.width,
+    sourceHeight: ref.asset.height,
+    slice: null,
+  };
+}
+
 function refFor(asset: TextureAsset, frame: TextureFrame | null): TextureRef | null {
   if (!isTextureFile(asset.file)) return null;
   return {

src/core/model/video.ts

@@ -0,0 +1,110 @@
+import { videoLoopsOf, videoSpanOf, videoVolumeOf } from '../properties/videos';
+import type { VideoAsset, VideoClip } from './Asset';
+import { findAsset, type Project } from './Project';
+import { clipsOf } from './sound';
+import { storedFilePattern } from './storedFile';
+
+const FILE_PATTERN = storedFilePattern('mp4');
+
+export function isVideoFile(name: string): boolean {
+  return FILE_PATTERN.test(name);
+}
+
+/** The still the worker cut from a video (#402), lying beside it. */
+export function posterFileOf(file: string): string {
+  return file.replace(/\.mp4$/, '.jpg');
+}
+
+/** What a texture field names when it names a video: the whole video, or one of its clips. */
+export interface VideoRef {
+  asset: VideoAsset;
+  clip: VideoClip | null;
+}
+
+/** A video or a clip of one, by id; clip ids share the generator with asset ids, as frames do. */
+export function findVideo(project: Project, id: string): VideoRef | null {
+  if (id === '') return null;
+  const asset = findAsset(project, id);
+  if (asset) return asset.type === 'video' ? { asset, clip: null } : null;
+  for (const video of project.assets) {
+    if (video.type !== 'video') continue;
+    const clip = clipsOf(video).find((c) => c.id === id);
+    if (clip) return { asset: video, clip };
+  }
+  return null;
+}
+
+/** Every id that means "this video": the asset itself and each of its clips. */
+export function videoRefIds(asset: VideoAsset): string[] {
+  return [asset.id, ...clipsOf(asset).map((c) => c.id)];
+}
+
+/**
+ * The stretch a clip plays, inside the video: a start before its end, an end of 0 or past the
+ * video's length being its end. Without a length (not known yet) the asked values stand.
+ */
+export function videoSpan(duration: number, start: number, end: number): { start: number; end: number } {
+  const length = duration > 0 ? duration : Infinity;
+  const from = Math.min(Math.max(0, start), Number.isFinite(length) ? Math.max(0, length - 0.001) : start);
+  const to = end > 0 ? Math.min(end, length) : length;
+  return { start: from, end: to > from ? to : length };
+}
+
+/** How a video is played: which file, which stretch of it, whether it repeats, how loud. */
+export interface VideoPlay {
+  file: string;
+  start: number;
+  /** Seconds; Infinity for the end of the video, wherever that is. */
+  end: number;
+  loop: boolean;
+  volume: number;
+}
+
+export function videoPlayOf(ref: VideoRef): VideoPlay {
+  const asked = ref.clip ? videoSpanOf(ref.clip.props) : { start: 0, end: 0 };
+  const span = videoSpan(ref.asset.duration, asked.start, asked.end);
+  return {
+    file: ref.asset.file,
+    start: span.start,
+    end: ref.asset.duration > 0 && span.end >= ref.asset.duration ? Infinity : span.end,
+    loop: videoLoopsOf(ref.asset.props),
+    volume: videoVolumeOf(ref.asset.props),
+  };
+}
+
+const PREFIX = 'video:';
+
+/**
+ * The name a playing video goes by where a picture is named by `TextureRef.file` (see
+ * `generated.ts`): `video:<file>#<start>~<end>~<loop>~<volume>`. No uploaded image looks like it,
+ * and one key is one player: two sprites wearing the same clip show the same picture.
+ */
+export function videoKey(play: VideoPlay): string {
+  const end = Number.isFinite(play.end) ? play.end : 0;
+  return `${PREFIX}${play.file}#${play.start}~${end}~${play.loop ? 1 : 0}~${play.volume}`;
+}
+
+export function isVideoKey(key: string): boolean {
+  return parseVideoKey(key) !== null;
+}
+
+export function parseVideoKey(key: string): VideoPlay | null {
+  if (!key.startsWith(PREFIX)) return null;
+  const hash = key.indexOf('#');
+  if (hash < 0) return null;
+  const file = key.slice(PREFIX.length, hash);
+  const parts = key.slice(hash + 1).split('~');
+  const [start, end, loop, volume] = parts.map(Number);
+  if (!isVideoFile(file) || parts.length !== 4 || ![start, end, loop, volume].every(Number.isFinite))
+    return null;
+  return { file, start: start!, end: end! > 0 ? end! : Infinity, loop: loop === 1, volume: volume! };
+}
+
+/** Where in its file a player is after `elapsed` seconds of playing (#402). */
+export function videoTimeAt(play: Pick<VideoPlay, 'start' | 'end' | 'loop'>, elapsed: number): number {
+  const length = play.end - play.start;
+  const run = Math.max(0, elapsed);
+  if (!Number.isFinite(length) || length <= 0) return play.start + run;
+  if (!play.loop) return play.start + Math.min(run, length);
+  return play.start + (run % length);
+}

src/core/properties/PropertySchema.ts

@@ -129,6 +129,7 @@ import {
 import { DEFAULT_SIGN, DEFAULT_TEMPO, MAX_TEMPO, MIN_TEMPO, SIGN_OPTIONS } from '../music/time';
 import { ASSET_FX_KEY } from './assetFx';
 import { BUS_FX_KEY, BUS_VOLUME_KEY } from './buses';
+import { VIDEO_ROWS } from './videos';
 import { Theme } from '../../ui';
 
 /**
@@ -138,7 +139,7 @@ import { Theme } from '../../ui';
  * `assetTypes` an asset picker offers every placeable asset; `none` is what it calls an empty pick,
  * `master` offers the master bus as a choice of its own, `noFx` "None" as one where empty means
  * "from the asset" (#352); `uploadedOnly` keeps drawn pictures out
- * of a texture picker (#186). `renderLayers` makes a layer row pick from the project's render layers
+ * of a texture picker (#186), `moving` lets videos and their clips in (#402). `renderLayers` makes a layer row pick from the project's render layers
  * instead of its collision layers (#302), with "Inherit" as the first choice of a single layer.
  */
 type Extras<K extends FlatKind> = K extends 'number'
@@ -153,7 +154,7 @@ type Extras<K extends FlatKind> = K extends 'number'
           noFx?: boolean;
         }
       : K extends 'texture'
-        ? { uploadedOnly?: boolean }
+        ? { uploadedOnly?: boolean; moving?: boolean }
         : K extends 'layer' | 'layers'
           ? { renderLayers?: boolean }
           : unknown;
@@ -529,7 +530,8 @@ const ASSET_SCHEMAS: Record<AssetType, PropertySchema> = {
     { key: 'height', label: 'Height', kind: 'number', default: 100, min: 1, step: 1 },
     // The fill while there is no picture, and a tint multiplier once there is one: white = no tint.
     { key: 'color', label: 'Color', kind: 'color', default: 0xffffff },
-    { key: 'texture', label: 'Texture', kind: 'texture', default: '' },
+    // A video or a clip of one plays on the sprite (#402).
+    { key: 'texture', label: 'Texture', kind: 'texture', default: '', moving: true },
     // The shader's own parameters follow this row – see `schemaForAsset`. Empty is no shader at all.
     { key: SHADER_KEY, label: 'Shader', kind: 'asset', default: '', assetTypes: ['shader'] },
   ],
@@ -760,6 +762,7 @@ const ASSET_SCHEMAS: Record<AssetType, PropertySchema> = {
   ],
   // Where sounds are mixed (#325): how loud and through which sound FX. Where it goes on is the
   // master routing's (#351).
+  video: [...VIDEO_ROWS],
   bus: [
     { key: BUS_VOLUME_KEY, label: 'Volume', kind: 'number', default: 1, min: 0, max: 1, step: 0.05 },
     { key: BUS_FX_KEY, label: 'Sound FX', kind: 'asset', default: '', assetTypes: ['soundfx'] },

src/core/properties/textureRef.ts

@@ -1,8 +1,10 @@
-import type { Asset, PixelFrame, TextureFrame } from '../model/Asset';
+import type { Asset, PixelFrame, TextureFrame, VideoClip } from '../model/Asset';
 import { generatesPictures } from '../model/generated';
 import { findPixelFrame, frameLabel } from '../model/pixelart';
 import { findAsset, type Project } from '../model/Project';
+import { clipsOf } from '../model/sound';
 import { findFrame } from '../model/texture';
+import { findVideo } from '../model/video';
 
 export const NO_TEXTURE_LABEL = 'None';
 
@@ -14,13 +16,13 @@ export interface TextureChoice {
   label: string;
   /** The texture, or the asset that draws its own picture (#185). */
   asset: Asset;
-  /** Null for the whole-picture entry. */
-  frame: TextureFrame | PixelFrame | null;
+  /** Null for the whole-picture entry; a clip for one of a video (#402). */
+  frame: TextureFrame | PixelFrame | VideoClip | null;
 }
 
-/** An asset a texture field may name: an uploaded picture or a drawn one. */
+/** An asset a texture field may name: an uploaded picture, a drawn one, or a video (#402). */
 function isPictureAsset(asset: Asset): boolean {
-  return asset.type === 'texture' || generatesPictures(asset.type);
+  return asset.type === 'texture' || asset.type === 'video' || generatesPictures(asset.type);
 }
 
 /**
@@ -36,7 +38,25 @@ export function textureRefLabel(project: Project | null, id: string): string {
   const found = findFrame(project, id);
   if (found) return `${found.asset.name}${FRAME_SEPARATOR}${found.frame.name}`;
   const pixel = findPixelFrame(project, id);
-  return pixel ? `${pixel.asset.name}${FRAME_SEPARATOR}${frameLabel(pixel.index)}` : `${id}?`;
+  if (pixel) return `${pixel.asset.name}${FRAME_SEPARATOR}${frameLabel(pixel.index)}`;
+  const clip = findVideo(project, id);
+  return clip?.clip ? `${clip.asset.name}${FRAME_SEPARATOR}${clip.clip.name}` : `${id}?`;
+}
+
+/** The videos and their clips (#402), what a texture field that plays them offers after the pictures. */
+export function videoChoices(project: Project): TextureChoice[] {
+  const videos = project.assets
+    .filter((a) => a.type === 'video')
+    .sort((a, b) => a.name.localeCompare(b.name));
+  return videos.flatMap((asset) => [
+    { id: asset.id, label: asset.name, asset, frame: null },
+    ...clipsOf<VideoClip>(asset).map((clip) => ({
+      id: clip.id,
+      label: `${asset.name}${FRAME_SEPARATOR}${clip.name}`,
+      asset,
+      frame: clip,
+    })),
+  ]);
 }
 
 /**
@@ -45,7 +65,9 @@ export function textureRefLabel(project: Project | null, id: string): string {
  * not depend on upload time; frames keep creation order, which for a sliced sheet is reading order.
  */
 export function pictureChoices(project: Project): TextureChoice[] {
-  const pictures = project.assets.filter(isPictureAsset).sort((a, b) => a.name.localeCompare(b.name));
+  const pictures = project.assets
+    .filter((asset) => asset.type !== 'video' && isPictureAsset(asset))
+    .sort((a, b) => a.name.localeCompare(b.name));
   const choices: TextureChoice[] = [];
   for (const asset of pictures) {
     choices.push({ id: asset.id, label: asset.name, asset, frame: null });

src/core/properties/videos.ts

@@ -0,0 +1,36 @@
+import { booleanOf, type PropertyBag } from '../model/Properties';
+import { coerceAs } from './kinds';
+import type { PropertyDef } from './PropertySchema';
+
+/** How loud the video's own sound is, 0–1 (#402). */
+export const VIDEO_VOLUME_KEY = 'volume';
+/** Whether it starts over at its end – a background loop is what a placed video mostly is. */
+export const VIDEO_LOOP_KEY = 'loop';
+/** Where a clip starts and ends, in seconds; an end of 0 is the end of the video. */
+export const VIDEO_START_KEY = 'start';
+export const VIDEO_END_KEY = 'end';
+
+export const VIDEO_ROWS: readonly PropertyDef[] = [
+  { key: VIDEO_VOLUME_KEY, label: 'Volume', kind: 'number', default: 1, min: 0, max: 1, step: 0.05 },
+  { key: VIDEO_LOOP_KEY, label: 'Loop', kind: 'boolean', default: true },
+];
+
+/** The rows of a clip of a video, set in the Video tab. */
+export const VIDEO_CLIP_ROWS: readonly PropertyDef[] = [
+  { key: VIDEO_START_KEY, label: 'Start (s)', kind: 'number', default: 0, min: 0, step: 0.01, decimals: 3 },
+  { key: VIDEO_END_KEY, label: 'End (s)', kind: 'number', default: 0, min: 0, step: 0.01, decimals: 3 },
+];
+
+export function videoVolumeOf(bag: PropertyBag): number {
+  return Math.min(1, Math.max(0, coerceAs(bag[VIDEO_VOLUME_KEY], 'number', 1)));
+}
+
+export function videoLoopsOf(bag: PropertyBag): boolean {
+  return bag[VIDEO_LOOP_KEY] === undefined ? true : booleanOf(bag[VIDEO_LOOP_KEY]);
+}
+
+/** Start and end of a clip in seconds, `end` 0 for the end of the video. */
+export function videoSpanOf(bag: PropertyBag): { start: number; end: number } {
+  const seconds = (key: string) => Math.max(0, coerceAs(bag[key], 'number', 0));
+  return { start: seconds(VIDEO_START_KEY), end: seconds(VIDEO_END_KEY) };
+}

src/core/store/HttpFileStore.ts

@@ -2,10 +2,11 @@ import type { FileStore, StoredFile, UploadedFile } from './FileStore';
 import { apiRequest } from './http';
 
 /**
- * The API's file collections, one route each; textures, fonts and sounds differ in nothing else.
+ * The API's file collections, one route each; textures, fonts, sounds and videos differ in nothing
+ * else.
  * Pixels (#130) are only ever uploaded – a cel is found by the name in the project, never listed.
  */
-export type FileRoute = 'textures' | 'fonts' | 'sounds' | 'pixels';
+export type FileRoute = 'textures' | 'fonts' | 'sounds' | 'videos' | 'pixels';
 
 export class HttpFileStore<
   U extends UploadedFile = UploadedFile,

src/core/store/VideoStore.ts

@@ -0,0 +1,28 @@
+import type { FileStore, StoredFile, UploadedFile } from './FileStore';
+
+/**
+ * Where an upload is on its way to `data/videos/` (#402): the worker converts one after the other,
+ * so a video waits, is converted, and is ready or failed.
+ */
+export type VideoStatus = 'queued' | 'converting' | 'ready' | 'failed';
+
+/**
+ * What the server says about a video. `file` is known the moment the upload is in, long before the
+ * MP4 exists; the details and the poster come with `ready`, `progress` (0–1) with `converting`
+ * and `error` with `failed`.
+ */
+export interface UploadedVideo extends UploadedFile {
+  status: VideoStatus;
+  progress?: number;
+  error?: string;
+  width?: number;
+  height?: number;
+  duration?: number;
+  fps?: number;
+  audio?: boolean;
+  /** `{folder}/{sha256}.jpg`, beside the MP4. */
+  poster?: string;
+}
+
+export type StoredVideo = StoredFile & UploadedVideo;
+export type VideoStore = FileStore<UploadedVideo, StoredVideo>;

src/core/store/serialize.ts

@@ -69,6 +69,7 @@ import { decodeCells, encodeCells } from './tilemapCells';
 import { isAssetType } from '../model/AssetTypes';
 import { isFontFile } from '../model/font';
 import { isSoundFile } from '../model/sound';
+import { isVideoFile } from '../model/video';
 import { emptyEdit, type InstanceEdit, isEmptyEdit } from '../model/instance';
 import { type PropertyBag, type PropertyValue, tagOf, type Vec2 } from '../model/Properties';
 import { PROJECT_FORMAT_VERSION, type Project } from '../model/Project';
@@ -117,6 +118,7 @@ import {
 } from '../properties/PropertySchema';
 import { FIELD_SHADER_KEY, particleParamDefs } from '../properties/particles';
 import { CLIP_ROWS } from '../properties/sounds';
+import { VIDEO_CLIP_ROWS } from '../properties/videos';
 import { SHADER_KEY, shaderParamDefs } from '../properties/shaders';
 
 /**
@@ -274,6 +276,23 @@ export function deserializeProject(raw: unknown, fallbackId = ''): Project {
         ...(clips.length > 0 && { clips }),
         ...(stream && { stream }),
       });
+    } else if (type === 'video') {
+      const clips = normalizeClips(a.clips, takenIds, VIDEO_CLIP_ROWS);
+      const size = (value: unknown) =>
+        typeof value === 'number' && Number.isFinite(value) && value > 0 ? value : 0;
+      assets.push({
+        id,
+        type,
+        name,
+        props,
+        file: typeof a.file === 'string' && isVideoFile(a.file) ? a.file : '',
+        width: Math.round(size(a.width)),
+        height: Math.round(size(a.height)),
+        duration: size(a.duration),
+        fps: size(a.fps),
+        audio: a.audio === true,
+        ...(clips.length > 0 && { clips: clips.map(({ id, name, props }) => ({ id, name, props })) }),
+      });
     } else if (type === 'tile') {
       // A picture naming nothing is kept, like an animation's dangling root: it reads as missing
       // in the tab and comes back when the texture it named is restored by an undo.
@@ -769,8 +788,15 @@ function normalizeTexture(
   return { file, width, height, frames };
 }
 
-/** The clips of a sound (#357); like a frame's, a clip's id may not repeat one the document has. */
-function normalizeClips(raw: unknown, takenIds: Set<string>): SoundClip[] {
+/**
+ * The clips of a sound (#357) or a video (#402); like a frame's, a clip's id may not repeat one the
+ * document has.
+ */
+function normalizeClips(
+  raw: unknown,
+  takenIds: Set<string>,
+  rows: readonly PropertyDef[] = CLIP_ROWS,
+): SoundClip[] {
   const clips: SoundClip[] = [];
   for (const item of Array.isArray(raw) ? raw : []) {
     const c = asRecord(item);
@@ -781,7 +807,7 @@ function normalizeClips(raw: unknown, takenIds: Set<string>): SoundClip[] {
     clips.push({
       id,
       name: typeof c.name === 'string' ? c.name : '',
-      props: normalizeBag(c.props, CLIP_ROWS, true),
+      props: normalizeBag(c.props, rows, true),
       ...(stream && { stream }),
     });
   }