Files
feuerwerk/src/editor/actions.ts
T
adminandClaude Opus 5.5 d48f174caf
game-maker / gate (push) Successful in 4m15s
game-maker / suite (push) Failing after 15m49s
Play videos on sprites, converted on the server with clips (#402)
A video becomes an asset like a sound: uploaded once, converted to one
format by a worker of its own, cut into clips in its tab, and worn by a
sprite through its Texture row. Only Chrome is a target, without alpha.

- Server: POST ?r=videos queues the upload under the hash of its bytes,
  so the MP4's name is known at once. The worker (VideoWorker over
  VideoFormat) writes H.264/AAC, yuv420p, constant frame rate, a
  keyframe every 15 frames, at most 1920x1080, with a poster and the
  details beside it. A failed job says why and is tried again on the
  next upload.
- An upload may be up to 2 GB. It is streamed from the request to disk
  and hashed there, so php-fpm holds none of it in memory. The other
  routes keep their 256 MB.
- The worker runs in the container game-maker-video on stein2, a
  compose project of its own (tools/video/), and in CI beside the PHP
  server. The CI image has ffmpeg for it.
- Model: asset type video with size, length, frame rate and sound, and
  clips like a sound's. The clip helpers and commands serve both.
- A sprite's Texture row offers videos and clips after the pictures.
  resolvePicture names a playing video by a key, and the texture library
  hands it to a VideoLibrary with one player per key.
- The editor shows a placed video as a quiet preview. Play starts it
  over at its clip's start and lets it be heard through the master bus,
  a pause holds it, Esc makes it quiet again.
- Video tab: the picture, a ruler, the clip strip and a track on the
  Sound tab's time axis. A drag spans a clip, a handle moves an edge,
  Play plays the picked clip.
- The picker shows a video on its way with its progress and adopts an
  upload from it once it is ready. Dropping a video file does the same.

Decided on my own:

- Two sprites wearing the same clip share one player and show the same
  picture. Control per instance belongs to the follow-up with nodes.
- A video plays in real time, the game's time scale does not slow it.
- The container may use 3 of the 4 cores of stein2.

Findings:

- The first real upload (279 MB) got 413: the API's own ceiling of
  256 MB for every request, not nginx or the Brandmauer.
- Chrome has no picture of a video that never played, and a Pixi
  VideoSource drawn before the first frame stays empty for good. The
  texture is made in the first frame callback (openVideo).
- A player nobody asks for in 2 s is let go. The inspector's PREVIEW
  asked only on an edit, so with the sprite not on screen its texture
  was destroyed under it and the batcher threw on every frame. The
  preview now redraws every frame while it shows a video, like a Time
  shader. A let-go texture keeps an empty source instead of none.
- VideoSource.load() reloads the element and cuts off an earlier
  play(). Playing starts after loadeddata.

Tests:

- PHP videos.test.php: 68 checks, with the real ffmpeg
- 300 MB upload through Brandmauer and nginx: 201 in 21 s, php-fpm at
  13.7 MB RSS at most
- Unit videos.test.ts: 20 tests
- e2e 66 (stein2, atx was off): ALL PASSED, 18 checks; e2e 04 with the
  new asset count green
- Gate (atx): 294 files, 4520 tests. serialize.test.ts used video as
  its example of an unknown type and now uses a made-up one.
- Counter-checks: a keyframe gap of 30 fails the PHP check, a key that
  keeps the video's end fails 2 unit tests, a clip without its end
  fails "plays only its stretch" in e2e 66

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:12:39 +02:00

303 lines
14 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Editor actions that several places trigger the same way (inspector, hierarchy context
* menu, keyboard): they run the command and put the selection where the user expects it.
*/
import { RemoveAnimationKeysCommand, RemoveAnimationTracksCommand } from '../core/commands/animationCommands';
import { annotationNoun, RemoveAnnotationCommand } from '../core/commands/annotationCommands';
import { DeleteAssetCommand, SetActiveSceneCommand } from '../core/commands/assetCommands';
import {
AddColliderCommand,
AddJointCommand,
AddRigidbodyCommand,
colliderRefusal,
jointRefusal,
rigidbodyRefusal,
} from '../core/commands/colliderCommands';
import { oneStep } from '../core/commands/CompoundCommand';
import { CreateContainerCommand } from '../core/commands/containerCommands';
import { RemoveClipCommand } from '../core/commands/clipCommands';
import { RemoveFrameCommand } from '../core/commands/frameCommands';
import { DisconnectCommand, FoldEdgeCommand, RemoveNodeCommand } from '../core/commands/graphCommands';
import { RemoveSceneObjectsCommand } from '../core/commands/sceneObjectCommands';
import { RemoveVectorShapeCommand, RemoveVectorShapesCommand } from '../core/commands/vectorArtCommands';
import type { AssetType } from '../core/model/AssetTypes';
import { curveUsages } from '../core/model/curve';
import { structTypeUsages } from '../core/model/structTypes';
import { generatesPictures } from '../core/model/generated';
import { isTemplateContent } from '../core/model/instance';
import type { Command } from '../core/commands/Command';
import { findAnnotation, findNode, nodeDef } from '../core/graph';
import { findAsset, findGraphAsset, type ObjectHolder, type Project } from '../core/model/Project';
import { ownerSong } from '../core/music/embedded';
import type { SceneObject } from '../core/model/SceneObject';
import { pixelArtRefIds } from '../core/model/pixelart';
import { textureRefIds } from '../core/model/texture';
import { videoRefIds } from '../core/model/video';
import {
animationsUsing,
cellsPainted,
tilesUsing,
findPropertyUsages,
findUsages,
} from '../core/model/usages';
import { Dialog } from '../ui';
import { removeSongItems, removeTrack } from './song/songActions';
import { type EditorState, graphSelection, objectSelection } from './state';
/**
* What a double-click on an asset does, wherever it happens (asset panel, "Open …" entries, a link
* in the inspector): open its tab – and a scene, whose tab is the scene view, becomes the active one.
*/
export function activateAsset(state: EditorState, assetId: string): void {
if (findAsset(state.project, assetId)?.type === 'scene') state.execute(new SetActiveSceneCommand(assetId));
state.openAsset(assetId);
}
/** Groups the objects into a new container asset placed in their stead; selects the container. */
export function createContainer(state: EditorState, holderId: string, objectIds: readonly string[]): boolean {
const command = new CreateContainerCommand(holderId, objectIds);
if (!state.execute(command)) return false;
state.select(objectSelection(holderId, [command.objectId]));
return true;
}
/** The test `AddColliderCommand` makes; template content has no structure of its own to change. */
export function canAddCollider(project: Project, holder: ObjectHolder, object: SceneObject): boolean {
return !isTemplateContent(holder, object.id) && colliderRefusal(project, holder, object) === null;
}
/**
* Gives the object a collider child – a new one sized like it, or the existing `assetId` (#222);
* selects the collider.
*/
export function addCollider(state: EditorState, holderId: string, objectId: string, assetId = ''): boolean {
const command = new AddColliderCommand(holderId, objectId, assetId);
if (!state.execute(command)) return false;
state.select(objectSelection(holderId, [command.objectId]));
return true;
}
/** The test `AddRigidbodyCommand` makes; template content has no structure of its own to change. */
export function canAddRigidbody(project: Project, holder: ObjectHolder, object: SceneObject): boolean {
return !isTemplateContent(holder, object.id) && rigidbodyRefusal(project, holder, object) === null;
}
/** Gives the object a rigidbody child, new or the existing `assetId`; selects the rigidbody. */
export function addRigidbody(state: EditorState, holderId: string, objectId: string, assetId = ''): boolean {
const command = new AddRigidbodyCommand(holderId, objectId, assetId);
if (!state.execute(command)) return false;
state.select(objectSelection(holderId, [command.objectId]));
return true;
}
/** The test `AddJointCommand` makes; template content has no structure of its own to change. */
export function canAddJoint(project: Project, holder: ObjectHolder, object: SceneObject): boolean {
return !isTemplateContent(holder, object.id) && jointRefusal(project, holder, object) === null;
}
/** Gives the object a joint child (#154), new or the existing `assetId`; selects the joint. */
export function addJoint(state: EditorState, holderId: string, objectId: string, assetId = ''): boolean {
const command = new AddJointCommand(holderId, objectId, assetId);
if (!state.execute(command)) return false;
state.select(objectSelection(holderId, [command.objectId]));
return true;
}
/** Removes the objects (with their subtrees) as one undo step. */
export function removeObjects(state: EditorState, holderId: string, objectIds: readonly string[]): boolean {
return state.execute(new RemoveSceneObjectsCommand(holderId, objectIds));
}
/** Takes shapes of a vector art out as one undo step; the drawing itself stays selected. */
export function removeShapes(state: EditorState, assetId: string, shapeIds: readonly string[]): boolean {
if (shapeIds.length === 0) return false;
const removed = state.execute(
shapeIds.length === 1
? new RemoveVectorShapeCommand(assetId, shapeIds[0]!)
: new RemoveVectorShapesCommand(assetId, shapeIds),
);
if (removed) state.select({ kind: 'asset', assetId });
return removed;
}
/** Deletes graph nodes as one undo step, however many were selected. */
export function removeNodes(state: EditorState, assetId: string, nodeIds: readonly string[]): boolean {
const commands = nodeIds.map((id) => new RemoveNodeCommand(assetId, id));
return commands.length > 0 && state.execute(oneStep(`Delete ${commands.length} nodes`, commands));
}
/** What a link is called in this graph: between states it is a transition. */
export function linkNoun(state: EditorState, assetId: string): string {
return findGraphAsset(state.project, assetId)?.type === 'statemachine' ? 'transition' : 'link';
}
/**
* Hides the links as one undo step and lets go of them: a hidden link has no line left to be
* picked on, its mark at the port is the handle from then on.
*/
export function hideLinks(state: EditorState, assetId: string, edgeIds: readonly string[]): boolean {
const commands = edgeIds.map((id) => new FoldEdgeCommand(assetId, id, true));
if (
commands.length === 0 ||
!state.execute(oneStep(commands.length === 1 ? 'Hide link' : 'Hide links', commands))
)
return false;
state.select(
graphSelection(assetId, state.selectedNodeIds, state.selectedAnnotationIds) ?? { kind: 'asset', assetId },
);
return true;
}
/**
* Deletes everything a graph selection holds – nodes, boxes and links alike – as one undo step,
* since one band caught them and one keypress asked for them to go. Boxes still take none of
* their nodes. Links go first, so one at a node that goes too is not asked for twice.
*/
export function removeGraphSelection(
state: EditorState,
assetId: string,
nodeIds: readonly string[],
annotationIds: readonly string[],
edgeIds: readonly string[] = [],
): boolean {
const graph = findGraphAsset(state.project, assetId)?.graph;
const links = (graph?.edges ?? []).filter((e) => edgeIds.includes(e.id));
// A node that comes and goes on its own – a track of a song (#350) – stays, the rest goes.
const removable = nodeIds.filter((id) => !nodeDef((graph && findNode(graph, id))?.type ?? '')?.fixed);
const commands: Command[] = [
...links.map((e) => new DisconnectCommand(assetId, e.id)),
...removable.map((id) => new RemoveNodeCommand(assetId, id)),
];
let noun = links.length === 1 ? linkNoun(state, assetId) : '';
for (const id of annotationIds) {
const found = graph && findAnnotation(graph, id);
if (!found) continue;
noun = annotationNoun(found.kind);
commands.push(new RemoveAnnotationCommand(assetId, found.kind, id));
}
if (commands.length === 0) {
if (nodeIds.length > 0) state.setNotice('These nodes come and go on their own and cannot be deleted');
return false;
}
const label =
commands.length === 1 && noun !== ''
? `Delete ${noun}`
: removable.length === commands.length
? `Delete ${commands.length} nodes`
: `Delete ${commands.length} items`;
return state.execute(oneStep(label, commands));
}
/**
* What Delete does to the selection; false when nothing is selected. Every kind is a case of its
* own, so a new kind of selection cannot slip through without saying what Delete takes away.
*/
export function removeSelection(state: EditorState): boolean {
const s = state.selection;
if (!s) return false;
switch (s.kind) {
case 'asset':
confirmDeleteAsset(state, s.assetId);
return true;
case 'object':
removeObjects(state, s.holderId, s.objectIds);
return true;
case 'node':
removeGraphSelection(state, s.assetId, s.nodeIds, s.annotationIds, s.edgeIds);
return true;
case 'annotation':
removeGraphSelection(state, s.assetId, [], s.annotationIds, s.edgeIds);
return true;
case 'edge':
removeGraphSelection(state, s.assetId, [], [], s.edgeIds);
return true;
case 'frame':
state.execute(new RemoveFrameCommand(s.assetId, s.frameId));
return true;
case 'clip':
state.execute(new RemoveClipCommand(s.assetId, s.clipId));
return true;
case 'shape':
removeShapes(state, s.assetId, s.shapeIds);
return true;
case 'track':
// The keys when some are picked, else the row itself – like a frame of a texture.
state.execute(
s.keys.length > 0
? new RemoveAnimationKeysCommand(s.assetId, s.keys)
: new RemoveAnimationTracksCommand(s.assetId, [s.trackId]),
);
return true;
case 'song':
// The items when some are picked, else the track itself and what lies on it.
if (s.itemIds.length > 0) removeSongItems(state, s.assetId, s.itemIds);
else removeTrack(state, s.assetId, s.trackId);
return true;
}
}
/** Asks before deleting an asset, spelling out what goes with it. */
export function confirmDeleteAsset(state: EditorState, assetId: string): void {
const asset = findAsset(state.project, assetId);
if (!asset) return;
const song = ownerSong(state.project, asset);
if (song) {
state.setNotice(`"${asset.name}" lives in the song "${song.name}" and goes with what uses it there`);
return;
}
const usages = findUsages(state.project, assetId).length;
// A texture is never placed but worn through a property, so counting placed instances would
// report "unused". Its frames count as the texture itself.
const ids =
asset.type === 'texture'
? textureRefIds(asset)
: asset.type === 'pixelart'
? pixelArtRefIds(asset)
: asset.type === 'video'
? videoRefIds(asset)
: [assetId];
const references = findPropertyUsages(state.project, ids).length;
// An asset that draws its own picture (#185) is worn the same way, only without frames.
const picture = asset.type === 'texture' || asset.type === 'video' || generatesPictures(asset.type);
const animations = picture ? animationsUsing(state.project, ids).length : 0;
const tiles = picture ? tilesUsing(state.project, ids).length : 0;
const eased = asset.type === 'curve' ? curveUsages(state.project, assetId) : { keys: 0, nodes: 0 };
const typed =
asset.type === 'struct' ? structTypeUsages(state.project, assetId) : { places: 0, lists: 0, nodes: 0 };
const lists = typed.lists > 0 ? `, ${typed.lists} list(s)` : '';
const painted = asset.type === 'tile' ? cellsPainted(state.project, assetId) : { tilemaps: 0, cells: 0 };
const detail =
asset.type === 'scene'
? `The scene and its ${asset.objects.length} object(s) are removed.`
: painted.cells > 0
? `${painted.cells} cell(s) on ${painted.tilemaps} tilemap(s) are painted with it and become empty.`
: eased.keys > 0 || eased.nodes > 0
? `${eased.keys} animation key(s) and ${eased.nodes} Lerp node(s) use it and blend linearly instead.`
: typed.places > 0 || typed.lists > 0 || typed.nodes > 0
? `${typed.places} struct(s)${lists} and ${typed.nodes} node(s) use it and keep its fields as their own.`
: references > 0 || animations > 0 || tiles > 0
? wornDetail(references, animations, tiles, asset.type)
: usages === 0
? 'This cannot be undone with Ctrl+Z after reload.'
: asset.type === 'container'
? `${usages} placed container(s) are removed together with their content.`
: `${usages} placed instance(s) are removed as well (scenes and container templates).`;
new Dialog(`Delete "${asset.name}"?`, detail, [
{ label: 'Cancel' },
{ label: 'Delete', danger: true, onClick: () => state.execute(new DeleteAssetCommand(assetId)) },
]).openCentered();
}
function wornDetail(sprites: number, animations: number, tiles: number, type: AssetType): string {
if (type === 'font') return `${sprites} text(s) reference it and fall back to the editor's own typeface.`;
const who = [
sprites > 0 ? `${sprites} sprite(s)` : '',
animations > 0 ? `${animations} animation(s)` : '',
tiles > 0 ? `${tiles} tile(s)` : '',
].filter(Boolean);
// "a, b and c" rather than "a and b and c", which three kinds of user made reachable.
const list =
who.length > 2 ? `${who.slice(0, -1).join(', ')} and ${who[who.length - 1]}` : who.join(' and ');
return `${list} reference it and show the plain color in its place.`;
}