feuerwerk

0 Fork 0

Table of Contents

Getting a video in
The asset
The Video tab
Wearing it
Following an animation
Recording the game view

Not in real time

Recording the editor
The worker

A video asset is a moving picture a sprite wears through its Texture row, as it wears an image (#402). It has clips like a sound, stretches of it with a name, and a sprite can wear one of those instead of the whole video. Only Chrome is a target, and a video has no alpha.

Getting a video in

+ → Video opens the picker: every video the server holds for you, each row with its poster, the name it was uploaded under and what it is ("1920x1080 · 12.5 s · 29.97 fps · with sound", the editor writes the x as a times sign). Use video makes the asset and opens its tab. Upload… is the way in for a new one, and dropping a video file onto the canvas does the same (.mp4, .mov, .m4v, .webm, .mkv, .avi, told apart by the extension).

An upload is converted first. The server turns every video into one format, whatever came in: MP4 with H.264 and AAC, yuv420p, at a constant frame rate, at most 3840x2160 (2160x3840 standing up, never made larger), with a keyframe every 15 frames, so jumping to a clip start is quick. Until 2026-10-12 the most was 1920x1080; a video converted before stays as it came out. A 4K video is heavy in a game: every frame is four times the pixels of 1080p on their way to the GPU. A phone video recorded sideways comes out standing up. The picker shows a video on its way as "Waiting to be converted…" or "Converting… 42 %", asks the server again every second, and makes the asset by itself once a video uploaded from it is ready. One that cannot be converted says why ("Could not be converted: …"); Delete takes it away, uploading it again tries again. Uploading the same file twice converts it once.

The upload is kept as it came. Download in the picker and in the Video tab saves that file, not the MP4, under the name it was uploaded with. A video uploaded before 2026-10-12 has none kept, and the Video tab saves its MP4. An editor recording can be uploaded without converting it (its info says "Not converted"): it can be downloaded, but no project can wear it until Convert in the picker sends it to the worker.

Where it goes. The upload is stored under the hash of its bytes in your folder, like every other file (see API): data/videos/{folder}/{sha256}.mp4, beside it the poster {sha256}.jpg, the details {sha256}.meta.json and the upload {sha256}.source.{ext} (the ending of its name, else the one its first bytes tell). Every video so takes about twice the room it did. The poster has the colours the video shows in Chrome: it is turned from the video's own colour matrix (BT.709, BT.601, …) into the BT.601 a JPEG is read as. A video that does not say its matrix is read as Chrome reads it, BT.709 from 720 lines up and BT.601 below. Posters made before 2026-10-13 are a little pale (a red of 255 came out as 229) and stay so. The name is known the moment the upload is in, long before the MP4 exists. A video can be up to 2 GB. It goes from the request straight to disk, so the API holds none of it in memory: 300 MB went up with php-fpm at 14 MB.

The asset

A video asset keeps the file and, copied in when it was picked, its width, height, length, frame rate and whether it has sound, so a sprite is measured before the file loads. Its rows:

Row
Volume	0 to 1, how loud its sound is in a game
Loop	on by default: at its end (or its clip's end) it starts over; off, it stops on its last frame

Create sprite from video in its context menu makes a sprite of its size wearing it.

The Video tab

The video on top, under it a ruler in seconds, the clip strip and a track:

The clip gestures are the same as in the Sound tab, with the track in place of the wave (#403):

With no clip picked, a drag in the strip or in the track spans a new clip, one undo step, and picks it.
A click on a band, or on a clip in the track, picks that clip. A click beside the clips picks the video again. The clips not picked stand faint in the track.
With a clip picked, it stands in the track between two handles. Dragging a handle moves that edge, dragging its band in the strip moves the whole clip, its length kept. Any other drag does nothing. The picture follows the edge or the start being dragged.
A click in the ruler shows that moment, and stops a play.
Play (or Space) plays the picked clip, else the whole video, with its sound and its Loop. It starts at the playhead when that stands inside, else at the start, so Stop and Play go on where it stopped. The playhead runs along the ruler. The sound goes straight into the master, past the video's links, as in every tab (see Sounds). So a recording of the editor has it.
The wheel zooms the time, the middle button pans, Fit shows the whole video. A wheel step and Fit are shown as the short zoom transition of the other tabs (see Concepts).
Download saves the video as it was uploaded, under the asset's name: a recording comes out as the .mkv the editor made. One uploaded before uploads were kept saves its MP4.

A clip's Start (s) and End (s) are in the inspector as well; an end of 0 is the end of the video. A clip is a frame long at least, its edges are stored to the millisecond.

Wearing it

A sprite's Texture row offers the videos and their clips after the pictures ("Intro", "Intro / Clip 1"). Size, Color as a tint, a shader, colliders and the draw order work as with an image. Only the sprite's Texture row offers videos; a tile, a particle source and a vector fill take pictures only.

In the editor a placed video plays in the scene view without sound, round and round, as a preview. The sprite's PREVIEW in the inspector plays it too, placed or not. A sprite destroyed or switched off holds its video still and quiet at once; worn again, it plays on from there. A sprite only out of view keeps it playing, heard in a game. A video nothing wears for 2 s is let go and starts over the next time it is shown.
Play starts every video over at its clip's start and lets it be heard at the video's Volume, along its links in the Master routing, or into the bus its sprite's Video bus names. A pause holds it, Esc makes it a quiet preview again. A released game does the same.
A video with sound has a node in the Master routing, like a sound: a new one goes where most other videos go, or to the master where none is linked yet, never by where the sounds go. A link sends it into a bus or the master. A video without sound has none. Where two video assets use the same file, the links of the first one count for both.
Two sprites wearing the same clip share one player and show the same picture at the same moment. A video plays in real time; the game's time scale does not slow it down.

Not yet: nodes to play, pause or jump in a video, or to hear when it ends. That is a follow-up.

Following an animation

A video that plays by itself runs on its own clock, so an animation cannot say which picture shows when. For a cut, the sprite's Video clock row says who keeps the time (#404). These rows show only while the Texture is a video or a clip of one.

Row
Video clock	Free (the default): the video plays by itself, as above. Driven: it shows the picture at its Video time
Video time	seconds into the clip, shown while the clock is Driven. A number like any other, so an animation keys it
Video volume	0 to 1, how loud the video is heard through this sprite, on top of the video's own Volume. Shown while the clock is Driven. The fades of a composition item set it
Video bus	where the video is heard through this sprite in a game (#418): From asset (empty, the default) along the links of the video in the Master routing, Master, or a bus. In a composition empty reads To composition, its bus, and From asset is a choice of its own. A sprite with a bus of its own plays its video in a player of its own

What the keys on Video time do:

Keys	Result
frame 0 at 0 s, frame 300 at 10 s (at 30 fps), linear	plays normally, in step with the animation
two keys with the same value, or `step`	a still
a flatter or a steeper line	slow motion or fast motion
an easing or a curve	ramps

1x to the end without arithmetic. Right-click the timeline on the Video time row and pick Video at 1x to its end from frame N (#407). From the playhead, with the value the row has there (0 on an empty row), it keys the last whole frame before the clip ends: a 57.27 s clip at 10 fps from frame 0 ends on frame 572 at 57.2 s, the rest of a frame is left out. The first key turns linear, keys between the two go, and it is one undo step. The clip is the one the sprite wears at the playhead, a keyed Texture included. The entry is greyed out on a relative row, without a video, while the video's length is not known yet, or when the playhead already stands at the end. A notice names the frame, the time and the clip's length.

Past the end of its clip the time goes round with Loop and holds the last picture without; before 0 it shows the first one. A cut is then: clip A and clip B each on a sprite, Enabled keyed to switch between them, Video time keyed on each, texts above, transitions over Alpha, Color or a shader's parameters. Or one sprite whose Texture is keyed from clip to clip.

Each driven sprite has a player of its own, per view: two sprites with the same clip can show it at different times.
The Scene tab and the inspector's PREVIEW show the still at the sprite's Video time.
The Animations tab follows the playhead: scrubbing shows the picture of that moment, Play plays the video along, heard while it runs at about 1x so a cut can be timed by its sound.
In a game Play animation sets Video time like any other row, and so can a behavior. A paused game holds the time, so the video stands too.
How it keeps up: while the time runs forward at 0.25x to 4x the video plays at that speed and is nudged by a few percent, and only seeks when it is more than 0.25 s off. A time that stands, jumps or runs back holds the video and seeks; while a seek runs, only the newest one waits. Seeking every frame would be too slow, a seek decodes from the last keyframe, up to 14 pictures.
The speed is read over up to 0.5 s: a game steps its time in 1/60 s, and over a shorter window one step more or less read as 17 % off, so the video was nudged and fell quiet in turn. The first reading comes after 0.1 s, a time that stands for 0.1 s stands, and a start is measured from where the time began to move. Until that first reading a start plays at 1x: held and sought frame by frame, the first pictures of a cut stuck and were then skipped. A step faster than 4x is a jump, not a start.
Heard only at about 1x (0.9x to 1.1x); a still, scrubbing, slow and fast motion are quiet.
Exact while it runs live in the browser, as far as Chrome keeps up: a picture that comes too late leaves the one before it on screen. To have a cut as a file, record the game view while it plays (see Recording the game view below).
Ready for the cut: a driven sprite that is switched off or out of view keeps its player, quiet, at its Video time, in the editor as in a game (#404). Switched on, the first frame shows its picture, not its plain colour while the video loads. Each one costs a player while it waits, so a cut with many clips holds many videos at once.
A cut into the middle of a clip: switched on at another picture than the one its player was held at (a scrub, or a Video time that jumps at the cut), the video has to seek first. Until the seek is done the view keeps its last frame, for at most 0.3 s, so the held picture never shows. A seek that takes longer leaves the sprite out until its picture is there.
Play waits for them: the Game tab opens players of its own at Play. Until each of its driven videos has its first picture, the run holds at second 0, for at most 3 s ("Loading the videos..."). A released game waits the same way. A driven sprite without its picture is not drawn at all, rather than in its plain colour. A composition started later waits for the videos of its own items.

Recording the game view

Two action nodes record what the game view shows, with what the master plays (#405):

Node
Start recording	name (empty: `Recording 2026-10-11 16:42:05`, when it started), resolution (Game view, 720p, 1080p, 1440p, 4K), fps (60, 30), Sound, Convert, Import after conversion (all three on, as in Start recording… below), Real time (on; off, see Not in real time)
Stop recording	ends it and saves it

The picture is the camera's, nothing of what lies over it: no notice, no fullscreen button, no performance overlay, no letterbox, no menu or window of the editor. Game view records at the size the picture has on screen when the first frame is taken, 720p to 4K at that height, the width from the camera's shape. Either is at most 3840x2160: a camera wider than 16:9 comes out at 4K scaled down to 3840 wide, a little lower. A window smaller than that is no loss: the frame is drawn into a picture of its own at its full size. Text and a camera shader are drawn at the size of the screen, though, and are scaled up with it, so they come out softer. The size holds for the whole recording.
Nothing of it shows on screen. The frame is never drawn into the editor's canvas: that canvas is low latency (without ?vsync) and shows what is drawn into it at once, so a frame drawn there flickered in its corner. The picture comes back from the graphics card a frame or two later, without the editor waiting for it.
The sound is what leaves the master behind its limiter, what you hear. Opus, 160 kbit/s. With Sound off the file has the picture alone.
It goes on through Load scene and Restart game. The editor's pause holds it and leaves the paused time out; a game that pauses itself is still seen, so that is recorded. Another tab in front of the Game tab does not stop it. Ending the run (Esc, Stop game) saves it.
Behind another browser tab it goes on. Chrome draws no frames for a tab that is not in front, nor for a minimized window, and the run stood still with them. While a recording runs, a beat in a worker moves the editor on instead, at the recording's fps, and stands aside once the page draws again. A game not being recorded still stands behind another tab. In a test a recording in real time had as many frames behind another tab as in front, and one with Real time off had every frame. Chrome may still put a tab to sleep that stays behind others for long, with its energy saver for one. The beat is src/viewport/backgroundBeat.ts; the waits of a recording use a worker's timers too (backgroundTimers.ts), since Chrome holds a hidden page's own to once a second. Behind another tab Chrome hands over no video frames (requestVideoFrameCallback), so a video that opens there takes its first picture as soon as it has data instead of waiting for one. Before, a video item that began behind another tab was missing from the recording, sound and all, until the tab was in front again, and every frame of it waited 3 s. Switching back and forth while recording works too.
It leaves out the wait for a video. While the game view leaves a video sprite out because its picture is not there yet, no frame is taken, the recorded time stands and the sound of that time is left out too, for at most 3 s at a time. A Start recording right after Play composition therefore begins with the composition's first picture, not with the empty frames while its video loads.
The game waits for the recording to start. Setting up the encoders takes a moment, in a test about 0.17 s, and no frame can be taken meanwhile. So the run holds until they are ready, for at most 3 s, as it holds for its videos at Play. A Start recording in On start therefore has the first frame of the run, and one later on starts at the frame it ran in. The game stands still on screen for that moment; a sound already playing goes on.
One at a time: a second Start recording while one records is ignored and says so in the log, and so does a Stop recording with none. In the editor run (Run in editor) the nodes only write to the log.
In the editor the recording goes up to the server like any upload and is converted to MP4. Once that is done, and the run is over, it is a video asset of the project with its name, ready for a sprite or for Download in its tab. The notice line says how far it is. Convert and Import after conversion stop it short of either step, as for the editor recording: it is always uploaded, and the video picker has it.
In a released game there is no project to put it in: the browser downloads it as an .mkv when it ends. That file is the one the browser made (H.264 and Opus), not converted; Convert and Import after conversion do nothing there.
A file of 2 GB, the most an upload may be, ends the recording and saves it, with a line in the log. At 1080p60 that is roughly half an hour.
The log says what it records: an info line when it starts, with its size, fps, codec and whether it has sound, such as Recording "Trailer" started: 3840x2160 at 60 fps, H.264, with sound.
When the computer cannot keep up, a frame is left out rather than everything falling behind: the editor drew too late for it, or the encoder was still busy. The time in the file stays right, but the picture stutters. The recording says so in the log twice, as a warning. Once while it records, the first time it misses more than a tenth of its frames over 2 s, such as Recording "Trailer" can't keep up: about 38 of 60 fps, so the video will stutter. And once when it is saved, if it missed more than 5 % in all, such as Recording "Trailer" has 517 of 827 frames: 310 missed (37 %), so it stutters, with a hint that 30 fps or a smaller resolution helps. Its first second is left out of both: a short stall while it starts is no stutter. The line at the end comes after the run, so in the editor it is in the console panel. 4K at 60 fps is too much for most computers: reading each frame back from the GPU takes longer than a frame. The converted MP4 has a constant frame rate again. A recording with Real time off has every frame instead.

How it is made: H.264 from Chrome's VideoEncoder (VP9 where there is no H.264), hardware where the computer has it, 10 Mbit/s at 1080p60, a keyframe every 2 s. The file is Matroska, written by the editor itself (src/core/recording/matroska.ts) in clusters as the frames come, with the frame rate and the colour (BT.709) in its head, so nothing has to guess them. The colour goes into the H.264 stream as well (src/core/recording/avcColour.ts writes it into the SPS of the encoder's avcC), for players that read only the stream, such as VLC. A game view recording is limited range like any video: the GPU turns the picture into I420, limited range BT.709, before it is read back, and the encoder gets the planes with that colour. Fed RGB, Chrome's encoder made full-range video, and VLC 3 showed that too dark and its colours off, even with the range in the stream. Game view recordings saved before 2026-10-13 are full range and stay so. A player that heeds the range shows them right. The rules (size, codec level, which frames are taken, where the picture goes in a frame) are plain functions in src/core/recording/recording.ts, the browser's side is src/viewport/GameRecorder.ts, and src/viewport/PictureReader.ts reads the frames back through pixel buffers with a fence. The encoders, the sound and the file are src/viewport/RecordingTake.ts, which the editor recording below shares.

Not in real time

With Real time off (#423) the game moves on by exactly one frame of the recording at a time, 1/60 or 1/30 s, however long the computer takes for it, and only once the frame before is taken. So the file has every frame, 4K at 60 fps too. The game view runs slower meanwhile, and the notice line says how far it has got, such as Recording "Trailer" not in real time: 4.2 s of video, 0.38x real time. The speed is seconds of video per second it took: 0.38x is a little under three times slower than real time. It can go above 1x, when the scene is cheap and the editor draws more frames a second than the recording has, on a 144 Hz screen say.

The sound is made apart, in an offline audio context: while it records, what the run plays (sounds, instruments, patterns, songs, the audio items of a composition) goes there instead of to the speakers, through the same mix, buses, sound FX and limiter. It is made up to each next frame before the game goes on, so it lies on the picture whatever the frames took. You hear none of it meanwhile. A sound that was playing before the recording started goes on at the speakers and is not in it. A composition playing then moves into it, at the place its picture is.
Videos are shown frame by frame: a driven video, which every video of a composition item is, holds at each picture asked for, and the frame waits until that picture is there. Its sound is decoded from the video's file and played in step, through its bus. A free video plays on in real time, so it does not fit such a recording: set its Video clock to Driven.
Input still reaches the game, but the game runs slower than your hands. Such a recording is for compositions and scenes that play by themselves.
At most ten minutes: the offline context holds all of its sound in memory, 230 MB for ten minutes. A recording that gets there stops and says so in the log.
When it is saved, the log says how long it took, such as Recording "Trailer" took 35.2 s for 13.8 s of video, 0.39x real time, and has every frame. The start line ends with not in real time.
Only the node records so. Start recording… in the main menu records the editor in real time.

The run moves on by a frame of the recording rather than by the frame the screen drew (SteadyPace in src/core/recording/recording.ts). The run's sound goes through an AudioSwitch (src/runtime/audioSwitch.ts), which diverts it to the library of a SteadySound (src/viewport/SteadySound.ts) while such a recording lasts: a SoundLibrary on an OfflineAudioContext, suspended at each next frame. A driven video sought while held hands over its picture before seeked, so the frame callback is asked for as the seek begins. And a seek right onto where a picture begins showed that picture or the one before, as its time came out in microseconds, so such a video is sought half a millisecond further.

Recording the editor

Start recording… in the main menu (top left) records the whole editor, for a tutorial or a bug report. While it records, the entry reads Stop recording, and a red dot with the time stands beside the menu button.

The dialog it opens starts on the last choice made in this browser (the name excepted):

Option
Name	empty: `Recording <date> <time>`
Resolution	Screen: the canvas in device pixels. 1080p, 1440p, 4K: fitted into 1920x1080, 2560x1440 or 3840x2160 with the window's shape. The line under it says the size this window comes out at
FPS	60 or 30
Mouse pointer	drawn in or not
Sound	the master, as for the game view, or none
Convert	the server makes an MP4 of it, which a project can wear. Off: it is only kept
Import after conversion	the MP4 becomes an asset of the project once it is ready. Grey while Convert is off

The recording is always uploaded; Download in the video picker saves it as it was recorded.

A larger frame is drawn for, not scaled up to. For 4K the editor draws finer while it records (from a 1920x1080 window twice as fine), so text and lines stay sharp; the screen shows the same as before. A window too small for its frame even at four times as fine is scaled up the rest of the way. A smaller frame keeps the screen's sharpness and is scaled down. Exactly 3840x2160 needs a window of that shape: open the editor with ?w=1920&h=1080.
Cost: 4K at 60 fps is about 40 Mbit/s, so the 2 GB a file may have last about 7 minutes. The encoder is whatever Chrome has; frames it cannot keep up with are dropped. 30 fps halves both. It warns of a stutter as the game view recording does, in the notice line. The line when it starts stays in the console panel: the notice would be in the recording.
The mouse pointer is drawn in when asked: the system draws it over the page, so it is not in the canvas; the recording draws its own in the shape the editor shows (arrow, hand, resize arrows, I-beam, crosshair).
The pointer shapes are SVG files in src/core/recording/cursors/, one per shape, to be opened and edited. The viewBox is the picture, data-hot="x y" on the root is the point that sits on the mouse. resize.svg is drawn turned for the other resize directions. A unit of the viewBox is shown CURSOR_SIZE (0.8) CSS pixels large, on screen and in the recording alike (core/recording/cursors.ts), so the files keep the units they were drawn in. Each is drawn sharp at the size of the recording; an edited file takes a build.
The editor shows the same pointer while you work, unless General settings… → Custom mouse pointer is off: the files are the canvas's CSS cursor (url(…) x y, keyword), so the system still draws the pointer and the canvas never does. What you see is what the recording draws.
The red dot is not in it. It is an HTML element over the canvas, the one thing on the page outside it, so the copy of the canvas never holds it; the room it needs beside the menu stays empty in the recording. Clicks go through it.
Everything else is, the menu that stops it included: cut the end off afterwards if it is in the way.
It costs no second drawing. The frame is copied from the canvas right after the editor drew it. An idle editor draws nothing, and the recording repeats its last picture.
A resized window is fitted into the size the recording started with, centred, the rest dark.
A size of its own: open the editor with ?w=1920&h=1080 and its canvas is laid out at that size, whatever the window. A smaller window shows it shrunk to fit, a larger one centred at full size. With Screen the recording is that size times the screen's scaling (up to 2): at 100 % exactly 1920*1080.
It runs on through Play and Stop, beside a recording of the game view if a game starts one. Saving is the same: up to the server, then a video asset once the editor is not playing.

The code is src/editor/EditorRecorder.ts with its dialog src/editor/EditorRecordingDialog.ts; the sizes and the options are plain functions in src/core/recording/editorRecording.ts, the pointer shapes are read in src/core/recording/cursors.ts.

The worker

Converting is a job, not a request. The API keeps the upload beside where the MP4 will be and puts a job for it into data/videos/queue/ (hidden from the web server), and a worker takes one after the other, oldest first. A failed job leaves the upload where it is, so it can still be downloaded. A job from before uploads were kept has its bytes hidden in the queue, and they go once it is done:

On stein2 that is the container feuerwerk-video: PHP and ffmpeg on Debian 12, the checkout mounted, running as www-data without a network, on at most 3 of the 4 cores. It is a compose project of its own beside the web stack: docker compose -f tools/video/compose.yml up -d. The image is built like the CI image (see the head of tools/video/Dockerfile); a change to the worker's PHP needs a restart, not a rebuild. docker logs feuerwerk-video shows one line per video, with how long it took.
In CI, tools/ci/serve.sh starts the same worker beside the PHP server; the CI image has ffmpeg.

The worker is tools/video-worker.php over api/lib/VideoWorker.php; the rules (size, frame rate, the ffmpeg arguments, the progress) are plain functions in api/lib/VideoFormat.php. A job cut off by a stopped worker is started again when it comes back. Measured on stein2, still with the 1080p ceiling: 5 s of 2560x1440 at 60 fps came out as 1080p60 in 5.0 s. A 4K video takes longer; that is not measured.

Using the editor

Working on the code