Draw notes for an instrument in a piano roll and loop them #333

Notes could only be played one at a time, by a key or a behavior. A
pattern is a stretch of notes for one instrument, drawn in a piano
roll and played in time through the scheduler of #332. Songs (#334)
will lay patterns out on tracks.

- New asset type pattern under the heading Patterns. Instruments are
  listed under Instruments now, since a heading is one per type.
- Notes are a start and a length in ticks, a note number and a
  velocity, kept to their rules on load and in every command. The
  instrument, the length in bars, the grid, the meter and the tempo
  are rows in the inspector. The loop is the asset's own data, the
  whole pattern while none is set.
- Grids from 1/1 to 1/32 and the triplets 1/4T to 1/32T, all whole
  ticks. The meter only moves the bar lines, the notes stay put.
- The tab shows the keys down the left, a ruler with bars, beats,
  loop and start along the top, the notes between them and a
  velocity lane below. Keys the instrument plays nothing for are
  grey, the names of a drum kit stand on its keys.
- A click draws a note on the grid line before the pointer, as long
  as the last one drawn or lengthened and as hard as the last
  velocity set. The body moves a note, the right end lengthens it,
  Alt leaves the grid. A drag on empty grid picks with a band, Shift
  adds, a right-click deletes.
- Keys: Space plays, Delete, Ctrl+C/V/D/A, the arrows by a semitone
  or a grid step and with Shift by an octave or a bar. A paste lands
  where playing starts. Every gesture is one undo step.
- A click on the ruler sets where playing starts, a drag the loop or
  one of its ends, its menu loops a bar or the whole pattern.
- The wheel scrolls, Shift across, on the ruler or with Alt it zooms.
  Ctrl+wheel stays with the browser, Pixi listens passively.
- Play runs the loop through the scheduler. An edit goes to it on
  the next window without a restart, a new loop or tempo counts from
  what is planned. Another pattern shown or a project opened stops it.
- The time axis and the ruler are their own pieces in editor/music
  for the song tab. The rules of the roll are pure functions in
  rollView and rollEdit.

Findings:
- Music.play could start after a stop that came while it was still
  loading its files. A run counter makes the late start give up.
  Music.edit now loads what a new note needs as well.
- A click on an end of the loop would have set the loop again. A
  press there only moves the end once it is dragged.

Tests:
- pattern.test.ts (new, 9) and piano-roll.test.ts (new, 16): model,
  grids, meter, held notes, sequence, commands, loop, round trip,
  drum names, time lines, rows, hits, bands, lane, edit rules.
- e2e 64 (new) on melektaus-atx: ALL PASSED, 20 checks. e2e 4 with
  29 asset types: ALL PASSED, 62 checks.
- Gate on melektaus-atx: CHECK EXIT 0, 247 test files, 3839 tests.
- Counter-checks: the move without its clamp, the sequence without
  the end of the pattern, the duplicate offset without whole grid
  steps each fail 1 test. Without passing edits to the scheduler the
  e2e check for a note drawn while playing fails, the rest passes.
- Probe on stein2: the tab with a drum kit, before and while playing,
  looked at in two screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/core/commands/assetCommands.ts

@@ -106,6 +106,8 @@ export function buildAsset(type: AssetType, name: string, id = newId()): Asset {
       return { ...base, type: 'soundfx', graph: createEmptyGraph() };
     case 'instrument':
       return { ...base, type: 'instrument', graph: starterInstrumentGraph() };
+    case 'pattern':
+      return { ...base, type: 'pattern', notes: [] };
     case 'particles':
       return { ...base, type: 'particles', graph: createEmptyGraph() };
   }

src/core/commands/patternCommands.ts

@@ -0,0 +1,63 @@
+/**
+ * Editing the notes of a pattern (#333). As with a curve, one gesture is one command: the whole list
+ * is written once the pointer lets go, so an undo takes the drag back. What is written is held to the
+ * rules of a note first (`heldPatternNotes`).
+ */
+import type { PatternAsset } from '../model/Asset';
+import { heldPatternNotes, type PatternNote } from '../music/pattern';
+import { heldLoop, type LoopRange } from '../music/sequence';
+import type { Project } from '../model/Project';
+import type { Command } from './Command';
+import { requirePattern } from './require';
+
+export class SetPatternNotesCommand implements Command {
+  private previous: PatternNote[] = [];
+
+  constructor(
+    private readonly assetId: string,
+    private readonly notes: readonly PatternNote[],
+    readonly label = 'Edit notes',
+  ) {}
+
+  /** Nothing to do when the notes already read exactly like this – no empty undo step. */
+  static changes(asset: PatternAsset, notes: readonly PatternNote[]): boolean {
+    return JSON.stringify(asset.notes) !== JSON.stringify(heldPatternNotes(notes));
+  }
+
+  execute(project: Project): void {
+    const asset = requirePattern(project, this.assetId);
+    this.previous = asset.notes;
+    asset.notes = heldPatternNotes(this.notes);
+  }
+
+  undo(project: Project): void {
+    requirePattern(project, this.assetId).notes = this.previous;
+  }
+}
+
+/** Sets the loop of the ruler, or takes it away (`null`): then the whole pattern is the loop. */
+export class SetPatternLoopCommand implements Command {
+  readonly label: string;
+  private previous: LoopRange | undefined;
+
+  constructor(
+    private readonly assetId: string,
+    private readonly loop: LoopRange | null,
+  ) {
+    this.label = loop ? 'Set loop' : 'Loop the whole pattern';
+  }
+
+  execute(project: Project): void {
+    const asset = requirePattern(project, this.assetId);
+    this.previous = asset.loop;
+    const held = heldLoop(this.loop);
+    if (held) asset.loop = { start: Math.round(held.start), end: Math.round(held.end) };
+    else delete asset.loop;
+  }
+
+  undo(project: Project): void {
+    const asset = requirePattern(project, this.assetId);
+    if (this.previous) asset.loop = this.previous;
+    else delete asset.loop;
+  }
+}

src/core/commands/require.ts

@@ -4,6 +4,7 @@ import { type NodeDef, nodeDef } from '../graph/NodeRegistry';
 import type {
   Asset,
   CurveAsset,
+  PatternAsset,
   TileAsset,
   BehaviorAsset,
   StateMachineAsset,
@@ -141,6 +142,12 @@ export function requireCurve(project: Project, assetId: string): CurveAsset {
   return asset;
 }
 
+export function requirePattern(project: Project, assetId: string): PatternAsset {
+  const asset = findAssetOf(project, assetId, 'pattern');
+  if (!asset) throw new CommandError(`Pattern ${assetId} not found`);
+  return asset;
+}
+
 export function requireVariablesAsset(project: Project, assetId: string): VariablesAsset {
   const asset = findAssetOf(project, assetId, 'variables');
   if (!asset) throw new CommandError(`Variables ${assetId} not found`);

src/core/instrument/compile.ts

@@ -44,6 +44,7 @@ import {
   InstrumentOutput,
   isInstrumentNodeDef,
   loops,
+  noteNameKey,
   noteOutputKey,
   noteParam,
   notesRange,
@@ -370,3 +371,17 @@ export function playedNotes(graph: Graph): Set<number> {
   }
   return notes;
 }
+
+/** The notes a Notes node with single notes has given a name of their own (a drum kit's). */
+export function namedNotes(graph: Graph): Map<number, string> {
+  const names = new Map<number, string>();
+  for (const node of graph.nodes) {
+    if (node.type !== INSTRUMENT_NOTES || !singleNotes(node.params)) continue;
+    const { from, to } = notesRange(node.params);
+    for (let note = from; note <= to; note++) {
+      const name = textOf(node.params[noteNameKey(note)] ?? '').trim();
+      if (name && !names.has(note)) names.set(note, name);
+    }
+  }
+  return names;
+}

src/core/model/Asset.ts

@@ -1,4 +1,6 @@
 import type { Graph } from '../graph/Graph';
+import type { PatternNote } from '../music/pattern';
+import type { LoopRange } from '../music/sequence';
 import type { PropertyDef } from '../properties/PropertySchema';
 import type { AssetType } from './AssetTypes';
 import type { PropertyBag, PropertyValue, Vec2 } from './Properties';
@@ -328,6 +330,17 @@ export interface InstrumentAsset extends AssetBase {
   graph: Graph;
 }
 
+/**
+ * Notes for one instrument (#333), drawn in a piano roll. The instrument, the length in bars, the
+ * grid, the meter and the tempo are properties; `loop` is the stretch the playhead runs round, the
+ * whole pattern while there is none.
+ */
+export interface PatternAsset extends AssetBase {
+  type: 'pattern';
+  notes: PatternNote[];
+  loop?: LoopRange;
+}
+
 /**
  * A tile a tilemap paints with: `pictures` names one block per animation step (a texture id for a
  * sheet that holds nothing else, a frame id for one block cut out of a sheet), and `props` say
@@ -582,6 +595,7 @@ export type Asset =
   | SoundAsset
   | SoundFxAsset
   | InstrumentAsset
+  | PatternAsset
   | ParticlesAsset;
 
 /** The assets whose `variables` are declared by the user – see `requireVariableOwner`. */

src/core/model/AssetTypes.ts

@@ -26,6 +26,7 @@ export const ASSET_TYPES = [
   'sound',
   'soundfx',
   'instrument',
+  'pattern',
   'particles',
 ] as const;
 
@@ -59,6 +60,7 @@ export const ASSET_TYPE_LABELS: Record<AssetType, string> = {
   sound: 'Sound',
   soundfx: 'Sound FX',
   instrument: 'Instrument',
+  pattern: 'Pattern',
   particles: 'Particles',
 };
 
@@ -71,8 +73,8 @@ export const ASSET_TYPE_LABELS: Record<AssetType, string> = {
  * is the tilemap. A curve is the way a key or a `Lerp` blends, picked by id like a state machine.
  * A struct is a shape that variables and fields take on (#253), nothing that exists on its own.
  * A shader function is part of other shaders (#303), worn by nothing itself. A sound FX is what a
- * sound is played through (#311), heard but never seen, and an instrument what a note is played
- * with (#331).
+ * sound is played through (#311), heard but never seen, an instrument what a note is played
+ * with (#331) and a pattern the notes it plays (#333).
  */
 export type PlaceableAssetType = Exclude<
   AssetType,
@@ -91,6 +93,7 @@ export type PlaceableAssetType = Exclude<
   | 'sound'
   | 'soundfx'
   | 'instrument'
+  | 'pattern'
   | 'variables'
   | 'struct'
 >;
@@ -111,6 +114,7 @@ const NOT_PLACEABLE: ReadonlySet<AssetType> = new Set([
   'sound',
   'soundfx',
   'instrument',
+  'pattern',
   'variables',
   'struct',
 ]);

src/core/music/pattern.ts

@@ -0,0 +1,157 @@
+/**
+ * A pattern (#333): notes for one instrument, edited in a piano roll and played by the scheduler
+ * (#332). The notes are counted in ticks; the meter only says where the bars of the ruler are, so a
+ * pattern changed from 4/4 to 6/8 keeps every note where it was.
+ */
+import { newId } from '../ids';
+import { HIGHEST_NOTE, LOWEST_NOTE } from '../instrument/notes';
+import type { PatternAsset } from '../model/Asset';
+import { type PropertyBag, textOf } from '../model/Properties';
+import { heldLoop, type LoopRange, type Sequence } from './sequence';
+import { heldMeter, heldTempo, type Meter, ticksPerBar } from './time';
+
+export interface PatternNote {
+  id: string;
+  /** Where it starts and how long it is held, in ticks. */
+  tick: number;
+  length: number;
+  /** As MIDI counts it: 60 is C4. */
+  note: number;
+  /** 0 to 1. */
+  velocity: number;
+}
+
+export const PATTERN_INSTRUMENT_KEY = 'instrument';
+export const PATTERN_BARS_KEY = 'bars';
+export const PATTERN_GRID_KEY = 'grid';
+export const PATTERN_BEATS_KEY = 'beats';
+export const PATTERN_BEAT_KEY = 'beat';
+export const PATTERN_TEMPO_KEY = 'tempo';
+
+export const DEFAULT_BARS = 4;
+export const MAX_BARS = 256;
+export const DEFAULT_GRID = '1/16';
+/** What a new note is struck with until another velocity has been set. */
+export const DEFAULT_VELOCITY = 0.8;
+
+/** The grids a note is drawn on, in ticks: with 480 to the quarter every triplet is a whole number. */
+const GRIDS: readonly (readonly [string, number])[] = [
+  ['1/1', 1920],
+  ['1/2', 960],
+  ['1/4', 480],
+  ['1/8', 240],
+  ['1/16', 120],
+  ['1/32', 60],
+  ['1/4T', 320],
+  ['1/8T', 160],
+  ['1/16T', 80],
+  ['1/32T', 40],
+];
+
+export const GRID_OPTIONS = GRIDS.map(([value]) => ({ value, label: value }));
+export const BEAT_OPTIONS = [1, 2, 4, 8, 16, 32].map((unit) => ({ value: String(unit), label: `1/${unit}` }));
+
+export function gridTicks(grid: string): number {
+  return (GRIDS.find(([value]) => value === grid) ?? GRIDS[4]!)[1];
+}
+
+function wholeNumber(
+  value: PropertyBag[string] | undefined,
+  fallback: number,
+  min: number,
+  max: number,
+): number {
+  const n = typeof value === 'number' ? Math.round(value) : NaN;
+  return Number.isFinite(n) ? Math.max(min, Math.min(max, n)) : fallback;
+}
+
+export function patternMeter(props: PropertyBag): Meter {
+  return heldMeter({
+    beats: wholeNumber(props[PATTERN_BEATS_KEY], 4, 1, 32),
+    unit: Number(textOf(props[PATTERN_BEAT_KEY] ?? '4')),
+  });
+}
+
+export function patternTempo(props: PropertyBag): number {
+  const tempo = props[PATTERN_TEMPO_KEY];
+  return heldTempo(typeof tempo === 'number' ? tempo : NaN);
+}
+
+export function patternBars(props: PropertyBag): number {
+  return wholeNumber(props[PATTERN_BARS_KEY], DEFAULT_BARS, 1, MAX_BARS);
+}
+
+export function patternGrid(props: PropertyBag): number {
+  return gridTicks(textOf(props[PATTERN_GRID_KEY] ?? DEFAULT_GRID));
+}
+
+export function patternInstrument(props: PropertyBag): string {
+  return textOf(props[PATTERN_INSTRUMENT_KEY] ?? '');
+}
+
+/** Where the pattern ends, in ticks: a note starting there or later is kept, but not played. */
+export function patternLength(asset: PatternAsset): number {
+  return patternBars(asset.props) * ticksPerBar(patternMeter(asset.props));
+}
+
+/** The loop the playhead runs round: as it was set, cut at the end of the pattern; else all of it. */
+export function patternLoop(asset: PatternAsset): LoopRange {
+  const length = patternLength(asset);
+  const held = heldLoop(asset.loop ?? null);
+  if (!held || held.start >= length) return { start: 0, end: length };
+  return { start: held.start, end: Math.min(held.end, length) };
+}
+
+/** What the scheduler plays of it: the notes before its end, with its instrument; none without one. */
+export function patternSequence(asset: PatternAsset): Sequence {
+  const instrument = patternInstrument(asset.props);
+  if (!instrument) return { notes: [] };
+  const length = patternLength(asset);
+  return {
+    notes: asset.notes
+      .filter((n) => n.tick < length)
+      .map((n) => ({
+        tick: n.tick,
+        length: Math.min(n.length, length - n.tick),
+        note: n.note,
+        velocity: n.velocity,
+        instrument,
+      })),
+  };
+}
+
+/** Notes in the order they are played, the lower one first where two start together. */
+export function byStart(a: PatternNote, b: PatternNote): number {
+  return a.tick - b.tick || a.note - b.note;
+}
+
+/**
+ * Held to what a note may be: a start in whole ticks from 0, at least a tick long, a note of the
+ * keyboard, a velocity from 0 to 1, an id no other note has. Anything else in the list is dropped.
+ */
+export function heldPatternNotes(raw: unknown): PatternNote[] {
+  if (!Array.isArray(raw)) return [];
+  const ids = new Set<string>();
+  const notes: PatternNote[] = [];
+  for (const item of raw as unknown[]) {
+    if (typeof item !== 'object' || item === null) continue;
+    const n = item as Record<string, unknown>;
+    const num = (value: unknown) => (typeof value === 'number' && Number.isFinite(value) ? value : NaN);
+    const tick = Math.round(num(n.tick));
+    const length = num(n.length) > 0 ? Math.max(1, Math.round(num(n.length))) : NaN;
+    const note = Math.round(num(n.note));
+    const velocity = num(n.velocity);
+    if (!(tick >= 0) || !(length >= 1) || !(note >= LOWEST_NOTE && note <= HIGHEST_NOTE)) continue;
+    let id = typeof n.id === 'string' && n.id !== '' ? n.id : newId();
+    while (ids.has(id)) id = newId();
+    ids.add(id);
+    notes.push({
+      id,
+      tick,
+      length,
+      note,
+      velocity: Number.isFinite(velocity) ? Math.max(0, Math.min(1, velocity)) : DEFAULT_VELOCITY,
+    });
+  }
+  return notes.sort(byStart);
+}

src/core/properties/PropertySchema.ts

@@ -104,6 +104,20 @@ import {
 import { SHADER_KEY, shaderOf, shaderParamDefs } from './shaders';
 import { SHADER_RESOLUTION_KEY, SHADER_RESOLUTION_OPTIONS } from './cameraShader';
 import { LOOP_KEY as SOUND_LOOP_KEY, VOLUME_KEY as SOUND_VOLUME_KEY } from './sounds';
+import {
+  BEAT_OPTIONS,
+  DEFAULT_BARS,
+  DEFAULT_GRID,
+  GRID_OPTIONS,
+  MAX_BARS,
+  PATTERN_BARS_KEY,
+  PATTERN_BEAT_KEY,
+  PATTERN_BEATS_KEY,
+  PATTERN_GRID_KEY,
+  PATTERN_INSTRUMENT_KEY,
+  PATTERN_TEMPO_KEY,
+} from '../music/pattern';
+import { DEFAULT_TEMPO, MAX_TEMPO, MIN_TEMPO } from '../music/time';
 import { Theme } from '../../ui';
 
 /**
@@ -659,6 +673,48 @@ const ASSET_SCHEMAS: Record<AssetType, PropertySchema> = {
   soundfx: [],
   // What a note sounds like is the asset's graph (see InstrumentAsset), not a property.
   instrument: [],
+  // The notes and the loop are intrinsic data on the asset (see PatternAsset). The meter and the
+  // tempo are for drawing and hearing it on its own; a song plays it at its own tempo (#334).
+  pattern: [
+    {
+      key: PATTERN_INSTRUMENT_KEY,
+      label: 'Instrument',
+      kind: 'asset',
+      default: '',
+      assetTypes: ['instrument'],
+    },
+    {
+      key: PATTERN_BARS_KEY,
+      label: 'Length (bars)',
+      kind: 'number',
+      default: DEFAULT_BARS,
+      min: 1,
+      max: MAX_BARS,
+      step: 1,
+      decimals: 0,
+    },
+    { key: PATTERN_GRID_KEY, label: 'Grid', kind: 'enum', default: DEFAULT_GRID, options: GRID_OPTIONS },
+    {
+      key: PATTERN_BEATS_KEY,
+      label: 'Beats per bar',
+      kind: 'number',
+      default: 4,
+      min: 1,
+      max: 32,
+      step: 1,
+      decimals: 0,
+    },
+    { key: PATTERN_BEAT_KEY, label: 'Beat', kind: 'enum', default: '4', options: BEAT_OPTIONS },
+    {
+      key: PATTERN_TEMPO_KEY,
+      label: 'Tempo (BPM)',
+      kind: 'number',
+      default: DEFAULT_TEMPO,
+      min: MIN_TEMPO,
+      max: MAX_TEMPO,
+      step: 1,
+    },
+  ],
   // How many particles and for how long; what one of them does is the asset's graph, whose Property
   // nodes add their rows after these – see `schemaForBag`.
   particles: [

src/core/store/serialize.ts

@@ -21,6 +21,8 @@ import { isParticleNodeDef } from '../particles/nodes';
 import { isShaderNodeDef } from '../shader/nodes';
 import { isSoundFxNodeDef } from '../soundfx/nodes';
 import { INSTRUMENT_SAMPLE, isInstrumentNodeDef, withSamplePlay } from '../instrument/nodes';
+import { heldPatternNotes } from '../music/pattern';
+import { heldLoop, type LoopRange } from '../music/sequence';
 import { isKeyInterpolation, normalizeKeys } from '../model/animation';
 import type {
   AnimationKey,
@@ -271,6 +273,9 @@ export function deserializeProject(raw: unknown, fallbackId = ''): Project {
       assets.push(drawing);
     } else if (type === 'palette') {
       assets.push({ id, type, name, props, colors: normalizePalette(a.colors) });
+    } else if (type === 'pattern') {
+      const loop = readLoop(a.loop);
+      assets.push({ id, type, name, props, notes: heldPatternNotes(a.notes), ...(loop && { loop }) });
     } else if (type === 'curve') {
       assets.push({ id, type, name, props, points: sanitizedCurve(normalizePoints(a.points)) });
     } else if (type === 'animation') {
@@ -650,6 +655,14 @@ function normalizePixelArt(
   return { width: canvasSide(raw.width), height: canvasSide(raw.height), layers, frames };
 }
 
+/** A loop of a pattern (#333): two numbers of ticks with room between them, else none. */
+function readLoop(raw: unknown): LoopRange | null {
+  const r = asRecord(raw);
+  const start = asFinite(r.start);
+  const end = asFinite(r.end);
+  return start === null || end === null ? null : heldLoop({ start: Math.round(start), end: Math.round(end) });
+}
+
 /** Colours as 0xRRGGBB, at most `PALETTE_MAX`; anything that is no colour falls away. */
 function normalizePalette(raw: unknown): number[] {
   const colors: number[] = [];