ZGaetano 9032853629 docs: finalize spec with user decisions (auto-save, multi-sequence, 59.94fps, gap-ok)

2026-05-18 19:23:01 -04:00

16 KiB

Raw Blame History

Wild Dragon MAM — Editor, Growing File Preview & Feature Expansion

Date: 2026-05-18
Status: APPROVED — ready for implementation planning

Resolved Decisions

Question	Decision
Save strategy	Auto-save (debounced 2s after any change)
Sequences per project	Multiple named sequences (like Premiere's project panel)
HLS temp disk	3 TB available — no constraint
Primary frame rate	59.94 fps
Program monitor fidelity	Brief gap at clip boundaries is acceptable for v1

Overview

Three sequenced phases:

NLE Editor (editor.html) — Premiere-style timeline editor with razor blade, in/out marking, and EDL export
Growing File Workflow — HLS live preview during SDI/SRT/RTMP capture, with rewind support
Feature Additions — Subclips, improved player, bulk ops, waveform, timecoded comments, smart bins, metadata templates

1. NLE Editor

1.1 Layout

A new page editor.html. player.html is kept as the lightweight browse/metadata view; the editor is the full-screen creative environment.

┌─ SOURCE MONITOR ──────┬─ PROGRAM MONITOR ──────┐   ← 40vh
│  [video preview]      │  [video preview]        │
│  TC: 00:00:00;00      │  TC: 00:00:00;00        │
│  [────scrub bar────]  │  [────scrub bar────]    │
│  [IN] [OUT] [Insert]  │  [J] [K] [L]  [⏩]     │
├─ MEDIA PANEL ─────────┴─ TIMELINE ─────────────┤   ← 60vh
│ [sequence picker]     │  [V] [C] [H]  [zoom]    │
│ [bin tree]            │  ruler: 00:00  00:05 …  │
│ [asset list]          │  V1 ░░░░[clip A]░░░░    │
│                       │  V2                     │
│                       │  A1 ░░░░[clip A]░░░░    │
│                       │  A2                     │
└───────────────────────┴─────────────────────────┘

Accessed from the library via an "Open in Editor" action on each asset card. Opens editor.html?project=<id>&asset=<id> — loads the asset into the source monitor and opens the project's most-recent sequence (or creates one named "Sequence 1" if none exists).

The sidebar nav gains an Editor link (between Library and Ingest).

1.2 Source Monitor

Displays the currently loaded clip (double-click in media panel, or via ?asset= param)
Native <video> element with a custom transport bar: scrub slider, current-time / duration label, play/pause button
Mark In (I): stores sourceIn as seconds; shown as left handle on the scrub bar
Mark Out (O): stores sourceOut as seconds; shown as right handle on the scrub bar
In/out range highlighted on scrub bar (accent color tint)
Insert: drops marked range at timeline playhead, shifts downstream clips right
Overwrite: drops marked range at timeline playhead, overwrites what's there
No marks set → uses full clip duration

1.3 Program Monitor

Plays the timeline from the current playhead position
Virtual playback: single <video> element. On each clip start, set src = that clip's signed proxy URL and currentTime = sourceIn. When currentTime reaches sourceOut, load the next clip. Brief load gap at clip boundaries is acceptable for v1.
Timecode display: HH:MM:SS;FF at 59.94 fps (drop-frame notation, semicolon separator)
Transport shortcuts: Space (play/pause), J (reverse), K (stop), L (forward), ←/→ (± 1 frame at 59.94), Home (jump to start)

1.4 Timeline

Ruler: Timecode-marked horizontal ruler. Default scale: 100px/s. Zoom: Ctrl + scroll wheel or zoom slider (range: 20px/s → 500px/s).

Tracks: V1, V2 (video), A1, A2 (audio). Each track row: 48px tall. Track header (40px wide, left): track label, lock toggle.

Clips: Absolutely positioned <div> elements within track rows.

left = timelineIn_seconds × scale
width = (timelineOut - timelineIn)_seconds × scale
Shows: clip name (truncated), source TC range
If width > 120px: asset thumbnail centered in clip body

Playhead: Red vertical line spanning all tracks. Draggable. Clicking the ruler jumps playhead.

Tools:

Tool	Key	Behavior
Select	`V`	Click to select (accent border). Drag body to move horizontally. Drag left/right edge to trim source in/out.
Razor	`C`	Click on a clip → splits into two clips at that exact frame.
Hand	`H`	Click-drag to pan timeline horizontally.

Editing:

Delete / Backspace: remove selected clip(s)
Ctrl+Z / Ctrl+Shift+Z: undo / redo (local history stack, max 50 steps)
Clips cannot overlap within the same track (enforced on move/razor)

Auto-save: Debounced 2s after any timeline change. Visual indicator: subtle "Saving…" / "Saved" text in the timeline toolbar.

1.5 Data Model

New migration: schema_patch_editor.sql

CREATE TABLE sequences (
  id          UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  project_id  UUID NOT NULL REFERENCES projects ON DELETE CASCADE,
  name        TEXT NOT NULL DEFAULT 'Sequence 1',
  frame_rate  NUMERIC(6,3) NOT NULL DEFAULT 59.94,
  width       INTEGER NOT NULL DEFAULT 1920,
  height      INTEGER NOT NULL DEFAULT 1080,
  created_at  TIMESTAMPTZ DEFAULT NOW(),
  updated_at  TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_sequences_project_id ON sequences(project_id);

CREATE TABLE sequence_clips (
  id                   UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  sequence_id          UUID NOT NULL REFERENCES sequences ON DELETE CASCADE,
  asset_id             UUID NOT NULL REFERENCES assets ON DELETE CASCADE,
  track                INTEGER NOT NULL DEFAULT 0,
  -- track encoding: 0=V1, 1=V2, 100=A1, 101=A2
  timeline_in_frames   BIGINT NOT NULL,
  timeline_out_frames  BIGINT NOT NULL,
  source_in_frames     BIGINT NOT NULL DEFAULT 0,
  source_out_frames    BIGINT NOT NULL,
  created_at           TIMESTAMPTZ DEFAULT NOW(),
  updated_at           TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_sequence_clips_sequence_id ON sequence_clips(sequence_id);

Frame math: All frame counts use 59.94 fps (= 60000/1001). Timecode display uses SMPTE drop-frame (; separator). Conversion helpers:

framesToSeconds(f) = f / 59.94
secondsToFrames(s) = Math.round(s * 59.94)
Drop-frame TC calculation uses standard SMPTE DF algorithm

1.6 API Routes

New file: services/mam-api/src/routes/sequences.js

Method	Path	Body / Params	Notes
GET	`/api/v1/sequences`	`?project_id=`	List sequences, ordered by `updated_at DESC`
POST	`/api/v1/sequences`	`{ project_id, name, frame_rate?, width?, height? }`	Create
GET	`/api/v1/sequences/:id`	—	Sequence + all clips joined with asset (`display_name`, `fps`, `duration_ms`, `proxy_s3_key`, `thumbnail_s3_key`)
PUT	`/api/v1/sequences/:id`	`{ name?, frame_rate?, width?, height? }`	Update metadata
DELETE	`/api/v1/sequences/:id`	—	Cascade deletes clips
PUT	`/api/v1/sequences/:id/clips`	`[ { asset_id, track, timeline_in_frames, timeline_out_frames, source_in_frames, source_out_frames } ]`	Full replace — deletes existing clips, inserts new array in one transaction
POST	`/api/v1/sequences/:id/export/edl`	—	Returns CMX3600 EDL as `text/plain; charset=utf-8`, `Content-Disposition: attachment`

1.7 EDL Export

generateEDL(sequenceName, clips, fps) function produces CMX3600. Timecode math reuses the logic from worker/src/edl/parser.js (copied into a shared util or duplicated in the API route).

TITLE: My Sequence

001  clip_filename  V  C  00:00:00;00 00:00:05;00 00:00:00;00 00:00:05;00
002  other_clip     V  C  00:00:02;00 00:00:08;00 00:00:05;00 00:00:11;00

Conforms via the existing conform.js BullMQ worker unchanged.

1.8 `api.js` Additions

getSequences(projectId)
createSequence(data)
getSequence(sequenceId)
updateSequence(sequenceId, data)
deleteSequence(sequenceId)
syncSequenceClips(sequenceId, clipsArray)
exportSequenceEDL(sequenceId)   // triggers file download

2. Growing File Workflow

2.1 Problem

During SDI capture, proxy is piped to S3 via multipart upload — invisible until CompleteMultipartUpload. No live preview is possible. For SRT/RTMP, no proxy exists at all until the BullMQ worker runs post-stop.

2.2 Solution: HLS Segments During Capture

Write proxy as HLS segments to local disk during recording. Serve them live from the capture service. On stop: stitch segments → single MP4 → upload to S3 as proxy.

Why HLS: Growing playlists are the browser-native live video mechanism. Accumulated segments give free rewind. hls.js is a small CDN-loadable polyfill for Chrome/Firefox. FFmpeg's hls muxer is proven in production.

2.3 FFmpeg Args Change

SDI (second FFmpeg process — proxy process):

# Before: fragmented MP4 → pipe → S3
ffmpeg [decklink] -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
  -movflags +frag_keyframe+empty_moov -f mp4 pipe:1

# After: HLS segments → local dir
ffmpeg [decklink] -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
  -hls_time 2 -hls_list_size 0 -hls_flags append_list \
  -hls_segment_filename '$HLS_DIR/<sessionId>/seg%05d.ts' \
  $HLS_DIR/<sessionId>/index.m3u8

SRT/RTMP (single FFmpeg process — two output pads):

ffmpeg [srt/rtmp input] \
  -c:v prores_ks -profile:v 3 -c:a pcm_s24le \
  -movflags +frag_keyframe+empty_moov -f mov pipe:1 \
  -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
  -hls_time 2 -hls_list_size 0 -hls_flags append_list \
  -hls_segment_filename '$HLS_DIR/<sessionId>/seg%05d.ts' \
  $HLS_DIR/<sessionId>/index.m3u8

2.4 Capture Service Changes

capture-manager.js:

New env var: HLS_SESSION_DIR (default: /tmp/wd-hls). 3 TB disk — no constraint.
start(): mkdir -p $HLS_SESSION_DIR/<sessionId>/, spawn FFmpeg with HLS output. Add liveUrl: /capture/live/<sessionId>/index.m3u8 to session state.
stop():
1. SIGINT FFmpeg (existing)
2. Wait for process exit (FFmpeg writes final segment + closes playlist before exiting)
3. ffmpeg -i $HLS_DIR/<sessionId>/index.m3u8 -c copy <tmp>.mp4 (stitch)
4. Upload stitched MP4 to S3 as proxy key (existing uploadToS3 helper)
5. rm -rf $HLS_DIR/<sessionId>/
Startup: scan $HLS_SESSION_DIR/ and delete dirs older than 24h

New capture routes:

GET /capture/live/:sessionId/index.m3u8  → serve HLS playlist file
GET /capture/live/:sessionId/:file       → serve .ts segment files

Both routes: check session exists (active or recently stopped with dir still present), stream file from disk with correct Content-Type (application/vnd.apple.mpegurl / video/mp2t).

Updated /capture/status response:

{
  "recording": true,
  "sessionId": "abc123",
  "liveUrl": "/capture/live/abc123/index.m3u8",
  "startedAt": "...",
  "duration": 42,
  ...
}

nginx: Add location block for /capture/live/ → proxy_pass to capture service (same pattern as existing /capture/ block).

2.5 Live Preview in `capture.html`

When status poll returns recording: true with liveUrl:

Show collapsible Live Preview panel beneath capture controls
Load hls.js from CDN: cdnjs.cloudflare.com/ajax/libs/hls.js/1.5.15/hls.min.js
new Hls() → hls.loadSource(status.liveUrl) → hls.attachMedia(videoEl)
videoEl.play() on MANIFEST_PARSED (plays at live edge)
⏮ Rewind button: videoEl.currentTime = 0
Elapsed time counter from status.startedAt

On recording stop: hls.destroy(), hide preview panel. Asset appears in library after proxy upload + thumbnail job complete.

3. Feature Additions (Prioritized)

P1 — Improved Player (`player.html` rebuild)

Migrate from old CSS variables (--color-bg-tertiary) to current design tokens (--bg-panel, etc.)
Replace browser default controls with custom transport bar: scrub slider, timecode display HH:MM:SS;FF at 59.94 fps, frame-step buttons, J/K/L shortcuts
Inline rename: click display_name to edit in place → auto-save
"Open in Editor" button → editor.html?asset=<id>

P2 — Subclips

Player shows In/Out markers → "Create Subclip" button
POST /api/v1/assets with { parent_asset_id, subclip_in_ms, subclip_out_ms, display_name }
New columns: parent_asset_id UUID REFERENCES assets, subclip_in_ms BIGINT, subclip_out_ms BIGINT
Library shows subclip cards with ✂ badge; player pre-seeks to subclip_in_ms
Subclips use parent's proxy S3 key — no re-transcode

P3 — Multi-select & Bulk Ops

Shift-click or checkbox (visible on hover) to select multiple asset cards
Floating action bar: Move to bin | Add tags | Delete
Move to bin: slide panel with bin tree, PATCH /assets/:id { bin_id } for each
Add tags: appends to all selected assets
Bulk delete: confirm modal → soft-delete

P4 — Waveform Display in Editor

In proxy worker (proxy.js): after transcode, run FFmpeg astats filter to generate per-second peak arrays → store as waveforms/<assetId>.json in S3
New waveform_s3_key TEXT column on assets
Editor timeline: fetch waveform JSON for audio track clips, render as SVG polyline within clip block

P5 — Timecoded Comments

CREATE TABLE asset_comments (
  id               UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  asset_id         UUID NOT NULL REFERENCES assets ON DELETE CASCADE,
  user_id          UUID NOT NULL REFERENCES users,
  timecode_seconds NUMERIC(10,3),
  body             TEXT NOT NULL,
  created_at       TIMESTAMPTZ DEFAULT NOW()
);

Player sidebar: "Comments" tab with "Post at current TC" button. Clickable entries seek video to that timecode.

P6 — Smart Bins

ALTER TABLE bins ADD COLUMN is_smart BOOLEAN DEFAULT false;
ALTER TABLE bins ADD COLUMN smart_query JSONB;
-- smart_query shape: { "tags": ["interview"], "media_type": "video", "created_after": "2026-01-01" }

Smart bin assets: dynamically queried from assets table. Shows ✦ icon in bin tree.

P7 — Metadata Templates

CREATE TABLE project_metadata_fields (
  id          UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  project_id  UUID NOT NULL REFERENCES projects ON DELETE CASCADE,
  field_key   TEXT NOT NULL,
  label       TEXT NOT NULL,
  field_type  TEXT NOT NULL DEFAULT 'text',
  options     JSONB,
  required    BOOLEAN DEFAULT false,
  sort_order  INTEGER DEFAULT 0
);
ALTER TABLE assets ADD COLUMN metadata JSONB DEFAULT '{}';

Implementation Sequencing

Phase 1 — Editor

DB migration (schema_patch_editor.sql)
API (sequences.js route — CRUD + clip sync + EDL export)
api.js — sequence helpers
editor.html — 4-panel shell + CSS (sidebar nav update)
Timeline engine — ruler, playhead, track rows, clip rendering
Select tool — click-select, drag-move, drag-edge trim
Razor tool — click-to-split
Source monitor — video + transport + in/out marking + Insert/Overwrite
Program monitor — virtual playback + 59.94 drop-frame timecode
Auto-save (debounced 2s) + sequence picker in media panel
Library "Open in Editor" action + sidebar link

Phase 2 — Growing File

Capture service: HLS output, $HLS_SESSION_DIR env var
Capture service: /capture/live/:sessionId/ routes + nginx config
Capture service: on-stop stitch + S3 upload + cleanup
capture.html: live preview panel (hls.js), rewind button

Phase 3 — Feature Additions

P1 → P2 → P3 → P4 → P5 → P6 → P7

16 KiB Raw Blame History Unescape Escape